deleted file mode 100644

@@ -1,189 +0,0 @@

-===========================

- Using Systemd in DevStack

-===========================

-

-.. note::

-

-   This is an in progress document as we work out the way forward here

-   with DevStack and systemd.

-

-DevStack can be run with all the services as systemd unit

-files. Systemd is now the default init system for nearly every Linux

-distro, and systemd encodes and solves many of the problems related to

-poorly running processes.

-

-Why this instead of screen?

-===========================

-

-The screen model for DevStack was invented when the number of services

-that a DevStack user was going to run was typically < 10. This made

-screen hot keys to jump around very easy. However, the landscape has

-changed (not all services are stoppable in screen as some are under

-Apache, there are typically at least 20 items)

-

-There is also a common developer workflow of changing code in more

-than one service, and needing to restart a bunch of services for that

-to take effect.

-

-To enable this add the following to your local.conf::

-

-  USE_SYSTEMD=True

-

-

-

-Unit Structure

-==============

-

-.. note::

-

-   Originally we actually wanted to do this as user units, however

-   there are issues with running this under non interactive

-   shells. For now, we'll be running as system units. Some user unit

-   code is left in place in case we can switch back later.

-

-All DevStack user units are created as a part of the DevStack slice

-given the name ``devstack@$servicename.service``. This lets us do

-certain operations at the slice level.

-

-Manipulating Units

-==================

-

-Assuming the unit ``n-cpu`` to make the examples more clear.

-

-Enable a unit (allows it to be started)::

-

-  sudo systemctl enable devstack@n-cpu.service

-

-Disable a unit::

-

-  sudo systemctl disable devstack@n-cpu.service

-

-Start a unit::

-

-  sudo systemctl start devstack@n-cpu.service

-

-Stop a unit::

-

-  sudo systemctl stop devstack@n-cpu.service

-

-Restart a unit::

-

-  sudo systemctl restart devstack@n-cpu.service

-

-See status of a unit::

-

-  sudo systemctl status devstack@n-cpu.service

-

-Operating on more than one unit at a time

-

-Systemd supports wildcarding for unit operations. To restart every

-service in devstack you can do that following::

-

-  sudo systemctl restart devstack@*

-

-Or to see the status of all Nova processes you can do::

-

-  sudo systemctl status devstack@n-*

-

-We'll eventually make the unit names a bit more meaningful so that

-it's easier to understand what you are restarting.

-

-Querying Logs

-=============

-

-One of the other major things that comes with systemd is journald, a

-consolidated way to access logs (including querying through structured

-metadata). This is accessed by the user via ``journalctl`` command.

-

-

-Logs can be accessed through ``journalctl``. journalctl has powerful

-query facilities. We'll start with some common options.

-

-Follow logs for a specific service::

-

-  journalctl -f --unit devstack@n-cpu.service

-

-Following logs for multiple services simultaneously::

-

-  journalctl -f --unit devstack@n-cpu.service --unit

-  devstack@n-cond.service

-

-or you can even do wild cards to follow all the nova services::

-

-  journalctl -f --unit devstack@n-*

-

-Use higher precision time stamps::

-

-  journalctl -f -o short-precise --unit devstack@n-cpu.service

-

-

-Known Issues

-============

-

-Be careful about systemd python libraries. There are 3 of them on

-pypi, and they are all very different. They unfortunately all install

-into the ``systemd`` namespace, which can cause some issues.

-

-- ``systemd-python`` - this is the upstream maintained library, it has

-  a version number like systemd itself (currently ``233``). This is

-  the one you want.

-- ``systemd`` - a python 3 only library, not what you want.

-- ``python-systemd`` - another library you don't want. Installing it

-  on a system will break ansible's ability to run.

-

-

-If we were using user units, the ``[Service]`` - ``Group=`` parameter

-doesn't seem to work with user units, even though the documentation

-says that it should. This means that we will need to do an explicit

-``/usr/bin/sg``. This has the downside of making the SYSLOG_IDENTIFIER

-be ``sg``. We can explicitly set that with ``SyslogIdentifier=``, but

-it's really unfortunate that we're going to need this work

-around. This is currently not a problem because we're only using

-system units.

-

-Future Work

-===========

-

-oslo.log journald

-

-Journald has an extremely rich mechanism for direct logging including

-structured metadata. We should enhance oslo.log to take advantage of

-that. It would let us do things like::

-

-  journalctl REQUEST_ID=......

-

-  journalctl INSTANCE_ID=......

-

-And get all lines related to the request id or instance id. (Note:

-this work has been started at https://review.openstack.org/#/c/451525/)

-

-log colorizing

-

-We lose log colorization through this process. We might want to build

-a custom colorizer that we could run journalctl output through

-optionally for people.

-

-user units

-

-It would be great if we could do services as user units, so that there

-is a clear separation of code being run as not root, to ensure running

-as root never accidentally gets baked in as an assumption to

-services. However, user units interact poorly with devstack-gate and

-the way that commands are run as users with ansible and su.

-

-Maybe someday we can figure that out.

-

-References

-==========

-

-- Arch Linux Wiki - https://wiki.archlinux.org/index.php/Systemd/User

-- Python interface to journald -

-  https://www.freedesktop.org/software/systemd/python-systemd/journal.html

-- Systemd documentation on service files -

-  https://www.freedesktop.org/software/systemd/man/systemd.service.html

-- Systemd documentation on exec (can be used to impact service runs) -

-  https://www.freedesktop.org/software/systemd/man/systemd.exec.html

@@ -278,43 +278,22 @@ number of days of old log files to keep.

         LOGDAYS=1

-The some of the project logs (Nova, Cinder, etc) will be colorized by

-default (if ``SYSLOG`` is not set below); this can be turned off by

-setting ``LOG_COLOR`` to ``False``.

-

-    ::

+Some coloring is used during the DevStack runs to make it easier to

+see what is going on. This can be disabled with::

         LOG_COLOR=False

 Logging the Service Output

 ~~~~~~~~~~~~~~~~~~~~~~~~~~

-DevStack will log the ``stdout`` output of the services it starts.

-When using ``screen`` this logs the output in the screen windows to a

-file.  Without ``screen`` this simply redirects stdout of the service

-process to a file in ``LOGDIR``.

-

-    ::

-

-        LOGDIR=$DEST/logs

+By default, services run under ``systemd`` and are natively logging to

+the systemd journal.

-Note the use of ``DEST`` to locate the main install directory; this

-is why we suggest setting it in ``local.conf``.

-

-Enabling Syslog

-~~~~~~~~~~~~~~~

-

-Logging all services to a single syslog can be convenient. Enable

-syslogging by setting ``SYSLOG`` to ``True``. If the destination log

-host is not localhost ``SYSLOG_HOST`` and ``SYSLOG_PORT`` can be used

-to direct the message stream to the log host.

-

-    ::

+To query the logs use the ``journalctl`` command, such as::

-        SYSLOG=True

-        SYSLOG_HOST=$HOST_IP

-        SYSLOG_PORT=516

+  journalctl --unit devstack@*

+More examples can be found in :ref:`journalctl-examples`.

 Example Logging Configuration

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -326,7 +305,6 @@ a file, keep service logs and disable color in the stored files.

        [[local|localrc]]

        DEST=/opt/stack/

-       LOGDIR=$DEST/logs

        LOGFILE=$LOGDIR/stack.sh.log

        LOG_COLOR=False

@@ -587,9 +565,7 @@ Swift

 Swift is disabled by default.  When enabled, it is configured with

 only one replica to avoid being IO/memory intensive on a small

-VM. When running with only one replica the account, container and

-object services will run directly in screen. The others services like

-replicator, updaters or auditor runs in background.

+VM.

 If you would like to enable Swift you can add this to your ``localrc``

 section:

@@ -630,7 +606,7 @@ install the swift3 middleware emulation. Swift will be configured to

 act as a S3 endpoint for Keystone so effectively replacing the

 ``nova-objectstore``.

-Only Swift proxy server is launched in the screen session all other

+Only Swift proxy server is launched in the systemd system all other

 services are started in background and managed by ``swift-init`` tool.

 Heat

@@ -8,56 +8,33 @@ with it?

 Inspecting Services

 ===================

-By default most services in DevStack are running in a `screen

-<https://www.gnu.org/software/screen/manual/screen.html>`_

-session.

+By default most services in DevStack are running as `systemd` units

+named `devstack@$servicename.service`. You can see running services

+with.

 .. code-block:: bash

-   os3:~> screen -list

-   There is a screen on:

-        28994.stack	(08/10/2016 09:01:33 PM)	(Detached)

-   1 Socket in /var/run/screen/S-sdague.

+   sudo systemctl status --unit="devstack@*"

-You can attach to this screen session using ``screen -r`` which gives

-you a view of the services in action.

-

-.. image:: assets/images/screen_session_1.png

-   :width: 100%

-

-Basic Screen Commands

-

-The following minimal commands will be useful to using screen:

-

-* ``ctrl-a n`` - go to next window. Next is assumed to be right of

-  current window.

-* ``ctrl-a p`` - go to previous window. Previous is assumed to be left

-  of current window.

-* ``ctrl-a [`` - entry copy/scrollback mode. This allows you to

-  navigate back through the logs with the up arrow.

-* ``ctrl-a d`` - detach from screen. Gets you back to a normal

-  terminal, while leaving everything running.

-

-For more about using screen, see the excellent `screen manual

-<https://www.gnu.org/software/screen/manual/screen.html>`_.

+To learn more about the basics of systemd, see :doc:`/systemd`

 Patching a Service

 ==================

 If you want to make a quick change to a running service the easiest

-way to do this is:

+way to do that is to change the code directly in /opt/stack/$service

+and then restart the affected daemons.

+

+.. code-block:: bash

+

+   sudo systemctl restart --unit=devstack@n-cpu.service

+

+If your change impacts more than one daemon you can restart by

+wildcard as well.

-* attach to screen

-* navigate to the window in question

-* ``ctrl-c`` to kill the service

-* make appropriate changes to the code

-* ``up arrow`` in the screen window to display the command used to run

-  that service

-* ``enter`` to restart the service

+.. code-block:: bash

-This works for services, except those running under Apache (currently

-just ``keystone`` by default).

+   sudo systemctl restart --unit="devstack@n-*"

 .. warning::

@@ -102,14 +79,6 @@ in gerrit by using the ref name that gerrit assigns to each change.

    NOVA_BRANCH=refs/changes/10/353710/1

-Testing Changes to Apache Based Services

-========================================

-

-When testing changes to Apache based services, such as ``keystone``,

-you can either use the Testing a Patch Series approach above, or make

-changes in the code tree and issue an apache restart.

-

-

 Testing Changes to Libraries

 ============================

@@ -132,9 +101,17 @@ your changes instead of just upstream master.

    OSLOPOLICY_REPO=/home/sdague/oslo.policy

    OSLOPOLICY_BRANCH=better_exception

-Because libraries are used by many services, library changes really

-need to go through a full ``./unstack.sh && ./stack.sh`` to see your

-changes in action.

+As libraries are not installed `editable` by pip, after you make any

+local changes you will need to:

+

+* cd to top of library path

+* sudo pip install -U .

+* restart all services you want to use the new library

+

+You can do that with wildcards such as

+

+.. code-block:: bash

+

+   sudo systemctl restart --unit="devstack@n-*"

-To figure out the repo / branch names for every library that's

-supported, you'll need to read the devstack source.

+which will restart all nova services.

@@ -41,8 +41,9 @@ Why not use packages?

 ~~~~~~~~~~~~~~~~~~~~~

 Unlike packages, DevStack leaves your cloud ready to develop -

-checkouts of the code and services running in screen. However, many

-people are doing the hard work of packaging and recipes for production

+checkouts of the code and services running locally under systemd,

+making it easy to hack on and test new patches. However, many people

+are doing the hard work of packaging and recipes for production

 deployments.

 Why isn't $MY\_FAVORITE\_DISTRO supported?

@@ -21,3 +21,4 @@

    development

    hacking

    guides

+   systemd

new file mode 100644

@@ -0,0 +1,167 @@

+===========================

+ Using Systemd in DevStack

+===========================

+

+By default DevStack is run with all the services as systemd unit

+files. Systemd is now the default init system for nearly every Linux

+distro, and systemd encodes and solves many of the problems related to

+poorly running processes.

+

+Why this instead of screen?

+===========================

+

+The screen model for DevStack was invented when the number of services

+that a DevStack user was going to run was typically < 10. This made

+screen hot keys to jump around very easy. However, the landscape has

+changed (not all services are stoppable in screen as some are under

+Apache, there are typically at least 20 items)

+

+There is also a common developer workflow of changing code in more

+than one service, and needing to restart a bunch of services for that

+to take effect.

+

+Unit Structure

+==============

+

+.. note::

+

+   Originally we actually wanted to do this as user units, however

+   there are issues with running this under non interactive

+   shells. For now, we'll be running as system units. Some user unit

+   code is left in place in case we can switch back later.

+

+All DevStack user units are created as a part of the DevStack slice

+given the name ``devstack@$servicename.service``. This makes it easy

+to understand which services are part of the devstack run, and lets us

+disable / stop them in a single command.

+

+Manipulating Units

+==================

+

+Assuming the unit ``n-cpu`` to make the examples more clear.

+

+Enable a unit (allows it to be started)::

+

+  sudo systemctl enable devstack@n-cpu.service

+

+Disable a unit::

+

+  sudo systemctl disable devstack@n-cpu.service

+

+Start a unit::

+

+  sudo systemctl start devstack@n-cpu.service

+

+Stop a unit::

+

+  sudo systemctl stop devstack@n-cpu.service

+

+Restart a unit::

+

+  sudo systemctl restart devstack@n-cpu.service

+

+See status of a unit::

+

+  sudo systemctl status devstack@n-cpu.service

+

+Operating on more than one unit at a time

+-----------------------------------------

+

+Systemd supports wildcarding for unit operations. To restart every

+service in devstack you can do that following::

+

+  sudo systemctl restart devstack@*

+

+Or to see the status of all Nova processes you can do::

+

+  sudo systemctl status devstack@n-*

+

+We'll eventually make the unit names a bit more meaningful so that

+it's easier to understand what you are restarting.

+

+.. _journalctl-examples:

+

+Querying Logs

+=============

+

+One of the other major things that comes with systemd is journald, a

+consolidated way to access logs (including querying through structured

+metadata). This is accessed by the user via ``journalctl`` command.

+

+

+Logs can be accessed through ``journalctl``. journalctl has powerful

+query facilities. We'll start with some common options.

+

+Follow logs for a specific service::

+

+  journalctl -f --unit devstack@n-cpu.service

+

+Following logs for multiple services simultaneously::

+

+  journalctl -f --unit devstack@n-cpu.service --unit

+  devstack@n-cond.service

+

+or you can even do wild cards to follow all the nova services::

+

+  journalctl -f --unit devstack@n-*

+

+Use higher precision time stamps::

+

+  journalctl -f -o short-precise --unit devstack@n-cpu.service

+

+

+Known Issues

+============

+

+Be careful about systemd python libraries. There are 3 of them on

+pypi, and they are all very different. They unfortunately all install

+into the ``systemd`` namespace, which can cause some issues.

+

+- ``systemd-python`` - this is the upstream maintained library, it has

+  a version number like systemd itself (currently ``234``). This is

+  the one you want.

+- ``systemd`` - a python 3 only library, not what you want.

+- ``python-systemd`` - another library you don't want. Installing it

+  on a system will break ansible's ability to run.

+

+

+If we were using user units, the ``[Service]`` - ``Group=`` parameter

+doesn't seem to work with user units, even though the documentation

+says that it should. This means that we will need to do an explicit

+``/usr/bin/sg``. This has the downside of making the SYSLOG_IDENTIFIER

+be ``sg``. We can explicitly set that with ``SyslogIdentifier=``, but

+it's really unfortunate that we're going to need this work

+around. This is currently not a problem because we're only using

+system units.

+

+Future Work

+===========

+

+log colorizing

+--------------

+

+We lose log colorization through this process. We might want to build

+a custom colorizer that we could run journalctl output through

+optionally for people.

+

+user units

+----------

+

+It would be great if we could do services as user units, so that there

+is a clear separation of code being run as not root, to ensure running

+as root never accidentally gets baked in as an assumption to

+services. However, user units interact poorly with devstack-gate and

+the way that commands are run as users with ansible and su.

+

+Maybe someday we can figure that out.

+

+References

+==========

+

+- Arch Linux Wiki - https://wiki.archlinux.org/index.php/Systemd/User

+- Python interface to journald -

+  https://www.freedesktop.org/software/systemd/python-systemd/journal.html

+- Systemd documentation on service files -

+  https://www.freedesktop.org/software/systemd/man/systemd.service.html

+- Systemd documentation on exec (can be used to impact service runs) -

+  https://www.freedesktop.org/software/systemd/man/systemd.exec.html

@@ -1474,6 +1474,13 @@ if [[ -n "$DEPRECATED_TEXT" ]]; then

     echo_summary "WARNING: $DEPRECATED_TEXT"

 fi

+# If USE_SYSTEMD is enabled, tell the user about using it.

+if [[ "$USE_SYSTEMD" == "True" ]]; then

+    echo "Services are running under systemd unit files."

+    echo "For more information see: "

+    echo "https://docs.openstack.org/developer/devstack/systemd.html"

+fi

+

 # Indicate how long this took to run (bash maintained variable ``SECONDS``)

 echo_summary "stack.sh completed in $SECONDS seconds."

@@ -80,12 +80,19 @@ NOVA_ENABLED_APIS=osapi_compute,metadata

 # Set the root URL for Horizon

 HORIZON_APACHE_ROOT="/dashboard"

+# TODO(sdague): Queens

+#

+# All the non systemd paths should be removed in queens, they only

+# exist in Pike to support testing from grenade. Ensure that all this

+# is cleaned up and purged, which should dramatically simplify the

+# devstack codebase.

+

 # Whether to use 'dev mode' for screen windows. Dev mode works by

 # stuffing text into the screen windows so that a developer can use

 # ctrl-c, up-arrow, enter to restart the service. Starting services

 # this way is slightly unreliable, and a bit slower, so this can

 # be disabled for automated testing by setting this value to False.

-USE_SCREEN=$(trueorfalse True USE_SCREEN)

+USE_SCREEN=$(trueorfalse False USE_SCREEN)

 # Whether to use SYSTEMD to manage services

 USE_SYSTEMD=$(trueorfalse False USE_SYSTEMD)

@@ -100,9 +107,6 @@ else

     JOURNALCTL_F="journalctl -f -o short-precise --unit"

 fi

-if [[ "$USE_SYSTEMD" == "True" ]]; then

-    USE_SCREEN=False

-fi

 # Whether or not to enable Kernel Samepage Merging (KSM) if available.

 # This allows programs that mark their memory as mergeable to share

@@ -157,6 +161,10 @@ elif [[ -f $RC_DIR/.localrc.auto ]]; then

     source $RC_DIR/.localrc.auto

 fi

+# TODO(sdague): Delete all this in Queens.

+if [[ "$USE_SYSTEMD" == "True" ]]; then

+    USE_SCREEN=False

+fi

 # if we are forcing off USE_SCREEN (as we do in the gate), force on

 # systemd. This allows us to drop one of 3 paths through the code.

 if [[ "$USE_SCREEN" == "False" ]]; then