Default developer use case to systemd
This moves the developer use case over to systemd, and updates all the
relevant docs to discuss the systemd workflow instead of screen. It
does so by defaulting USE_SCREEN=False, so will not impact people that
set it explicitly.
Change-Id: I6d664612bc2b850eb7f56852afbc841867223ab7
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 53ae82f..318e044 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -278,43 +278,22 @@
LOGDAYS=1
-The some of the project logs (Nova, Cinder, etc) will be colorized by
-default (if ``SYSLOG`` is not set below); this can be turned off by
-setting ``LOG_COLOR`` to ``False``.
-
- ::
+Some coloring is used during the DevStack runs to make it easier to
+see what is going on. This can be disabled with::
LOG_COLOR=False
Logging the Service Output
~~~~~~~~~~~~~~~~~~~~~~~~~~
-DevStack will log the ``stdout`` output of the services it starts.
-When using ``screen`` this logs the output in the screen windows to a
-file. Without ``screen`` this simply redirects stdout of the service
-process to a file in ``LOGDIR``.
+By default, services run under ``systemd`` and are natively logging to
+the systemd journal.
- ::
+To query the logs use the ``journalctl`` command, such as::
- LOGDIR=$DEST/logs
+ journalctl --unit devstack@*
-Note the use of ``DEST`` to locate the main install directory; this
-is why we suggest setting it in ``local.conf``.
-
-Enabling Syslog
-~~~~~~~~~~~~~~~
-
-Logging all services to a single syslog can be convenient. Enable
-syslogging by setting ``SYSLOG`` to ``True``. If the destination log
-host is not localhost ``SYSLOG_HOST`` and ``SYSLOG_PORT`` can be used
-to direct the message stream to the log host.
-
- ::
-
- SYSLOG=True
- SYSLOG_HOST=$HOST_IP
- SYSLOG_PORT=516
-
+More examples can be found in :ref:`journalctl-examples`.
Example Logging Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -326,7 +305,6 @@
[[local|localrc]]
DEST=/opt/stack/
- LOGDIR=$DEST/logs
LOGFILE=$LOGDIR/stack.sh.log
LOG_COLOR=False
@@ -587,9 +565,7 @@
Swift is disabled by default. When enabled, it is configured with
only one replica to avoid being IO/memory intensive on a small
-VM. When running with only one replica the account, container and
-object services will run directly in screen. The others services like
-replicator, updaters or auditor runs in background.
+VM.
If you would like to enable Swift you can add this to your ``localrc``
section:
@@ -630,7 +606,7 @@
act as a S3 endpoint for Keystone so effectively replacing the
``nova-objectstore``.
-Only Swift proxy server is launched in the screen session all other
+Only Swift proxy server is launched in the systemd system all other
services are started in background and managed by ``swift-init`` tool.
Heat
diff --git a/doc/source/development.rst b/doc/source/development.rst
index 776ac6c..a3f2747 100644
--- a/doc/source/development.rst
+++ b/doc/source/development.rst
@@ -8,56 +8,33 @@
Inspecting Services
===================
-By default most services in DevStack are running in a `screen
-<https://www.gnu.org/software/screen/manual/screen.html>`_
-session.
+By default most services in DevStack are running as `systemd` units
+named `devstack@$servicename.service`. You can see running services
+with.
.. code-block:: bash
- os3:~> screen -list
- There is a screen on:
- 28994.stack (08/10/2016 09:01:33 PM) (Detached)
- 1 Socket in /var/run/screen/S-sdague.
+ sudo systemctl status --unit="devstack@*"
-You can attach to this screen session using ``screen -r`` which gives
-you a view of the services in action.
-
-.. image:: assets/images/screen_session_1.png
- :width: 100%
-
-Basic Screen Commands
----------------------
-
-The following minimal commands will be useful to using screen:
-
-* ``ctrl-a n`` - go to next window. Next is assumed to be right of
- current window.
-* ``ctrl-a p`` - go to previous window. Previous is assumed to be left
- of current window.
-* ``ctrl-a [`` - entry copy/scrollback mode. This allows you to
- navigate back through the logs with the up arrow.
-* ``ctrl-a d`` - detach from screen. Gets you back to a normal
- terminal, while leaving everything running.
-
-For more about using screen, see the excellent `screen manual
-<https://www.gnu.org/software/screen/manual/screen.html>`_.
+To learn more about the basics of systemd, see :doc:`/systemd`
Patching a Service
==================
If you want to make a quick change to a running service the easiest
-way to do this is:
+way to do that is to change the code directly in /opt/stack/$service
+and then restart the affected daemons.
-* attach to screen
-* navigate to the window in question
-* ``ctrl-c`` to kill the service
-* make appropriate changes to the code
-* ``up arrow`` in the screen window to display the command used to run
- that service
-* ``enter`` to restart the service
+.. code-block:: bash
-This works for services, except those running under Apache (currently
-just ``keystone`` by default).
+ sudo systemctl restart --unit=devstack@n-cpu.service
+
+If your change impacts more than one daemon you can restart by
+wildcard as well.
+
+.. code-block:: bash
+
+ sudo systemctl restart --unit="devstack@n-*"
.. warning::
@@ -102,14 +79,6 @@
NOVA_BRANCH=refs/changes/10/353710/1
-Testing Changes to Apache Based Services
-========================================
-
-When testing changes to Apache based services, such as ``keystone``,
-you can either use the Testing a Patch Series approach above, or make
-changes in the code tree and issue an apache restart.
-
-
Testing Changes to Libraries
============================
@@ -132,9 +101,17 @@
OSLOPOLICY_REPO=/home/sdague/oslo.policy
OSLOPOLICY_BRANCH=better_exception
-Because libraries are used by many services, library changes really
-need to go through a full ``./unstack.sh && ./stack.sh`` to see your
-changes in action.
+As libraries are not installed `editable` by pip, after you make any
+local changes you will need to:
-To figure out the repo / branch names for every library that's
-supported, you'll need to read the devstack source.
+* cd to top of library path
+* sudo pip install -U .
+* restart all services you want to use the new library
+
+You can do that with wildcards such as
+
+.. code-block:: bash
+
+ sudo systemctl restart --unit="devstack@n-*"
+
+which will restart all nova services.
diff --git a/doc/source/faq.rst b/doc/source/faq.rst
index f03304f..cb2f328 100644
--- a/doc/source/faq.rst
+++ b/doc/source/faq.rst
@@ -41,8 +41,9 @@
~~~~~~~~~~~~~~~~~~~~~
Unlike packages, DevStack leaves your cloud ready to develop -
-checkouts of the code and services running in screen. However, many
-people are doing the hard work of packaging and recipes for production
+checkouts of the code and services running locally under systemd,
+making it easy to hack on and test new patches. However, many people
+are doing the hard work of packaging and recipes for production
deployments.
Why isn't $MY\_FAVORITE\_DISTRO supported?
diff --git a/doc/source/site-map.rst b/doc/source/site-map.rst
index 801fc66..022cc73 100644
--- a/doc/source/site-map.rst
+++ b/doc/source/site-map.rst
@@ -21,3 +21,4 @@
development
hacking
guides
+ systemd
diff --git a/doc/source/systemd.rst b/doc/source/systemd.rst
new file mode 100644
index 0000000..efe79e4
--- /dev/null
+++ b/doc/source/systemd.rst
@@ -0,0 +1,167 @@
+===========================
+ Using Systemd in DevStack
+===========================
+
+By default DevStack is run with all the services as systemd unit
+files. Systemd is now the default init system for nearly every Linux
+distro, and systemd encodes and solves many of the problems related to
+poorly running processes.
+
+Why this instead of screen?
+===========================
+
+The screen model for DevStack was invented when the number of services
+that a DevStack user was going to run was typically < 10. This made
+screen hot keys to jump around very easy. However, the landscape has
+changed (not all services are stoppable in screen as some are under
+Apache, there are typically at least 20 items)
+
+There is also a common developer workflow of changing code in more
+than one service, and needing to restart a bunch of services for that
+to take effect.
+
+Unit Structure
+==============
+
+.. note::
+
+ Originally we actually wanted to do this as user units, however
+ there are issues with running this under non interactive
+ shells. For now, we'll be running as system units. Some user unit
+ code is left in place in case we can switch back later.
+
+All DevStack user units are created as a part of the DevStack slice
+given the name ``devstack@$servicename.service``. This makes it easy
+to understand which services are part of the devstack run, and lets us
+disable / stop them in a single command.
+
+Manipulating Units
+==================
+
+Assuming the unit ``n-cpu`` to make the examples more clear.
+
+Enable a unit (allows it to be started)::
+
+ sudo systemctl enable devstack@n-cpu.service
+
+Disable a unit::
+
+ sudo systemctl disable devstack@n-cpu.service
+
+Start a unit::
+
+ sudo systemctl start devstack@n-cpu.service
+
+Stop a unit::
+
+ sudo systemctl stop devstack@n-cpu.service
+
+Restart a unit::
+
+ sudo systemctl restart devstack@n-cpu.service
+
+See status of a unit::
+
+ sudo systemctl status devstack@n-cpu.service
+
+Operating on more than one unit at a time
+-----------------------------------------
+
+Systemd supports wildcarding for unit operations. To restart every
+service in devstack you can do that following::
+
+ sudo systemctl restart devstack@*
+
+Or to see the status of all Nova processes you can do::
+
+ sudo systemctl status devstack@n-*
+
+We'll eventually make the unit names a bit more meaningful so that
+it's easier to understand what you are restarting.
+
+.. _journalctl-examples:
+
+Querying Logs
+=============
+
+One of the other major things that comes with systemd is journald, a
+consolidated way to access logs (including querying through structured
+metadata). This is accessed by the user via ``journalctl`` command.
+
+
+Logs can be accessed through ``journalctl``. journalctl has powerful
+query facilities. We'll start with some common options.
+
+Follow logs for a specific service::
+
+ journalctl -f --unit devstack@n-cpu.service
+
+Following logs for multiple services simultaneously::
+
+ journalctl -f --unit devstack@n-cpu.service --unit
+ devstack@n-cond.service
+
+or you can even do wild cards to follow all the nova services::
+
+ journalctl -f --unit devstack@n-*
+
+Use higher precision time stamps::
+
+ journalctl -f -o short-precise --unit devstack@n-cpu.service
+
+
+Known Issues
+============
+
+Be careful about systemd python libraries. There are 3 of them on
+pypi, and they are all very different. They unfortunately all install
+into the ``systemd`` namespace, which can cause some issues.
+
+- ``systemd-python`` - this is the upstream maintained library, it has
+ a version number like systemd itself (currently ``234``). This is
+ the one you want.
+- ``systemd`` - a python 3 only library, not what you want.
+- ``python-systemd`` - another library you don't want. Installing it
+ on a system will break ansible's ability to run.
+
+
+If we were using user units, the ``[Service]`` - ``Group=`` parameter
+doesn't seem to work with user units, even though the documentation
+says that it should. This means that we will need to do an explicit
+``/usr/bin/sg``. This has the downside of making the SYSLOG_IDENTIFIER
+be ``sg``. We can explicitly set that with ``SyslogIdentifier=``, but
+it's really unfortunate that we're going to need this work
+around. This is currently not a problem because we're only using
+system units.
+
+Future Work
+===========
+
+log colorizing
+--------------
+
+We lose log colorization through this process. We might want to build
+a custom colorizer that we could run journalctl output through
+optionally for people.
+
+user units
+----------
+
+It would be great if we could do services as user units, so that there
+is a clear separation of code being run as not root, to ensure running
+as root never accidentally gets baked in as an assumption to
+services. However, user units interact poorly with devstack-gate and
+the way that commands are run as users with ansible and su.
+
+Maybe someday we can figure that out.
+
+References
+==========
+
+- Arch Linux Wiki - https://wiki.archlinux.org/index.php/Systemd/User
+- Python interface to journald -
+ https://www.freedesktop.org/software/systemd/python-systemd/journal.html
+- Systemd documentation on service files -
+ https://www.freedesktop.org/software/systemd/man/systemd.service.html
+- Systemd documentation on exec (can be used to impact service runs) -
+ https://www.freedesktop.org/software/systemd/man/systemd.exec.html