diff --git a/.gitignore b/.gitignore
index d1781bc..d2c127d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,8 @@
 files/pip-*
 files/get-pip.py*
 files/ir-deploy*
+files/ironic-inspector*
+files/etcd*
 local.conf
 local.sh
 localrc
diff --git a/HACKING.rst b/HACKING.rst
index b76cb6c..fc67f09 100644
--- a/HACKING.rst
+++ b/HACKING.rst
@@ -322,7 +322,7 @@
 
 
 Review Criteria
-===============
+---------------
 
 There are some broad criteria that will be followed when reviewing
 your change
@@ -364,3 +364,26 @@
 
 * **Reviewers** -- please see ``MAINTAINERS.rst`` for a list of people
   that should be added to reviews of various sub-systems.
+
+
+Making Changes, Testing, and CI
+-------------------------------
+
+Changes to Devstack are tested by automated continuous integration jobs
+that run on a variety of Linux Distros using a handful of common
+configurations. What this means is that every change to Devstack is
+self testing. One major benefit of this is that developers do not
+typically need to add new non voting test jobs to add features to
+Devstack. Instead the features can be added, then if testing passes
+with the feature enabled the change is ready to merge (pending code
+review).
+
+A concrete example of this was the switch from screen based service
+management to systemd based service management. No new jobs were
+created for this. Instead the features were added to devstack, tested
+locally and in CI using a change that enabled the feature, then once
+the enabling change was passing and the new behavior communicated and
+documented it was merged.
+
+Using this process has been proven to be effective and leads to
+quicker implementation of desired features.
diff --git a/README.md b/README.rst
similarity index 88%
rename from README.md
rename to README.rst
index ff5598b..b4240bd 100644
--- a/README.md
+++ b/README.rst
@@ -1,6 +1,7 @@
 DevStack is a set of scripts and utilities to quickly deploy an OpenStack cloud.
 
-# Goals
+Goals
+=====
 
 * To quickly build dev OpenStack environments in a clean Ubuntu or Fedora
   environment
@@ -20,12 +21,13 @@
 networking configuration.  We strongly recommend that you run `stack.sh`
 in a clean and disposable vm when you are first getting started.
 
-# Versions
+Versions
+========
 
 The DevStack master branch generally points to trunk versions of OpenStack
 components.  For older, stable versions, look for branches named
 stable/[release] in the DevStack repo.  For example, you can do the
-following to create a Newton OpenStack cloud:
+following to create a Newton OpenStack cloud::
 
     git checkout stable/newton
     ./stack.sh
@@ -38,7 +40,8 @@
     GLANCE_REPO=git://git.openstack.org/openstack/glance.git
     GLANCE_BRANCH=milestone-proposed
 
-# Start A Dev Cloud
+Start A Dev Cloud
+=================
 
 Installing in a dedicated disposable VM is safer than installing on your
 dev machine!  Plus you can pick one of the supported Linux distros for
@@ -54,14 +57,15 @@
 * Keystone: http://myhost:5000/v2.0/
 
 We also provide an environment file that you can use to interact with your
-cloud via CLI:
+cloud via CLI::
 
     # source openrc file to load your environment with OpenStack CLI creds
     . openrc
     # list instances
-    nova list
+    openstack server list
 
-# DevStack Execution Environment
+DevStack Execution Environment
+==============================
 
 DevStack runs rampant over the system it runs on, installing things and
 uninstalling other things.  Running this on a system you care about is a recipe
@@ -81,10 +85,12 @@
 it runs under.  Many people simply use their usual login (the default
 'ubuntu' login on a UEC image for example).
 
-# Customizing
+Customizing
+===========
 
 DevStack can be extensively configured via the configuration file
 `local.conf`.  It is likely that you will need to provide and modify
 this file if you want anything other than the most basic setup.  Start
-by reading the [configuration guide](doc/source/configuration.rst) for
-details of the configuration file and the many available options.
+by reading the `configuration guide
+<https://docs.openstack.org/developer/devstack/configuration.html>`_
+for details of the configuration file and the many available options.
diff --git a/clean.sh b/clean.sh
index 90b21eb..9ffe3be 100755
--- a/clean.sh
+++ b/clean.sh
@@ -64,13 +64,8 @@
     done
 fi
 
-# See if there is anything running...
-# need to adapt when run_service is merged
-SESSION=$(screen -ls | awk '/[0-9].stack/ { print $1 }')
-if [[ -n "$SESSION" ]]; then
-    # Let unstack.sh do its thing first
-    $TOP_DIR/unstack.sh --all
-fi
+# Let unstack.sh do its thing first
+$TOP_DIR/unstack.sh --all
 
 # Run extras
 # ==========
@@ -130,6 +125,13 @@
     sudo rm -rf $SCREEN_LOGDIR
 fi
 
+# Clean out the sytemd user unit files if systemd was used.
+if [[ "$USE_SYSTEMD" = "True" ]]; then
+    sudo find $SYSTEMD_DIR -type f -name '*devstack@*service' -delete
+    # Make systemd aware of the deletion.
+    $SYSTEMCTL daemon-reload
+fi
+
 # Clean up venvs
 DIRS_TO_CLEAN="$WHEELHOUSE ${PROJECT_VENV[@]} .config/openstack"
 rm -rf $DIRS_TO_CLEAN
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 53ae82f..064bf51 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -195,6 +195,9 @@
 Setting it here also makes it available for ``openrc`` to set ``OS_AUTH_URL``.
 ``HOST_IPV6`` is not set by default.
 
+For architecture specific configurations which differ from the x86 default
+here, see `arch-configuration`_.
+
 Historical Notes
 ================
 
@@ -278,43 +281,22 @@
 
         LOGDAYS=1
 
-The some of the project logs (Nova, Cinder, etc) will be colorized by
-default (if ``SYSLOG`` is not set below); this can be turned off by
-setting ``LOG_COLOR`` to ``False``.
-
-    ::
+Some coloring is used during the DevStack runs to make it easier to
+see what is going on. This can be disabled with::
 
         LOG_COLOR=False
 
 Logging the Service Output
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-DevStack will log the ``stdout`` output of the services it starts.
-When using ``screen`` this logs the output in the screen windows to a
-file.  Without ``screen`` this simply redirects stdout of the service
-process to a file in ``LOGDIR``.
+By default, services run under ``systemd`` and are natively logging to
+the systemd journal.
 
-    ::
+To query the logs use the ``journalctl`` command, such as::
 
-        LOGDIR=$DEST/logs
+  journalctl --unit devstack@*
 
-Note the use of ``DEST`` to locate the main install directory; this
-is why we suggest setting it in ``local.conf``.
-
-Enabling Syslog
-~~~~~~~~~~~~~~~
-
-Logging all services to a single syslog can be convenient. Enable
-syslogging by setting ``SYSLOG`` to ``True``. If the destination log
-host is not localhost ``SYSLOG_HOST`` and ``SYSLOG_PORT`` can be used
-to direct the message stream to the log host.
-
-    ::
-
-        SYSLOG=True
-        SYSLOG_HOST=$HOST_IP
-        SYSLOG_PORT=516
-
+More examples can be found in :ref:`journalctl-examples`.
 
 Example Logging Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -326,7 +308,6 @@
 
        [[local|localrc]]
        DEST=/opt/stack/
-       LOGDIR=$DEST/logs
        LOGFILE=$LOGDIR/stack.sh.log
        LOG_COLOR=False
 
@@ -587,9 +568,7 @@
 
 Swift is disabled by default.  When enabled, it is configured with
 only one replica to avoid being IO/memory intensive on a small
-VM. When running with only one replica the account, container and
-object services will run directly in screen. The others services like
-replicator, updaters or auditor runs in background.
+VM.
 
 If you would like to enable Swift you can add this to your ``localrc``
 section:
@@ -630,32 +609,9 @@
 act as a S3 endpoint for Keystone so effectively replacing the
 ``nova-objectstore``.
 
-Only Swift proxy server is launched in the screen session all other
+Only Swift proxy server is launched in the systemd system all other
 services are started in background and managed by ``swift-init`` tool.
 
-Heat
-~~~~
-
-Heat is disabled by default (see ``stackrc`` file). To enable it
-explicitly you'll need the following settings in your ``localrc``
-section
-
-::
-
-    enable_service heat h-api h-api-cfn h-api-cw h-eng
-
-Heat can also run in standalone mode, and be configured to orchestrate
-on an external OpenStack cloud. To launch only Heat in standalone mode
-you'll need the following settings in your ``localrc`` section
-
-::
-
-    disable_all_services
-    enable_service rabbit mysql heat h-api h-api-cfn h-api-cw h-eng
-    HEAT_STANDALONE=True
-    KEYSTONE_SERVICE_HOST=...
-    KEYSTONE_AUTH_HOST=...
-
 Tempest
 ~~~~~~~
 
@@ -796,3 +752,60 @@
     ::
 
         TERMINATE_TIMEOUT=30
+
+
+.. _arch-configuration:
+
+Architectures
+-------------
+
+The upstream CI runs exclusively on nodes with x86 architectures, but
+OpenStack supports even more architectures. Some of them need to configure
+Devstack in a certain way.
+
+KVM on s390x (IBM z Systems)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+KVM on s390x (IBM z Systems) is supported since the *Kilo* release. For
+an all-in-one setup, these minimal settings in the ``local.conf`` file
+are needed::
+
+    [[local|localrc]]
+    ADMIN_PASSWORD=secret
+    DATABASE_PASSWORD=$ADMIN_PASSWORD
+    RABBIT_PASSWORD=$ADMIN_PASSWORD
+    SERVICE_PASSWORD=$ADMIN_PASSWORD
+
+    DOWNLOAD_DEFAULT_IMAGES=False
+    IMAGE_URLS="https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-s390x-disk1.img"
+
+    enable_service n-sproxy
+    disable_service n-novnc
+    disable_service etcd3  # https://bugs.launchpad.net/devstack/+bug/1693192
+
+    [[post-config|$NOVA_CONF]]
+
+    [serial_console]
+    base_url=ws://$HOST_IP:6083/  # optional
+
+Reasoning:
+
+* The default image of Devstack is x86 only, so we deactivate the download
+  with ``DOWNLOAD_DEFAULT_IMAGES``. The referenced guest image
+  in the code above (``IMAGE_URLS``) serves as an example. The list of
+  possible s390x guest images is not limited to that.
+
+* This platform doesn't support a graphical console like VNC or SPICE.
+  The technical reason is the missing framebuffer on the platform. This
+  means we rely on the substitute feature *serial console* which needs the
+  proxy service ``n-sproxy``. We also disable VNC's proxy ``n-novnc`` for
+  that reason . The configuration in the ``post-config`` section is only
+  needed if you want to use the *serial console* outside of the all-in-one
+  setup.
+
+* The service ``etcd3`` needs to be disabled as long as bug report
+  https://bugs.launchpad.net/devstack/+bug/1693192 is not resolved.
+
+.. note:: To run *Tempest* against this *Devstack* all-in-one, you'll need
+   to use a guest image which is smaller than 1GB when uncompressed.
+   The example image from above is bigger than that!
diff --git a/doc/source/development.rst b/doc/source/development.rst
index 776ac6c..957de9b 100644
--- a/doc/source/development.rst
+++ b/doc/source/development.rst
@@ -8,56 +8,33 @@
 Inspecting Services
 ===================
 
-By default most services in DevStack are running in a `screen
-<https://www.gnu.org/software/screen/manual/screen.html>`_
-session.
+By default most services in DevStack are running as `systemd` units
+named `devstack@$servicename.service`. You can see running services
+with.
 
 .. code-block:: bash
 
-   os3:~> screen -list
-   There is a screen on:
-        28994.stack	(08/10/2016 09:01:33 PM)	(Detached)
-   1 Socket in /var/run/screen/S-sdague.
+   sudo systemctl status "devstack@*"
 
-You can attach to this screen session using ``screen -r`` which gives
-you a view of the services in action.
-
-.. image:: assets/images/screen_session_1.png
-   :width: 100%
-
-Basic Screen Commands
----------------------
-
-The following minimal commands will be useful to using screen:
-
-* ``ctrl-a n`` - go to next window. Next is assumed to be right of
-  current window.
-* ``ctrl-a p`` - go to previous window. Previous is assumed to be left
-  of current window.
-* ``ctrl-a [`` - entry copy/scrollback mode. This allows you to
-  navigate back through the logs with the up arrow.
-* ``ctrl-a d`` - detach from screen. Gets you back to a normal
-  terminal, while leaving everything running.
-
-For more about using screen, see the excellent `screen manual
-<https://www.gnu.org/software/screen/manual/screen.html>`_.
+To learn more about the basics of systemd, see :doc:`/systemd`
 
 Patching a Service
 ==================
 
 If you want to make a quick change to a running service the easiest
-way to do this is:
+way to do that is to change the code directly in /opt/stack/$service
+and then restart the affected daemons.
 
-* attach to screen
-* navigate to the window in question
-* ``ctrl-c`` to kill the service
-* make appropriate changes to the code
-* ``up arrow`` in the screen window to display the command used to run
-  that service
-* ``enter`` to restart the service
+.. code-block:: bash
 
-This works for services, except those running under Apache (currently
-just ``keystone`` by default).
+   sudo systemctl restart devstack@n-cpu.service
+
+If your change impacts more than one daemon you can restart by
+wildcard as well.
+
+.. code-block:: bash
+
+   sudo systemctl restart "devstack@n-*"
 
 .. warning::
 
@@ -102,14 +79,6 @@
    NOVA_BRANCH=refs/changes/10/353710/1
 
 
-Testing Changes to Apache Based Services
-========================================
-
-When testing changes to Apache based services, such as ``keystone``,
-you can either use the Testing a Patch Series approach above, or make
-changes in the code tree and issue an apache restart.
-
-
 Testing Changes to Libraries
 ============================
 
@@ -132,9 +101,17 @@
    OSLOPOLICY_REPO=/home/sdague/oslo.policy
    OSLOPOLICY_BRANCH=better_exception
 
-Because libraries are used by many services, library changes really
-need to go through a full ``./unstack.sh && ./stack.sh`` to see your
-changes in action.
+As libraries are not installed `editable` by pip, after you make any
+local changes you will need to:
 
-To figure out the repo / branch names for every library that's
-supported, you'll need to read the devstack source.
+* cd to top of library path
+* sudo pip install -U .
+* restart all services you want to use the new library
+
+You can do that with wildcards such as
+
+.. code-block:: bash
+
+   sudo systemctl restart "devstack@n-*"
+
+which will restart all nova services.
diff --git a/doc/source/faq.rst b/doc/source/faq.rst
index f03304f..a186336 100644
--- a/doc/source/faq.rst
+++ b/doc/source/faq.rst
@@ -41,8 +41,9 @@
 ~~~~~~~~~~~~~~~~~~~~~
 
 Unlike packages, DevStack leaves your cloud ready to develop -
-checkouts of the code and services running in screen. However, many
-people are doing the hard work of packaging and recipes for production
+checkouts of the code and services running locally under systemd,
+making it easy to hack on and test new patches. However, many people
+are doing the hard work of packaging and recipes for production
 deployments.
 
 Why isn't $MY\_FAVORITE\_DISTRO supported?
@@ -158,16 +159,6 @@
 often good enough for a single-node installation. And in an extreme
 case, use ``clean.sh`` to eradicate it and try again.
 
-Configure ``local.conf`` thusly:
-
-    ::
-
-        [[local|localrc]]
-        HEAT_STANDALONE=True
-        ENABLED_SERVICES=rabbit,mysql,heat,h-api,h-api-cfn,h-api-cw,h-eng
-        KEYSTONE_SERVICE_HOST=<keystone-host>
-        KEYSTONE_AUTH_HOST=<keystone-host>
-
 Why are my configuration changes ignored?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/doc/source/guides/devstack-with-lbaas-v2.rst b/doc/source/guides/devstack-with-lbaas-v2.rst
index 21bea99..4ed64bf 100644
--- a/doc/source/guides/devstack-with-lbaas-v2.rst
+++ b/doc/source/guides/devstack-with-lbaas-v2.rst
@@ -45,7 +45,7 @@
     # Horizon
     ENABLED_SERVICES+=,horizon
     # Nova
-    ENABLED_SERVICES+=,n-api,n-crt,n-cpu,n-cond,n-sch
+    ENABLED_SERVICES+=,n-api,n-cpu,n-cond,n-sch
     # Glance
     ENABLED_SERVICES+=,g-api,g-reg
     # Neutron
diff --git a/doc/source/index.rst b/doc/source/index.rst
index cbd6971..47087c5 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -142,3 +142,12 @@
 Get :doc:`the big picture <overview>` of what we are trying to do
 with devstack, and help us by :doc:`contributing to the project
 <hacking>`.
+
+Contents
+--------
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   *
diff --git a/doc/source/overview.rst b/doc/source/overview.rst
index d245035..c07a8e6 100644
--- a/doc/source/overview.rst
+++ b/doc/source/overview.rst
@@ -20,11 +20,11 @@
 
 *The OpenStack Technical Committee (TC) has defined the current CI
 strategy to include the latest Ubuntu release and the latest RHEL
-release (for Python 2.6 testing).*
+release.*
 
 -  Ubuntu: current LTS release plus current development release
 -  Fedora: current release plus previous release
--  RHEL: current major release
+-  RHEL/Centos: current major release
 -  Other OS platforms may continue to be included but the maintenance of
    those platforms shall not be assumed simply due to their presence.
    Having a listed point-of-contact for each additional OS will greatly
@@ -38,7 +38,6 @@
 *As packaged by the host OS*
 
 -  MySQL
--  PostgreSQL
 
 Queues
 ------
@@ -46,7 +45,6 @@
 *As packaged by the host OS*
 
 -  Rabbit
--  Qpid
 
 Web Server
 ----------
@@ -58,9 +56,6 @@
 OpenStack Network
 -----------------
 
-*Defaults to nova network, optionally use neutron*
-
--  Nova Network: FlatDHCP
 -  Neutron: A basic configuration approximating the original FlatDHCP
    mode using linuxbridge or OpenVSwitch.
 
@@ -68,9 +63,8 @@
 --------
 
 The default services configured by DevStack are Identity (keystone),
-Object Storage (swift), Image Service (glance), Block Storage (cinder),
-Compute (nova), Networking (nova), Dashboard (horizon), Orchestration
-(heat)
+Object Storage (swift), Image Service (glance), Block Storage
+(cinder), Compute (nova), Networking (neutron), Dashboard (horizon)
 
 Additional services not included directly in DevStack can be tied in to
 ``stack.sh`` using the :doc:`plugin mechanism <plugins>` to call
@@ -80,8 +74,7 @@
 -------------------
 
 -  single node
--  multi-node is not tested regularly by the core team, and even then
-   only minimal configurations are reviewed
+-  multi-node configurations as are tested by the gate
 
 Exercises
 ---------
diff --git a/doc/source/plugin-registry.rst b/doc/source/plugin-registry.rst
index 96a2733..92e5ecd 100644
--- a/doc/source/plugin-registry.rst
+++ b/doc/source/plugin-registry.rst
@@ -48,10 +48,12 @@
 devstack-plugin-glusterfs              `git://git.openstack.org/openstack/devstack-plugin-glusterfs <https://git.openstack.org/cgit/openstack/devstack-plugin-glusterfs>`__
 devstack-plugin-hdfs                   `git://git.openstack.org/openstack/devstack-plugin-hdfs <https://git.openstack.org/cgit/openstack/devstack-plugin-hdfs>`__
 devstack-plugin-kafka                  `git://git.openstack.org/openstack/devstack-plugin-kafka <https://git.openstack.org/cgit/openstack/devstack-plugin-kafka>`__
+devstack-plugin-libvirt-qemu           `git://git.openstack.org/openstack/devstack-plugin-libvirt-qemu <https://git.openstack.org/cgit/openstack/devstack-plugin-libvirt-qemu>`__
 devstack-plugin-mariadb                `git://git.openstack.org/openstack/devstack-plugin-mariadb <https://git.openstack.org/cgit/openstack/devstack-plugin-mariadb>`__
 devstack-plugin-nfs                    `git://git.openstack.org/openstack/devstack-plugin-nfs <https://git.openstack.org/cgit/openstack/devstack-plugin-nfs>`__
 devstack-plugin-pika                   `git://git.openstack.org/openstack/devstack-plugin-pika <https://git.openstack.org/cgit/openstack/devstack-plugin-pika>`__
 devstack-plugin-sheepdog               `git://git.openstack.org/openstack/devstack-plugin-sheepdog <https://git.openstack.org/cgit/openstack/devstack-plugin-sheepdog>`__
+devstack-plugin-vmax                   `git://git.openstack.org/openstack/devstack-plugin-vmax <https://git.openstack.org/cgit/openstack/devstack-plugin-vmax>`__
 devstack-plugin-zmq                    `git://git.openstack.org/openstack/devstack-plugin-zmq <https://git.openstack.org/cgit/openstack/devstack-plugin-zmq>`__
 dragonflow                             `git://git.openstack.org/openstack/dragonflow <https://git.openstack.org/cgit/openstack/dragonflow>`__
 drbd-devstack                          `git://git.openstack.org/openstack/drbd-devstack <https://git.openstack.org/cgit/openstack/drbd-devstack>`__
@@ -62,7 +64,6 @@
 fuxi                                   `git://git.openstack.org/openstack/fuxi <https://git.openstack.org/cgit/openstack/fuxi>`__
 gce-api                                `git://git.openstack.org/openstack/gce-api <https://git.openstack.org/cgit/openstack/gce-api>`__
 glare                                  `git://git.openstack.org/openstack/glare <https://git.openstack.org/cgit/openstack/glare>`__
-gnocchi                                `git://git.openstack.org/openstack/gnocchi <https://git.openstack.org/cgit/openstack/gnocchi>`__
 group-based-policy                     `git://git.openstack.org/openstack/group-based-policy <https://git.openstack.org/cgit/openstack/group-based-policy>`__
 heat                                   `git://git.openstack.org/openstack/heat <https://git.openstack.org/cgit/openstack/heat>`__
 horizon-mellanox                       `git://git.openstack.org/openstack/horizon-mellanox <https://git.openstack.org/cgit/openstack/horizon-mellanox>`__
@@ -123,9 +124,11 @@
 neutron                                `git://git.openstack.org/openstack/neutron <https://git.openstack.org/cgit/openstack/neutron>`__
 neutron-dynamic-routing                `git://git.openstack.org/openstack/neutron-dynamic-routing <https://git.openstack.org/cgit/openstack/neutron-dynamic-routing>`__
 neutron-fwaas                          `git://git.openstack.org/openstack/neutron-fwaas <https://git.openstack.org/cgit/openstack/neutron-fwaas>`__
+neutron-fwaas-dashboard                `git://git.openstack.org/openstack/neutron-fwaas-dashboard <https://git.openstack.org/cgit/openstack/neutron-fwaas-dashboard>`__
 neutron-lbaas                          `git://git.openstack.org/openstack/neutron-lbaas <https://git.openstack.org/cgit/openstack/neutron-lbaas>`__
 neutron-lbaas-dashboard                `git://git.openstack.org/openstack/neutron-lbaas-dashboard <https://git.openstack.org/cgit/openstack/neutron-lbaas-dashboard>`__
 neutron-vpnaas                         `git://git.openstack.org/openstack/neutron-vpnaas <https://git.openstack.org/cgit/openstack/neutron-vpnaas>`__
+neutron-vpnaas-dashboard               `git://git.openstack.org/openstack/neutron-vpnaas-dashboard <https://git.openstack.org/cgit/openstack/neutron-vpnaas-dashboard>`__
 nova-dpm                               `git://git.openstack.org/openstack/nova-dpm <https://git.openstack.org/cgit/openstack/nova-dpm>`__
 nova-lxd                               `git://git.openstack.org/openstack/nova-lxd <https://git.openstack.org/cgit/openstack/nova-lxd>`__
 nova-mksproxy                          `git://git.openstack.org/openstack/nova-mksproxy <https://git.openstack.org/cgit/openstack/nova-mksproxy>`__
@@ -136,6 +139,7 @@
 os-xenapi                              `git://git.openstack.org/openstack/os-xenapi <https://git.openstack.org/cgit/openstack/os-xenapi>`__
 osprofiler                             `git://git.openstack.org/openstack/osprofiler <https://git.openstack.org/cgit/openstack/osprofiler>`__
 panko                                  `git://git.openstack.org/openstack/panko <https://git.openstack.org/cgit/openstack/panko>`__
+patrole                                `git://git.openstack.org/openstack/patrole <https://git.openstack.org/cgit/openstack/patrole>`__
 picasso                                `git://git.openstack.org/openstack/picasso <https://git.openstack.org/cgit/openstack/picasso>`__
 rally                                  `git://git.openstack.org/openstack/rally <https://git.openstack.org/cgit/openstack/rally>`__
 sahara                                 `git://git.openstack.org/openstack/sahara <https://git.openstack.org/cgit/openstack/sahara>`__
@@ -145,12 +149,15 @@
 searchlight-ui                         `git://git.openstack.org/openstack/searchlight-ui <https://git.openstack.org/cgit/openstack/searchlight-ui>`__
 senlin                                 `git://git.openstack.org/openstack/senlin <https://git.openstack.org/cgit/openstack/senlin>`__
 solum                                  `git://git.openstack.org/openstack/solum <https://git.openstack.org/cgit/openstack/solum>`__
+stackube                               `git://git.openstack.org/openstack/stackube <https://git.openstack.org/cgit/openstack/stackube>`__
 tacker                                 `git://git.openstack.org/openstack/tacker <https://git.openstack.org/cgit/openstack/tacker>`__
 tap-as-a-service                       `git://git.openstack.org/openstack/tap-as-a-service <https://git.openstack.org/cgit/openstack/tap-as-a-service>`__
+tap-as-a-service-dashboard             `git://git.openstack.org/openstack/tap-as-a-service-dashboard <https://git.openstack.org/cgit/openstack/tap-as-a-service-dashboard>`__
 tricircle                              `git://git.openstack.org/openstack/tricircle <https://git.openstack.org/cgit/openstack/tricircle>`__
 trio2o                                 `git://git.openstack.org/openstack/trio2o <https://git.openstack.org/cgit/openstack/trio2o>`__
 trove                                  `git://git.openstack.org/openstack/trove <https://git.openstack.org/cgit/openstack/trove>`__
 trove-dashboard                        `git://git.openstack.org/openstack/trove-dashboard <https://git.openstack.org/cgit/openstack/trove-dashboard>`__
+valet                                  `git://git.openstack.org/openstack/valet <https://git.openstack.org/cgit/openstack/valet>`__
 vitrage                                `git://git.openstack.org/openstack/vitrage <https://git.openstack.org/cgit/openstack/vitrage>`__
 vitrage-dashboard                      `git://git.openstack.org/openstack/vitrage-dashboard <https://git.openstack.org/cgit/openstack/vitrage-dashboard>`__
 vmware-nsx                             `git://git.openstack.org/openstack/vmware-nsx <https://git.openstack.org/cgit/openstack/vmware-nsx>`__
diff --git a/doc/source/site-map.rst b/doc/source/site-map.rst
deleted file mode 100644
index 801fc66..0000000
--- a/doc/source/site-map.rst
+++ /dev/null
@@ -1,23 +0,0 @@
-:orphan:
-
-.. the TOC on the front page actually makes the document a lot more
-   confusing. This lets us bury a toc which we can link in when
-   appropriate.
-
-==========
- Site Map
-==========
-
-.. toctree::
-   :glob:
-   :maxdepth: 3
-
-   overview
-   configuration
-   networking
-   plugins
-   plugin-registry
-   faq
-   development
-   hacking
-   guides
diff --git a/SYSTEMD.rst b/doc/source/systemd.rst
similarity index 81%
rename from SYSTEMD.rst
rename to doc/source/systemd.rst
index 729fdf4..60a7719 100644
--- a/SYSTEMD.rst
+++ b/doc/source/systemd.rst
@@ -2,12 +2,7 @@
  Using Systemd in DevStack
 ===========================
 
-.. note::
-
-   This is an in progress document as we work out the way forward here
-   with DevStack and systemd.
-
-DevStack can be run with all the services as systemd unit
+By default DevStack is run with all the services as systemd unit
 files. Systemd is now the default init system for nearly every Linux
 distro, and systemd encodes and solves many of the problems related to
 poorly running processes.
@@ -25,12 +20,6 @@
 than one service, and needing to restart a bunch of services for that
 to take effect.
 
-To enable this add the following to your local.conf::
-
-  USE_SYSTEMD=True
-
-
-
 Unit Structure
 ==============
 
@@ -42,8 +31,9 @@
    code is left in place in case we can switch back later.
 
 All DevStack user units are created as a part of the DevStack slice
-given the name ``devstack@$servicename.service``. This lets us do
-certain operations at the slice level.
+given the name ``devstack@$servicename.service``. This makes it easy
+to understand which services are part of the devstack run, and lets us
+disable / stop them in a single command.
 
 Manipulating Units
 ==================
@@ -89,6 +79,8 @@
 We'll eventually make the unit names a bit more meaningful so that
 it's easier to understand what you are restarting.
 
+.. _journalctl-examples:
+
 Querying Logs
 =============
 
@@ -117,6 +109,17 @@
 
   journalctl -f -o short-precise --unit devstack@n-cpu.service
 
+By default, journalctl strips out "unprintable" characters, including
+ASCII color codes. To keep the color codes (which can be interpreted by
+an appropriate terminal/pager - e.g. ``less``, the default)::
+
+  journalctl -a --unit devstack@n-cpu.service
+
+When outputting to the terminal using the default pager, long lines
+appear to be truncated, but horizontal scrolling is supported via the
+left/right arrow keys.
+
+See ``man 1 journalctl`` for more.
 
 Known Issues
 ============
@@ -126,7 +129,7 @@
 into the ``systemd`` namespace, which can cause some issues.
 
 - ``systemd-python`` - this is the upstream maintained library, it has
-  a version number like systemd itself (currently ``233``). This is
+  a version number like systemd itself (currently ``234``). This is
   the one you want.
 - ``systemd`` - a python 3 only library, not what you want.
 - ``python-systemd`` - another library you don't want. Installing it
@@ -145,27 +148,6 @@
 Future Work
 ===========
 
-oslo.log journald
------------------
-
-Journald has an extremely rich mechanism for direct logging including
-structured metadata. We should enhance oslo.log to take advantage of
-that. It would let us do things like::
-
-  journalctl REQUEST_ID=......
-
-  journalctl INSTANCE_ID=......
-
-And get all lines related to the request id or instance id. (Note:
-this work has been started at https://review.openstack.org/#/c/451525/)
-
-log colorizing
---------------
-
-We lose log colorization through this process. We might want to build
-a custom colorizer that we could run journalctl output through
-optionally for people.
-
 user units
 ----------
 
diff --git a/files/debs/general b/files/debs/general
index 20490c6..1dde03b 100644
--- a/files/debs/general
+++ b/files/debs/general
@@ -17,6 +17,7 @@
 libmysqlclient-dev  # MySQL-python
 libpq-dev  # psycopg2
 libssl-dev # for pyOpenSSL
+libsystemd-dev # for systemd-python
 libxml2-dev  # lxml
 libxslt1-dev  # lxml
 libyaml-dev
@@ -26,10 +27,8 @@
 pkg-config
 psmisc
 python2.7
-python3-systemd
 python-dev
 python-gdbm # needed for testr
-python-systemd
 screen
 tar
 tcpdump
diff --git a/files/debs/n-api b/files/debs/n-api
deleted file mode 100644
index 0928cd5..0000000
--- a/files/debs/n-api
+++ /dev/null
@@ -1 +0,0 @@
-fping
diff --git a/files/debs/zookeeper b/files/debs/zookeeper
deleted file mode 100644
index f41b559..0000000
--- a/files/debs/zookeeper
+++ /dev/null
@@ -1 +0,0 @@
-zookeeperd
diff --git a/files/rpms-suse/general b/files/rpms-suse/general
index 1044c25..370f240 100644
--- a/files/rpms-suse/general
+++ b/files/rpms-suse/general
@@ -1,3 +1,5 @@
+apache2
+apache2-devel
 bc
 bridge-utils
 ca-certificates-mozilla
@@ -23,9 +25,11 @@
 python-devel  # pyOpenSSL
 python-xml
 screen
+systemd-devel # for systemd-python
 tar
 tcpdump
 unzip
 util-linux
 wget
+which
 zlib-devel
diff --git a/files/rpms-suse/n-api b/files/rpms-suse/n-api
index af5ac2f..0f08daa 100644
--- a/files/rpms-suse/n-api
+++ b/files/rpms-suse/n-api
@@ -1,2 +1 @@
-fping
 python-dateutil
diff --git a/files/rpms/general b/files/rpms/general
index 106aa6a..1393d18 100644
--- a/files/rpms/general
+++ b/files/rpms/general
@@ -29,7 +29,7 @@
 python-devel
 redhat-rpm-config # missing dep for gcc hardening flags, see rhbz#1217376
 screen
-systemd-python
+systemd-devel # for systemd-python
 tar
 tcpdump
 unzip
diff --git a/files/rpms/n-api b/files/rpms/n-api
deleted file mode 100644
index 0928cd5..0000000
--- a/files/rpms/n-api
+++ /dev/null
@@ -1 +0,0 @@
-fping
diff --git a/files/rpms/zookeeper b/files/rpms/zookeeper
deleted file mode 100644
index 1bfac53..0000000
--- a/files/rpms/zookeeper
+++ /dev/null
@@ -1 +0,0 @@
-zookeeper
diff --git a/files/zookeeper/environment b/files/zookeeper/environment
deleted file mode 100644
index afa2d2f..0000000
--- a/files/zookeeper/environment
+++ /dev/null
@@ -1,36 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Modified from http://packages.ubuntu.com/saucy/zookeeperd
-NAME=zookeeper
-ZOOCFGDIR=/etc/zookeeper/conf
-
-# seems, that log4j requires the log4j.properties file to be in the classpath
-CLASSPATH="$ZOOCFGDIR:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar"
-
-ZOOCFG="$ZOOCFGDIR/zoo.cfg"
-ZOO_LOG_DIR=/var/log/zookeeper
-USER=$NAME
-GROUP=$NAME
-PIDDIR=/var/run/$NAME
-PIDFILE=$PIDDIR/$NAME.pid
-SCRIPTNAME=/etc/init.d/$NAME
-JAVA=/usr/bin/java
-ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
-ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
-JMXLOCALONLY=false
-JAVA_OPTS=""
diff --git a/files/zookeeper/log4j.properties b/files/zookeeper/log4j.properties
deleted file mode 100644
index 6c45a4a..0000000
--- a/files/zookeeper/log4j.properties
+++ /dev/null
@@ -1,69 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# From http://packages.ubuntu.com/saucy/zookeeperd
-
-# ZooKeeper Logging Configuration
-#
-
-# Format is "<default threshold> (, <appender>)+
-
-log4j.rootLogger=${zookeeper.root.logger}
-
-# Example: console appender only
-# log4j.rootLogger=INFO, CONSOLE
-
-# Example with rolling log file
-#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE
-
-# Example with rolling log file and tracing
-#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE
-
-#
-# Log INFO level and above messages to the console
-#
-log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
-log4j.appender.CONSOLE.Threshold=INFO
-log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
-log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
-
-#
-# Add ROLLINGFILE to rootLogger to get log file output
-#    Log DEBUG level and above messages to a log file
-log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
-log4j.appender.ROLLINGFILE.Threshold=WARN
-log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log
-
-# Max log file size of 10MB
-log4j.appender.ROLLINGFILE.MaxFileSize=10MB
-# uncomment the next line to limit number of backup files
-#log4j.appender.ROLLINGFILE.MaxBackupIndex=10
-
-log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
-log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
-
-
-#
-# Add TRACEFILE to rootLogger to get log file output
-#    Log DEBUG level and above messages to a log file
-log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
-log4j.appender.TRACEFILE.Threshold=TRACE
-log4j.appender.TRACEFILE.File=${zookeeper.log.dir}/zookeeper_trace.log
-
-log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
-### Notice we are including log4j's NDC here (%x)
-log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n
diff --git a/files/zookeeper/myid b/files/zookeeper/myid
deleted file mode 100644
index c227083..0000000
--- a/files/zookeeper/myid
+++ /dev/null
@@ -1 +0,0 @@
-0
\ No newline at end of file
diff --git a/files/zookeeper/zoo.cfg b/files/zookeeper/zoo.cfg
deleted file mode 100644
index b8f5582..0000000
--- a/files/zookeeper/zoo.cfg
+++ /dev/null
@@ -1,74 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
-
-# The number of milliseconds of each tick
-tickTime=2000
-# The number of ticks that the initial
-# synchronization phase can take
-initLimit=10
-# The number of ticks that can pass between
-# sending a request and getting an acknowledgement
-syncLimit=5
-# the directory where the snapshot is stored.
-dataDir=/var/lib/zookeeper
-# Place the dataLogDir to a separate physical disc for better performance
-# dataLogDir=/disk2/zookeeper
-
-# the port at which the clients will connect
-clientPort=2181
-
-# Maximum number of clients that can connect from one client
-maxClientCnxns=60
-
-# specify all zookeeper servers
-# The fist port is used by followers to connect to the leader
-# The second one is used for leader election
-
-server.0=127.0.0.1:2888:3888
-
-# To avoid seeks ZooKeeper allocates space in the transaction log file in
-# blocks of preAllocSize kilobytes. The default block size is 64M. One reason
-# for changing the size of the blocks is to reduce the block size if snapshots
-# are taken more often. (Also, see snapCount).
-#preAllocSize=65536
-
-# Clients can submit requests faster than ZooKeeper can process them,
-# especially if there are a lot of clients. To prevent ZooKeeper from running
-# out of memory due to queued requests, ZooKeeper will throttle clients so that
-# there is no more than globalOutstandingLimit outstanding requests in the
-# system. The default limit is 1,000.ZooKeeper logs transactions to a
-# transaction log. After snapCount transactions are written to a log file a
-# snapshot is started and a new transaction log file is started. The default
-# snapCount is 10,000.
-#snapCount=1000
-
-# If this option is defined, requests will be will logged to a trace file named
-# traceFile.year.month.day.
-#traceFile=
-
-# Leader accepts client connections. Default value is "yes". The leader machine
-# coordinates updates. For higher update throughput at thes slight expense of
-# read throughput the leader can be configured to not accept clients and focus
-# on coordination.
-#leaderServes=yes
-
-# Autopurge every hour to avoid using lots of disk in bursts
-# Order of the next 2 properties matters.
-# autopurge.snapRetainCount must be before autopurge.purgeInterval.
-autopurge.snapRetainCount=3
-autopurge.purgeInterval=1
\ No newline at end of file
diff --git a/functions b/functions
index c99e435..738040d 100644
--- a/functions
+++ b/functions
@@ -310,6 +310,11 @@
             disk_format=qcow2
             container_format=bare
             ;;
+        *.raw)
+            image_name=$(basename "$image" ".raw")
+            disk_format=raw
+            container_format=bare
+            ;;
         *.iso)
             image_name=$(basename "$image" ".iso")
             disk_format=iso
@@ -318,7 +323,7 @@
         *.vhd|*.vhdx|*.vhd.gz|*.vhdx.gz)
             local extension="${image_fname#*.}"
             image_name=$(basename "$image" ".$extension")
-            disk_format=vhd
+            disk_format=$(echo $image_fname | grep -oP '(?<=\.)vhdx?(?=\.|$)')
             container_format=bare
             if [ "${image_fname##*.}" == "gz" ]; then
                 unpack=zcat
@@ -606,14 +611,23 @@
 function setup_systemd_logging {
     local conf_file=$1
     local conf_section="DEFAULT"
-    iniset $conf_file $conf_section use_journal "True"
-    iniset $conf_file $conf_section logging_context_format_string \
-           "%(levelname)s %(name)s [%(request_id)s %(project_name)s %(user_name)s] %(instance)s%(message)s"
-    iniset $conf_file $conf_section logging_default_format_string \
-           "%(levelname)s %(name)s [-] %(instance)s%(color)s%(message)s"
-    iniset $conf_file $conf_section logging_debug_format_suffix \
-           "from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d"
-    iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s %(instance)s"
+    # NOTE(sdague): this is a nice to have, and means we're using the
+    # native systemd path, which provides for things like search on
+    # request-id. However, there may be an eventlet interaction here,
+    # so going off for now.
+    USE_JOURNAL=$(trueorfalse False USE_JOURNAL)
+    local pidstr=""
+    if [[ "$USE_JOURNAL" == "True" ]]; then
+        iniset $conf_file $conf_section use_journal "True"
+        # if we are using the journal directly, our process id is already correct
+    else
+        pidstr="(pid=%(process)d) "
+    fi
+    iniset $conf_file $conf_section logging_debug_format_suffix "[00;33m{{${pidstr}%(funcName)s %(pathname)s:%(lineno)d}}[00m"
+
+    iniset $conf_file $conf_section logging_context_format_string "%(color)s%(levelname)s %(name)s [[01;36m%(global_request_id)s %(request_id)s [00;36m%(project_name)s %(user_name)s%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m"
+    iniset $conf_file $conf_section logging_default_format_string "%(color)s%(levelname)s %(name)s [[00;36m-%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m"
+    iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s [01;35m%(instance)s[00m"
 }
 
 function setup_standard_logging_identity {
@@ -703,6 +717,55 @@
 }
 
 
+# Set a systemd system override
+#
+# This sets a system-side override in system.conf. A per-service
+# override would be /etc/systemd/system/${service}.service/override.conf
+function set_systemd_override {
+    local key="$1"
+    local value="$2"
+
+    local sysconf="/etc/systemd/system.conf"
+    iniset -sudo "${sysconf}" "Manager" "$key" "$value"
+    echo "Set systemd system override for ${key}=${value}"
+
+    sudo systemctl daemon-reload
+}
+
+# Get a random port from the local port range
+#
+# This function returns an available port in the local port range. The search
+# order is not truly random, but should be considered a random value by the
+# user because it depends on the state of your local system.
+function get_random_port {
+    read lower_port upper_port < /proc/sys/net/ipv4/ip_local_port_range
+    while true; do
+        for (( port = upper_port ; port >= lower_port ; port-- )); do
+            sudo lsof -i ":$port" &> /dev/null
+            if [[ $? > 0 ]] ; then
+                break 2
+            fi
+        done
+    done
+    echo $port
+}
+
+
+function write_devstack_version {
+    pushd $TOP_DIR
+    local git_version=""
+    git_version=$(git log --format="%H %s %ci" -1)
+    cat - > /tmp/devstack-version <<EOF
+#!/bin/bash
+
+echo "DevStack Version: ${DEVSTACK_SERIES} - ${git_version}"
+echo "OS Version: ${os_VENDOR} ${os_RELEASE} ${os_CODENAME}"
+
+EOF
+    sudo install -m 755 /tmp/devstack-version /usr/local/bin/devstack-version
+    rm /tmp/devstack-version
+}
+
 # Restore xtrace
 $_XTRACE_FUNCTIONS
 
diff --git a/functions-common b/functions-common
index 35b4860..660df79 100644
--- a/functions-common
+++ b/functions-common
@@ -48,8 +48,8 @@
 
 # Save these variables to .stackenv
 STACK_ENV_VARS="BASE_SQL_CONN DATA_DIR DEST ENABLED_SERVICES HOST_IP \
-    KEYSTONE_AUTH_PROTOCOL KEYSTONE_AUTH_URI KEYSTONE_SERVICE_URI \
-    LOGFILE OS_CACERT SERVICE_HOST SERVICE_PROTOCOL STACK_USER TLS_IP \
+    KEYSTONE_AUTH_URI KEYSTONE_SERVICE_URI \
+    LOGFILE OS_CACERT SERVICE_HOST STACK_USER TLS_IP \
     HOST_IPV6 SERVICE_IP_VERSION"
 
 
@@ -93,7 +93,7 @@
         --os-region-name $REGION_NAME \
         --os-identity-api-version 3 \
         $CA_CERT_ARG \
-        --os-auth-url $KEYSTONE_AUTH_URI \
+        --os-auth-url $KEYSTONE_SERVICE_URI \
         --os-username demo \
         --os-password $ADMIN_PASSWORD \
         --os-project-name demo
@@ -105,7 +105,7 @@
         --os-region-name $REGION_NAME \
         --os-identity-api-version 3 \
         $CA_CERT_ARG \
-        --os-auth-url $KEYSTONE_AUTH_URI \
+        --os-auth-url $KEYSTONE_SERVICE_URI \
         --os-username alt_demo \
         --os-password $ADMIN_PASSWORD \
         --os-project-name alt_demo
@@ -117,7 +117,7 @@
         --os-region-name $REGION_NAME \
         --os-identity-api-version 3 \
         $CA_CERT_ARG \
-        --os-auth-url $KEYSTONE_AUTH_URI \
+        --os-auth-url $KEYSTONE_SERVICE_URI \
         --os-username admin \
         --os-password $ADMIN_PASSWORD \
         --os-project-name admin
@@ -864,10 +864,11 @@
 
     # Gets user role id
     user_role_id=$(openstack role assignment list \
+        --role $1 \
         --user $2 \
         --project $3 \
         $domain_args \
-        | grep " $1 " | get_field 1)
+        | grep '^|\s[a-f0-9]\+' | get_field 1)
     if [[ -z "$user_role_id" ]]; then
         # Adds role to user and get it
         openstack role add $1 \
@@ -875,10 +876,11 @@
             --project $3 \
             $domain_args
         user_role_id=$(openstack role assignment list \
+            --role $1 \
             --user $2 \
             --project $3 \
             $domain_args \
-            | grep " $1 " | get_field 1)
+            | grep '^|\s[a-f0-9]\+' | get_field 1)
     fi
     echo $user_role_id
 }
@@ -889,18 +891,20 @@
     local user_role_id
     # Gets user role id
     user_role_id=$(openstack role assignment list \
+        --role $1 \
         --user $2 \
         --domain $3 \
-        | grep " $1 " | get_field 1)
+        | grep '^|\s[a-f0-9]\+' | get_field 1)
     if [[ -z "$user_role_id" ]]; then
         # Adds role to user and get it
         openstack role add $1 \
             --user $2 \
             --domain $3
         user_role_id=$(openstack role assignment list \
+            --role $1 \
             --user $2 \
             --domain $3 \
-            | grep " $1 " | get_field 1)
+            | grep '^|\s[a-f0-9]\+' | get_field 1)
     fi
     echo $user_role_id
 }
@@ -911,6 +915,7 @@
     local group_role_id
     # Gets group role id
     group_role_id=$(openstack role assignment list \
+        --role $1 \
         --group $2 \
         --project $3 \
         -f value)
@@ -920,6 +925,7 @@
             --group $2 \
             --project $3
         group_role_id=$(openstack role assignment list \
+            --role $1 \
             --group $2 \
             --project $3 \
             -f value)
@@ -1463,11 +1469,12 @@
     mkdir -p $SYSTEMD_DIR
 
     iniset -sudo $unitfile "Unit" "Description" "Devstack $service"
+    iniset -sudo $unitfile "Service" "SyslogIdentifier" "$service"
     iniset -sudo $unitfile "Service" "User" "$user"
     iniset -sudo $unitfile "Service" "ExecStart" "$command"
     iniset -sudo $unitfile "Service" "Type" "notify"
     iniset -sudo $unitfile "Service" "KillSignal" "SIGQUIT"
-    iniset -sudo $unitfile "Service" "Restart" "Always"
+    iniset -sudo $unitfile "Service" "Restart" "always"
     iniset -sudo $unitfile "Service" "NotifyAccess" "all"
     iniset -sudo $unitfile "Service" "RestartForceExitStatus" "100"
 
@@ -1480,10 +1487,41 @@
     $SYSTEMCTL daemon-reload
 }
 
+function _common_systemd_pitfalls {
+    local cmd=$1
+    # do some sanity checks on $cmd to see things we don't expect to work
+
+    if [[ "$cmd" =~ "sudo" ]]; then
+        local msg=<<EOF
+You are trying to use run_process with sudo, this is not going to work under systemd.
+
+If you need to run a service as a user other than $STACK_USER call it with:
+
+   run_process \$name \$cmd \$group \$user
+EOF
+        die $LINENO $msg
+    fi
+
+    if [[ ! "$cmd" =~ ^/ ]]; then
+        local msg=<<EOF
+The cmd="$cmd" does not start with an absolute path. It will fail to
+start under systemd.
+
+Please update your run_process stanza to have an absolute path.
+EOF
+        die $LINENO $msg
+    fi
+
+}
+
+# Helper function to build a basic unit file and run it under systemd.
 function _run_under_systemd {
     local service=$1
     local command="$2"
     local cmd=$command
+    # sanity check the command
+    _common_systemd_pitfalls "$cmd"
+
     local systemd_service="devstack@$service.service"
     local group=$3
     local user=${4:-$STACK_USER}
@@ -1495,22 +1533,6 @@
 
     $SYSTEMCTL enable $systemd_service
     $SYSTEMCTL start $systemd_service
-    _journal_log $service $systemd_service
-}
-
-function _journal_log {
-    local service=$1
-    local unit=$2
-    local logfile="${service}.log.${CURRENT_LOG_TIME}"
-    local real_logfile="${LOGDIR}/${logfile}"
-    if [[ -n ${LOGDIR} ]]; then
-        $JOURNALCTL_F $2 > "$real_logfile" &
-        bash -c "cd '$LOGDIR' && ln -sf '$logfile' ${service}.log"
-        if [[ -n ${SCREEN_LOGDIR} ]]; then
-            # Drop the backward-compat symlink
-            ln -sf "$real_logfile" ${SCREEN_LOGDIR}/screen-${service}.log
-        fi
-    fi
 }
 
 # Helper to remove the ``*.failure`` files under ``$SERVICE_DIR/$SCREEN_NAME``.
@@ -1543,7 +1565,7 @@
 # If an optional group is provided sg will be used to run the
 # command as that group.
 # Uses globals ``USE_SCREEN``
-# run_process service "command-line" [group]
+# run_process service "command-line" [group] [user]
 function run_process {
     local service=$1
     local command="$2"
@@ -1695,11 +1717,10 @@
     SERVICE_DIR=${SERVICE_DIR:-${DEST}/status}
 
     if is_service_enabled $service; then
-        # Kill via pid if we have one available
-        if [[ "$USE_SYSTEMD" == "True" ]]; then
-            # Only do this for units which appear enabled, this also
-            # catches units that don't really exist for cases like
-            # keystone without a failure.
+        # Only do this for units which appear enabled, this also
+        # catches units that don't really exist for cases like
+        # keystone without a failure.
+        if $SYSTEMCTL is-enabled devstack@$service.service; then
             $SYSTEMCTL stop devstack@$service.service
             $SYSTEMCTL disable devstack@$service.service
         fi
@@ -2476,7 +2497,7 @@
     if [[ -n "$start_time" ]]; then
         die $LINENO "Trying to start the clock on $name, but it's already been started"
     fi
-    _TIME_START[$name]=$(date +%s)
+    _TIME_START[$name]=$(date +%s%3N)
 }
 
 # time_stop $name
@@ -2497,7 +2518,7 @@
     if [[ -z "$start_time" ]]; then
         die $LINENO "Trying to stop the clock on $name, but it was never started"
     fi
-    end_time=$(date +%s)
+    end_time=$(date +%s%3N)
     elapsed_time=$(($end_time - $start_time))
     total=${_TIME_TOTAL[$name]:-0}
     # reset the clock so we can start it in the future
@@ -2505,6 +2526,49 @@
     _TIME_TOTAL[$name]=$(($total + $elapsed_time))
 }
 
+function oscwrap {
+    local out
+    local rc
+    local start
+    local end
+    # Cannot use timer_start and timer_stop as we run in subshells
+    # and those rely on modifying vars in the same process (which cannot
+    # happen from a subshell.
+    start=$(date +%s%3N)
+    out=$(command openstack "$@")
+    rc=$?
+    end=$(date +%s%3N)
+    echo $((end - start)) >> $OSCWRAP_TIMER_FILE
+
+    echo "$out"
+    return $rc
+}
+
+function install_oscwrap {
+    # File to accumulate our timing data
+    OSCWRAP_TIMER_FILE=$(mktemp)
+    # Bash by default doesn't expand aliases, allow it for the aliases
+    # we want to whitelist.
+    shopt -s expand_aliases
+    # Remove all aliases that might be expanded to preserve old unexpanded
+    # behavior
+    unalias -a
+    # Add only the alias we want for openstack
+    alias openstack=oscwrap
+}
+
+function cleanup_oscwrap {
+    local total=0
+    if python3_enabled ; then
+        local python=python3
+    else
+        local python=python
+    fi
+    total=$(cat $OSCWRAP_TIMER_FILE | $python -c "import sys; print(sum(int(l) for l in sys.stdin))")
+    _TIME_TOTAL["osc"]=$total
+    rm $OSCWRAP_TIMER_FILE
+}
+
 # time_totals
 #  Print out total time summary
 function time_totals {
@@ -2523,6 +2587,8 @@
         fi
     done
 
+    cleanup_oscwrap
+
     xtrace=$(set +o | grep xtrace)
     set +o xtrace
 
@@ -2534,6 +2600,8 @@
     echo
     for t in ${!_TIME_TOTAL[*]}; do
         local v=${_TIME_TOTAL[$t]}
+        # because we're recording in milliseconds
+        v=$(($v / 1000))
         printf "%-${len}s %3d\n" "$t" "$v"
     done
     echo "========================="
diff --git a/inc/python b/inc/python
index 2443c4d..718cbb2 100644
--- a/inc/python
+++ b/inc/python
@@ -320,6 +320,14 @@
     fi
 
     $xtrace
+
+    # Also install test requirements
+    local install_test_reqs=""
+    local test_req="${!#}/test-requirements.txt"
+    if [[ -e "$test_req" ]]; then
+        install_test_reqs="-r $test_req"
+    fi
+
     # adding SETUPTOOLS_SYS_PATH_TECHNIQUE is a workaround to keep
     # the same behaviour of setuptools before version 25.0.0.
     # related issue: https://github.com/pypa/pip/issues/3874
@@ -329,28 +337,28 @@
         no_proxy="${no_proxy:-}" \
         PIP_FIND_LINKS=$PIP_FIND_LINKS \
         SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite \
-        $cmd_pip $upgrade \
+        $cmd_pip $upgrade $install_test_reqs \
         $@
     result=$?
 
-    # Also install test requirements
-    local test_req="${!#}/test-requirements.txt"
-    if [[ $result == 0 ]] && [[ -e "$test_req" ]]; then
-        echo "Installing test-requirements for $test_req"
-        $sudo_pip \
-            http_proxy=${http_proxy:-} \
-            https_proxy=${https_proxy:-} \
-            no_proxy=${no_proxy:-} \
-            PIP_FIND_LINKS=$PIP_FIND_LINKS \
-            $cmd_pip $upgrade \
-            -r $test_req
-        result=$?
-    fi
-
     time_stop "pip_install"
     return $result
 }
 
+function pip_uninstall {
+    local name=$1
+    if [[ -n ${PIP_VIRTUAL_ENV:=} && -d ${PIP_VIRTUAL_ENV} ]]; then
+        local cmd_pip=$PIP_VIRTUAL_ENV/bin/pip
+        local sudo_pip="env"
+    else
+        local cmd_pip
+        cmd_pip=$(get_pip_command $PYTHON2_VERSION)
+        local sudo_pip="sudo -H"
+    fi
+    # don't error if we can't uninstall, it might not be there
+    $sudo_pip $cmd_pip uninstall -y $name || /bin/true
+}
+
 # get version of a package from global requirements file
 # get_from_global_requirements <package>
 function get_from_global_requirements {
diff --git a/lib/apache b/lib/apache
index afeac15..25c65fe 100644
--- a/lib/apache
+++ b/lib/apache
@@ -53,8 +53,15 @@
 function enable_apache_mod {
     local mod=$1
     # Apache installation, because we mark it NOPRIME
-    if is_ubuntu || is_suse ; then
-        if ! a2query -m $mod ; then
+    if is_ubuntu; then
+        # Skip mod_version as it is not a valid mod to enable
+        # on debuntu, instead it is built in.
+        if [[ "$mod" != "version" ]] && ! a2query -m $mod ; then
+            sudo a2enmod $mod
+            restart_apache_server
+        fi
+    elif is_suse; then
+        if ! a2enmod -q $mod ; then
             sudo a2enmod $mod
             restart_apache_server
         fi
@@ -96,7 +103,7 @@
     # delete the temp directory
     sudo rm -rf $dir
 
-    if is_ubuntu; then
+    if is_ubuntu || is_suse ; then
         # we've got to enable proxy and proxy_uwsgi for this to work
         sudo a2enmod proxy
         sudo a2enmod proxy_uwsgi
@@ -171,6 +178,8 @@
 # enable_apache_site() - Enable a particular apache site
 function enable_apache_site {
     local site=$@
+    # Many of our sites use mod version. Just enable it.
+    enable_apache_mod version
     if is_ubuntu; then
         sudo a2ensite ${site}
     elif is_fedora || is_suse; then
@@ -218,11 +227,6 @@
     restart_service $APACHE_NAME
 }
 
-# reload_apache_server
-function reload_apache_server {
-    reload_service $APACHE_NAME
-}
-
 function write_uwsgi_config {
     local file=$1
     local wsgi=$2
@@ -234,7 +238,13 @@
     # create a home for the sockets; note don't use /tmp -- apache has
     # a private view of it on some platforms.
     local socket_dir='/var/run/uwsgi'
-    sudo install -d -o $STACK_USER -m 755 $socket_dir
+
+    # /var/run will be empty on ubuntu after reboot, so we can use systemd-temptiles
+    # to automatically create $socket_dir.
+    sudo mkdir -p /etc/tmpfiles.d/
+    echo "d $socket_dir 0755 $STACK_USER root" | sudo tee /etc/tmpfiles.d/uwsgi.conf
+    sudo systemd-tmpfiles --create /etc/tmpfiles.d/uwsgi.conf
+
     local socket="$socket_dir/${name}.socket"
 
     # always cleanup given that we are using iniset here
@@ -265,12 +275,64 @@
     else
         local apache_conf=""
         apache_conf=$(apache_site_config_for $name)
-        echo "ProxyPass \"${url}\" \"unix:${socket}|uwsgi://uwsgi-uds-${name}/\" retry=0 " | sudo tee $apache_conf
+        echo "SetEnv proxy-sendcl 1" | sudo tee $apache_conf
+        echo "ProxyPass \"${url}\" \"unix:${socket}|uwsgi://uwsgi-uds-${name}/\" retry=0 " | sudo tee -a $apache_conf
         enable_apache_site $name
-        reload_apache_server
+        restart_apache_server
     fi
 }
 
+# For services using chunked encoding, the only services known to use this
+# currently are Glance and Swift, we need to use an http proxy instead of
+# mod_proxy_uwsgi because the chunked encoding gets dropped. See:
+# https://github.com/unbit/uwsgi/issues/1540 You can workaround this on python2
+# but that involves having apache buffer the request before sending it to
+# uswgi.
+function write_local_uwsgi_http_config {
+    local file=$1
+    local wsgi=$2
+    local url=$3
+    name=$(basename $wsgi)
+
+    # create a home for the sockets; note don't use /tmp -- apache has
+    # a private view of it on some platforms.
+
+    # always cleanup given that we are using iniset here
+    rm -rf $file
+    iniset "$file" uwsgi wsgi-file "$wsgi"
+    port=$(get_random_port)
+    iniset "$file" uwsgi http "127.0.0.1:$port"
+    iniset "$file" uwsgi processes $API_WORKERS
+    # This is running standalone
+    iniset "$file" uwsgi master true
+    # Set die-on-term & exit-on-reload so that uwsgi shuts down
+    iniset "$file" uwsgi die-on-term true
+    iniset "$file" uwsgi exit-on-reload true
+    iniset "$file" uwsgi enable-threads true
+    iniset "$file" uwsgi plugins python
+    # uwsgi recommends this to prevent thundering herd on accept.
+    iniset "$file" uwsgi thunder-lock true
+    # Override the default size for headers from the 4k default.
+    iniset "$file" uwsgi buffer-size 65535
+    # Make sure the client doesn't try to re-use the connection.
+    iniset "$file" uwsgi add-header "Connection: close"
+    # This ensures that file descriptors aren't shared between processes.
+    iniset "$file" uwsgi lazy-apps true
+    iniset "$file" uwsgi chmod-socket 666
+    iniset "$file" uwsgi http-raw-body true
+    iniset "$file" uwsgi http-chunked-input true
+    iniset "$file" uwsgi http-auto-chunked true
+
+    enable_apache_mod proxy
+    enable_apache_mod proxy_http
+    local apache_conf=""
+    apache_conf=$(apache_site_config_for $name)
+    echo "KeepAlive Off" | sudo tee $apache_conf
+    echo "ProxyPass \"${url}\" \"http://127.0.0.1:$port\" retry=0 " | sudo tee -a $apache_conf
+    enable_apache_site $name
+    restart_apache_server
+}
+
 function remove_uwsgi_config {
     local file=$1
     local wsgi=$2
diff --git a/lib/cinder b/lib/cinder
index 3c182cc..b585416 100644
--- a/lib/cinder
+++ b/lib/cinder
@@ -84,20 +84,6 @@
 # CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-lvm:lvmdriver-1,lvm:lvmdriver-2}
 CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-lvm:lvmdriver-1}
 
-
-# Should cinder perform secure deletion of volumes?
-# Defaults to zero. Can also be set to none or shred.
-# This was previously CINDER_SECURE_DELETE (True or False).
-# Equivalents using CINDER_VOLUME_CLEAR are zero and none, respectively.
-# Set to none to avoid this bug when testing:
-# https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1023755
-if [[ -n $CINDER_SECURE_DELETE ]]; then
-    CINDER_SECURE_DELETE=$(trueorfalse True CINDER_SECURE_DELETE)
-    if [[ $CINDER_SECURE_DELETE == "False" ]]; then
-        CINDER_VOLUME_CLEAR_DEFAULT="none"
-    fi
-    deprecated "Configure secure Cinder volume deletion using CINDER_VOLUME_CLEAR instead of CINDER_SECURE_DELETE."
-fi
 CINDER_VOLUME_CLEAR=${CINDER_VOLUME_CLEAR:-${CINDER_VOLUME_CLEAR_DEFAULT:-zero}}
 CINDER_VOLUME_CLEAR=$(echo ${CINDER_VOLUME_CLEAR} | tr '[:upper:]' '[:lower:]')
 
@@ -152,6 +138,7 @@
 # Test if any Cinder services are enabled
 # is_cinder_enabled
 function is_cinder_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"cinder" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"c-" ]] && return 0
     return 1
 }
@@ -253,6 +240,10 @@
 
     configure_rootwrap cinder
 
+    if [[ -f "$CINDER_DIR/etc/cinder/resource_filters.json" ]]; then
+        cp -p "$CINDER_DIR/etc/cinder/resource_filters.json" "$CINDER_CONF_DIR/resource_filters.json"
+    fi
+
     cp $CINDER_DIR/etc/cinder/api-paste.ini $CINDER_API_PASTE_INI
 
     inicomment $CINDER_API_PASTE_INI filter:authtoken auth_host
@@ -308,6 +299,9 @@
                 default_name=$be_name
             fi
             enabled_backends+=$be_name,
+
+            iniset $CINDER_CONF $be_name volume_clear $CINDER_VOLUME_CLEAR
+
         done
         iniset $CINDER_CONF DEFAULT enabled_backends ${enabled_backends%,*}
         if [[ -n "$default_name" ]]; then
@@ -328,7 +322,6 @@
         # Set the service port for a proxy to take the original
         iniset $CINDER_CONF DEFAULT osapi_volume_listen_port $CINDER_SERVICE_PORT_INT
         iniset $CINDER_CONF DEFAULT public_endpoint $CINDER_SERVICE_PROTOCOL://$CINDER_SERVICE_HOST:$CINDER_SERVICE_PORT
-        iniset $CINDER_CONF DEFAULT osapi_volume_base_URL $CINDER_SERVICE_PROTOCOL://$CINDER_SERVICE_HOST:$CINDER_SERVICE_PORT
     fi
 
     if [ "$SYSLOG" != "False" ]; then
@@ -337,8 +330,6 @@
 
     iniset_rpc_backend cinder $CINDER_CONF
 
-    iniset $CINDER_CONF DEFAULT volume_clear $CINDER_VOLUME_CLEAR
-
     # Format logging
     setup_logging $CINDER_CONF $CINDER_USE_MOD_WSGI
 
@@ -352,7 +343,7 @@
 
     iniset $CINDER_CONF DEFAULT osapi_volume_workers "$API_WORKERS"
 
-    iniset $CINDER_CONF DEFAULT glance_api_servers "${GLANCE_SERVICE_PROTOCOL}://${GLANCE_HOSTPORT}"
+    iniset $CINDER_CONF DEFAULT glance_api_servers "$GLANCE_URL"
     if is_service_enabled tls-proxy; then
         iniset $CINDER_CONF DEFAULT glance_protocol https
         iniset $CINDER_CONF DEFAULT glance_ca_certificates_file $SSL_BUNDLE_FILE
@@ -368,11 +359,10 @@
     iniset $CINDER_CONF DEFAULT os_privileged_user_tenant "$SERVICE_PROJECT_NAME"
     iniset $CINDER_CONF DEFAULT graceful_shutdown_timeout "$SERVICE_GRACEFUL_SHUTDOWN_TIMEOUT"
 
-    # Set the backend url according to the configured dlm backend
-    if is_dlm_enabled; then
-        if [[ "$(dlm_backend)" == "zookeeper" ]]; then
-            iniset $CINDER_CONF coordination backend_url "zookeeper://${SERVICE_HOST}:2181"
-        fi
+    if [[ ! -z "$CINDER_COORDINATION_URL" ]]; then
+        iniset $CINDER_CONF coordination backend_url "$CINDER_COORDINATION_URL"
+    elif is_service_enabled etcd3; then
+        iniset $CINDER_CONF coordination backend_url "etcd3+http://${SERVICE_HOST}:2379"
     fi
 }
 
@@ -425,8 +415,10 @@
         # (Re)create cinder database
         recreate_database cinder
 
+        time_start "dbsync"
         # Migrate cinder database
         $CINDER_BIN_DIR/cinder-manage --config-file $CINDER_CONF db sync
+        time_stop "dbsync"
     fi
 
     if is_service_enabled c-vol && [[ -n "$CINDER_ENABLED_BACKENDS" ]]; then
@@ -515,17 +507,24 @@
         fi
     fi
 
-    if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
-        enable_apache_site osapi-volume
-        restart_apache_server
-        tail_log c-api /var/log/$APACHE_NAME/c-api.log
-    else
-        run_process c-api "$CINDER_BIN_DIR/cinder-api --config-file $CINDER_CONF"
-    fi
+    if is_service_enabled c-api ; then
+        if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
+            enable_apache_site osapi-volume
+            restart_apache_server
+            tail_log c-api /var/log/$APACHE_NAME/c-api.log
+        else
+            run_process c-api "$CINDER_BIN_DIR/cinder-api --config-file $CINDER_CONF"
+        fi
 
-    echo "Waiting for Cinder API to start..."
-    if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$CINDER_SERVICE_HOST:$service_port; then
-        die $LINENO "c-api did not start"
+        echo "Waiting for Cinder API to start..."
+        if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$CINDER_SERVICE_HOST:$service_port; then
+            die $LINENO "c-api did not start"
+        fi
+
+        # Start proxies if enabled
+        if is_service_enabled tls-proxy; then
+            start_tls_proxy cinder '*' $CINDER_SERVICE_PORT $CINDER_SERVICE_HOST $CINDER_SERVICE_PORT_INT
+        fi
     fi
 
     run_process c-sch "$CINDER_BIN_DIR/cinder-scheduler --config-file $CINDER_CONF"
@@ -535,11 +534,6 @@
     # NOTE(jdg): For cinder, startup order matters.  To ensure that repor_capabilities is received
     # by the scheduler start the cinder-volume service last (or restart it) after the scheduler
     # has started.  This is a quick fix for lp bug/1189595
-
-    # Start proxies if enabled
-    if is_service_enabled c-api && is_service_enabled tls-proxy; then
-        start_tls_proxy cinder '*' $CINDER_SERVICE_PORT $CINDER_SERVICE_HOST $CINDER_SERVICE_PORT_INT
-    fi
 }
 
 # stop_cinder() - Stop running processes
diff --git a/lib/cinder_backends/lvm b/lib/cinder_backends/lvm
index d927f9c..03e1880 100644
--- a/lib/cinder_backends/lvm
+++ b/lib/cinder_backends/lvm
@@ -53,9 +53,6 @@
     iniset $CINDER_CONF $be_name iscsi_helper "$CINDER_ISCSI_HELPER"
     iniset $CINDER_CONF $be_name lvm_type "$CINDER_LVM_TYPE"
 
-    if [[ "$CINDER_SECURE_DELETE" == "False" ]]; then
-        iniset $CINDER_CONF $be_name volume_clear none
-    fi
 }
 
 # init_cinder_backend_lvm - Initialize volume group
diff --git a/lib/databases/mysql b/lib/databases/mysql
index 7bbcace..a0cf7a4 100644
--- a/lib/databases/mysql
+++ b/lib/databases/mysql
@@ -71,6 +71,10 @@
     elif is_fedora; then
         mysql=mariadb
         my_conf=/etc/my.cnf
+        local cracklib_conf=/etc/my.cnf.d/cracklib_password_check.cnf
+        if [ -f "$cracklib_conf" ]; then
+            inicomment -sudo "$cracklib_conf" "mariadb" "plugin-load-add"
+        fi
     else
         exit_distro_not_supported "mysql configuration"
     fi
diff --git a/lib/dlm b/lib/dlm
deleted file mode 100644
index b5ac0f5..0000000
--- a/lib/dlm
+++ /dev/null
@@ -1,111 +0,0 @@
-#!/bin/bash
-#
-# lib/dlm
-#
-# Functions to control the installation and configuration of software
-# that provides a dlm (and possibly other functions). The default is
-# **zookeeper**, and is going to be the only backend supported in the
-# devstack tree.
-
-# Dependencies:
-#
-# - ``functions`` file
-
-# ``stack.sh`` calls the entry points in this order:
-#
-# - is_dlm_enabled
-# - install_dlm
-# - configure_dlm
-# - cleanup_dlm
-
-# Save trace setting
-_XTRACE_DLM=$(set +o | grep xtrace)
-set +o xtrace
-
-
-# Defaults
-# --------
-
-# <define global variables here that belong to this project>
-
-# Set up default directories
-ZOOKEEPER_DATA_DIR=$DEST/data/zookeeper
-ZOOKEEPER_CONF_DIR=/etc/zookeeper
-
-
-# Entry Points
-# ------------
-#
-# NOTE(sdague): it is expected that when someone wants to implement
-# another one of these out of tree, they'll implement the following
-# functions:
-#
-# - dlm_backend
-# - install_dlm
-# - configure_dlm
-# - cleanup_dlm
-
-# This should be declared in the settings file of any plugin or
-# service that needs to have a dlm in their environment.
-function use_dlm {
-    enable_service $(dlm_backend)
-}
-
-# A function to return the name of the backend in question, some users
-# are going to need to know this.
-function dlm_backend {
-    echo "zookeeper"
-}
-
-# Test if a dlm is enabled (defaults to a zookeeper specific check)
-function is_dlm_enabled {
-    [[ ,${ENABLED_SERVICES}, =~ ,"$(dlm_backend)", ]] && return 0
-    return 1
-}
-
-# cleanup_dlm() - Remove residual data files, anything left over from previous
-# runs that a clean run would need to clean up
-function cleanup_dlm {
-    # NOTE(sdague): we don't check for is_enabled here because we
-    # should just delete this regardless. Some times users updated
-    # their service list before they run cleanup.
-    sudo rm -rf $ZOOKEEPER_DATA_DIR
-}
-
-# configure_dlm() - Set config files, create data dirs, etc
-function configure_dlm {
-    if is_dlm_enabled; then
-        sudo cp $FILES/zookeeper/* $ZOOKEEPER_CONF_DIR
-        sudo sed -i -e 's|.*dataDir.*|dataDir='$ZOOKEEPER_DATA_DIR'|' $ZOOKEEPER_CONF_DIR/zoo.cfg
-        # clean up from previous (possibly aborted) runs
-        # create required data files
-        sudo rm -rf $ZOOKEEPER_DATA_DIR
-        sudo mkdir -p $ZOOKEEPER_DATA_DIR
-        # restart after configuration, there is no reason to make this
-        # another step, because having data files that don't match the
-        # zookeeper running is just going to cause tears.
-        restart_service zookeeper
-    fi
-}
-
-# install_dlm() - Collect source and prepare
-function install_dlm {
-    if is_dlm_enabled; then
-        pip_install_gr_extras tooz zookeeper
-        if is_ubuntu; then
-            install_package zookeeperd
-        elif is_fedora; then
-            install_package zookeeper
-        else
-            die $LINENO "Don't know how to install zookeeper on this platform"
-        fi
-    fi
-}
-
-# Restore xtrace
-$_XTRACE_DLM
-
-# Tell emacs to use shell-script-mode
-## Local variables:
-## mode: shell-script
-## End:
diff --git a/lib/etcd3 b/lib/etcd3
new file mode 100644
index 0000000..0e1fbd5
--- /dev/null
+++ b/lib/etcd3
@@ -0,0 +1,139 @@
+#!/bin/bash
+#
+# lib/etcd3
+#
+# Functions to control the installation and configuration of etcd 3.x
+# that provides a key-value store (and possibly other functions).
+
+# Dependencies:
+#
+# - ``functions`` file
+
+# ``stack.sh`` calls the entry points in this order:
+#
+# - start_etcd3
+# - stop_etcd3
+# - cleanup_etcd3
+
+# Save trace setting
+_XTRACE_ETCD3=$(set +o | grep xtrace)
+set +o xtrace
+
+
+# Defaults
+# --------
+
+# Set up default values for etcd
+ETCD_DOWNLOAD_URL=${ETCD_DOWNLOAD_URL:-https://github.com/coreos/etcd/releases/download}
+ETCD_VERSION=${ETCD_VERSION:-v3.1.7}
+ETCD_DATA_DIR="$DEST/data/etcd"
+ETCD_SYSTEMD_SERVICE="devstack@etcd.service"
+ETCD_BIN_DIR="$DEST/bin"
+ETCD_SHA256_AMD64="4fde194bbcd259401e2b5c462dfa579ee7f6af539f13f130b8f5b4f52e3b3c52"
+# NOTE(sdague): etcd v3.1.7 doesn't have anything for these architectures, though 3.2.0 does.
+ETCD_SHA256_ARM64=""
+ETCD_SHA256_PPC64=""
+ETCD_PORT=2379
+
+if is_ubuntu ; then
+    UBUNTU_RELEASE_BASE_NUM=`lsb_release -r | awk '{print $2}' | cut -d '.' -f 1`
+fi
+
+# start_etcd3() - Starts to run the etcd process
+function start_etcd3 {
+    local cmd="$ETCD_BIN_DIR/etcd"
+    cmd+=" --name $HOSTNAME --data-dir $ETCD_DATA_DIR"
+    cmd+=" --initial-cluster-state new --initial-cluster-token etcd-cluster-01"
+    cmd+=" --initial-cluster $HOSTNAME=http://$SERVICE_HOST:2380"
+    cmd+=" --initial-advertise-peer-urls http://$SERVICE_HOST:2380"
+    cmd+=" --advertise-client-urls http://${HOST_IP}:$ETCD_PORT"
+    cmd+=" --listen-peer-urls http://0.0.0.0:2380 "
+    cmd+=" --listen-client-urls http://${HOST_IP}:$ETCD_PORT"
+
+    local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
+    write_user_unit_file $ETCD_SYSTEMD_SERVICE "$cmd" "" "root"
+
+    iniset -sudo $unitfile "Unit" "After" "network.target"
+    iniset -sudo $unitfile "Service" "Type" "notify"
+    iniset -sudo $unitfile "Service" "Restart" "on-failure"
+    iniset -sudo $unitfile "Service" "LimitNOFILE" "65536"
+
+    $SYSTEMCTL daemon-reload
+    $SYSTEMCTL enable $ETCD_SYSTEMD_SERVICE
+    $SYSTEMCTL start $ETCD_SYSTEMD_SERVICE
+}
+
+# stop_etcd3() stops the etcd3 process
+function stop_etcd3 {
+    # Don't install in sub nodes (multinode scenario)
+    if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+        return
+    fi
+
+    $SYSTEMCTL stop $ETCD_SYSTEMD_SERVICE
+}
+
+function cleanup_etcd3 {
+    # Don't install in sub nodes (multinode scenario)
+    if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+        return
+    fi
+
+    $SYSTEMCTL disable $ETCD_SYSTEMD_SERVICE
+
+    local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
+    sudo rm -f $unitfile
+
+    $SYSTEMCTL daemon-reload
+
+    sudo rm -rf $ETCD_DATA_DIR
+}
+
+function install_etcd3 {
+    echo "Installing etcd"
+
+    # Make sure etcd3 downloads the correct architecture
+    if is_arch "x86_64"; then
+        ETCD_ARCH="amd64"
+        ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_AMD64}
+    elif is_arch "aarch64"; then
+        ETCD_ARCH="arm64"
+        ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_ARM64}
+    elif is_arch "ppc64le"; then
+        ETCD_ARCH="ppc64le"
+        ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_PPC64}
+    else
+        exit_distro_not_supported "invalid hardware type - $ETCD_ARCH"
+    fi
+
+    ETCD_NAME=etcd-$ETCD_VERSION-linux-$ETCD_ARCH
+
+    # Create the necessary directories
+    sudo mkdir -p $ETCD_BIN_DIR
+    sudo mkdir -p $ETCD_DATA_DIR
+
+    # Download and cache the etcd tgz for subsequent use
+    if [ ! -f "$FILES/etcd-$ETCD_VERSION-linux-$ETCD_ARCH/etcd" ]; then
+        ETCD_DOWNLOAD_FILE=$ETCD_NAME.tar.gz
+        if [ ! -f "$FILES/$ETCD_DOWNLOAD_FILE" ]; then
+            wget $ETCD_DOWNLOAD_URL/$ETCD_VERSION/$ETCD_DOWNLOAD_FILE -O $FILES/$ETCD_DOWNLOAD_FILE
+        fi
+        echo "${ETCD_SHA256} $FILES/${ETCD_DOWNLOAD_FILE}" > $FILES/etcd.sha256sum
+        # NOTE(sdague): this should go fatal if this fails
+        sha256sum -c $FILES/etcd.sha256sum
+
+        tar xzvf $FILES/$ETCD_DOWNLOAD_FILE -C $FILES
+        sudo cp $FILES/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+    fi
+    if [ ! -f "$ETCD_BIN_DIR/etcd" ]; then
+        sudo cp $FILES/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+    fi
+}
+
+# Restore xtrace
+$_XTRACE_ETCD3
+
+# Tell emacs to use shell-script-mode
+## Local variables:
+## mode: shell-script
+## End:
diff --git a/lib/glance b/lib/glance
index 23a1cbf..41145f9 100644
--- a/lib/glance
+++ b/lib/glance
@@ -43,6 +43,7 @@
 
 GLANCE_CACHE_DIR=${GLANCE_CACHE_DIR:=$DATA_DIR/glance/cache}
 GLANCE_IMAGE_DIR=${GLANCE_IMAGE_DIR:=$DATA_DIR/glance/images}
+GLANCE_LOCK_DIR=${GLANCE_LOCK_DIR:=$DATA_DIR/glance/locks}
 GLANCE_AUTH_CACHE_DIR=${GLANCE_AUTH_CACHE_DIR:-/var/cache/glance}
 
 GLANCE_CONF_DIR=${GLANCE_CONF_DIR:-/etc/glance}
@@ -70,6 +71,16 @@
 GLANCE_SERVICE_PROTOCOL=${GLANCE_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL}
 GLANCE_REGISTRY_PORT=${GLANCE_REGISTRY_PORT:-9191}
 GLANCE_REGISTRY_PORT_INT=${GLANCE_REGISTRY_PORT_INT:-19191}
+GLANCE_UWSGI=$GLANCE_BIN_DIR/glance-wsgi-api
+GLANCE_UWSGI_CONF=$GLANCE_CONF_DIR/glance-uswgi.ini
+# If wsgi mode is uwsgi run glance under uwsgi, else default to eventlet
+# TODO(mtreinish): Remove the eventlet path here and in all the similar
+# conditionals below after the Pike release
+if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+    GLANCE_URL="$GLANCE_SERVICE_PROTOCOL://$GLANCE_SERVICE_HOST/image"
+else
+    GLANCE_URL="$GLANCE_SERVICE_PROTOCOL://$GLANCE_HOSTPORT"
+fi
 
 # Functions
 # ---------
@@ -77,6 +88,7 @@
 # Test if any Glance services are enabled
 # is_glance_enabled
 function is_glance_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"glance" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"g-" ]] && return 0
     return 1
 }
@@ -102,20 +114,18 @@
     dburl=`database_connection_url glance`
     iniset $GLANCE_REGISTRY_CONF database connection $dburl
     iniset $GLANCE_REGISTRY_CONF DEFAULT use_syslog $SYSLOG
-    iniset $GLANCE_REGISTRY_CONF DEFAULT workers "$API_WORKERS"
     iniset $GLANCE_REGISTRY_CONF paste_deploy flavor keystone
     configure_auth_token_middleware $GLANCE_REGISTRY_CONF glance $GLANCE_AUTH_CACHE_DIR/registry
     iniset $GLANCE_REGISTRY_CONF oslo_messaging_notifications driver messagingv2
     iniset_rpc_backend glance $GLANCE_REGISTRY_CONF
     iniset $GLANCE_REGISTRY_CONF DEFAULT graceful_shutdown_timeout "$SERVICE_GRACEFUL_SHUTDOWN_TIMEOUT"
 
-    cp $GLANCE_DIR/etc/glance-api.conf $GLANCE_API_CONF
     iniset $GLANCE_API_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
-    iniset $GLANCE_API_CONF DEFAULT bind_host $GLANCE_SERVICE_LISTEN_ADDRESS
     inicomment $GLANCE_API_CONF DEFAULT log_file
     iniset $GLANCE_API_CONF database connection $dburl
     iniset $GLANCE_API_CONF DEFAULT use_syslog $SYSLOG
     iniset $GLANCE_API_CONF DEFAULT image_cache_dir $GLANCE_CACHE_DIR/
+    iniset $GLANCE_API_CONF DEFAULT lock_path $GLANCE_LOCK_DIR
     iniset $GLANCE_API_CONF paste_deploy flavor keystone+cachemanagement
     configure_auth_token_middleware $GLANCE_API_CONF glance $GLANCE_AUTH_CACHE_DIR/api
     iniset $GLANCE_API_CONF oslo_messaging_notifications driver messagingv2
@@ -138,8 +148,6 @@
     iniset $GLANCE_API_CONF glance_store filesystem_store_datadir $GLANCE_IMAGE_DIR/
     iniset $GLANCE_API_CONF DEFAULT registry_host $GLANCE_SERVICE_HOST
 
-    iniset $GLANCE_API_CONF DEFAULT workers "$API_WORKERS"
-
     # CORS feature support - to allow calls from Horizon by default
     if [ -n "$GLANCE_CORS_ALLOWED_ORIGIN" ]; then
         iniset $GLANCE_API_CONF cors allowed_origin "$GLANCE_CORS_ALLOWED_ORIGIN"
@@ -178,9 +186,12 @@
         inicomment $GLANCE_API_CONF glance_store swift_store_auth_address
     fi
 
+    # We need to tell glance what it's public endpoint is so that the version
+    # discovery document will be correct
+    iniset $GLANCE_API_CONF DEFAULT public_endpoint $GLANCE_URL
+
     if is_service_enabled tls-proxy; then
         iniset $GLANCE_API_CONF DEFAULT bind_port $GLANCE_SERVICE_PORT_INT
-        iniset $GLANCE_API_CONF DEFAULT public_endpoint $GLANCE_SERVICE_PROTOCOL://$GLANCE_HOSTPORT
         iniset $GLANCE_REGISTRY_CONF DEFAULT bind_port $GLANCE_REGISTRY_PORT_INT
 
         iniset $GLANCE_API_CONF keystone_authtoken identity_uri $KEYSTONE_AUTH_URI
@@ -196,7 +207,6 @@
     setup_logging $GLANCE_REGISTRY_CONF
 
     cp -p $GLANCE_DIR/etc/glance-registry-paste.ini $GLANCE_REGISTRY_PASTE_INI
-
     cp -p $GLANCE_DIR/etc/glance-api-paste.ini $GLANCE_API_PASTE_INI
 
     cp $GLANCE_DIR/etc/glance-cache.conf $GLANCE_CACHE_CONF
@@ -205,7 +215,7 @@
     iniset $GLANCE_CACHE_CONF DEFAULT use_syslog $SYSLOG
     iniset $GLANCE_CACHE_CONF DEFAULT image_cache_dir $GLANCE_CACHE_DIR/
     iniuncomment $GLANCE_CACHE_CONF DEFAULT auth_url
-    iniset $GLANCE_CACHE_CONF DEFAULT auth_url $KEYSTONE_AUTH_URI/v3
+    iniset $GLANCE_CACHE_CONF DEFAULT auth_url $KEYSTONE_AUTH_URI
     iniuncomment $GLANCE_CACHE_CONF DEFAULT auth_tenant_name
     iniset $GLANCE_CACHE_CONF DEFAULT admin_tenant_name $SERVICE_PROJECT_NAME
     iniuncomment $GLANCE_CACHE_CONF DEFAULT auth_user
@@ -229,6 +239,13 @@
         iniset $GLANCE_API_CONF DEFAULT cinder_endpoint_template "https://$CINDER_SERVICE_HOST:$CINDER_SERVICE_PORT/v1/%(project_id)s"
         iniset $GLANCE_CACHE_CONF DEFAULT cinder_endpoint_template "https://$CINDER_SERVICE_HOST:$CINDER_SERVICE_PORT/v1/%(project_id)s"
     fi
+
+    if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+        write_local_uwsgi_http_config "$GLANCE_UWSGI_CONF" "$GLANCE_UWSGI" "/image"
+    else
+        iniset $GLANCE_API_CONF DEFAULT bind_host $GLANCE_SERVICE_LISTEN_ADDRESS
+        iniset $GLANCE_API_CONF DEFAULT workers "$API_WORKERS"
+    fi
 }
 
 # create_glance_accounts() - Set up common required glance accounts
@@ -253,7 +270,7 @@
         get_or_create_endpoint \
             "image" \
             "$REGION_NAME" \
-            "$GLANCE_SERVICE_PROTOCOL://$GLANCE_HOSTPORT"
+            "$GLANCE_URL"
 
         # Note(frickler): Crude workaround for https://bugs.launchpad.net/glance-store/+bug/1620999
         service_domain_id=$(get_or_create_domain $SERVICE_DOMAIN_NAME)
@@ -282,11 +299,13 @@
     # (Re)create glance database
     recreate_database glance
 
+    time_start "dbsync"
     # Migrate glance database
     $GLANCE_BIN_DIR/glance-manage --config-file $GLANCE_CONF_DIR/glance-api.conf db_sync
 
     # Load metadata definitions
     $GLANCE_BIN_DIR/glance-manage --config-file $GLANCE_CONF_DIR/glance-api.conf db_load_metadefs
+    time_stop "dbsync"
 
     create_glance_cache_dir
 }
@@ -318,15 +337,21 @@
 function start_glance {
     local service_protocol=$GLANCE_SERVICE_PROTOCOL
     if is_service_enabled tls-proxy; then
-        start_tls_proxy glance-service '*' $GLANCE_SERVICE_PORT $GLANCE_SERVICE_HOST $GLANCE_SERVICE_PORT_INT
+        if [[ "$WSGI_MODE" != "uwsgi" ]]; then
+            start_tls_proxy glance-service '*' $GLANCE_SERVICE_PORT $GLANCE_SERVICE_HOST $GLANCE_SERVICE_PORT_INT
+        fi
         start_tls_proxy glance-registry '*' $GLANCE_REGISTRY_PORT $GLANCE_SERVICE_HOST $GLANCE_REGISTRY_PORT_INT
     fi
 
     run_process g-reg "$GLANCE_BIN_DIR/glance-registry --config-file=$GLANCE_CONF_DIR/glance-registry.conf"
-    run_process g-api "$GLANCE_BIN_DIR/glance-api --config-file=$GLANCE_CONF_DIR/glance-api.conf"
+    if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+        run_process g-api "$GLANCE_BIN_DIR/uwsgi --ini $GLANCE_UWSGI_CONF"
+    else
+        run_process g-api "$GLANCE_BIN_DIR/glance-api --config-file=$GLANCE_CONF_DIR/glance-api.conf"
+    fi
 
-    echo "Waiting for g-api ($GLANCE_HOSTPORT) to start..."
-    if ! wait_for_service $SERVICE_TIMEOUT $GLANCE_SERVICE_PROTOCOL://$GLANCE_HOSTPORT; then
+    echo "Waiting for g-api ($GLANCE_SERVICE_HOST) to start..."
+    if ! wait_for_service $SERVICE_TIMEOUT $GLANCE_URL; then
         die $LINENO "g-api did not start"
     fi
 }
diff --git a/lib/keystone b/lib/keystone
index a26ef8a..eb46526 100644
--- a/lib/keystone
+++ b/lib/keystone
@@ -113,8 +113,9 @@
     KEYSTONE_SERVICE_PROTOCOL="https"
 fi
 
-KEYSTONE_AUTH_URI=${KEYSTONE_AUTH_PROTOCOL}://${KEYSTONE_AUTH_HOST}/identity_admin
 KEYSTONE_SERVICE_URI=${KEYSTONE_SERVICE_PROTOCOL}://${KEYSTONE_SERVICE_HOST}/identity
+# for compat
+KEYSTONE_AUTH_URI=$KEYSTONE_SERVICE_URI
 
 # V3 URIs
 KEYSTONE_AUTH_URI_V3=$KEYSTONE_AUTH_URI/v3
@@ -123,9 +124,15 @@
 # Security compliance
 KEYSTONE_SECURITY_COMPLIANCE_ENABLED=${KEYSTONE_SECURITY_COMPLIANCE_ENABLED:-True}
 KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS=${KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS:-2}
-KEYSTONE_LOCKOUT_DURATION=${KEYSTONE_LOCKOUT_DURATION:-5}
+KEYSTONE_LOCKOUT_DURATION=${KEYSTONE_LOCKOUT_DURATION:-10}
 KEYSTONE_UNIQUE_LAST_PASSWORD_COUNT=${KEYSTONE_UNIQUE_LAST_PASSWORD_COUNT:-2}
 
+# Number of bcrypt hashing rounds, increasing number exponentially increases required
+# resources to generate password hash. This is very effective way to protect from
+# bruteforce attacks. 4 is minimal value that can be specified for bcrypt and
+# it works way faster than default 12. Minimal value is great for CI and development
+# however may not be suitable for real production.
+KEYSTONE_PASSWORD_HASH_ROUNDS=${KEYSTONE_PASSWORD_HASH_ROUNDS:-4}
 
 # Functions
 # ---------
@@ -133,6 +140,7 @@
 # Test if Keystone is enabled
 # is_keystone_enabled
 function is_keystone_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"keystone" ]] && return 1
     [[ ,${ENABLED_SERVICES}, =~ ,"key", ]] && return 0
     return 1
 }
@@ -140,15 +148,16 @@
 # cleanup_keystone() - Remove residual data files, anything left over from previous
 # runs that a clean run would need to clean up
 function cleanup_keystone {
-    if [[ "$WSGI_MODE" == "uwsgi" ]]; then
-        remove_uwsgi_config "$KEYSTONE_PUBLIC_UWSGI_CONF" "$KEYSTONE_PUBLIC_UWSGI"
-        remove_uwsgi_config "$KEYSTONE_ADMIN_UWSGI_CONF" "$KEYSTONE_ADMIN_UWSGI"
-        sudo rm -f $(apache_site_config_for keystone-wsgi-public)
-        sudo rm -f $(apache_site_config_for keystone-wsgi-admin)
-    else
-        disable_apache_site keystone
-        sudo rm -f $(apache_site_config_for keystone)
-    fi
+    # TODO: remove admin at pike-2
+    # These files will be created if we are running WSGI_MODE="uwsgi"
+    remove_uwsgi_config "$KEYSTONE_PUBLIC_UWSGI_CONF" "$KEYSTONE_PUBLIC_UWSGI"
+    remove_uwsgi_config "$KEYSTONE_ADMIN_UWSGI_CONF" "$KEYSTONE_ADMIN_UWSGI"
+    sudo rm -f $(apache_site_config_for keystone-wsgi-public)
+    sudo rm -f $(apache_site_config_for keystone-wsgi-admin)
+
+    # These files will be created if we are running WSGI_MODE="mod_wsgi"
+    disable_apache_site keystone
+    sudo rm -f $(apache_site_config_for keystone)
 }
 
 # _config_keystone_apache_wsgi() - Set WSGI config files of Keystone
@@ -222,6 +231,7 @@
     fi
 
     iniset $KEYSTONE_CONF identity driver "$KEYSTONE_IDENTITY_BACKEND"
+    iniset $KEYSTONE_CONF identity password_hash_rounds $KEYSTONE_PASSWORD_HASH_ROUNDS
     iniset $KEYSTONE_CONF assignment driver "$KEYSTONE_ASSIGNMENT_BACKEND"
     iniset $KEYSTONE_CONF role driver "$KEYSTONE_ROLE_BACKEND"
     iniset $KEYSTONE_CONF resource driver "$KEYSTONE_RESOURCE_BACKEND"
@@ -265,9 +275,7 @@
     fi
 
     # Format logging
-    if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ] && [ "$KEYSTONE_DEPLOY" != "mod_wsgi" ] ; then
-        setup_colorized_logging $KEYSTONE_CONF
-    fi
+    setup_logging $KEYSTONE_CONF
 
     iniset $KEYSTONE_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
 
@@ -434,14 +442,13 @@
     local section=${4:-keystone_authtoken}
 
     iniset $conf_file $section auth_type password
-    iniset $conf_file $section auth_url $KEYSTONE_AUTH_URI
+    iniset $conf_file $section auth_url $KEYSTONE_SERVICE_URI
     iniset $conf_file $section username $admin_user
     iniset $conf_file $section password $SERVICE_PASSWORD
     iniset $conf_file $section user_domain_name "$SERVICE_DOMAIN_NAME"
     iniset $conf_file $section project_name $SERVICE_PROJECT_NAME
     iniset $conf_file $section project_domain_name "$SERVICE_DOMAIN_NAME"
 
-    iniset $conf_file $section auth_uri $KEYSTONE_SERVICE_URI
     iniset $conf_file $section cafile $SSL_BUNDLE_FILE
     iniset $conf_file $section signing_dir $signing_dir
     iniset $conf_file $section memcached_servers $SERVICE_HOST:11211
@@ -458,8 +465,10 @@
         recreate_database keystone
     fi
 
+    time_start "dbsync"
     # Initialize keystone database
     $KEYSTONE_BIN_DIR/keystone-manage --config-file $KEYSTONE_CONF db_sync
+    time_stop "dbsync"
 
     if [[ "$KEYSTONE_TOKEN_FORMAT" == "pki" || "$KEYSTONE_TOKEN_FORMAT" == "pkiz" ]]; then
         # Set up certificates
@@ -544,11 +553,7 @@
         tail_log key /var/log/$APACHE_NAME/keystone.log
         tail_log key-access /var/log/$APACHE_NAME/keystone_access.log
     else # uwsgi
-        # TODO(sdague): we should really get down to a single keystone here
-        enable_service key-p
-        enable_service key-a
-        run_process key-p "$KEYSTONE_BIN_DIR/uwsgi --ini $KEYSTONE_PUBLIC_UWSGI_CONF" ""
-        run_process key-a "$KEYSTONE_BIN_DIR/uwsgi --ini $KEYSTONE_ADMIN_UWSGI_CONF" ""
+        run_process keystone "$KEYSTONE_BIN_DIR/uwsgi --ini $KEYSTONE_PUBLIC_UWSGI_CONF" ""
     fi
 
     echo "Waiting for keystone to start..."
@@ -579,9 +584,9 @@
         disable_apache_site keystone
         restart_apache_server
     else
-        stop_process key-p
-        stop_process key-a
+        stop_process keystone
         remove_uwsgi_config "$KEYSTONE_PUBLIC_UWSGI_CONF" "$KEYSTONE_PUBLIC_UWSGI"
+        # TODO(remove in at pike-2)
         remove_uwsgi_config "$KEYSTONE_ADMIN_UWSGI_CONF" "$KEYSTONE_ADMIN_UWSGI"
     fi
     # Kill the Keystone screen window
diff --git a/lib/libraries b/lib/libraries
new file mode 100644
index 0000000..4ceb804
--- /dev/null
+++ b/lib/libraries
@@ -0,0 +1,143 @@
+#!/bin/bash
+#
+# lib/oslo
+#
+# Functions to install libraries from git
+#
+# We need this to handle the fact that projects would like to use
+# pre-released versions of oslo libraries.
+
+# Dependencies:
+#
+# - ``functions`` file
+
+# ``stack.sh`` calls the entry points in this order:
+#
+# - install_libraries
+
+# Save trace setting
+_XTRACE_LIB_LIBRARIES=$(set +o | grep xtrace)
+set +o xtrace
+
+
+# Defaults
+# --------
+GITDIR["automaton"]=$DEST/automaton
+GITDIR["castellan"]=$DEST/castellan
+GITDIR["cliff"]=$DEST/cliff
+GITDIR["cursive"]=$DEST/cursive
+GITDIR["debtcollector"]=$DEST/debtcollector
+GITDIR["futurist"]=$DEST/futurist
+GITDIR["os-client-config"]=$DEST/os-client-config
+GITDIR["osc-lib"]=$DEST/osc-lib
+GITDIR["oslo.cache"]=$DEST/oslo.cache
+GITDIR["oslo.concurrency"]=$DEST/oslo.concurrency
+GITDIR["oslo.config"]=$DEST/oslo.config
+GITDIR["oslo.context"]=$DEST/oslo.context
+GITDIR["oslo.db"]=$DEST/oslo.db
+GITDIR["oslo.i18n"]=$DEST/oslo.i18n
+GITDIR["oslo.log"]=$DEST/oslo.log
+GITDIR["oslo.messaging"]=$DEST/oslo.messaging
+GITDIR["oslo.middleware"]=$DEST/oslo.middleware
+GITDIR["oslo.policy"]=$DEST/oslo.policy
+GITDIR["oslo.privsep"]=$DEST/oslo.privsep
+GITDIR["oslo.reports"]=$DEST/oslo.reports
+GITDIR["oslo.rootwrap"]=$DEST/oslo.rootwrap
+GITDIR["oslo.serialization"]=$DEST/oslo.serialization
+GITDIR["oslo.service"]=$DEST/oslo.service
+GITDIR["oslo.utils"]=$DEST/oslo.utils
+GITDIR["oslo.versionedobjects"]=$DEST/oslo.versionedobjects
+GITDIR["oslo.vmware"]=$DEST/oslo.vmware
+GITDIR["osprofiler"]=$DEST/osprofiler
+GITDIR["pycadf"]=$DEST/pycadf
+GITDIR["python-openstacksdk"]=$DEST/python-openstacksdk
+GITDIR["stevedore"]=$DEST/stevedore
+GITDIR["taskflow"]=$DEST/taskflow
+GITDIR["tooz"]=$DEST/tooz
+
+# Non oslo libraries are welcomed below as well, this prevents
+# duplication of this code.
+GITDIR["os-brick"]=$DEST/os-brick
+GITDIR["os-traits"]=$DEST/os-traits
+
+# Support entry points installation of console scripts
+OSLO_BIN_DIR=$(get_python_exec_prefix)
+
+
+# Functions
+# ---------
+
+function _install_lib_from_source {
+    local name=$1
+    if use_library_from_git "$name"; then
+        git_clone_by_name "$name"
+        setup_dev_lib "$name"
+    fi
+}
+
+# install_oslo - install libraries that oslo needs
+function install_oslo {
+    install_libs
+}
+
+# install_libs() - Install additional libraries that we need and want
+# on all environments. Some will only install here if from source,
+# others will always install.
+function install_libs {
+    _install_lib_from_source "automaton"
+    _install_lib_from_source "castellan"
+    _install_lib_from_source "cliff"
+    _install_lib_from_source "cursive"
+    _install_lib_from_source "debtcollector"
+    _install_lib_from_source "futurist"
+    _install_lib_from_source "osc-lib"
+    _install_lib_from_source "os-client-config"
+    _install_lib_from_source "oslo.cache"
+    _install_lib_from_source "oslo.concurrency"
+    _install_lib_from_source "oslo.config"
+    _install_lib_from_source "oslo.context"
+    _install_lib_from_source "oslo.db"
+    _install_lib_from_source "oslo.i18n"
+    _install_lib_from_source "oslo.log"
+    _install_lib_from_source "oslo.messaging"
+    _install_lib_from_source "oslo.middleware"
+    _install_lib_from_source "oslo.policy"
+    _install_lib_from_source "oslo.privsep"
+    _install_lib_from_source "oslo.reports"
+    _install_lib_from_source "oslo.rootwrap"
+    _install_lib_from_source "oslo.serialization"
+    _install_lib_from_source "oslo.service"
+    _install_lib_from_source "oslo.utils"
+    _install_lib_from_source "oslo.versionedobjects"
+    _install_lib_from_source "oslo.vmware"
+    _install_lib_from_source "osprofiler"
+    _install_lib_from_source "pycadf"
+    _install_lib_from_source "python-openstacksdk"
+    _install_lib_from_source "stevedore"
+    _install_lib_from_source "taskflow"
+    _install_lib_from_source "tooz"
+    # installation of additional libraries
+    #
+    # os-traits for nova
+    _install_lib_from_source "os-brick"
+    _install_lib_from_source "os-traits"
+    #
+    # python client libraries we might need from git can go here
+    _install_lib_from_source "python-barbicanclient"
+
+
+    # etcd (because tooz does not have a hard dependency on these)
+    #
+    # NOTE(sdague): this is currently a work around because tooz
+    # doesn't pull in etcd3.
+    pip_install etcd3
+    pip_install etcd3gw
+}
+
+# Restore xtrace
+$_XTRACE_LIB_LIBRARIES
+
+# Tell emacs to use shell-script-mode
+## Local variables:
+## mode: shell-script
+## End:
diff --git a/lib/neutron b/lib/neutron
index 492a0ee..2a660ec 100644
--- a/lib/neutron
+++ b/lib/neutron
@@ -91,6 +91,7 @@
 # Test if any Neutron services are enabled
 # is_neutron_enabled
 function is_neutron_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"neutron" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"neutron-" || ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
     return 1
 }
@@ -98,6 +99,7 @@
 # Test if any Neutron services are enabled
 # is_neutron_enabled
 function is_neutron_legacy_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"neutron" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
     return 1
 }
@@ -135,7 +137,11 @@
 
     mkdir -p $NEUTRON_CORE_PLUGIN_CONF_PATH
 
-    cp $NEUTRON_DIR/etc/neutron/plugins/$NEUTRON_CORE_PLUGIN/$NEUTRON_CORE_PLUGIN_CONF_FILENAME.sample $NEUTRON_CORE_PLUGIN_CONF
+    # NOTE(yamamoto): A decomposed plugin should prepare the config file in
+    # its devstack plugin.
+    if [ -f $NEUTRON_DIR/etc/neutron/plugins/$NEUTRON_CORE_PLUGIN/$NEUTRON_CORE_PLUGIN_CONF_FILENAME.sample ]; then
+        cp $NEUTRON_DIR/etc/neutron/plugins/$NEUTRON_CORE_PLUGIN/$NEUTRON_CORE_PLUGIN_CONF_FILENAME.sample $NEUTRON_CORE_PLUGIN_CONF
+    fi
 
     iniset $NEUTRON_CONF database connection `database_connection_url neutron`
     iniset $NEUTRON_CONF DEFAULT state_path $NEUTRON_STATE_PATH
@@ -226,17 +232,12 @@
         iniset $NEUTRON_META_CONF agent root_helper_daemon "$NEUTRON_ROOTWRAP_DAEMON_CMD"
 
         # TODO(dtroyer): remove the v2.0 hard code below
-        iniset $NEUTRON_META_CONF DEFAULT auth_url $KEYSTONE_SERVICE_URI/v2.0
+        iniset $NEUTRON_META_CONF DEFAULT auth_url $KEYSTONE_SERVICE_URI
         configure_auth_token_middleware $NEUTRON_META_CONF neutron $NEUTRON_AUTH_CACHE_DIR DEFAULT
     fi
 
     # Format logging
-    if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ]; then
-        setup_colorized_logging $NEUTRON_CONF DEFAULT project_id
-    else
-        # Show user_name and project_name by default
-        iniset $NEUTRON_CONF DEFAULT logging_context_format_string "%(asctime)s.%(msecs)03d %(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s] %(instance)s%(message)s"
-    fi
+    setup_logging $NEUTRON_CONF
 
     if is_service_enabled tls-proxy; then
         # Set the service port for a proxy to take the original
@@ -284,7 +285,7 @@
 function configure_neutron_nova_new {
     iniset $NOVA_CONF DEFAULT use_neutron True
     iniset $NOVA_CONF neutron auth_type "password"
-    iniset $NOVA_CONF neutron auth_url "$KEYSTONE_SERVICE_URI/v3"
+    iniset $NOVA_CONF neutron auth_url "$KEYSTONE_SERVICE_URI"
     iniset $NOVA_CONF neutron username neutron
     iniset $NOVA_CONF neutron password "$SERVICE_PASSWORD"
     iniset $NOVA_CONF neutron user_domain_name "Default"
@@ -335,8 +336,10 @@
 
     recreate_database neutron
 
+    time_start "dbsync"
     # Run Neutron db migrations
     $NEUTRON_BIN_DIR/neutron-db-manage upgrade heads
+    time_stop "dbsync"
 
     create_neutron_cache_dir
 }
diff --git a/lib/neutron-legacy b/lib/neutron-legacy
index af9a3d2..784f3a8 100644
--- a/lib/neutron-legacy
+++ b/lib/neutron-legacy
@@ -368,7 +368,7 @@
 function create_nova_conf_neutron {
     iniset $NOVA_CONF DEFAULT use_neutron True
     iniset $NOVA_CONF neutron auth_type "password"
-    iniset $NOVA_CONF neutron auth_url "$KEYSTONE_AUTH_URI/v3"
+    iniset $NOVA_CONF neutron auth_url "$KEYSTONE_AUTH_URI"
     iniset $NOVA_CONF neutron username "$Q_ADMIN_USERNAME"
     iniset $NOVA_CONF neutron password "$SERVICE_PASSWORD"
     iniset $NOVA_CONF neutron user_domain_name "$SERVICE_DOMAIN_NAME"
@@ -417,8 +417,10 @@
 # init_mutnauq() - Initialize databases, etc.
 function init_mutnauq {
     recreate_database $Q_DB_NAME
+    time_start "dbsync"
     # Run Neutron db migrations
     $NEUTRON_BIN_DIR/neutron-db-manage --config-file $NEUTRON_CONF --config-file /$Q_PLUGIN_CONF_FILE upgrade head
+    time_stop "dbsync"
 }
 
 # install_mutnauq() - Collect source and prepare
@@ -577,7 +579,7 @@
         local IP_DEL=""
         local IP_UP=""
         local DEFAULT_ROUTE_GW
-        DEFAULT_ROUTE_GW=$(ip -f $af r | awk "/default.+$from_intf/ { print \$3; exit }")
+        DEFAULT_ROUTE_GW=$(ip -f $af r | awk "/default.+$from_intf\s/ { print \$3; exit }")
         local ADD_OVS_PORT=""
         local DEL_OVS_PORT=""
         local ARP_CMD=""
diff --git a/lib/nova b/lib/nova
index de053ab..f56509a 100644
--- a/lib/nova
+++ b/lib/nova
@@ -175,6 +175,7 @@
 # Test if any Nova services are enabled
 # is_nova_enabled
 function is_nova_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"nova" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"n-" ]] && return 0
     return 1
 }
@@ -453,8 +454,8 @@
     fi
     iniset $NOVA_CONF wsgi api_paste_config "$NOVA_API_PASTE_INI"
     iniset $NOVA_CONF DEFAULT rootwrap_config "$NOVA_CONF_DIR/rootwrap.conf"
-    iniset $NOVA_CONF DEFAULT scheduler_driver "$SCHEDULER"
-    iniset $NOVA_CONF DEFAULT scheduler_default_filters "$FILTERS"
+    iniset $NOVA_CONF scheduler driver "$SCHEDULER"
+    iniset $NOVA_CONF filter_scheduler enabled_filters "$FILTERS"
     iniset $NOVA_CONF DEFAULT default_floating_pool "$PUBLIC_NETWORK_NAME"
     if [[ $SERVICE_IP_VERSION == 6 ]]; then
         iniset $NOVA_CONF DEFAULT my_ip "$HOST_IPV6"
@@ -574,7 +575,7 @@
     # enable notifications, but it will allow them to function when enabled.
     iniset $NOVA_CONF oslo_messaging_notifications driver "messagingv2"
     iniset_rpc_backend nova $NOVA_CONF
-    iniset $NOVA_CONF glance api_servers "${GLANCE_SERVICE_PROTOCOL}://${GLANCE_HOSTPORT}"
+    iniset $NOVA_CONF glance api_servers "$GLANCE_URL"
 
     iniset $NOVA_CONF DEFAULT osapi_compute_workers "$API_WORKERS"
     iniset $NOVA_CONF DEFAULT metadata_workers "$API_WORKERS"
@@ -648,7 +649,9 @@
             iniset $NOVA_CELLS_CONF DEFAULT enabled_apis metadata
         fi
 
+        time_start "dbsync"
         $NOVA_BIN_DIR/nova-manage --config-file $NOVA_CELLS_CONF db sync
+        time_stop "dbsync"
         $NOVA_BIN_DIR/nova-manage --config-file $NOVA_CELLS_CONF cell create --name=region --cell_type=parent --username=$RABBIT_USERID --hostname=$RABBIT_HOST --port=5672 --password=$RABBIT_PASSWORD --virtual_host=/ --woffset=0 --wscale=1
         $NOVA_BIN_DIR/nova-manage cell create --name=child --cell_type=child --username=$RABBIT_USERID --hostname=$RABBIT_HOST --port=5672 --password=$RABBIT_PASSWORD --virtual_host=child_cell --woffset=0 --wscale=1
 
@@ -878,7 +881,6 @@
     run_process n-cond "$NOVA_BIN_DIR/nova-conductor --config-file $compute_cell_conf"
     run_process n-cell-region "$NOVA_BIN_DIR/nova-cells --config-file $api_cell_conf"
     run_process n-cell-child "$NOVA_BIN_DIR/nova-cells --config-file $compute_cell_conf"
-    run_process n-crt "$NOVA_BIN_DIR/nova-cert --config-file $api_cell_conf"
 
     if is_service_enabled n-net; then
         if ! running_in_container; then
@@ -929,7 +931,7 @@
     # Kill the nova screen windows
     # Some services are listed here twice since more than one instance
     # of a service may be running in certain configs.
-    for serv in n-api n-crt n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cell n-cell n-api-meta n-sproxy; do
+    for serv in n-api n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cell n-cell n-api-meta n-sproxy; do
         stop_process $serv
     done
 }
diff --git a/lib/nova_plugins/functions-libvirt b/lib/nova_plugins/functions-libvirt
index 47605af..3e38b89 100644
--- a/lib/nova_plugins/functions-libvirt
+++ b/lib/nova_plugins/functions-libvirt
@@ -67,6 +67,8 @@
         else
             install_package libvirt-clients libvirt-daemon-system libvirt-dev
         fi
+        # uninstall in case the libvirt version changed
+        pip_uninstall libvirt-python
         pip_install_gr libvirt-python
         #pip_install_gr <there-si-no-guestfs-in-pypi>
     elif is_fedora || is_suse; then
@@ -84,6 +86,7 @@
         fi
 
         install_package libvirt libvirt-devel
+        pip_uninstall libvirt-python
         pip_install_gr libvirt-python
     fi
 
@@ -95,7 +98,7 @@
 # Configures the installed libvirt system so that is accessible by
 # STACK_USER via qemu:///system with management capabilities.
 function configure_libvirt {
-    if is_service_enabled neutron && is_neutron_ovs_base_plugin && ! sudo grep -q '^cgroup_device_acl' $QEMU_CONF; then
+    if is_service_enabled neutron && ! sudo grep -q '^cgroup_device_acl' $QEMU_CONF; then
         # Add /dev/net/tun to cgroup_device_acls, needed for type=ethernet interfaces
         cat <<EOF | sudo tee -a $QEMU_CONF
 cgroup_device_acl = [
diff --git a/lib/nova_plugins/hypervisor-fake b/lib/nova_plugins/hypervisor-fake
index f9b95c1..49c8dee 100644
--- a/lib/nova_plugins/hypervisor-fake
+++ b/lib/nova_plugins/hypervisor-fake
@@ -49,7 +49,7 @@
     iniset $NOVA_CONF DEFAULT quota_security_groups -1
     iniset $NOVA_CONF DEFAULT quota_security_group_rules -1
     iniset $NOVA_CONF DEFAULT quota_key_pairs -1
-    iniset $NOVA_CONF DEFAULT scheduler_default_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"
+    iniset $NOVA_CONF filter_scheduler enabled_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"
 }
 
 # install_nova_hypervisor() - Install external components
diff --git a/lib/nova_plugins/hypervisor-ironic b/lib/nova_plugins/hypervisor-ironic
index c9544fe..7d47ef0 100644
--- a/lib/nova_plugins/hypervisor-ironic
+++ b/lib/nova_plugins/hypervisor-ironic
@@ -49,7 +49,7 @@
     iniset $NOVA_CONF ironic auth_type password
     iniset $NOVA_CONF ironic username admin
     iniset $NOVA_CONF ironic password $ADMIN_PASSWORD
-    iniset $NOVA_CONF ironic auth_url $KEYSTONE_AUTH_URI/v3
+    iniset $NOVA_CONF ironic auth_url $KEYSTONE_AUTH_URI
     iniset $NOVA_CONF ironic project_domain_id default
     iniset $NOVA_CONF ironic user_domain_id default
     iniset $NOVA_CONF ironic project_name demo
diff --git a/lib/nova_plugins/hypervisor-xenserver b/lib/nova_plugins/hypervisor-xenserver
index 880b87f..6f79e4f 100644
--- a/lib/nova_plugins/hypervisor-xenserver
+++ b/lib/nova_plugins/hypervisor-xenserver
@@ -84,14 +84,6 @@
 * * * * * /root/rotate_xen_guest_logs.sh >/dev/null 2>&1
 CRONTAB
 
-    # Create directories for kernels and images
-    {
-        echo "set -eux"
-        cat $TOP_DIR/tools/xen/functions
-        echo "create_directory_for_images"
-        echo "create_directory_for_kernels"
-        echo "install_conntrack_tools"
-    } | $ssh_dom0
 }
 
 # install_nova_hypervisor() - Install external components
diff --git a/lib/os_brick b/lib/os_brick
deleted file mode 100644
index d1cca4a..0000000
--- a/lib/os_brick
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash
-#
-# lib/os_brick
-# Install **os-brick** python module from source
-
-# Dependencies:
-#
-# - functions
-# - DEST, DATA_DIR must be defined
-
-# stack.sh
-# ---------
-# - install_os_brick
-
-# Save trace setting
-_XTRACE_OS_BRICK=$(set +o | grep xtrace)
-set +o xtrace
-
-
-GITDIR["os-brick"]=$DEST/os-brick
-
-# Install os_brick from git only if requested, otherwise it will be pulled from
-# pip repositories by requirements of projects that need it.
-function install_os_brick {
-    if use_library_from_git "os-brick"; then
-        git_clone_by_name "os-brick"
-        setup_dev_lib "os-brick"
-    fi
-}
-
-# Restore xtrace
-$_XTRACE_OS_BRICK
\ No newline at end of file
diff --git a/lib/oslo b/lib/oslo
index 2895503..3ae64c8 100644
--- a/lib/oslo
+++ b/lib/oslo
@@ -6,110 +6,6 @@
 #
 # We need this to handle the fact that projects would like to use
 # pre-released versions of oslo libraries.
-
-# Dependencies:
 #
-# - ``functions`` file
-
-# ``stack.sh`` calls the entry points in this order:
-#
-# - install_oslo
-
-# Save trace setting
-_XTRACE_LIB_OSLO=$(set +o | grep xtrace)
-set +o xtrace
-
-
-# Defaults
-# --------
-GITDIR["automaton"]=$DEST/automaton
-GITDIR["castellan"]=$DEST/castellan
-GITDIR["cliff"]=$DEST/cliff
-GITDIR["cursive"]=$DEST/cursive
-GITDIR["debtcollector"]=$DEST/debtcollector
-GITDIR["futurist"]=$DEST/futurist
-GITDIR["os-client-config"]=$DEST/os-client-config
-GITDIR["osc-lib"]=$DEST/osc-lib
-GITDIR["oslo.cache"]=$DEST/oslo.cache
-GITDIR["oslo.concurrency"]=$DEST/oslo.concurrency
-GITDIR["oslo.config"]=$DEST/oslo.config
-GITDIR["oslo.context"]=$DEST/oslo.context
-GITDIR["oslo.db"]=$DEST/oslo.db
-GITDIR["oslo.i18n"]=$DEST/oslo.i18n
-GITDIR["oslo.log"]=$DEST/oslo.log
-GITDIR["oslo.messaging"]=$DEST/oslo.messaging
-GITDIR["oslo.middleware"]=$DEST/oslo.middleware
-GITDIR["oslo.policy"]=$DEST/oslo.policy
-GITDIR["oslo.privsep"]=$DEST/oslo.privsep
-GITDIR["oslo.reports"]=$DEST/oslo.reports
-GITDIR["oslo.rootwrap"]=$DEST/oslo.rootwrap
-GITDIR["oslo.serialization"]=$DEST/oslo.serialization
-GITDIR["oslo.service"]=$DEST/oslo.service
-GITDIR["oslo.utils"]=$DEST/oslo.utils
-GITDIR["oslo.versionedobjects"]=$DEST/oslo.versionedobjects
-GITDIR["oslo.vmware"]=$DEST/oslo.vmware
-GITDIR["osprofiler"]=$DEST/osprofiler
-GITDIR["pycadf"]=$DEST/pycadf
-GITDIR["python-openstacksdk"]=$DEST/python-openstacksdk
-GITDIR["stevedore"]=$DEST/stevedore
-GITDIR["taskflow"]=$DEST/taskflow
-GITDIR["tooz"]=$DEST/tooz
-
-# Support entry points installation of console scripts
-OSLO_BIN_DIR=$(get_python_exec_prefix)
-
-
-# Functions
-# ---------
-
-function _do_install_oslo_lib {
-    local name=$1
-    if use_library_from_git "$name"; then
-        git_clone_by_name "$name"
-        setup_dev_lib "$name"
-    fi
-}
-
-# install_oslo() - Collect source and prepare
-function install_oslo {
-    _do_install_oslo_lib "automaton"
-    _do_install_oslo_lib "castellan"
-    _do_install_oslo_lib "cliff"
-    _do_install_oslo_lib "cursive"
-    _do_install_oslo_lib "debtcollector"
-    _do_install_oslo_lib "futurist"
-    _do_install_oslo_lib "osc-lib"
-    _do_install_oslo_lib "os-client-config"
-    _do_install_oslo_lib "oslo.cache"
-    _do_install_oslo_lib "oslo.concurrency"
-    _do_install_oslo_lib "oslo.config"
-    _do_install_oslo_lib "oslo.context"
-    _do_install_oslo_lib "oslo.db"
-    _do_install_oslo_lib "oslo.i18n"
-    _do_install_oslo_lib "oslo.log"
-    _do_install_oslo_lib "oslo.messaging"
-    _do_install_oslo_lib "oslo.middleware"
-    _do_install_oslo_lib "oslo.policy"
-    _do_install_oslo_lib "oslo.privsep"
-    _do_install_oslo_lib "oslo.reports"
-    _do_install_oslo_lib "oslo.rootwrap"
-    _do_install_oslo_lib "oslo.serialization"
-    _do_install_oslo_lib "oslo.service"
-    _do_install_oslo_lib "oslo.utils"
-    _do_install_oslo_lib "oslo.versionedobjects"
-    _do_install_oslo_lib "oslo.vmware"
-    _do_install_oslo_lib "osprofiler"
-    _do_install_oslo_lib "pycadf"
-    _do_install_oslo_lib "python-openstacksdk"
-    _do_install_oslo_lib "stevedore"
-    _do_install_oslo_lib "taskflow"
-    _do_install_oslo_lib "tooz"
-}
-
-# Restore xtrace
-$_XTRACE_LIB_OSLO
-
-# Tell emacs to use shell-script-mode
-## Local variables:
-## mode: shell-script
-## End:
+# Included for compatibility with grenade, remove in Queens
+source $TOP_DIR/lib/libraries
diff --git a/lib/placement b/lib/placement
index 4755a58..8adbbde 100644
--- a/lib/placement
+++ b/lib/placement
@@ -69,6 +69,7 @@
 # cleanup_placement() - Remove residual data files, anything left over from previous
 # runs that a clean run would need to clean up
 function cleanup_placement {
+    sudo rm -f $(apache_site_config_for nova-placement-api)
     sudo rm -f $(apache_site_config_for placement-api)
 }
 
@@ -102,7 +103,7 @@
 
 function configure_placement_nova_compute {
     iniset $NOVA_CONF placement auth_type "password"
-    iniset $NOVA_CONF placement auth_url "$KEYSTONE_SERVICE_URI/v3"
+    iniset $NOVA_CONF placement auth_url "$KEYSTONE_SERVICE_URI"
     iniset $NOVA_CONF placement username placement
     iniset $NOVA_CONF placement password "$SERVICE_PASSWORD"
     iniset $NOVA_CONF placement user_domain_name "$SERVICE_DOMAIN_NAME"
@@ -148,7 +149,9 @@
 function init_placement {
     if [ "$PLACEMENT_DB_ENABLED" != False ]; then
         recreate_database placement
+        time_start "dbsync"
         $NOVA_BIN_DIR/nova-manage --config-file $NOVA_CONF api_db sync
+        time_stop "dbsync"
     fi
     create_placement_accounts
 }
diff --git a/lib/swift b/lib/swift
index 8fad6b8..e247f15 100644
--- a/lib/swift
+++ b/lib/swift
@@ -174,6 +174,7 @@
 # Test if any Swift services are enabled
 # is_swift_enabled
 function is_swift_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"swift" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"s-" ]] && return 0
     return 1
 }
@@ -541,6 +542,10 @@
         else
             iniset ${testfile} func_test auth_prefix /identity/v2.0/
         fi
+        if is_service_enabled tls-proxy; then
+            iniset ${testfile} func_test cafile ${SSL_BUNDLE_FILE}
+            iniset ${testfile} func_test web_front_end apache2
+        fi
     fi
 
     local user_group
diff --git a/lib/tempest b/lib/tempest
index f19686a..cc65ec7 100644
--- a/lib/tempest
+++ b/lib/tempest
@@ -11,6 +11,7 @@
 #   - ``DEST``, ``FILES``
 #   - ``ADMIN_PASSWORD``
 #   - ``DEFAULT_IMAGE_NAME``
+#   - ``DEFAULT_IMAGE_FILE_NAME``
 #   - ``S3_SERVICE_PORT``
 #   - ``SERVICE_HOST``
 #   - ``BASE_SQL_CONN`` ``lib/database`` declares
@@ -278,6 +279,10 @@
         iniset $TEMPEST_CONFIG identity-feature-enabled api_v2 False
     fi
     iniset $TEMPEST_CONFIG identity auth_version ${TEMPEST_AUTH_VERSION:-v3}
+    if [[ "$TEMPEST_AUTH_VERSION" != "v2.0" ]]; then
+        # we're going to disable v2 admin unless we're using v2.0 by default.
+        iniset $TEMPEST_CONFIG identity-feature-enabled api_v2_admin False
+    fi
 
     if is_service_enabled tls-proxy; then
         iniset $TEMPEST_CONFIG identity ca_certificates_file $SSL_BUNDLE_FILE
@@ -389,24 +394,6 @@
     iniset $TEMPEST_CONFIG network-feature-enabled ipv6_subnet_attributes "$IPV6_SUBNET_ATTRIBUTES_ENABLED"
     iniset $TEMPEST_CONFIG network-feature-enabled port_security $NEUTRON_PORT_SECURITY
 
-    # Orchestration Tests
-    if is_service_enabled heat; then
-        if [[ ! -z "$HEAT_CFN_IMAGE_URL" ]]; then
-            iniset $TEMPEST_CONFIG orchestration image_ref $(basename "${HEAT_CFN_IMAGE_URL%.*}")
-        fi
-        # Nova might not be enabled, especially when we want to test tempest scenario/API that only create Neutron resources
-        if is_service_enabled nova; then
-            # build a specialized heat flavor
-            available_flavors=$(nova flavor-list)
-            if [[ ! ( $available_flavors =~ 'm1.heat' ) ]]; then
-                openstack flavor create --id 451 --ram 512 --disk 0 --vcpus 1 m1.heat
-            fi
-            iniset $TEMPEST_CONFIG orchestration instance_type "m1.heat"
-        fi
-        iniset $TEMPEST_CONFIG orchestration build_timeout 900
-        iniset $TEMPEST_CONFIG orchestration stack_owner_role Member
-    fi
-
     # Scenario
     if [ "$VIRT_DRIVER" = "xenserver" ]; then
         SCENARIO_IMAGE_DIR=${SCENARIO_IMAGE_DIR:-$FILES}
@@ -415,7 +402,7 @@
         iniset $TEMPEST_CONFIG scenario img_container_format ovf
     else
         SCENARIO_IMAGE_DIR=${SCENARIO_IMAGE_DIR:-$FILES}
-        SCENARIO_IMAGE_FILE=$DEFAULT_IMAGE_NAME
+        SCENARIO_IMAGE_FILE=$DEFAULT_IMAGE_FILE_NAME
     fi
     iniset $TEMPEST_CONFIG scenario img_dir $SCENARIO_IMAGE_DIR
     iniset $TEMPEST_CONFIG scenario img_file $SCENARIO_IMAGE_FILE
@@ -426,7 +413,7 @@
         TEMPEST_SSH_NETWORK_NAME=$PHYSICAL_NETWORK
     fi
     # Validation
-    iniset $TEMPEST_CONFIG validation run_validation ${TEMPEST_RUN_VALIDATION:-False}
+    iniset $TEMPEST_CONFIG validation run_validation ${TEMPEST_RUN_VALIDATION:-True}
     iniset $TEMPEST_CONFIG validation ip_version_for_ssh 4
     iniset $TEMPEST_CONFIG validation ssh_timeout $BUILD_TIMEOUT
     iniset $TEMPEST_CONFIG validation image_ssh_user ${DEFAULT_INSTANCE_USER:-cirros}
diff --git a/lib/template b/lib/template
index b92fb40..25d653c 100644
--- a/lib/template
+++ b/lib/template
@@ -41,6 +41,7 @@
 # Test if any XXXX services are enabled
 # is_XXXX_enabled
 function is_XXXX_enabled {
+    [[ ,${DISABLED_SERVICES} =~ ,"XXXX" ]] && return 1
     [[ ,${ENABLED_SERVICES} =~ ,"XX-" ]] && return 0
     return 1
 }
diff --git a/lib/tls b/lib/tls
index 7a7b104..6d67c90 100644
--- a/lib/tls
+++ b/lib/tls
@@ -113,11 +113,11 @@
 certificate             = \$dir/cacert.pem
 private_key             = \$dir/private/cacert.key
 RANDFILE                = \$dir/private/.rand
-default_md              = default
+default_md              = sha256
 
 [ req ]
-default_bits            = 1024
-default_md              = sha1
+default_bits            = 2048
+default_md              = sha256
 
 prompt                  = no
 distinguished_name      = ca_distinguished_name
@@ -212,6 +212,9 @@
     if is_fedora; then
         sudo cp $INT_CA_DIR/ca-chain.pem /usr/share/pki/ca-trust-source/anchors/devstack-chain.pem
         sudo update-ca-trust
+    elif is_suse; then
+        sudo cp $INT_CA_DIR/ca-chain.pem /usr/share/pki/trust/anchors/devstack-chain.pem
+        sudo update-ca-certificates
     elif is_ubuntu; then
         sudo cp $INT_CA_DIR/ca-chain.pem /usr/local/share/ca-certificates/devstack-int.crt
         sudo cp $ROOT_CA_DIR/cacert.pem /usr/local/share/ca-certificates/devstack-root.crt
@@ -345,7 +348,8 @@
 function fix_system_ca_bundle_path {
     if is_service_enabled tls-proxy; then
         local capath
-        capath=$(python -c $'try:\n from requests import certs\n print certs.where()\nexcept ImportError: pass')
+        local python_cmd=${1:-python}
+        capath=$($python_cmd -c $'try:\n from requests import certs\n print (certs.where())\nexcept ImportError: pass')
 
         if [[ ! $capath == "" && ! $capath =~ ^/etc/.* && ! -L $capath ]]; then
             if is_fedora; then
@@ -354,6 +358,9 @@
             elif is_ubuntu; then
                 sudo rm -f $capath
                 sudo ln -s /etc/ssl/certs/ca-certificates.crt $capath
+            elif is_suse; then
+                sudo rm -f $capath
+                sudo ln -s /etc/ssl/ca-bundle.pem $capath
             else
                 echo "Don't know how to set the CA bundle, expect the install to fail."
             fi
@@ -416,6 +423,9 @@
 
     if is_ubuntu; then
         sudo a2enmod ssl
+    elif is_suse; then
+        sudo a2enmod ssl
+        sudo a2enflag SSL
     elif is_fedora; then
         # Fedora enables mod_ssl by default
         :
@@ -522,16 +532,14 @@
     LogFormat "%v %h %l %u %t \"%r\" %>s %b"
 </VirtualHost>
 EOF
+    if is_suse ; then
+        sudo a2enflag SSL
+    fi
     for mod in ssl proxy proxy_http; do
         enable_apache_mod $mod
     done
     enable_apache_site $b_service
-    # Only a reload is required to pull in new vhosts
-    # Note that a restart reliably fails on centos7 and trusty
-    # because apache can't open port 80 because the old apache
-    # still has it open. Using reload fixes trusty but centos7
-    # still doesn't work.
-    reload_apache_server
+    restart_apache_server
 }
 
 # Follow TLS proxy
diff --git a/openrc b/openrc
index 4cdb50e..23c173c 100644
--- a/openrc
+++ b/openrc
@@ -72,17 +72,23 @@
     GLANCE_HOST=${GLANCE_HOST:-$HOST_IP}
 fi
 
-SERVICE_PROTOCOL=${SERVICE_PROTOCOL:-http}
-
 # Identity API version
 export OS_IDENTITY_API_VERSION=${IDENTITY_API_VERSION:-3}
 
+# Ask keystoneauth1 to use keystone
+export OS_AUTH_TYPE=password
+
 # Authenticating against an OpenStack cloud using Keystone returns a **Token**
 # and **Service Catalog**.  The catalog contains the endpoints for all services
 # the user/project has access to - including nova, glance, keystone, swift, ...
 # We currently recommend using the version 3 *identity api*.
 #
-export OS_AUTH_URL=$KEYSTONE_AUTH_URI
+
+# If you don't have a working .stackenv, this is the backup possition
+KEYSTONE_BACKUP=$SERVICE_PROTOCOL://$SERVICE_HOST:5000
+KEYSTONE_AUTH_URI=${KEYSTONE_AUTH_URI:-$KEYSTONE_BACKUP}
+
+export OS_AUTH_URL=${OS_AUTH_URL:-$KEYSTONE_AUTH_URI}
 
 # Currently, in order to use openstackclient with Identity API v3,
 # we need to set the domain which the user and project belong to.
diff --git a/setup.cfg b/setup.cfg
index e4b2888..73d22b5 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -2,7 +2,7 @@
 name = DevStack
 summary = OpenStack DevStack
 description-file =
-    README.md
+    README.rst
 author = OpenStack
 author-email = openstack-dev@lists.openstack.org
 home-page = http://docs.openstack.org/developer/devstack
@@ -15,6 +15,7 @@
 all_files = 1
 build-dir = doc/build
 source-dir = doc/source
+warning-is-error = 1
 
 [pbr]
 warnerrors = True
diff --git a/stack.sh b/stack.sh
index 7a24ed9..fd18651 100755
--- a/stack.sh
+++ b/stack.sh
@@ -2,7 +2,7 @@
 
 # ``stack.sh`` is an opinionated OpenStack developer installation.  It
 # installs and configures various combinations of **Cinder**, **Glance**,
-# **Heat**, **Horizon**, **Keystone**, **Nova**, **Neutron**, and **Swift**
+# **Horizon**, **Keystone**, **Nova**, **Neutron**, and **Swift**
 
 # This script's options can be changed by setting appropriate environment
 # variables.  You can configure things like which git repositories to use,
@@ -27,11 +27,37 @@
 # Make sure custom grep options don't get in the way
 unset GREP_OPTIONS
 
-# Sanitize language settings to avoid commands bailing out
-# with "unsupported locale setting" errors.
+# NOTE(sdague): why do we explicitly set locale when running stack.sh?
+#
+# Devstack is written in bash, and many functions used throughout
+# devstack process text comming off a command (like the ip command)
+# and do transforms using grep, sed, cut, awk on the strings that are
+# returned. Many of these programs are interationalized, which is
+# great for end users, but means that the strings that devstack
+# functions depend upon might not be there in other locales. We thus
+# need to pin the world to an english basis during the runs.
+#
+# Previously we used the C locale for this, every system has it, and
+# it gives us a stable sort order. It does however mean that we
+# effectively drop unicode support.... boo!  :(
+#
+# With python3 being more unicode aware by default, that's not the
+# right option. While there is a C.utf8 locale, some distros are
+# shipping it as C.UTF8 for extra confusingness. And it's support
+# isn't super clear across distros. This is made more challenging when
+# trying to support both out of the box distros, and the gate which
+# uses diskimage builder to build disk images in a different way than
+# the distros do.
+#
+# So... en_US.utf8 it is. That's existed for a very long time. It is a
+# compromise position, but it is the least worse idea at the time of
+# this comment.
+#
+# We also have to unset other variables that might impact LC_ALL
+# taking effect.
 unset LANG
 unset LANGUAGE
-LC_ALL=C
+LC_ALL=en_US.utf8
 export LC_ALL
 
 # Make sure umask is sane
@@ -190,9 +216,12 @@
 fi
 source $TOP_DIR/stackrc
 
+# this installs a devstack-version script to make it easy to report the version back
+write_devstack_version
+
 # Warn users who aren't on an explicitly supported distro, but allow them to
 # override check and attempt installation with ``FORCE=yes ./stack``
-if [[ ! ${DISTRO} =~ (xenial|yakkety|zesty|stretch|jessie|f24|f25|rhel7|kvmibm1) ]]; then
+if [[ ! ${DISTRO} =~ (xenial|yakkety|zesty|stretch|jessie|f24|f25|opensuse-42.2|rhel7|kvmibm1) ]]; then
     echo "WARNING: this script has not been tested on $DISTRO"
     if [[ "$FORCE" != "yes" ]]; then
         die $LINENO "If you wish to run this script anyway run with FORCE=yes"
@@ -498,6 +527,11 @@
         kill 2>&1 $jobs
     fi
 
+    #Remove timing data file
+    if [ -f "$OSCWRAP_TIMER_FILE" ] ; then
+        rm "$OSCWRAP_TIMER_FILE"
+    fi
+
     # Kill the last spinner process
     kill_spinner
 
@@ -561,7 +595,7 @@
 
 # Source project function libraries
 source $TOP_DIR/lib/infra
-source $TOP_DIR/lib/oslo
+source $TOP_DIR/lib/libraries
 source $TOP_DIR/lib/lvm
 source $TOP_DIR/lib/horizon
 source $TOP_DIR/lib/keystone
@@ -573,8 +607,7 @@
 source $TOP_DIR/lib/neutron
 source $TOP_DIR/lib/ldap
 source $TOP_DIR/lib/dstat
-source $TOP_DIR/lib/dlm
-source $TOP_DIR/lib/os_brick
+source $TOP_DIR/lib/etcd3
 
 # Extras Source
 # --------------
@@ -746,6 +779,13 @@
 # Do the ugly hacks for broken packages and distros
 source $TOP_DIR/tools/fixup_stuff.sh
 
+if [[ "$USE_SYSTEMD" == "True" ]]; then
+    pip_install_gr systemd-python
+    # the default rate limit of 1000 messages / 30 seconds is not
+    # sufficient given how verbose our logging is.
+    iniset -sudo /etc/systemd/journald.conf "Journal" "RateLimitBurst" "0"
+    sudo systemctl restart systemd-journald
+fi
 
 # Virtual Environment
 # -------------------
@@ -758,13 +798,12 @@
 # Phase: pre-install
 run_phase stack pre-install
 
+# NOTE(danms): Set global limits before installing anything
+set_systemd_override DefaultLimitNOFILE ${ULIMIT_NOFILE}
+
 install_rpc_backend
 restart_rpc_backend
 
-# NOTE(sdague): dlm install is conditional on one being enabled by configuration
-install_dlm
-configure_dlm
-
 if is_service_enabled $DATABASE_BACKENDS; then
     install_database
 fi
@@ -776,13 +815,17 @@
     install_neutron_agent_packages
 fi
 
+if is_service_enabled etcd3; then
+    install_etcd3
+fi
+
 # Check Out and Install Source
 # ----------------------------
 
 echo_summary "Installing OpenStack project source"
 
-# Install Oslo libraries
-install_oslo
+# Install additional libraries
+install_libs
 
 # Install uwsgi
 install_apache_uwsgi
@@ -800,11 +843,6 @@
     install_neutronclient
 fi
 
-# Install shared libraries
-if is_service_enabled cinder nova; then
-    install_os_brick
-fi
-
 # Setup TLS certs
 if is_service_enabled tls-proxy; then
     configure_CA
@@ -888,6 +926,9 @@
 
 if is_service_enabled tls-proxy; then
     fix_system_ca_bundle_path
+    if python3_enabled ; then
+        fix_system_ca_bundle_path python3
+    fi
 fi
 
 # Extras Install
@@ -904,6 +945,10 @@
     pip_install_gr python-openstackclient
 fi
 
+# Installs alias for osc so that we can collect timing for all
+# osc commands. Alias dies with stack.sh.
+install_oscwrap
+
 if [[ $TRACK_DEPENDS = True ]]; then
     $DEST/.venv/bin/pip freeze > $DEST/requires-post-pip
     if ! diff -Nru $DEST/requires-pre-pip $DEST/requires-post-pip > $DEST/requires.diff; then
@@ -1033,6 +1078,13 @@
 # A better kind of sysstat, with the top process per time slice
 start_dstat
 
+# Etcd
+# -----
+
+# etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines
+if is_service_enabled etcd3; then
+    start_etcd3
+fi
 
 # Keystone
 # --------
@@ -1246,8 +1298,13 @@
 fi
 
 # Create a randomized default value for the key manager's fixed_key
+# NOTE(lyarwood): This is currently set to 36 as a workaround to the following
+# libvirt bug that incorrectly pads passphrases that are a multiple of 16 bytes
+# in length.
+# Unable to use LUKS passphrase that is exactly 16 bytes long
+# https://bugzilla.redhat.com/show_bug.cgi?id=1447297
 if is_service_enabled nova; then
-    iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 32)
+    iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 36)
 fi
 
 # Launch the nova-api and wait for it to answer before continuing
@@ -1279,6 +1336,13 @@
     $NOVA_BIN_DIR/nova-manage --config-file $NM_CONF floating create --ip_range=$TEST_FLOATING_RANGE --pool=$TEST_FLOATING_POOL
 fi
 
+# Start placement before any of the service that are likely to want
+# to use it to manage resource providers.
+if is_service_enabled placement; then
+    echo_summary "Starting Placement"
+    start_placement
+fi
+
 if is_service_enabled neutron; then
     start_neutron
 fi
@@ -1293,10 +1357,6 @@
     start_nova
     create_flavors
 fi
-if is_service_enabled placement; then
-    echo_summary "Starting Placement"
-    start_placement
-fi
 if is_service_enabled cinder; then
     echo_summary "Starting Cinder"
     start_cinder
@@ -1325,10 +1385,6 @@
         USERRC_PARAMS="$USERRC_PARAMS --os-cacert $SSL_BUNDLE_FILE"
     fi
 
-    if [[ "$HEAT_STANDALONE" = "True" ]]; then
-        USERRC_PARAMS="$USERRC_PARAMS --heat-url http://$HEAT_API_HOST:$HEAT_API_PORT/v1"
-    fi
-
     $TOP_DIR/tools/create_userrc.sh $USERRC_PARAMS
 fi
 
@@ -1468,12 +1524,28 @@
 
 # Warn that a deprecated feature was used
 if [[ -n "$DEPRECATED_TEXT" ]]; then
-    echo_summary "WARNING: $DEPRECATED_TEXT"
+    echo
+    echo -e "WARNING: $DEPRECATED_TEXT"
+    echo
 fi
 
+# If USE_SYSTEMD is enabled, tell the user about using it.
+if [[ "$USE_SYSTEMD" == "True" ]]; then
+    echo
+    echo "Services are running under systemd unit files."
+    echo "For more information see: "
+    echo "https://docs.openstack.org/developer/devstack/systemd.html"
+    echo
+fi
+
+# devstack version
+devstack-version
+echo
+
 # Indicate how long this took to run (bash maintained variable ``SECONDS``)
 echo_summary "stack.sh completed in $SECONDS seconds."
 
+
 # Restore/close logging file descriptors
 exec 1>&3
 exec 2>&3
diff --git a/stackrc b/stackrc
index ed1cf6e..5da9a2a 100644
--- a/stackrc
+++ b/stackrc
@@ -65,7 +65,7 @@
     # Dashboard
     ENABLED_SERVICES+=,horizon
     # Additional services
-    ENABLED_SERVICES+=,rabbit,tempest,mysql,dstat
+    ENABLED_SERVICES+=,rabbit,tempest,mysql,etcd3,dstat
 fi
 
 # Global toggle for enabling services under mod_wsgi. If this is set to
@@ -80,12 +80,19 @@
 # Set the root URL for Horizon
 HORIZON_APACHE_ROOT="/dashboard"
 
+# TODO(sdague): Queens
+#
+# All the non systemd paths should be removed in queens, they only
+# exist in Pike to support testing from grenade. Ensure that all this
+# is cleaned up and purged, which should dramatically simplify the
+# devstack codebase.
+
 # Whether to use 'dev mode' for screen windows. Dev mode works by
 # stuffing text into the screen windows so that a developer can use
 # ctrl-c, up-arrow, enter to restart the service. Starting services
 # this way is slightly unreliable, and a bit slower, so this can
 # be disabled for automated testing by setting this value to False.
-USE_SCREEN=$(trueorfalse True USE_SCREEN)
+USE_SCREEN=$(trueorfalse False USE_SCREEN)
 
 # Whether to use SYSTEMD to manage services
 USE_SYSTEMD=$(trueorfalse False USE_SYSTEMD)
@@ -93,16 +100,11 @@
 if [[ "$USER_UNITS" == "True" ]]; then
     SYSTEMD_DIR="$HOME/.local/share/systemd/user"
     SYSTEMCTL="systemctl --user"
-    JOURNALCTL_F="journalctl -f -o short-precise --user-unit"
 else
     SYSTEMD_DIR="/etc/systemd/system"
     SYSTEMCTL="sudo systemctl"
-    JOURNALCTL_F="journalctl -f -o short-precise --unit"
 fi
 
-if [[ "$USE_SYSTEMD" == "True" ]]; then
-    USE_SCREEN=False
-fi
 
 # Whether or not to enable Kernel Samepage Merging (KSM) if available.
 # This allows programs that mark their memory as mergeable to share
@@ -157,6 +159,19 @@
     source $RC_DIR/.localrc.auto
 fi
 
+# TODO(sdague): Delete all this in Queens.
+if [[ "$USE_SYSTEMD" == "True" ]]; then
+    USE_SCREEN=False
+fi
+# if we are forcing off USE_SCREEN (as we do in the gate), force on
+# systemd. This allows us to drop one of 3 paths through the code.
+if [[ "$USE_SCREEN" == "False" ]]; then
+    # Remove in Pike: this gets us through grenade upgrade
+    if [[ "$GRENADE_PHASE" != "target" ]]; then
+        USE_SYSTEMD="True"
+    fi
+fi
+
 # Default for log coloring is based on interactive-or-not.
 # Baseline assumption is that non-interactive invocations are for CI,
 # where logs are to be presented as browsable text files; hence color
@@ -257,6 +272,7 @@
 # Setting the variable to 'ALL' will activate the download for all
 # libraries.
 
+DEVSTACK_SERIES="pike"
 
 ##############
 #
@@ -326,6 +342,11 @@
 GITREPO["python-brick-cinderclient-ext"]=${BRICK_CINDERCLIENT_REPO:-${GIT_BASE}/openstack/python-brick-cinderclient-ext.git}
 GITBRANCH["python-brick-cinderclient-ext"]=${BRICK_CINDERCLIENT_BRANCH:-master}
 
+# python barbican client library
+GITREPO["python-barbicanclient"]=${BARBICANCLIENT_REPO:-${GIT_BASE}/openstack/python-barbicanclient.git}
+GITBRANCH["python-barbicanclient"]=${BARBICANCLIENT_BRANCH:-master}
+GITDIR["python-barbicanclient"]=$DEST/python-barbicanclient
+
 # python glance client library
 GITREPO["python-glanceclient"]=${GLANCECLIENT_REPO:-${GIT_BASE}/openstack/python-glanceclient.git}
 GITBRANCH["python-glanceclient"]=${GLANCECLIENT_BRANCH:-master}
@@ -501,14 +522,6 @@
 GITREPO["glance_store"]=${GLANCE_STORE_REPO:-${GIT_BASE}/openstack/glance_store.git}
 GITBRANCH["glance_store"]=${GLANCE_STORE_BRANCH:-master}
 
-# heat-cfntools server agent
-HEAT_CFNTOOLS_REPO=${HEAT_CFNTOOLS_REPO:-${GIT_BASE}/openstack/heat-cfntools.git}
-HEAT_CFNTOOLS_BRANCH=${HEAT_CFNTOOLS_BRANCH:-master}
-
-# heat example templates and elements
-HEAT_TEMPLATES_REPO=${HEAT_TEMPLATES_REPO:-${GIT_BASE}/openstack/heat-templates.git}
-HEAT_TEMPLATES_BRANCH=${HEAT_TEMPLATES_BRANCH:-master}
-
 # django openstack_auth library
 GITREPO["django_openstack_auth"]=${HORIZONAUTH_REPO:-${GIT_BASE}/openstack/django_openstack_auth.git}
 GITBRANCH["django_openstack_auth"]=${HORIZONAUTH_BRANCH:-master}
@@ -563,6 +576,10 @@
 GITBRANCH["neutron-lib"]=${NEUTRON_LIB_BRANCH:-master}
 GITDIR["neutron-lib"]=$DEST/neutron-lib
 
+# os-traits library for resource provider traits in the placement service
+GITREPO["os-traits"]=${OS_TRAITS_REPO:-${GIT_BASE}/openstack/os-traits.git}
+GITBRANCH["os-traits"]=${OS_TRAITS_BRANCH:-master}
+
 ##################
 #
 #  TripleO / Heat Agent Components
@@ -600,7 +617,7 @@
 
 # a websockets/html5 or flash powered VNC console for vm instances
 NOVNC_REPO=${NOVNC_REPO:-https://github.com/kanaka/noVNC.git}
-NOVNC_BRANCH=${NOVNC_BRANCH:-master}
+NOVNC_BRANCH=${NOVNC_BRANCH:-stable/v0.6}
 
 # a websockets/html5 or flash powered SPICE console for vm instances
 SPICE_REPO=${SPICE_REPO:-http://anongit.freedesktop.org/git/spice/spice-html5.git}
@@ -648,6 +665,8 @@
         ;;
 esac
 
+# By default, devstack will use Ubuntu Cloud Archive.
+ENABLE_UBUNTU_CLOUD_ARCHIVE=$(trueorfalse True ENABLE_UBUNTU_CLOUD_ARCHIVE)
 
 # Images
 # ------
@@ -687,17 +706,21 @@
             case "$LIBVIRT_TYPE" in
                 lxc) # the cirros root disk in the uec tarball is empty, so it will not work for lxc
                     DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs}
-                    IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img.gz";;
+                    DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_FILE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img.gz}
+                    IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/${DEFAULT_IMAGE_FILE_NAME}";;
                 *) # otherwise, use the qcow image
-                    DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img}
-                    IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img";;
+                    DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk}
+                    DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_FILE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img}
+                    IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/${DEFAULT_IMAGE_FILE_NAME}";;
                 esac
             ;;
         vsphere)
             DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-0.3.2-i386-disk.vmdk}
-            IMAGE_URLS+="http://partnerweb.vmware.com/programs/vmdkimage/cirros-0.3.2-i386-disk.vmdk";;
+            DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_FILE_NAME:-$DEFAULT_IMAGE_NAME}
+            IMAGE_URLS+="http://partnerweb.vmware.com/programs/vmdkimage/${DEFAULT_IMAGE_FILE_NAME}";;
         xenserver)
             DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-0.3.4-x86_64-disk}
+            DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_NAME:-cirros-0.3.4-x86_64-disk.vhd.tgz}
             IMAGE_URLS+="http://ca.downloads.xensource.com/OpenStack/cirros-0.3.4-x86_64-disk.vhd.tgz"
             IMAGE_URLS+=",http://download.cirros-cloud.net/${CIRROS_VERSION}/cirros-${CIRROS_VERSION}-x86_64-uec.tar.gz";;
     esac
@@ -912,6 +935,9 @@
 # ``LOGDIR`` is always set at this point so it is not useful as a 'enable' for service logs
 # ``SCREEN_LOGDIR`` may be set, it is useful to enable the compat symlinks
 
+# System-wide ulimit file descriptors override
+ULIMIT_NOFILE=${ULIMIT_NOFILE:-2048}
+
 # Local variables:
 # mode: shell-script
 # End:
diff --git a/tests/test_libs_from_pypi.sh b/tests/test_libs_from_pypi.sh
index 608ef6a..5b4ff32 100755
--- a/tests/test_libs_from_pypi.sh
+++ b/tests/test_libs_from_pypi.sh
@@ -39,11 +39,11 @@
 ALL_LIBS+=" python-openstackclient osc-lib os-client-config oslo.rootwrap"
 ALL_LIBS+=" oslo.i18n oslo.utils python-openstacksdk python-swiftclient"
 ALL_LIBS+=" python-neutronclient tooz ceilometermiddleware oslo.policy"
-ALL_LIBS+=" debtcollector os-brick automaton futurist oslo.service"
+ALL_LIBS+=" debtcollector os-brick os-traits automaton futurist oslo.service"
 ALL_LIBS+=" oslo.cache oslo.reports osprofiler cursive"
 ALL_LIBS+=" keystoneauth ironic-lib neutron-lib oslo.privsep"
 ALL_LIBS+=" diskimage-builder os-vif python-brick-cinderclient-ext"
-ALL_LIBS+=" castellan"
+ALL_LIBS+=" castellan python-barbicanclient"
 
 # Generate the above list with
 # echo ${!GITREPO[@]}
diff --git a/tests/test_refs.sh b/tests/test_refs.sh
index bccca5d..65848cd 100755
--- a/tests/test_refs.sh
+++ b/tests/test_refs.sh
@@ -15,7 +15,7 @@
 
 echo "Ensuring we don't have crazy refs"
 
-REFS=`grep BRANCH stackrc | grep -v -- '-master'`
+REFS=`grep BRANCH stackrc | grep -v -- '-master' | grep -v 'NOVNC_BRANCH'`
 rc=$?
 if [[ $rc -eq 0 ]]; then
     echo "Branch defaults must be master. Found:"
diff --git a/tools/dstat.sh b/tools/dstat.sh
index ae7306e..01c6d9b 100755
--- a/tools/dstat.sh
+++ b/tools/dstat.sh
@@ -13,10 +13,10 @@
 LOGDIR=$1
 
 # Command line arguments for primary DStat process.
-DSTAT_OPTS="-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap"
+DSTAT_OPTS="-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap --tcp"
 
 # Command-line arguments for secondary background DStat process.
-DSTAT_CSV_OPTS="-tcmndrylpg --output $LOGDIR/dstat-csv.log"
+DSTAT_CSV_OPTS="-tcmndrylpg --tcp --output $LOGDIR/dstat-csv.log"
 
 # Execute and background the secondary dstat process and discard its output.
 dstat $DSTAT_CSV_OPTS >& /dev/null &
diff --git a/tools/fixup_stuff.sh b/tools/fixup_stuff.sh
index f3ba702..0b78bde 100755
--- a/tools/fixup_stuff.sh
+++ b/tools/fixup_stuff.sh
@@ -72,7 +72,10 @@
 # We've found that Libvirt on Xenial is flaky and crashes enough to be
 # a regular top e-r bug. Opt into Ubuntu Cloud Archive if on Xenial to
 # get newer Libvirt.
-if [[ "$DISTRO" = "xenial" ]]; then
+# Make it possible to switch this based on an environment variable as
+# libvirt 2.5.0 doesn't handle nested virtualization quite well and this
+# is required for the trove development environment.
+if [[ "${ENABLE_UBUNTU_CLOUD_ARCHIVE}" == "True" && "$DISTRO" = "xenial" ]]; then
     # This pulls in apt-add-repository
     install_package "software-properties-common"
     # Use UCA for newer libvirt. Should give us libvirt 2.5.0.
@@ -82,15 +85,17 @@
         source /etc/ci/mirror_info.sh
 
         sudo apt-add-repository -y "deb $NODEPOOL_UCA_MIRROR xenial-updates/ocata main"
-
-        # Disable use of libvirt wheel here as presence of mirror implies
-        # presence of cached wheel build against older libvirt binary.
-        # TODO(clarkb) figure out how to use wheel again.
-        sudo bash -c 'echo "no-binary = libvirt-python" >> /etc/pip.conf'
     else
         # Otherwise use upstream UCA
         sudo add-apt-repository -y cloud-archive:ocata
     fi
+
+    # Disable use of libvirt wheel since a cached wheel build might be
+    # against older libvirt binary.  Particularly a problem if using
+    # the openstack wheel mirrors, but can hit locally too.
+    # TODO(clarkb) figure out how to use upstream wheel again.
+    iniset -sudo /etc/pip.conf "global" "no-binary" "libvirt-python"
+
     # Force update our APT repos, since we added UCA above.
     REPOS_UPDATED=False
     apt_get_update
diff --git a/tox.ini b/tox.ini
index 55a06d0..cc7c544 100644
--- a/tox.ini
+++ b/tox.ini
@@ -37,8 +37,8 @@
 deps =
    Pygments
    docutils
-   sphinx>=1.1.2,<1.2
-   pbr>=0.6,!=0.7,<1.0
+   sphinx>=1.5.1,<1.6.1
+   pbr>=2.0.0,!=2.1.0
    oslosphinx
    nwdiag
    blockdiag
@@ -52,8 +52,8 @@
 
 [testenv:venv]
 deps =
-   pbr>=0.6,!=0.7,<1.0
-   sphinx>=1.1.2,<1.2
+   pbr>=2.0.0,!=2.1.0
+   sphinx>=1.5.1,<1.6.1
    oslosphinx
    blockdiag
    sphinxcontrib-blockdiag
diff --git a/unstack.sh b/unstack.sh
index 485fed7..77a151f 100755
--- a/unstack.sh
+++ b/unstack.sh
@@ -69,7 +69,7 @@
 source $TOP_DIR/lib/neutron
 source $TOP_DIR/lib/ldap
 source $TOP_DIR/lib/dstat
-source $TOP_DIR/lib/dlm
+source $TOP_DIR/lib/etcd3
 
 # Extras Source
 # --------------
@@ -162,6 +162,11 @@
     cleanup_neutron
 fi
 
+if is_service_enabled etcd3; then
+    stop_etcd3
+    cleanup_etcd3
+fi
+
 if is_service_enabled dstat; then
     stop_dstat
 fi