Merge "Provide timings for OSC during devstack run"
diff --git a/.gitignore b/.gitignore
index d1781bc..7967e14 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,7 @@
files/pip-*
files/get-pip.py*
files/ir-deploy*
+files/ironic-inspector*
local.conf
local.sh
localrc
diff --git a/HACKING.rst b/HACKING.rst
index b76cb6c..fc67f09 100644
--- a/HACKING.rst
+++ b/HACKING.rst
@@ -322,7 +322,7 @@
Review Criteria
-===============
+---------------
There are some broad criteria that will be followed when reviewing
your change
@@ -364,3 +364,26 @@
* **Reviewers** -- please see ``MAINTAINERS.rst`` for a list of people
that should be added to reviews of various sub-systems.
+
+
+Making Changes, Testing, and CI
+-------------------------------
+
+Changes to Devstack are tested by automated continuous integration jobs
+that run on a variety of Linux Distros using a handful of common
+configurations. What this means is that every change to Devstack is
+self testing. One major benefit of this is that developers do not
+typically need to add new non voting test jobs to add features to
+Devstack. Instead the features can be added, then if testing passes
+with the feature enabled the change is ready to merge (pending code
+review).
+
+A concrete example of this was the switch from screen based service
+management to systemd based service management. No new jobs were
+created for this. Instead the features were added to devstack, tested
+locally and in CI using a change that enabled the feature, then once
+the enabling change was passing and the new behavior communicated and
+documented it was merged.
+
+Using this process has been proven to be effective and leads to
+quicker implementation of desired features.
diff --git a/README.md b/README.rst
similarity index 88%
rename from README.md
rename to README.rst
index ff5598b..b4240bd 100644
--- a/README.md
+++ b/README.rst
@@ -1,6 +1,7 @@
DevStack is a set of scripts and utilities to quickly deploy an OpenStack cloud.
-# Goals
+Goals
+=====
* To quickly build dev OpenStack environments in a clean Ubuntu or Fedora
environment
@@ -20,12 +21,13 @@
networking configuration. We strongly recommend that you run `stack.sh`
in a clean and disposable vm when you are first getting started.
-# Versions
+Versions
+========
The DevStack master branch generally points to trunk versions of OpenStack
components. For older, stable versions, look for branches named
stable/[release] in the DevStack repo. For example, you can do the
-following to create a Newton OpenStack cloud:
+following to create a Newton OpenStack cloud::
git checkout stable/newton
./stack.sh
@@ -38,7 +40,8 @@
GLANCE_REPO=git://git.openstack.org/openstack/glance.git
GLANCE_BRANCH=milestone-proposed
-# Start A Dev Cloud
+Start A Dev Cloud
+=================
Installing in a dedicated disposable VM is safer than installing on your
dev machine! Plus you can pick one of the supported Linux distros for
@@ -54,14 +57,15 @@
* Keystone: http://myhost:5000/v2.0/
We also provide an environment file that you can use to interact with your
-cloud via CLI:
+cloud via CLI::
# source openrc file to load your environment with OpenStack CLI creds
. openrc
# list instances
- nova list
+ openstack server list
-# DevStack Execution Environment
+DevStack Execution Environment
+==============================
DevStack runs rampant over the system it runs on, installing things and
uninstalling other things. Running this on a system you care about is a recipe
@@ -81,10 +85,12 @@
it runs under. Many people simply use their usual login (the default
'ubuntu' login on a UEC image for example).
-# Customizing
+Customizing
+===========
DevStack can be extensively configured via the configuration file
`local.conf`. It is likely that you will need to provide and modify
this file if you want anything other than the most basic setup. Start
-by reading the [configuration guide](doc/source/configuration.rst) for
-details of the configuration file and the many available options.
+by reading the `configuration guide
+<https://docs.openstack.org/developer/devstack/configuration.html>`_
+for details of the configuration file and the many available options.
diff --git a/clean.sh b/clean.sh
index 90b21eb..9ffe3be 100755
--- a/clean.sh
+++ b/clean.sh
@@ -64,13 +64,8 @@
done
fi
-# See if there is anything running...
-# need to adapt when run_service is merged
-SESSION=$(screen -ls | awk '/[0-9].stack/ { print $1 }')
-if [[ -n "$SESSION" ]]; then
- # Let unstack.sh do its thing first
- $TOP_DIR/unstack.sh --all
-fi
+# Let unstack.sh do its thing first
+$TOP_DIR/unstack.sh --all
# Run extras
# ==========
@@ -130,6 +125,13 @@
sudo rm -rf $SCREEN_LOGDIR
fi
+# Clean out the sytemd user unit files if systemd was used.
+if [[ "$USE_SYSTEMD" = "True" ]]; then
+ sudo find $SYSTEMD_DIR -type f -name '*devstack@*service' -delete
+ # Make systemd aware of the deletion.
+ $SYSTEMCTL daemon-reload
+fi
+
# Clean up venvs
DIRS_TO_CLEAN="$WHEELHOUSE ${PROJECT_VENV[@]} .config/openstack"
rm -rf $DIRS_TO_CLEAN
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 53ae82f..66b8702 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -278,43 +278,22 @@
LOGDAYS=1
-The some of the project logs (Nova, Cinder, etc) will be colorized by
-default (if ``SYSLOG`` is not set below); this can be turned off by
-setting ``LOG_COLOR`` to ``False``.
-
- ::
+Some coloring is used during the DevStack runs to make it easier to
+see what is going on. This can be disabled with::
LOG_COLOR=False
Logging the Service Output
~~~~~~~~~~~~~~~~~~~~~~~~~~
-DevStack will log the ``stdout`` output of the services it starts.
-When using ``screen`` this logs the output in the screen windows to a
-file. Without ``screen`` this simply redirects stdout of the service
-process to a file in ``LOGDIR``.
+By default, services run under ``systemd`` and are natively logging to
+the systemd journal.
- ::
+To query the logs use the ``journalctl`` command, such as::
- LOGDIR=$DEST/logs
+ journalctl --unit devstack@*
-Note the use of ``DEST`` to locate the main install directory; this
-is why we suggest setting it in ``local.conf``.
-
-Enabling Syslog
-~~~~~~~~~~~~~~~
-
-Logging all services to a single syslog can be convenient. Enable
-syslogging by setting ``SYSLOG`` to ``True``. If the destination log
-host is not localhost ``SYSLOG_HOST`` and ``SYSLOG_PORT`` can be used
-to direct the message stream to the log host.
-
- ::
-
- SYSLOG=True
- SYSLOG_HOST=$HOST_IP
- SYSLOG_PORT=516
-
+More examples can be found in :ref:`journalctl-examples`.
Example Logging Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -326,7 +305,6 @@
[[local|localrc]]
DEST=/opt/stack/
- LOGDIR=$DEST/logs
LOGFILE=$LOGDIR/stack.sh.log
LOG_COLOR=False
@@ -587,9 +565,7 @@
Swift is disabled by default. When enabled, it is configured with
only one replica to avoid being IO/memory intensive on a small
-VM. When running with only one replica the account, container and
-object services will run directly in screen. The others services like
-replicator, updaters or auditor runs in background.
+VM.
If you would like to enable Swift you can add this to your ``localrc``
section:
@@ -630,32 +606,9 @@
act as a S3 endpoint for Keystone so effectively replacing the
``nova-objectstore``.
-Only Swift proxy server is launched in the screen session all other
+Only Swift proxy server is launched in the systemd system all other
services are started in background and managed by ``swift-init`` tool.
-Heat
-~~~~
-
-Heat is disabled by default (see ``stackrc`` file). To enable it
-explicitly you'll need the following settings in your ``localrc``
-section
-
-::
-
- enable_service heat h-api h-api-cfn h-api-cw h-eng
-
-Heat can also run in standalone mode, and be configured to orchestrate
-on an external OpenStack cloud. To launch only Heat in standalone mode
-you'll need the following settings in your ``localrc`` section
-
-::
-
- disable_all_services
- enable_service rabbit mysql heat h-api h-api-cfn h-api-cw h-eng
- HEAT_STANDALONE=True
- KEYSTONE_SERVICE_HOST=...
- KEYSTONE_AUTH_HOST=...
-
Tempest
~~~~~~~
diff --git a/doc/source/development.rst b/doc/source/development.rst
index 776ac6c..957de9b 100644
--- a/doc/source/development.rst
+++ b/doc/source/development.rst
@@ -8,56 +8,33 @@
Inspecting Services
===================
-By default most services in DevStack are running in a `screen
-<https://www.gnu.org/software/screen/manual/screen.html>`_
-session.
+By default most services in DevStack are running as `systemd` units
+named `devstack@$servicename.service`. You can see running services
+with.
.. code-block:: bash
- os3:~> screen -list
- There is a screen on:
- 28994.stack (08/10/2016 09:01:33 PM) (Detached)
- 1 Socket in /var/run/screen/S-sdague.
+ sudo systemctl status "devstack@*"
-You can attach to this screen session using ``screen -r`` which gives
-you a view of the services in action.
-
-.. image:: assets/images/screen_session_1.png
- :width: 100%
-
-Basic Screen Commands
----------------------
-
-The following minimal commands will be useful to using screen:
-
-* ``ctrl-a n`` - go to next window. Next is assumed to be right of
- current window.
-* ``ctrl-a p`` - go to previous window. Previous is assumed to be left
- of current window.
-* ``ctrl-a [`` - entry copy/scrollback mode. This allows you to
- navigate back through the logs with the up arrow.
-* ``ctrl-a d`` - detach from screen. Gets you back to a normal
- terminal, while leaving everything running.
-
-For more about using screen, see the excellent `screen manual
-<https://www.gnu.org/software/screen/manual/screen.html>`_.
+To learn more about the basics of systemd, see :doc:`/systemd`
Patching a Service
==================
If you want to make a quick change to a running service the easiest
-way to do this is:
+way to do that is to change the code directly in /opt/stack/$service
+and then restart the affected daemons.
-* attach to screen
-* navigate to the window in question
-* ``ctrl-c`` to kill the service
-* make appropriate changes to the code
-* ``up arrow`` in the screen window to display the command used to run
- that service
-* ``enter`` to restart the service
+.. code-block:: bash
-This works for services, except those running under Apache (currently
-just ``keystone`` by default).
+ sudo systemctl restart devstack@n-cpu.service
+
+If your change impacts more than one daemon you can restart by
+wildcard as well.
+
+.. code-block:: bash
+
+ sudo systemctl restart "devstack@n-*"
.. warning::
@@ -102,14 +79,6 @@
NOVA_BRANCH=refs/changes/10/353710/1
-Testing Changes to Apache Based Services
-========================================
-
-When testing changes to Apache based services, such as ``keystone``,
-you can either use the Testing a Patch Series approach above, or make
-changes in the code tree and issue an apache restart.
-
-
Testing Changes to Libraries
============================
@@ -132,9 +101,17 @@
OSLOPOLICY_REPO=/home/sdague/oslo.policy
OSLOPOLICY_BRANCH=better_exception
-Because libraries are used by many services, library changes really
-need to go through a full ``./unstack.sh && ./stack.sh`` to see your
-changes in action.
+As libraries are not installed `editable` by pip, after you make any
+local changes you will need to:
-To figure out the repo / branch names for every library that's
-supported, you'll need to read the devstack source.
+* cd to top of library path
+* sudo pip install -U .
+* restart all services you want to use the new library
+
+You can do that with wildcards such as
+
+.. code-block:: bash
+
+ sudo systemctl restart "devstack@n-*"
+
+which will restart all nova services.
diff --git a/doc/source/faq.rst b/doc/source/faq.rst
index f03304f..a186336 100644
--- a/doc/source/faq.rst
+++ b/doc/source/faq.rst
@@ -41,8 +41,9 @@
~~~~~~~~~~~~~~~~~~~~~
Unlike packages, DevStack leaves your cloud ready to develop -
-checkouts of the code and services running in screen. However, many
-people are doing the hard work of packaging and recipes for production
+checkouts of the code and services running locally under systemd,
+making it easy to hack on and test new patches. However, many people
+are doing the hard work of packaging and recipes for production
deployments.
Why isn't $MY\_FAVORITE\_DISTRO supported?
@@ -158,16 +159,6 @@
often good enough for a single-node installation. And in an extreme
case, use ``clean.sh`` to eradicate it and try again.
-Configure ``local.conf`` thusly:
-
- ::
-
- [[local|localrc]]
- HEAT_STANDALONE=True
- ENABLED_SERVICES=rabbit,mysql,heat,h-api,h-api-cfn,h-api-cw,h-eng
- KEYSTONE_SERVICE_HOST=<keystone-host>
- KEYSTONE_AUTH_HOST=<keystone-host>
-
Why are my configuration changes ignored?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/doc/source/guides/devstack-with-lbaas-v2.rst b/doc/source/guides/devstack-with-lbaas-v2.rst
index 21bea99..4ed64bf 100644
--- a/doc/source/guides/devstack-with-lbaas-v2.rst
+++ b/doc/source/guides/devstack-with-lbaas-v2.rst
@@ -45,7 +45,7 @@
# Horizon
ENABLED_SERVICES+=,horizon
# Nova
- ENABLED_SERVICES+=,n-api,n-crt,n-cpu,n-cond,n-sch
+ ENABLED_SERVICES+=,n-api,n-cpu,n-cond,n-sch
# Glance
ENABLED_SERVICES+=,g-api,g-reg
# Neutron
diff --git a/doc/source/guides/multinode-lab.rst b/doc/source/guides/multinode-lab.rst
index 484ebba..1a8ddbc 100644
--- a/doc/source/guides/multinode-lab.rst
+++ b/doc/source/guides/multinode-lab.rst
@@ -175,7 +175,7 @@
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
- ENABLED_SERVICES=n-cpu,q-agt,n-api-meta,c-vol
+ ENABLED_SERVICES=n-cpu,q-agt,n-api-meta,c-vol,placement-client
NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"
VNCSERVER_LISTEN=$HOST_IP
diff --git a/doc/source/index.rst b/doc/source/index.rst
index c3bac9d..47087c5 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -63,7 +63,7 @@
::
- $ sudo tee <<<"stack ALL=(ALL) NOPASSWD: ALL" /etc/sudoers
+ $ echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
$ sudo su - stack
Download DevStack
@@ -142,3 +142,12 @@
Get :doc:`the big picture <overview>` of what we are trying to do
with devstack, and help us by :doc:`contributing to the project
<hacking>`.
+
+Contents
+--------
+
+.. toctree::
+ :glob:
+ :maxdepth: 2
+
+ *
diff --git a/doc/source/overview.rst b/doc/source/overview.rst
index d245035..c07a8e6 100644
--- a/doc/source/overview.rst
+++ b/doc/source/overview.rst
@@ -20,11 +20,11 @@
*The OpenStack Technical Committee (TC) has defined the current CI
strategy to include the latest Ubuntu release and the latest RHEL
-release (for Python 2.6 testing).*
+release.*
- Ubuntu: current LTS release plus current development release
- Fedora: current release plus previous release
-- RHEL: current major release
+- RHEL/Centos: current major release
- Other OS platforms may continue to be included but the maintenance of
those platforms shall not be assumed simply due to their presence.
Having a listed point-of-contact for each additional OS will greatly
@@ -38,7 +38,6 @@
*As packaged by the host OS*
- MySQL
-- PostgreSQL
Queues
------
@@ -46,7 +45,6 @@
*As packaged by the host OS*
- Rabbit
-- Qpid
Web Server
----------
@@ -58,9 +56,6 @@
OpenStack Network
-----------------
-*Defaults to nova network, optionally use neutron*
-
-- Nova Network: FlatDHCP
- Neutron: A basic configuration approximating the original FlatDHCP
mode using linuxbridge or OpenVSwitch.
@@ -68,9 +63,8 @@
--------
The default services configured by DevStack are Identity (keystone),
-Object Storage (swift), Image Service (glance), Block Storage (cinder),
-Compute (nova), Networking (nova), Dashboard (horizon), Orchestration
-(heat)
+Object Storage (swift), Image Service (glance), Block Storage
+(cinder), Compute (nova), Networking (neutron), Dashboard (horizon)
Additional services not included directly in DevStack can be tied in to
``stack.sh`` using the :doc:`plugin mechanism <plugins>` to call
@@ -80,8 +74,7 @@
-------------------
- single node
-- multi-node is not tested regularly by the core team, and even then
- only minimal configurations are reviewed
+- multi-node configurations as are tested by the gate
Exercises
---------
diff --git a/doc/source/plugin-registry.rst b/doc/source/plugin-registry.rst
index cc55c0b..9bfedcf 100644
--- a/doc/source/plugin-registry.rst
+++ b/doc/source/plugin-registry.rst
@@ -44,9 +44,11 @@
devstack-plugin-amqp1 `git://git.openstack.org/openstack/devstack-plugin-amqp1 <https://git.openstack.org/cgit/openstack/devstack-plugin-amqp1>`__
devstack-plugin-bdd `git://git.openstack.org/openstack/devstack-plugin-bdd <https://git.openstack.org/cgit/openstack/devstack-plugin-bdd>`__
devstack-plugin-ceph `git://git.openstack.org/openstack/devstack-plugin-ceph <https://git.openstack.org/cgit/openstack/devstack-plugin-ceph>`__
+devstack-plugin-container `git://git.openstack.org/openstack/devstack-plugin-container <https://git.openstack.org/cgit/openstack/devstack-plugin-container>`__
devstack-plugin-glusterfs `git://git.openstack.org/openstack/devstack-plugin-glusterfs <https://git.openstack.org/cgit/openstack/devstack-plugin-glusterfs>`__
devstack-plugin-hdfs `git://git.openstack.org/openstack/devstack-plugin-hdfs <https://git.openstack.org/cgit/openstack/devstack-plugin-hdfs>`__
devstack-plugin-kafka `git://git.openstack.org/openstack/devstack-plugin-kafka <https://git.openstack.org/cgit/openstack/devstack-plugin-kafka>`__
+devstack-plugin-libvirt-qemu `git://git.openstack.org/openstack/devstack-plugin-libvirt-qemu <https://git.openstack.org/cgit/openstack/devstack-plugin-libvirt-qemu>`__
devstack-plugin-mariadb `git://git.openstack.org/openstack/devstack-plugin-mariadb <https://git.openstack.org/cgit/openstack/devstack-plugin-mariadb>`__
devstack-plugin-nfs `git://git.openstack.org/openstack/devstack-plugin-nfs <https://git.openstack.org/cgit/openstack/devstack-plugin-nfs>`__
devstack-plugin-pika `git://git.openstack.org/openstack/devstack-plugin-pika <https://git.openstack.org/cgit/openstack/devstack-plugin-pika>`__
@@ -61,13 +63,14 @@
fuxi `git://git.openstack.org/openstack/fuxi <https://git.openstack.org/cgit/openstack/fuxi>`__
gce-api `git://git.openstack.org/openstack/gce-api <https://git.openstack.org/cgit/openstack/gce-api>`__
glare `git://git.openstack.org/openstack/glare <https://git.openstack.org/cgit/openstack/glare>`__
-gnocchi `git://git.openstack.org/openstack/gnocchi <https://git.openstack.org/cgit/openstack/gnocchi>`__
group-based-policy `git://git.openstack.org/openstack/group-based-policy <https://git.openstack.org/cgit/openstack/group-based-policy>`__
heat `git://git.openstack.org/openstack/heat <https://git.openstack.org/cgit/openstack/heat>`__
horizon-mellanox `git://git.openstack.org/openstack/horizon-mellanox <https://git.openstack.org/cgit/openstack/horizon-mellanox>`__
ironic `git://git.openstack.org/openstack/ironic <https://git.openstack.org/cgit/openstack/ironic>`__
ironic-inspector `git://git.openstack.org/openstack/ironic-inspector <https://git.openstack.org/cgit/openstack/ironic-inspector>`__
ironic-staging-drivers `git://git.openstack.org/openstack/ironic-staging-drivers <https://git.openstack.org/cgit/openstack/ironic-staging-drivers>`__
+ironic-ui `git://git.openstack.org/openstack/ironic-ui <https://git.openstack.org/cgit/openstack/ironic-ui>`__
+k8s-cloud-provider `git://git.openstack.org/openstack/k8s-cloud-provider <https://git.openstack.org/cgit/openstack/k8s-cloud-provider>`__
karbor `git://git.openstack.org/openstack/karbor <https://git.openstack.org/cgit/openstack/karbor>`__
karbor-dashboard `git://git.openstack.org/openstack/karbor-dashboard <https://git.openstack.org/cgit/openstack/karbor-dashboard>`__
keystone `git://git.openstack.org/openstack/keystone <https://git.openstack.org/cgit/openstack/keystone>`__
@@ -84,6 +87,7 @@
mistral `git://git.openstack.org/openstack/mistral <https://git.openstack.org/cgit/openstack/mistral>`__
mixmatch `git://git.openstack.org/openstack/mixmatch <https://git.openstack.org/cgit/openstack/mixmatch>`__
mogan `git://git.openstack.org/openstack/mogan <https://git.openstack.org/cgit/openstack/mogan>`__
+mogan-ui `git://git.openstack.org/openstack/mogan-ui <https://git.openstack.org/cgit/openstack/mogan-ui>`__
monasca-analytics `git://git.openstack.org/openstack/monasca-analytics <https://git.openstack.org/cgit/openstack/monasca-analytics>`__
monasca-api `git://git.openstack.org/openstack/monasca-api <https://git.openstack.org/cgit/openstack/monasca-api>`__
monasca-ceilometer `git://git.openstack.org/openstack/monasca-ceilometer <https://git.openstack.org/cgit/openstack/monasca-ceilometer>`__
@@ -132,6 +136,7 @@
os-xenapi `git://git.openstack.org/openstack/os-xenapi <https://git.openstack.org/cgit/openstack/os-xenapi>`__
osprofiler `git://git.openstack.org/openstack/osprofiler <https://git.openstack.org/cgit/openstack/osprofiler>`__
panko `git://git.openstack.org/openstack/panko <https://git.openstack.org/cgit/openstack/panko>`__
+patrole `git://git.openstack.org/openstack/patrole <https://git.openstack.org/cgit/openstack/patrole>`__
picasso `git://git.openstack.org/openstack/picasso <https://git.openstack.org/cgit/openstack/picasso>`__
rally `git://git.openstack.org/openstack/rally <https://git.openstack.org/cgit/openstack/rally>`__
sahara `git://git.openstack.org/openstack/sahara <https://git.openstack.org/cgit/openstack/sahara>`__
@@ -141,6 +146,7 @@
searchlight-ui `git://git.openstack.org/openstack/searchlight-ui <https://git.openstack.org/cgit/openstack/searchlight-ui>`__
senlin `git://git.openstack.org/openstack/senlin <https://git.openstack.org/cgit/openstack/senlin>`__
solum `git://git.openstack.org/openstack/solum <https://git.openstack.org/cgit/openstack/solum>`__
+stackube `git://git.openstack.org/openstack/stackube <https://git.openstack.org/cgit/openstack/stackube>`__
tacker `git://git.openstack.org/openstack/tacker <https://git.openstack.org/cgit/openstack/tacker>`__
tap-as-a-service `git://git.openstack.org/openstack/tap-as-a-service <https://git.openstack.org/cgit/openstack/tap-as-a-service>`__
tricircle `git://git.openstack.org/openstack/tricircle <https://git.openstack.org/cgit/openstack/tricircle>`__
diff --git a/doc/source/site-map.rst b/doc/source/site-map.rst
deleted file mode 100644
index 801fc66..0000000
--- a/doc/source/site-map.rst
+++ /dev/null
@@ -1,23 +0,0 @@
-:orphan:
-
-.. the TOC on the front page actually makes the document a lot more
- confusing. This lets us bury a toc which we can link in when
- appropriate.
-
-==========
- Site Map
-==========
-
-.. toctree::
- :glob:
- :maxdepth: 3
-
- overview
- configuration
- networking
- plugins
- plugin-registry
- faq
- development
- hacking
- guides
diff --git a/SYSTEMD.rst b/doc/source/systemd.rst
similarity index 74%
rename from SYSTEMD.rst
rename to doc/source/systemd.rst
index b6ed193..60a7719 100644
--- a/SYSTEMD.rst
+++ b/doc/source/systemd.rst
@@ -2,12 +2,7 @@
Using Systemd in DevStack
===========================
-.. note::
-
- This is an in progress document as we work out the way forward here
- with DevStack and systemd.
-
-DevStack can be run with all the services as systemd unit
+By default DevStack is run with all the services as systemd unit
files. Systemd is now the default init system for nearly every Linux
distro, and systemd encodes and solves many of the problems related to
poorly running processes.
@@ -25,12 +20,6 @@
than one service, and needing to restart a bunch of services for that
to take effect.
-To enable this add the following to your local.conf::
-
- USE_SYSTEMD=True
-
-
-
Unit Structure
==============
@@ -42,8 +31,9 @@
code is left in place in case we can switch back later.
All DevStack user units are created as a part of the DevStack slice
-given the name ``devstack@$servicename.service``. This lets us do
-certain operations at the slice level.
+given the name ``devstack@$servicename.service``. This makes it easy
+to understand which services are part of the devstack run, and lets us
+disable / stop them in a single command.
Manipulating Units
==================
@@ -74,6 +64,22 @@
sudo systemctl status devstack@n-cpu.service
+Operating on more than one unit at a time
+-----------------------------------------
+
+Systemd supports wildcarding for unit operations. To restart every
+service in devstack you can do that following::
+
+ sudo systemctl restart devstack@*
+
+Or to see the status of all Nova processes you can do::
+
+ sudo systemctl status devstack@n-*
+
+We'll eventually make the unit names a bit more meaningful so that
+it's easier to understand what you are restarting.
+
+.. _journalctl-examples:
Querying Logs
=============
@@ -92,13 +98,28 @@
Following logs for multiple services simultaneously::
- journalctl -f --unit devstack@n-cpu.service --user-unit
+ journalctl -f --unit devstack@n-cpu.service --unit
devstack@n-cond.service
+or you can even do wild cards to follow all the nova services::
+
+ journalctl -f --unit devstack@n-*
+
Use higher precision time stamps::
journalctl -f -o short-precise --unit devstack@n-cpu.service
+By default, journalctl strips out "unprintable" characters, including
+ASCII color codes. To keep the color codes (which can be interpreted by
+an appropriate terminal/pager - e.g. ``less``, the default)::
+
+ journalctl -a --unit devstack@n-cpu.service
+
+When outputting to the terminal using the default pager, long lines
+appear to be truncated, but horizontal scrolling is supported via the
+left/right arrow keys.
+
+See ``man 1 journalctl`` for more.
Known Issues
============
@@ -108,7 +129,7 @@
into the ``systemd`` namespace, which can cause some issues.
- ``systemd-python`` - this is the upstream maintained library, it has
- a version number like systemd itself (currently ``233``). This is
+ a version number like systemd itself (currently ``234``). This is
the one you want.
- ``systemd`` - a python 3 only library, not what you want.
- ``python-systemd`` - another library you don't want. Installing it
@@ -127,33 +148,6 @@
Future Work
===========
-oslo.log journald
------------------
-
-Journald has an extremely rich mechanism for direct logging including
-structured metadata. We should enhance oslo.log to take advantage of
-that. It would let us do things like::
-
- journalctl REQUEST_ID=......
-
- journalctl INSTANCE_ID=......
-
-And get all lines related to the request id or instance id.
-
-sub targets/slices
-------------------
-
-We might want to create per project slices so that it's easy to
-follow, restart all services of a single project (like swift) without
-impacting other services.
-
-log colorizing
---------------
-
-We lose log colorization through this process. We might want to build
-a custom colorizer that we could run journalctl output through
-optionally for people.
-
user units
----------
diff --git a/files/debs/general b/files/debs/general
index c121770..1dde03b 100644
--- a/files/debs/general
+++ b/files/debs/general
@@ -1,3 +1,5 @@
+apache2
+apache2-dev
bc
bridge-utils
bsdmainutils
@@ -9,11 +11,13 @@
git
graphviz # needed for docs
iputils-ping
+libapache2-mod-proxy-uwsgi
libffi-dev # for pyOpenSSL
libjpeg-dev # Pillow 3.0.0
libmysqlclient-dev # MySQL-python
libpq-dev # psycopg2
libssl-dev # for pyOpenSSL
+libsystemd-dev # for systemd-python
libxml2-dev # lxml
libxslt1-dev # lxml
libyaml-dev
diff --git a/files/debs/n-api b/files/debs/n-api
deleted file mode 100644
index 0928cd5..0000000
--- a/files/debs/n-api
+++ /dev/null
@@ -1 +0,0 @@
-fping
diff --git a/files/debs/n-cpu b/files/debs/n-cpu
index 69ac430..d8bbf59 100644
--- a/files/debs/n-cpu
+++ b/files/debs/n-cpu
@@ -2,6 +2,7 @@
genisoimage
gir1.2-libosinfo-1.0
lvm2 # NOPRIME
+netcat-openbsd
open-iscsi
python-guestfs # NOPRIME
qemu-utils
diff --git a/files/debs/nova b/files/debs/nova
index 58dad41..5e14aec 100644
--- a/files/debs/nova
+++ b/files/debs/nova
@@ -10,7 +10,9 @@
kpartx
libjs-jquery-tablesorter # Needed for coverage html reports
libmysqlclient-dev
-libvirt-bin # NOPRIME
+libvirt-bin # dist:xenial NOPRIME
+libvirt-clients # not:xenial NOPRIME
+libvirt-daemon-system # not:xenial NOPRIME
libvirt-dev # NOPRIME
mysql-server # NOPRIME
parted
diff --git a/files/debs/zookeeper b/files/debs/zookeeper
deleted file mode 100644
index f41b559..0000000
--- a/files/debs/zookeeper
+++ /dev/null
@@ -1 +0,0 @@
-zookeeperd
diff --git a/files/ebtables.workaround b/files/ebtables.workaround
deleted file mode 100644
index c8af51f..0000000
--- a/files/ebtables.workaround
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/bash
-#
-# Copyright 2015 Hewlett-Packard Development Company, L.P.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License. You may obtain
-# a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations
-# under the License.
-#
-#
-# This is a terrible, terrible, truly terrible work around for
-# environments that have libvirt < 1.2.11. ebtables requires that you
-# specifically tell it you would like to not race and get punched in
-# the face when 2 run at the same time with a --concurrent flag.
-
-flock -w 300 /var/lock/ebtables.nova /sbin/ebtables.real $@
diff --git a/files/rpms-suse/general b/files/rpms-suse/general
index 1044c25..370f240 100644
--- a/files/rpms-suse/general
+++ b/files/rpms-suse/general
@@ -1,3 +1,5 @@
+apache2
+apache2-devel
bc
bridge-utils
ca-certificates-mozilla
@@ -23,9 +25,11 @@
python-devel # pyOpenSSL
python-xml
screen
+systemd-devel # for systemd-python
tar
tcpdump
unzip
util-linux
wget
+which
zlib-devel
diff --git a/files/rpms-suse/n-api b/files/rpms-suse/n-api
index af5ac2f..0f08daa 100644
--- a/files/rpms-suse/n-api
+++ b/files/rpms-suse/n-api
@@ -1,2 +1 @@
-fping
python-dateutil
diff --git a/files/rpms/general b/files/rpms/general
index 77d2fa5..1393d18 100644
--- a/files/rpms/general
+++ b/files/rpms/general
@@ -7,6 +7,8 @@
gettext # used for compiling message catalogs
git-core
graphviz # needed only for docs
+httpd
+httpd-devel
iptables-services # NOPRIME f23,f24,f25
java-1.7.0-openjdk-headless # NOPRIME rhel7
java-1.8.0-openjdk-headless # NOPRIME f23,f24,f25
@@ -27,6 +29,7 @@
python-devel
redhat-rpm-config # missing dep for gcc hardening flags, see rhbz#1217376
screen
+systemd-devel # for systemd-python
tar
tcpdump
unzip
diff --git a/files/rpms/n-api b/files/rpms/n-api
deleted file mode 100644
index 0928cd5..0000000
--- a/files/rpms/n-api
+++ /dev/null
@@ -1 +0,0 @@
-fping
diff --git a/files/rpms/zookeeper b/files/rpms/zookeeper
deleted file mode 100644
index 1bfac53..0000000
--- a/files/rpms/zookeeper
+++ /dev/null
@@ -1 +0,0 @@
-zookeeper
diff --git a/files/zookeeper/environment b/files/zookeeper/environment
deleted file mode 100644
index afa2d2f..0000000
--- a/files/zookeeper/environment
+++ /dev/null
@@ -1,36 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Modified from http://packages.ubuntu.com/saucy/zookeeperd
-NAME=zookeeper
-ZOOCFGDIR=/etc/zookeeper/conf
-
-# seems, that log4j requires the log4j.properties file to be in the classpath
-CLASSPATH="$ZOOCFGDIR:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar"
-
-ZOOCFG="$ZOOCFGDIR/zoo.cfg"
-ZOO_LOG_DIR=/var/log/zookeeper
-USER=$NAME
-GROUP=$NAME
-PIDDIR=/var/run/$NAME
-PIDFILE=$PIDDIR/$NAME.pid
-SCRIPTNAME=/etc/init.d/$NAME
-JAVA=/usr/bin/java
-ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
-ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
-JMXLOCALONLY=false
-JAVA_OPTS=""
diff --git a/files/zookeeper/log4j.properties b/files/zookeeper/log4j.properties
deleted file mode 100644
index 6c45a4a..0000000
--- a/files/zookeeper/log4j.properties
+++ /dev/null
@@ -1,69 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# From http://packages.ubuntu.com/saucy/zookeeperd
-
-# ZooKeeper Logging Configuration
-#
-
-# Format is "<default threshold> (, <appender>)+
-
-log4j.rootLogger=${zookeeper.root.logger}
-
-# Example: console appender only
-# log4j.rootLogger=INFO, CONSOLE
-
-# Example with rolling log file
-#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE
-
-# Example with rolling log file and tracing
-#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE
-
-#
-# Log INFO level and above messages to the console
-#
-log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
-log4j.appender.CONSOLE.Threshold=INFO
-log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
-log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
-
-#
-# Add ROLLINGFILE to rootLogger to get log file output
-# Log DEBUG level and above messages to a log file
-log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
-log4j.appender.ROLLINGFILE.Threshold=WARN
-log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log
-
-# Max log file size of 10MB
-log4j.appender.ROLLINGFILE.MaxFileSize=10MB
-# uncomment the next line to limit number of backup files
-#log4j.appender.ROLLINGFILE.MaxBackupIndex=10
-
-log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
-log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
-
-
-#
-# Add TRACEFILE to rootLogger to get log file output
-# Log DEBUG level and above messages to a log file
-log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
-log4j.appender.TRACEFILE.Threshold=TRACE
-log4j.appender.TRACEFILE.File=${zookeeper.log.dir}/zookeeper_trace.log
-
-log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
-### Notice we are including log4j's NDC here (%x)
-log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n
diff --git a/files/zookeeper/myid b/files/zookeeper/myid
deleted file mode 100644
index c227083..0000000
--- a/files/zookeeper/myid
+++ /dev/null
@@ -1 +0,0 @@
-0
\ No newline at end of file
diff --git a/files/zookeeper/zoo.cfg b/files/zookeeper/zoo.cfg
deleted file mode 100644
index b8f5582..0000000
--- a/files/zookeeper/zoo.cfg
+++ /dev/null
@@ -1,74 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
-
-# The number of milliseconds of each tick
-tickTime=2000
-# The number of ticks that the initial
-# synchronization phase can take
-initLimit=10
-# The number of ticks that can pass between
-# sending a request and getting an acknowledgement
-syncLimit=5
-# the directory where the snapshot is stored.
-dataDir=/var/lib/zookeeper
-# Place the dataLogDir to a separate physical disc for better performance
-# dataLogDir=/disk2/zookeeper
-
-# the port at which the clients will connect
-clientPort=2181
-
-# Maximum number of clients that can connect from one client
-maxClientCnxns=60
-
-# specify all zookeeper servers
-# The fist port is used by followers to connect to the leader
-# The second one is used for leader election
-
-server.0=127.0.0.1:2888:3888
-
-# To avoid seeks ZooKeeper allocates space in the transaction log file in
-# blocks of preAllocSize kilobytes. The default block size is 64M. One reason
-# for changing the size of the blocks is to reduce the block size if snapshots
-# are taken more often. (Also, see snapCount).
-#preAllocSize=65536
-
-# Clients can submit requests faster than ZooKeeper can process them,
-# especially if there are a lot of clients. To prevent ZooKeeper from running
-# out of memory due to queued requests, ZooKeeper will throttle clients so that
-# there is no more than globalOutstandingLimit outstanding requests in the
-# system. The default limit is 1,000.ZooKeeper logs transactions to a
-# transaction log. After snapCount transactions are written to a log file a
-# snapshot is started and a new transaction log file is started. The default
-# snapCount is 10,000.
-#snapCount=1000
-
-# If this option is defined, requests will be will logged to a trace file named
-# traceFile.year.month.day.
-#traceFile=
-
-# Leader accepts client connections. Default value is "yes". The leader machine
-# coordinates updates. For higher update throughput at thes slight expense of
-# read throughput the leader can be configured to not accept clients and focus
-# on coordination.
-#leaderServes=yes
-
-# Autopurge every hour to avoid using lots of disk in bursts
-# Order of the next 2 properties matters.
-# autopurge.snapRetainCount must be before autopurge.purgeInterval.
-autopurge.snapRetainCount=3
-autopurge.purgeInterval=1
\ No newline at end of file
diff --git a/functions b/functions
index f6679fd..3ca3717 100644
--- a/functions
+++ b/functions
@@ -310,6 +310,11 @@
disk_format=qcow2
container_format=bare
;;
+ *.raw)
+ image_name=$(basename "$image" ".raw")
+ disk_format=raw
+ container_format=bare
+ ;;
*.iso)
image_name=$(basename "$image" ".iso")
disk_format=iso
@@ -318,7 +323,7 @@
*.vhd|*.vhdx|*.vhd.gz|*.vhdx.gz)
local extension="${image_fname#*.}"
image_name=$(basename "$image" ".$extension")
- disk_format=vhd
+ disk_format=$(echo $image_fname | grep -oP '(?<=\.)vhdx?(?=\.|$)')
container_format=bare
if [ "${image_fname##*.}" == "gz" ]; then
unpack=zcat
@@ -606,12 +611,23 @@
function setup_systemd_logging {
local conf_file=$1
local conf_section="DEFAULT"
- local project_var="project_name"
- local user_var="user_name"
- iniset $conf_file $conf_section logging_context_format_string "%(levelname)s %(name)s [%(request_id)s %("$project_var")s %("$user_var")s] %(instance)s%(message)s"
- iniset $conf_file $conf_section logging_default_format_string "%(levelname)s %(name)s [-] %(instance)s%(color)s%(message)s"
- iniset $conf_file $conf_section logging_debug_format_suffix "from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d"
- iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s %(instance)s"
+ # NOTE(sdague): this is a nice to have, and means we're using the
+ # native systemd path, which provides for things like search on
+ # request-id. However, there may be an eventlet interaction here,
+ # so going off for now.
+ USE_JOURNAL=$(trueorfalse False USE_JOURNAL)
+ local pidstr=""
+ if [[ "$USE_JOURNAL" == "True" ]]; then
+ iniset $conf_file $conf_section use_journal "True"
+ # if we are using the journal directly, our process id is already correct
+ else
+ pidstr="(pid=%(process)d) "
+ fi
+ iniset $conf_file $conf_section logging_debug_format_suffix "[00;33m{{${pidstr}%(funcName)s %(pathname)s:%(lineno)d}}[00m"
+
+ iniset $conf_file $conf_section logging_context_format_string "%(color)s%(levelname)s %(name)s [[01;36m%(global_request_id)s %(request_id)s [00;36m%(project_name)s %(user_name)s%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m"
+ iniset $conf_file $conf_section logging_default_format_string "%(color)s%(levelname)s %(name)s [[00;36m-%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m"
+ iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s [01;35m%(instance)s[00m"
}
function setup_standard_logging_identity {
@@ -701,6 +717,22 @@
}
+# Set a systemd system override
+#
+# This sets a system-side override in system.conf. A per-service
+# override would be /etc/systemd/system/${service}.service/override.conf
+function set_systemd_override {
+ local key="$1"
+ local value="$2"
+
+ local sysconf="/etc/systemd/system.conf"
+ iniset -sudo "${sysconf}" "Manager" "$key" "$value"
+ echo "Set systemd system override for ${key}=${value}"
+
+ sudo systemctl daemon-reload
+}
+
+
# Restore xtrace
$_XTRACE_FUNCTIONS
diff --git a/functions-common b/functions-common
index 98e9a4b..65db681 100644
--- a/functions-common
+++ b/functions-common
@@ -48,8 +48,8 @@
# Save these variables to .stackenv
STACK_ENV_VARS="BASE_SQL_CONN DATA_DIR DEST ENABLED_SERVICES HOST_IP \
- KEYSTONE_AUTH_PROTOCOL KEYSTONE_AUTH_URI KEYSTONE_SERVICE_URI \
- LOGFILE OS_CACERT SERVICE_HOST SERVICE_PROTOCOL STACK_USER TLS_IP \
+ KEYSTONE_AUTH_URI KEYSTONE_SERVICE_URI \
+ LOGFILE OS_CACERT SERVICE_HOST STACK_USER TLS_IP \
HOST_IPV6 SERVICE_IP_VERSION"
@@ -93,7 +93,7 @@
--os-region-name $REGION_NAME \
--os-identity-api-version 3 \
$CA_CERT_ARG \
- --os-auth-url $KEYSTONE_AUTH_URI \
+ --os-auth-url $KEYSTONE_SERVICE_URI \
--os-username demo \
--os-password $ADMIN_PASSWORD \
--os-project-name demo
@@ -105,7 +105,7 @@
--os-region-name $REGION_NAME \
--os-identity-api-version 3 \
$CA_CERT_ARG \
- --os-auth-url $KEYSTONE_AUTH_URI \
+ --os-auth-url $KEYSTONE_SERVICE_URI \
--os-username alt_demo \
--os-password $ADMIN_PASSWORD \
--os-project-name alt_demo
@@ -117,7 +117,7 @@
--os-region-name $REGION_NAME \
--os-identity-api-version 3 \
$CA_CERT_ARG \
- --os-auth-url $KEYSTONE_AUTH_URI \
+ --os-auth-url $KEYSTONE_SERVICE_URI \
--os-username admin \
--os-password $ADMIN_PASSWORD \
--os-project-name admin
@@ -905,34 +905,6 @@
echo $user_role_id
}
-# Gets or adds user role to domain
-# Usage: get_or_add_user_domain_role <role> <user> <domain>
-function get_or_add_user_domain_role {
- local user_role_id
- # Gets user role id
- user_role_id=$(openstack role assignment list \
- --user $2 \
- --os-url=$KEYSTONE_SERVICE_URI_V3 \
- --os-identity-api-version=3 \
- --domain $3 \
- | grep " $1 " | get_field 1)
- if [[ -z "$user_role_id" ]]; then
- # Adds role to user and get it
- openstack role add $1 \
- --user $2 \
- --domain $3 \
- --os-url=$KEYSTONE_SERVICE_URI_V3 \
- --os-identity-api-version=3
- user_role_id=$(openstack role assignment list \
- --user $2 \
- --os-url=$KEYSTONE_SERVICE_URI_V3 \
- --os-identity-api-version=3 \
- --domain $3 \
- | grep " $1 " | get_field 1)
- fi
- echo $user_role_id
-}
-
# Gets or adds group role to project
# Usage: get_or_add_group_project_role <role> <group> <project>
function get_or_add_group_project_role {
@@ -1148,6 +1120,19 @@
fi
fi
+ # Look for # not:xxx in comment
+ if [[ $line =~ (.*)#.*not:([^ ]*) ]]; then
+ # We are using BASH regexp matching feature.
+ package=${BASH_REMATCH[1]}
+ distros=${BASH_REMATCH[2]}
+ # In bash ${VAR,,} will lowercase VAR
+ # Look for a match in the distro list
+ if [[ ${distros,,} =~ ${DISTRO,,} ]]; then
+ # If match then skip this package
+ inst_pkg=0
+ fi
+ fi
+
if [[ $inst_pkg = 1 ]]; then
echo $package
fi
@@ -1166,6 +1151,8 @@
# - ``# NOPRIME`` defers installation to be performed later in `stack.sh`
# - ``# dist:DISTRO`` or ``dist:DISTRO1,DISTRO2`` limits the selection
# of the package to the distros listed. The distro names are case insensitive.
+# - ``# not:DISTRO`` or ``not:DISTRO1,DISTRO2`` limits the selection
+# of the package to the distros not listed. The distro names are case insensitive.
function get_packages {
local xtrace
xtrace=$(set +o | grep xtrace)
@@ -1467,33 +1454,82 @@
$SYSTEMCTL daemon-reload
}
-function _run_under_systemd {
+function write_uwsgi_user_unit_file {
+ local service=$1
+ local command="$2"
+ local group=$3
+ local user=$4
+ local unitfile="$SYSTEMD_DIR/$service"
+ mkdir -p $SYSTEMD_DIR
+
+ iniset -sudo $unitfile "Unit" "Description" "Devstack $service"
+ iniset -sudo $unitfile "Service" "SyslogIdentifier" "$service"
+ iniset -sudo $unitfile "Service" "User" "$user"
+ iniset -sudo $unitfile "Service" "ExecStart" "$command"
+ iniset -sudo $unitfile "Service" "Type" "notify"
+ iniset -sudo $unitfile "Service" "KillSignal" "SIGQUIT"
+ iniset -sudo $unitfile "Service" "Restart" "Always"
+ iniset -sudo $unitfile "Service" "NotifyAccess" "all"
+ iniset -sudo $unitfile "Service" "RestartForceExitStatus" "100"
+
+ if [[ -n "$group" ]]; then
+ iniset -sudo $unitfile "Service" "Group" "$group"
+ fi
+ iniset -sudo $unitfile "Install" "WantedBy" "multi-user.target"
+
+ # changes to existing units sometimes need a refresh
+ $SYSTEMCTL daemon-reload
+}
+
+function _common_systemd_pitfalls {
+ local cmd=$1
+ # do some sanity checks on $cmd to see things we don't expect to work
+
+ if [[ "$cmd" =~ "sudo" ]]; then
+ local msg=<<EOF
+You are trying to use run_process with sudo, this is not going to work under systemd.
+
+If you need to run a service as a user other than $STACK_USER call it with:
+
+ run_process \$name \$cmd \$group \$user
+EOF
+ die $LINENO $msg
+ fi
+
+ if [[ ! "$cmd" =~ ^/ ]]; then
+ local msg=<<EOF
+The cmd="$cmd" does not start with an absolute path. It will fail to
+start under systemd.
+
+Please update your run_process stanza to have an absolute path.
+EOF
+ die $LINENO $msg
+ fi
+
+}
+
+# Defines a systemd service which can be enabled and started later on.
+# arg1: The openstack service name ('n-cpu', 'c-sch', ...).
+# arg2: The command to start (e.g. path to service binary + config files).
+# arg3: The group which owns the process.
+# arg4: The user which owns the process.
+# Returns: The systemd service name which got defined.
+function _define_systemd_service {
local service=$1
local command="$2"
local cmd=$command
+ # sanity check the command
+ _common_systemd_pitfalls "$cmd"
+
local systemd_service="devstack@$service.service"
local group=$3
local user=${4:-$STACK_USER}
- write_user_unit_file $systemd_service "$cmd" "$group" "$user"
-
- $SYSTEMCTL enable $systemd_service
- $SYSTEMCTL start $systemd_service
- _journal_log $service $systemd_service
-}
-
-function _journal_log {
- local service=$1
- local unit=$2
- local logfile="${service}.log.${CURRENT_LOG_TIME}"
- local real_logfile="${LOGDIR}/${logfile}"
- if [[ -n ${LOGDIR} ]]; then
- $JOURNALCTL_F $2 > "$real_logfile" &
- bash -c "cd '$LOGDIR' && ln -sf '$logfile' ${service}.log"
- if [[ -n ${SCREEN_LOGDIR} ]]; then
- # Drop the backward-compat symlink
- ln -sf "$real_logfile" ${SCREEN_LOGDIR}/screen-${service}.log
- fi
+ if [[ "$command" =~ "uwsgi" ]] ; then
+ write_uwsgi_user_unit_file $systemd_service "$cmd" "$group" "$user"
+ else
+ write_user_unit_file $systemd_service "$cmd" "$group" "$user"
fi
+ echo $systemd_service
}
# Helper to remove the ``*.failure`` files under ``$SERVICE_DIR/$SCREEN_NAME``.
@@ -1526,7 +1562,7 @@
# If an optional group is provided sg will be used to run the
# command as that group.
# Uses globals ``USE_SCREEN``
-# run_process service "command-line" [group]
+# run_process service "command-line" [group] [user]
function run_process {
local service=$1
local command="$2"
@@ -1534,11 +1570,19 @@
local user=$4
local name=$service
+ local systemd_service
time_start "run_process"
+ # Note we deliberately make all service files, even if the service
+ # isn't enabled, so it can be enabled by a dev manually on command
+ # line.
+ if [[ "$USE_SYSTEMD" = "True" ]]; then
+ systemd_service=$(_define_systemd_service "$name" "$command" "$group" "$user")
+ fi
if is_service_enabled $service; then
if [[ "$USE_SYSTEMD" = "True" ]]; then
- _run_under_systemd "$name" "$command" "$group" "$user"
+ $SYSTEMCTL enable $systemd_service
+ $SYSTEMCTL start $systemd_service
elif [[ "$USE_SCREEN" = "True" ]]; then
if [[ "$user" == "root" ]]; then
command="sudo $command"
@@ -1678,11 +1722,10 @@
SERVICE_DIR=${SERVICE_DIR:-${DEST}/status}
if is_service_enabled $service; then
- # Kill via pid if we have one available
- if [[ "$USE_SYSTEMD" == "True" ]]; then
- # Only do this for units which appear enabled, this also
- # catches units that don't really exist for cases like
- # keystone without a failure.
+ # Only do this for units which appear enabled, this also
+ # catches units that don't really exist for cases like
+ # keystone without a failure.
+ if $SYSTEMCTL is-enabled devstack@$service.service; then
$SYSTEMCTL stop devstack@$service.service
$SYSTEMCTL disable devstack@$service.service
fi
diff --git a/inc/python b/inc/python
index 2443c4d..718cbb2 100644
--- a/inc/python
+++ b/inc/python
@@ -320,6 +320,14 @@
fi
$xtrace
+
+ # Also install test requirements
+ local install_test_reqs=""
+ local test_req="${!#}/test-requirements.txt"
+ if [[ -e "$test_req" ]]; then
+ install_test_reqs="-r $test_req"
+ fi
+
# adding SETUPTOOLS_SYS_PATH_TECHNIQUE is a workaround to keep
# the same behaviour of setuptools before version 25.0.0.
# related issue: https://github.com/pypa/pip/issues/3874
@@ -329,28 +337,28 @@
no_proxy="${no_proxy:-}" \
PIP_FIND_LINKS=$PIP_FIND_LINKS \
SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite \
- $cmd_pip $upgrade \
+ $cmd_pip $upgrade $install_test_reqs \
$@
result=$?
- # Also install test requirements
- local test_req="${!#}/test-requirements.txt"
- if [[ $result == 0 ]] && [[ -e "$test_req" ]]; then
- echo "Installing test-requirements for $test_req"
- $sudo_pip \
- http_proxy=${http_proxy:-} \
- https_proxy=${https_proxy:-} \
- no_proxy=${no_proxy:-} \
- PIP_FIND_LINKS=$PIP_FIND_LINKS \
- $cmd_pip $upgrade \
- -r $test_req
- result=$?
- fi
-
time_stop "pip_install"
return $result
}
+function pip_uninstall {
+ local name=$1
+ if [[ -n ${PIP_VIRTUAL_ENV:=} && -d ${PIP_VIRTUAL_ENV} ]]; then
+ local cmd_pip=$PIP_VIRTUAL_ENV/bin/pip
+ local sudo_pip="env"
+ else
+ local cmd_pip
+ cmd_pip=$(get_pip_command $PYTHON2_VERSION)
+ local sudo_pip="sudo -H"
+ fi
+ # don't error if we can't uninstall, it might not be there
+ $sudo_pip $cmd_pip uninstall -y $name || /bin/true
+}
+
# get version of a package from global requirements file
# get_from_global_requirements <package>
function get_from_global_requirements {
diff --git a/lib/apache b/lib/apache
index d1a11ae..c1b6bf8 100644
--- a/lib/apache
+++ b/lib/apache
@@ -53,8 +53,15 @@
function enable_apache_mod {
local mod=$1
# Apache installation, because we mark it NOPRIME
- if is_ubuntu || is_suse ; then
- if ! a2query -m $mod ; then
+ if is_ubuntu; then
+ # Skip mod_version as it is not a valid mod to enable
+ # on debuntu, instead it is built in.
+ if [[ "$mod" != "version" ]] && ! a2query -m $mod ; then
+ sudo a2enmod $mod
+ restart_apache_server
+ fi
+ elif is_suse; then
+ if ! a2enmod -q $mod ; then
sudo a2enmod $mod
restart_apache_server
fi
@@ -66,6 +73,48 @@
fi
}
+# NOTE(sdague): Install uwsgi including apache module, we need to get
+# to 2.0.6+ to get a working mod_proxy_uwsgi. We can probably build a
+# check for that and do it differently for different platforms.
+function install_apache_uwsgi {
+ local apxs="apxs2"
+ if is_fedora; then
+ apxs="apxs"
+ fi
+
+ # Ubuntu xenial is back level on uwsgi so the proxy doesn't
+ # actually work. Hence we have to build from source for now.
+ #
+ # Centos 7 actually has the module in epel, but there was a big
+ # push to disable epel by default. As such, compile from source
+ # there as well.
+
+ local dir
+ dir=$(mktemp -d)
+ pushd $dir
+ pip_install uwsgi
+ pip download uwsgi -c $REQUIREMENTS_DIR/upper-constraints.txt
+ local uwsgi
+ uwsgi=$(ls uwsgi*)
+ tar xvf $uwsgi
+ cd uwsgi*/apache2
+ sudo $apxs -i -c mod_proxy_uwsgi.c
+ popd
+ # delete the temp directory
+ sudo rm -rf $dir
+
+ if is_ubuntu || is_suse ; then
+ # we've got to enable proxy and proxy_uwsgi for this to work
+ sudo a2enmod proxy
+ sudo a2enmod proxy_uwsgi
+ elif is_fedora; then
+ # redhat is missing a nice way to turn on/off modules
+ echo "LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so" \
+ | sudo tee /etc/httpd/conf.modules.d/02-proxy-uwsgi.conf
+ fi
+ restart_apache_server
+}
+
# install_apache_wsgi() - Install Apache server and wsgi module
function install_apache_wsgi {
# Apache installation, because we mark it NOPRIME
@@ -90,49 +139,15 @@
fi
# WSGI isn't enabled by default, enable it
enable_apache_mod wsgi
-
- # ensure mod_version enabled for <IfVersion ...>. This is
- # built-in statically on anything recent, but precise (2.2)
- # doesn't have it enabled
- sudo a2enmod version || true
-}
-
-# get_apache_version() - return the version of Apache installed
-# This function is used to determine the Apache version installed. There are
-# various differences between Apache 2.2 and 2.4 that warrant special handling.
-function get_apache_version {
- if is_ubuntu; then
- local version_str
- version_str=$(sudo /usr/sbin/apache2ctl -v | awk '/Server version/ {print $3}' | cut -f2 -d/)
- elif is_fedora; then
- local version_str
- version_str=$(rpm -qa --queryformat '%{VERSION}' httpd)
- elif is_suse; then
- local version_str
- version_str=$(rpm -qa --queryformat '%{VERSION}' apache2)
- else
- exit_distro_not_supported "cannot determine apache version"
- fi
- if [[ "$version_str" =~ ^2\.2\. ]]; then
- echo "2.2"
- elif [[ "$version_str" =~ ^2\.4\. ]]; then
- echo "2.4"
- else
- exit_distro_not_supported "apache version not supported"
- fi
}
# apache_site_config_for() - The filename of the site's configuration file.
# This function uses the global variables APACHE_NAME and APACHE_CONF_DIR.
#
-# On Ubuntu 14.04, the site configuration file must have a .conf suffix for a2ensite and a2dissite to
+# On Ubuntu 14.04+, the site configuration file must have a .conf suffix for a2ensite and a2dissite to
# recognise it. a2ensite and a2dissite ignore the .conf suffix used as parameter. The default sites'
# files are 000-default.conf and default-ssl.conf.
#
-# On Ubuntu 12.04, the site configuration file may have any format, as long as it is in
-# /etc/apache2/sites-available/. a2ensite and a2dissite need the entire file name to work. The default
-# sites' files are default and default-ssl.
-#
# On Fedora and openSUSE, any file in /etc/httpd/conf.d/ whose name ends with .conf is enabled.
#
# On RHEL and CentOS, things should hopefully work as in Fedora.
@@ -141,22 +156,14 @@
# +----------------------+--------------------+--------------------------+--------------------------+
# | Distribution | File name | Site enabling command | Site disabling command |
# +----------------------+--------------------+--------------------------+--------------------------+
-# | Ubuntu 12.04 | site | a2ensite site | a2dissite site |
# | Ubuntu 14.04 | site.conf | a2ensite site | a2dissite site |
# | Fedora, RHEL, CentOS | site.conf.disabled | mv site.conf{.disabled,} | mv site.conf{,.disabled} |
# +----------------------+--------------------+--------------------------+--------------------------+
function apache_site_config_for {
local site=$@
if is_ubuntu; then
- local apache_version
- apache_version=$(get_apache_version)
- if [[ "$apache_version" == "2.2" ]]; then
- # Ubuntu 12.04 - Apache 2.2
- echo $APACHE_CONF_DIR/${site}
- else
- # Ubuntu 14.04 - Apache 2.4
- echo $APACHE_CONF_DIR/${site}.conf
- fi
+ # Ubuntu 14.04 - Apache 2.4
+ echo $APACHE_CONF_DIR/${site}.conf
elif is_fedora || is_suse; then
# fedora conf.d is only imported if it ends with .conf so this is approx the same
local enabled_site_file="$APACHE_CONF_DIR/${site}.conf"
@@ -171,6 +178,8 @@
# enable_apache_site() - Enable a particular apache site
function enable_apache_site {
local site=$@
+ # Many of our sites use mod version. Just enable it.
+ enable_apache_mod version
if is_ubuntu; then
sudo a2ensite ${site}
elif is_fedora || is_suse; then
@@ -186,7 +195,7 @@
function disable_apache_site {
local site=$@
if is_ubuntu; then
- sudo a2dissite ${site}
+ sudo a2dissite ${site} || true
elif is_fedora || is_suse; then
local enabled_site_file="$APACHE_CONF_DIR/${site}.conf"
# Do nothing if no site config exists
@@ -215,16 +224,71 @@
# Apache can be slow to stop, doing an explicit stop, sleep, start helps
# to mitigate issues where apache will claim a port it's listening on is
# still in use and fail to start.
- time_start "restart_apache_server"
- stop_service $APACHE_NAME
- sleep 3
- start_service $APACHE_NAME
- time_stop "restart_apache_server"
+ restart_service $APACHE_NAME
}
-# reload_apache_server
-function reload_apache_server {
- reload_service $APACHE_NAME
+function write_uwsgi_config {
+ local file=$1
+ local wsgi=$2
+ local url=$3
+ local http=$4
+ local name=""
+ name=$(basename $wsgi)
+
+ # create a home for the sockets; note don't use /tmp -- apache has
+ # a private view of it on some platforms.
+ local socket_dir='/var/run/uwsgi'
+
+ # /var/run will be empty on ubuntu after reboot, so we can use systemd-temptiles
+ # to automatically create $socket_dir.
+ sudo mkdir -p /etc/tmpfiles.d/
+ echo "d $socket_dir 0755 $STACK_USER root" | sudo tee /etc/tmpfiles.d/uwsgi.conf
+ sudo systemd-tmpfiles --create /etc/tmpfiles.d/uwsgi.conf
+
+ local socket="$socket_dir/${name}.socket"
+
+ # always cleanup given that we are using iniset here
+ rm -rf $file
+ iniset "$file" uwsgi wsgi-file "$wsgi"
+ iniset "$file" uwsgi socket "$socket"
+ iniset "$file" uwsgi processes $API_WORKERS
+ # This is running standalone
+ iniset "$file" uwsgi master true
+ # Set die-on-term & exit-on-reload so that uwsgi shuts down
+ iniset "$file" uwsgi die-on-term true
+ iniset "$file" uwsgi exit-on-reload true
+ iniset "$file" uwsgi enable-threads true
+ iniset "$file" uwsgi plugins python
+ # uwsgi recommends this to prevent thundering herd on accept.
+ iniset "$file" uwsgi thunder-lock true
+ # Override the default size for headers from the 4k default.
+ iniset "$file" uwsgi buffer-size 65535
+ # Make sure the client doesn't try to re-use the connection.
+ iniset "$file" uwsgi add-header "Connection: close"
+ # This ensures that file descriptors aren't shared between processes.
+ iniset "$file" uwsgi lazy-apps true
+ iniset "$file" uwsgi chmod-socket 666
+
+ # If we said bind directly to http, then do that and don't start the apache proxy
+ if [[ -n "$http" ]]; then
+ iniset "$file" uwsgi http $http
+ else
+ local apache_conf=""
+ apache_conf=$(apache_site_config_for $name)
+ echo "ProxyPass \"${url}\" \"unix:${socket}|uwsgi://uwsgi-uds-${name}/\" retry=0 " | sudo tee $apache_conf
+ enable_apache_site $name
+ restart_apache_server
+ fi
+}
+
+function remove_uwsgi_config {
+ local file=$1
+ local wsgi=$2
+ local name=""
+ name=$(basename $wsgi)
+
+ rm -rf $file
+ disable_apache_site $name
}
# Restore xtrace
diff --git a/lib/cinder b/lib/cinder
index c17cea0..2068812 100644
--- a/lib/cinder
+++ b/lib/cinder
@@ -58,7 +58,7 @@
CINDER_API_PASTE_INI=$CINDER_CONF_DIR/api-paste.ini
# Public facing bits
-if is_ssl_enabled_service "cinder" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
CINDER_SERVICE_PROTOCOL="https"
fi
CINDER_SERVICE_HOST=${CINDER_SERVICE_HOST:-$SERVICE_HOST}
@@ -84,20 +84,6 @@
# CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-lvm:lvmdriver-1,lvm:lvmdriver-2}
CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-lvm:lvmdriver-1}
-
-# Should cinder perform secure deletion of volumes?
-# Defaults to zero. Can also be set to none or shred.
-# This was previously CINDER_SECURE_DELETE (True or False).
-# Equivalents using CINDER_VOLUME_CLEAR are zero and none, respectively.
-# Set to none to avoid this bug when testing:
-# https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1023755
-if [[ -n $CINDER_SECURE_DELETE ]]; then
- CINDER_SECURE_DELETE=$(trueorfalse True CINDER_SECURE_DELETE)
- if [[ $CINDER_SECURE_DELETE == "False" ]]; then
- CINDER_VOLUME_CLEAR_DEFAULT="none"
- fi
- deprecated "Configure secure Cinder volume deletion using CINDER_VOLUME_CLEAR instead of CINDER_SECURE_DELETE."
-fi
CINDER_VOLUME_CLEAR=${CINDER_VOLUME_CLEAR:-${CINDER_VOLUME_CLEAR_DEFAULT:-zero}}
CINDER_VOLUME_CLEAR=$(echo ${CINDER_VOLUME_CLEAR} | tr '[:upper:]' '[:lower:]')
@@ -143,6 +129,7 @@
# Test if any Cinder services are enabled
# is_cinder_enabled
function is_cinder_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"cinder" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"c-" ]] && return 0
return 1
}
@@ -215,11 +202,6 @@
local cinder_api_port=$CINDER_SERVICE_PORT
local venv_path=""
- if is_ssl_enabled_service c-api; then
- cinder_ssl="SSLEngine On"
- cinder_certfile="SSLCertificateFile $CINDER_SSL_CERT"
- cinder_keyfile="SSLCertificateKeyFile $CINDER_SSL_KEY"
- fi
if [[ ${USE_VENV} = True ]]; then
venv_path="python-path=${PROJECT_VENV["cinder"]}/lib/python2.7/site-packages"
fi
@@ -249,6 +231,10 @@
configure_rootwrap cinder
+ if [[ -f "$CINDER_DIR/etc/cinder/resource_filters.json" ]]; then
+ cp -p "$CINDER_DIR/etc/cinder/resource_filters.json" "$CINDER_CONF_DIR/resource_filters.json"
+ fi
+
cp $CINDER_DIR/etc/cinder/api-paste.ini $CINDER_API_PASTE_INI
inicomment $CINDER_API_PASTE_INI filter:authtoken auth_host
@@ -288,6 +274,8 @@
iniset $CINDER_CONF DEFAULT os_region_name "$REGION_NAME"
+ iniset $CINDER_CONF key_manager api_class cinder.keymgr.conf_key_mgr.ConfKeyManager
+
if is_service_enabled c-vol && [[ -n "$CINDER_ENABLED_BACKENDS" ]]; then
local enabled_backends=""
local default_name=""
@@ -302,6 +290,9 @@
default_name=$be_name
fi
enabled_backends+=$be_name,
+
+ iniset $CINDER_CONF $be_name volume_clear $CINDER_VOLUME_CLEAR
+
done
iniset $CINDER_CONF DEFAULT enabled_backends ${enabled_backends%,*}
if [[ -n "$default_name" ]]; then
@@ -331,8 +322,6 @@
iniset_rpc_backend cinder $CINDER_CONF
- iniset $CINDER_CONF DEFAULT volume_clear $CINDER_VOLUME_CLEAR
-
# Format logging
setup_logging $CINDER_CONF $CINDER_USE_MOD_WSGI
@@ -347,7 +336,7 @@
iniset $CINDER_CONF DEFAULT osapi_volume_workers "$API_WORKERS"
iniset $CINDER_CONF DEFAULT glance_api_servers "${GLANCE_SERVICE_PROTOCOL}://${GLANCE_HOSTPORT}"
- if is_ssl_enabled_service glance || is_service_enabled tls-proxy; then
+ if is_service_enabled tls-proxy; then
iniset $CINDER_CONF DEFAULT glance_protocol https
iniset $CINDER_CONF DEFAULT glance_ca_certificates_file $SSL_BUNDLE_FILE
fi
@@ -356,25 +345,16 @@
iniset $CINDER_CONF DEFAULT glance_api_version 2
fi
- # Register SSL certificates if provided
- if is_ssl_enabled_service cinder; then
- ensure_certificates CINDER
-
- iniset $CINDER_CONF DEFAULT ssl_cert_file "$CINDER_SSL_CERT"
- iniset $CINDER_CONF DEFAULT ssl_key_file "$CINDER_SSL_KEY"
- fi
-
# Set os_privileged_user credentials (used for os-assisted-snapshots)
iniset $CINDER_CONF DEFAULT os_privileged_user_name nova
iniset $CINDER_CONF DEFAULT os_privileged_user_password "$SERVICE_PASSWORD"
iniset $CINDER_CONF DEFAULT os_privileged_user_tenant "$SERVICE_PROJECT_NAME"
iniset $CINDER_CONF DEFAULT graceful_shutdown_timeout "$SERVICE_GRACEFUL_SHUTDOWN_TIMEOUT"
- # Set the backend url according to the configured dlm backend
- if is_dlm_enabled; then
- if [[ "$(dlm_backend)" == "zookeeper" ]]; then
- iniset $CINDER_CONF coordination backend_url "zookeeper://${SERVICE_HOST}:2181"
- fi
+ if [[ ! -z "$CINDER_COORDINATION_URL" ]]; then
+ iniset $CINDER_CONF coordination backend_url "$CINDER_COORDINATION_URL"
+ elif is_service_enabled etcd3; then
+ iniset $CINDER_CONF coordination backend_url "etcd3+http://${SERVICE_HOST}:2379"
fi
}
@@ -427,8 +407,10 @@
# (Re)create cinder database
recreate_database cinder
+ time_start "dbsync"
# Migrate cinder database
$CINDER_BIN_DIR/cinder-manage --config-file $CINDER_CONF db sync
+ time_stop "dbsync"
fi
if is_service_enabled c-vol && [[ -n "$CINDER_ENABLED_BACKENDS" ]]; then
@@ -464,9 +446,6 @@
if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
install_apache_wsgi
- if is_ssl_enabled_service "c-api"; then
- enable_mod_ssl
- fi
fi
}
@@ -522,16 +501,24 @@
fi
fi
- if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
- enable_apache_site osapi-volume
- restart_apache_server
- tail_log c-api /var/log/$APACHE_NAME/c-api.log
- else
- run_process c-api "$CINDER_BIN_DIR/cinder-api --config-file $CINDER_CONF"
+ if is_service_enabled c-api ; then
+ if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
+ enable_apache_site osapi-volume
+ restart_apache_server
+ tail_log c-api /var/log/$APACHE_NAME/c-api.log
+ else
+ run_process c-api "$CINDER_BIN_DIR/cinder-api --config-file $CINDER_CONF"
+ fi
+
echo "Waiting for Cinder API to start..."
if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$CINDER_SERVICE_HOST:$service_port; then
die $LINENO "c-api did not start"
fi
+
+ # Start proxies if enabled
+ if is_service_enabled tls-proxy; then
+ start_tls_proxy cinder '*' $CINDER_SERVICE_PORT $CINDER_SERVICE_HOST $CINDER_SERVICE_PORT_INT
+ fi
fi
run_process c-sch "$CINDER_BIN_DIR/cinder-scheduler --config-file $CINDER_CONF"
@@ -541,11 +528,6 @@
# NOTE(jdg): For cinder, startup order matters. To ensure that repor_capabilities is received
# by the scheduler start the cinder-volume service last (or restart it) after the scheduler
# has started. This is a quick fix for lp bug/1189595
-
- # Start proxies if enabled
- if is_service_enabled c-api && is_service_enabled tls-proxy; then
- start_tls_proxy cinder '*' $CINDER_SERVICE_PORT $CINDER_SERVICE_HOST $CINDER_SERVICE_PORT_INT
- fi
}
# stop_cinder() - Stop running processes
diff --git a/lib/cinder_backends/lvm b/lib/cinder_backends/lvm
index d927f9c..03e1880 100644
--- a/lib/cinder_backends/lvm
+++ b/lib/cinder_backends/lvm
@@ -53,9 +53,6 @@
iniset $CINDER_CONF $be_name iscsi_helper "$CINDER_ISCSI_HELPER"
iniset $CINDER_CONF $be_name lvm_type "$CINDER_LVM_TYPE"
- if [[ "$CINDER_SECURE_DELETE" == "False" ]]; then
- iniset $CINDER_CONF $be_name volume_clear none
- fi
}
# init_cinder_backend_lvm - Initialize volume group
diff --git a/lib/dlm b/lib/dlm
deleted file mode 100644
index b5ac0f5..0000000
--- a/lib/dlm
+++ /dev/null
@@ -1,111 +0,0 @@
-#!/bin/bash
-#
-# lib/dlm
-#
-# Functions to control the installation and configuration of software
-# that provides a dlm (and possibly other functions). The default is
-# **zookeeper**, and is going to be the only backend supported in the
-# devstack tree.
-
-# Dependencies:
-#
-# - ``functions`` file
-
-# ``stack.sh`` calls the entry points in this order:
-#
-# - is_dlm_enabled
-# - install_dlm
-# - configure_dlm
-# - cleanup_dlm
-
-# Save trace setting
-_XTRACE_DLM=$(set +o | grep xtrace)
-set +o xtrace
-
-
-# Defaults
-# --------
-
-# <define global variables here that belong to this project>
-
-# Set up default directories
-ZOOKEEPER_DATA_DIR=$DEST/data/zookeeper
-ZOOKEEPER_CONF_DIR=/etc/zookeeper
-
-
-# Entry Points
-# ------------
-#
-# NOTE(sdague): it is expected that when someone wants to implement
-# another one of these out of tree, they'll implement the following
-# functions:
-#
-# - dlm_backend
-# - install_dlm
-# - configure_dlm
-# - cleanup_dlm
-
-# This should be declared in the settings file of any plugin or
-# service that needs to have a dlm in their environment.
-function use_dlm {
- enable_service $(dlm_backend)
-}
-
-# A function to return the name of the backend in question, some users
-# are going to need to know this.
-function dlm_backend {
- echo "zookeeper"
-}
-
-# Test if a dlm is enabled (defaults to a zookeeper specific check)
-function is_dlm_enabled {
- [[ ,${ENABLED_SERVICES}, =~ ,"$(dlm_backend)", ]] && return 0
- return 1
-}
-
-# cleanup_dlm() - Remove residual data files, anything left over from previous
-# runs that a clean run would need to clean up
-function cleanup_dlm {
- # NOTE(sdague): we don't check for is_enabled here because we
- # should just delete this regardless. Some times users updated
- # their service list before they run cleanup.
- sudo rm -rf $ZOOKEEPER_DATA_DIR
-}
-
-# configure_dlm() - Set config files, create data dirs, etc
-function configure_dlm {
- if is_dlm_enabled; then
- sudo cp $FILES/zookeeper/* $ZOOKEEPER_CONF_DIR
- sudo sed -i -e 's|.*dataDir.*|dataDir='$ZOOKEEPER_DATA_DIR'|' $ZOOKEEPER_CONF_DIR/zoo.cfg
- # clean up from previous (possibly aborted) runs
- # create required data files
- sudo rm -rf $ZOOKEEPER_DATA_DIR
- sudo mkdir -p $ZOOKEEPER_DATA_DIR
- # restart after configuration, there is no reason to make this
- # another step, because having data files that don't match the
- # zookeeper running is just going to cause tears.
- restart_service zookeeper
- fi
-}
-
-# install_dlm() - Collect source and prepare
-function install_dlm {
- if is_dlm_enabled; then
- pip_install_gr_extras tooz zookeeper
- if is_ubuntu; then
- install_package zookeeperd
- elif is_fedora; then
- install_package zookeeper
- else
- die $LINENO "Don't know how to install zookeeper on this platform"
- fi
- fi
-}
-
-# Restore xtrace
-$_XTRACE_DLM
-
-# Tell emacs to use shell-script-mode
-## Local variables:
-## mode: shell-script
-## End:
diff --git a/lib/etcd3 b/lib/etcd3
new file mode 100644
index 0000000..f8b113b
--- /dev/null
+++ b/lib/etcd3
@@ -0,0 +1,148 @@
+#!/bin/bash
+#
+# lib/etcd3
+#
+# Functions to control the installation and configuration of etcd 3.x
+# that provides a key-value store (and possibly other functions).
+
+# Dependencies:
+#
+# - ``functions`` file
+
+# ``stack.sh`` calls the entry points in this order:
+#
+# - start_etcd3
+# - stop_etcd3
+# - cleanup_etcd3
+
+# Save trace setting
+_XTRACE_ETCD3=$(set +o | grep xtrace)
+set +o xtrace
+
+
+# Defaults
+# --------
+
+# Set up default values for etcd
+ETCD_DOWNLOAD_URL=${ETCD_DOWNLOAD_URL:-https://github.com/coreos/etcd/releases/download}
+ETCD_VERSION=${ETCD_VERSION:-v3.1.7}
+ETCD_DATA_DIR="$DEST/data/etcd"
+ETCD_SYSTEMD_SERVICE="devstack@etcd.service"
+ETCD_BIN_DIR="$DEST/bin"
+ETCD_SHA256_AMD64="4fde194bbcd259401e2b5c462dfa579ee7f6af539f13f130b8f5b4f52e3b3c52"
+# NOTE(sdague): etcd v3.1.7 doesn't have anything for these architectures, though 3.2.0 does.
+ETCD_SHA256_ARM64=""
+ETCD_SHA256_PPC64=""
+ETCD_PORT=2379
+
+if is_ubuntu ; then
+ UBUNTU_RELEASE_BASE_NUM=`lsb_release -r | awk '{print $2}' | cut -d '.' -f 1`
+fi
+
+# start_etcd3() - Starts to run the etcd process
+function start_etcd3 {
+ # Don't install in sub nodes (multinode scenario)
+ if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+ return
+ fi
+
+ _install_etcd
+
+ local cmd="$ETCD_BIN_DIR/etcd"
+ cmd+=" --name $HOSTNAME --data-dir $ETCD_DATA_DIR"
+ cmd+=" --initial-cluster-state new --initial-cluster-token etcd-cluster-01"
+ cmd+=" --initial-cluster $HOSTNAME=http://$SERVICE_HOST:2380"
+ cmd+=" --initial-advertise-peer-urls http://$SERVICE_HOST:2380"
+ cmd+=" --advertise-client-urls http://$SERVICE_HOST:$ETCD_PORT"
+ cmd+=" --listen-peer-urls http://0.0.0.0:2380 "
+ cmd+=" --listen-client-urls http://$SERVICE_HOST:$ETCD_PORT"
+
+ local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
+ write_user_unit_file $ETCD_SYSTEMD_SERVICE "$cmd" "" "root"
+
+ iniset -sudo $unitfile "Unit" "After" "network.target"
+ iniset -sudo $unitfile "Service" "Type" "notify"
+ iniset -sudo $unitfile "Service" "Restart" "on-failure"
+ iniset -sudo $unitfile "Service" "LimitNOFILE" "65536"
+
+ $SYSTEMCTL daemon-reload
+ $SYSTEMCTL enable $ETCD_SYSTEMD_SERVICE
+ $SYSTEMCTL start $ETCD_SYSTEMD_SERVICE
+}
+
+# stop_etcd3() stops the etcd3 process
+function stop_etcd3 {
+ # Don't install in sub nodes (multinode scenario)
+ if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+ return
+ fi
+
+ $SYSTEMCTL stop $ETCD_SYSTEMD_SERVICE
+}
+
+function cleanup_etcd3 {
+ # Don't install in sub nodes (multinode scenario)
+ if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+ return
+ fi
+
+ $SYSTEMCTL disable $ETCD_SYSTEMD_SERVICE
+
+ local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
+ sudo rm -f $unitfile
+
+ $SYSTEMCTL daemon-reload
+
+ sudo rm -rf $ETCD_DATA_DIR
+}
+
+function _install_etcd {
+ echo "Installing etcd"
+
+ # Make sure etcd3 downloads the correct architecture
+ if is_arch "x86_64"; then
+ ETCD_ARCH="amd64"
+ ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_AMD64}
+ elif is_arch "aarch64"; then
+ ETCD_ARCH="arm64"
+ ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_ARM64}
+ elif is_arch "ppc64le"; then
+ ETCD_ARCH="ppc64le"
+ ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_PPC64}
+ else
+ exit_distro_not_supported "invalid hardware type - $ETCD_ARCH"
+ fi
+
+ ETCD_NAME=etcd-$ETCD_VERSION-linux-$ETCD_ARCH
+
+ # Install the libraries needed. Note: tooz for example does not have a hard dependency on these libraries
+ pip_install etcd3
+ pip_install etcd3gw
+
+ # Create the necessary directories
+ sudo mkdir -p $ETCD_BIN_DIR
+ sudo mkdir -p $ETCD_DATA_DIR
+
+ # Download and cache the etcd tgz for subsequent use
+ if [ ! -f "$FILES/etcd-$ETCD_VERSION-linux-$ETCD_ARCH/etcd" ]; then
+ ETCD_DOWNLOAD_FILE=$ETCD_NAME.tar.gz
+ wget $ETCD_DOWNLOAD_URL/$ETCD_VERSION/$ETCD_DOWNLOAD_FILE -O $FILES/$ETCD_DOWNLOAD_FILE
+ echo "${ETCD_SHA256} $FILES/${ETCD_DOWNLOAD_FILE}" > $FILES/etcd.sha256sum
+ # NOTE(sdague): this should go fatal if this fails
+ sha256sum -c $FILES/etcd.sha256sum
+
+ tar xzvf $FILES/$ETCD_DOWNLOAD_FILE -C $FILES
+ sudo cp $FILES/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+ fi
+ if [ ! -f "$ETCD_BIN_DIR/etcd" ]; then
+ sudo cp $FILES/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+ fi
+}
+
+# Restore xtrace
+$_XTRACE_ETCD3
+
+# Tell emacs to use shell-script-mode
+## Local variables:
+## mode: shell-script
+## End:
diff --git a/lib/glance b/lib/glance
index 2f4aa5f..baf8c61 100644
--- a/lib/glance
+++ b/lib/glance
@@ -43,6 +43,7 @@
GLANCE_CACHE_DIR=${GLANCE_CACHE_DIR:=$DATA_DIR/glance/cache}
GLANCE_IMAGE_DIR=${GLANCE_IMAGE_DIR:=$DATA_DIR/glance/images}
+GLANCE_LOCK_DIR=${GLANCE_LOCK_DIR:=$DATA_DIR/glance/locks}
GLANCE_AUTH_CACHE_DIR=${GLANCE_AUTH_CACHE_DIR:-/var/cache/glance}
GLANCE_CONF_DIR=${GLANCE_CONF_DIR:-/etc/glance}
@@ -57,7 +58,7 @@
GLANCE_SWIFT_STORE_CONF=$GLANCE_CONF_DIR/glance-swift-store.conf
GLANCE_V1_ENABLED=${GLANCE_V1_ENABLED:-False}
-if is_ssl_enabled_service "glance" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
GLANCE_SERVICE_PROTOCOL="https"
fi
@@ -77,6 +78,7 @@
# Test if any Glance services are enabled
# is_glance_enabled
function is_glance_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"glance" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"g-" ]] && return 0
return 1
}
@@ -116,6 +118,7 @@
iniset $GLANCE_API_CONF database connection $dburl
iniset $GLANCE_API_CONF DEFAULT use_syslog $SYSLOG
iniset $GLANCE_API_CONF DEFAULT image_cache_dir $GLANCE_CACHE_DIR/
+ iniset $GLANCE_API_CONF DEFAULT lock_path $GLANCE_LOCK_DIR
iniset $GLANCE_API_CONF paste_deploy flavor keystone+cachemanagement
configure_auth_token_middleware $GLANCE_API_CONF glance $GLANCE_AUTH_CACHE_DIR/api
iniset $GLANCE_API_CONF oslo_messaging_notifications driver messagingv2
@@ -187,18 +190,7 @@
iniset $GLANCE_REGISTRY_CONF keystone_authtoken identity_uri $KEYSTONE_AUTH_URI
fi
- # Register SSL certificates if provided
- if is_ssl_enabled_service glance; then
- ensure_certificates GLANCE
-
- iniset $GLANCE_API_CONF DEFAULT cert_file "$GLANCE_SSL_CERT"
- iniset $GLANCE_API_CONF DEFAULT key_file "$GLANCE_SSL_KEY"
-
- iniset $GLANCE_REGISTRY_CONF DEFAULT cert_file "$GLANCE_SSL_CERT"
- iniset $GLANCE_REGISTRY_CONF DEFAULT key_file "$GLANCE_SSL_KEY"
- fi
-
- if is_ssl_enabled_service glance || is_service_enabled tls-proxy; then
+ if is_service_enabled tls-proxy; then
iniset $GLANCE_API_CONF DEFAULT registry_client_protocol https
fi
@@ -216,7 +208,7 @@
iniset $GLANCE_CACHE_CONF DEFAULT use_syslog $SYSLOG
iniset $GLANCE_CACHE_CONF DEFAULT image_cache_dir $GLANCE_CACHE_DIR/
iniuncomment $GLANCE_CACHE_CONF DEFAULT auth_url
- iniset $GLANCE_CACHE_CONF DEFAULT auth_url $KEYSTONE_AUTH_URI/v3
+ iniset $GLANCE_CACHE_CONF DEFAULT auth_url $KEYSTONE_AUTH_URI
iniuncomment $GLANCE_CACHE_CONF DEFAULT auth_tenant_name
iniset $GLANCE_CACHE_CONF DEFAULT admin_tenant_name $SERVICE_PROJECT_NAME
iniuncomment $GLANCE_CACHE_CONF DEFAULT auth_user
@@ -233,7 +225,7 @@
cp -p $GLANCE_DIR/etc/metadefs/*.json $GLANCE_METADEF_DIR
- if is_ssl_enabled_service "cinder" || is_service_enabled tls-proxy; then
+ if is_service_enabled tls-proxy; then
CINDER_SERVICE_HOST=${CINDER_SERVICE_HOST:-$SERVICE_HOST}
CINDER_SERVICE_PORT=${CINDER_SERVICE_PORT:-8776}
@@ -293,11 +285,13 @@
# (Re)create glance database
recreate_database glance
+ time_start "dbsync"
# Migrate glance database
$GLANCE_BIN_DIR/glance-manage --config-file $GLANCE_CONF_DIR/glance-api.conf db_sync
# Load metadata definitions
$GLANCE_BIN_DIR/glance-manage --config-file $GLANCE_CONF_DIR/glance-api.conf db_load_metadefs
+ time_stop "dbsync"
create_glance_cache_dir
}
diff --git a/lib/keystone b/lib/keystone
index af607c3..eb46526 100644
--- a/lib/keystone
+++ b/lib/keystone
@@ -50,22 +50,18 @@
KEYSTONE_CONF_DIR=${KEYSTONE_CONF_DIR:-/etc/keystone}
KEYSTONE_CONF=$KEYSTONE_CONF_DIR/keystone.conf
KEYSTONE_PASTE_INI=${KEYSTONE_PASTE_INI:-$KEYSTONE_CONF_DIR/keystone-paste.ini}
-
-# Toggle for deploying Keystone under HTTPD + mod_wsgi
-# Deprecated in Mitaka, use KEYSTONE_DEPLOY instead.
-KEYSTONE_USE_MOD_WSGI=${KEYSTONE_USE_MOD_WSGI:-${ENABLE_HTTPD_MOD_WSGI_SERVICES}}
+KEYSTONE_PUBLIC_UWSGI_CONF=$KEYSTONE_CONF_DIR/keystone-uwsgi-public.ini
+KEYSTONE_ADMIN_UWSGI_CONF=$KEYSTONE_CONF_DIR/keystone-uwsgi-admin.ini
+KEYSTONE_PUBLIC_UWSGI=$KEYSTONE_BIN_DIR/keystone-wsgi-public
+KEYSTONE_ADMIN_UWSGI=$KEYSTONE_BIN_DIR/keystone-wsgi-admin
# KEYSTONE_DEPLOY defines how keystone is deployed, allowed values:
# - mod_wsgi : Run keystone under Apache HTTPd mod_wsgi
# - uwsgi : Run keystone under uwsgi
-if [ -z "$KEYSTONE_DEPLOY" ]; then
- if [ -z "$KEYSTONE_USE_MOD_WSGI" ]; then
- KEYSTONE_DEPLOY=mod_wsgi
- elif [ "$KEYSTONE_USE_MOD_WSGI" == True ]; then
- KEYSTONE_DEPLOY=mod_wsgi
- else
- KEYSTONE_DEPLOY=uwsgi
- fi
+if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+ KEYSTONE_DEPLOY=uwsgi
+else
+ KEYSTONE_DEPLOY=mod_wsgi
fi
# Select the token persistence backend driver
@@ -112,20 +108,14 @@
SERVICE_TENANT_NAME=${SERVICE_PROJECT_NAME:-service}
# if we are running with SSL use https protocols
-if is_ssl_enabled_service "key" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
KEYSTONE_AUTH_PROTOCOL="https"
KEYSTONE_SERVICE_PROTOCOL="https"
fi
-# complete URIs
-if [ "$KEYSTONE_DEPLOY" == "mod_wsgi" ]; then
- # If running in Apache, use path access rather than port.
- KEYSTONE_AUTH_URI=${KEYSTONE_AUTH_PROTOCOL}://${KEYSTONE_AUTH_HOST}/identity_admin
- KEYSTONE_SERVICE_URI=${KEYSTONE_SERVICE_PROTOCOL}://${KEYSTONE_SERVICE_HOST}/identity
-else
- KEYSTONE_AUTH_URI=${KEYSTONE_AUTH_PROTOCOL}://${KEYSTONE_AUTH_HOST}:${KEYSTONE_AUTH_PORT}
- KEYSTONE_SERVICE_URI=${KEYSTONE_SERVICE_PROTOCOL}://${KEYSTONE_SERVICE_HOST}:${KEYSTONE_SERVICE_PORT}
-fi
+KEYSTONE_SERVICE_URI=${KEYSTONE_SERVICE_PROTOCOL}://${KEYSTONE_SERVICE_HOST}/identity
+# for compat
+KEYSTONE_AUTH_URI=$KEYSTONE_SERVICE_URI
# V3 URIs
KEYSTONE_AUTH_URI_V3=$KEYSTONE_AUTH_URI/v3
@@ -134,9 +124,15 @@
# Security compliance
KEYSTONE_SECURITY_COMPLIANCE_ENABLED=${KEYSTONE_SECURITY_COMPLIANCE_ENABLED:-True}
KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS=${KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS:-2}
-KEYSTONE_LOCKOUT_DURATION=${KEYSTONE_LOCKOUT_DURATION:-5}
+KEYSTONE_LOCKOUT_DURATION=${KEYSTONE_LOCKOUT_DURATION:-10}
KEYSTONE_UNIQUE_LAST_PASSWORD_COUNT=${KEYSTONE_UNIQUE_LAST_PASSWORD_COUNT:-2}
+# Number of bcrypt hashing rounds, increasing number exponentially increases required
+# resources to generate password hash. This is very effective way to protect from
+# bruteforce attacks. 4 is minimal value that can be specified for bcrypt and
+# it works way faster than default 12. Minimal value is great for CI and development
+# however may not be suitable for real production.
+KEYSTONE_PASSWORD_HASH_ROUNDS=${KEYSTONE_PASSWORD_HASH_ROUNDS:-4}
# Functions
# ---------
@@ -144,6 +140,7 @@
# Test if Keystone is enabled
# is_keystone_enabled
function is_keystone_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"keystone" ]] && return 1
[[ ,${ENABLED_SERVICES}, =~ ,"key", ]] && return 0
return 1
}
@@ -151,6 +148,14 @@
# cleanup_keystone() - Remove residual data files, anything left over from previous
# runs that a clean run would need to clean up
function cleanup_keystone {
+ # TODO: remove admin at pike-2
+ # These files will be created if we are running WSGI_MODE="uwsgi"
+ remove_uwsgi_config "$KEYSTONE_PUBLIC_UWSGI_CONF" "$KEYSTONE_PUBLIC_UWSGI"
+ remove_uwsgi_config "$KEYSTONE_ADMIN_UWSGI_CONF" "$KEYSTONE_ADMIN_UWSGI"
+ sudo rm -f $(apache_site_config_for keystone-wsgi-public)
+ sudo rm -f $(apache_site_config_for keystone-wsgi-admin)
+
+ # These files will be created if we are running WSGI_MODE="mod_wsgi"
disable_apache_site keystone
sudo rm -f $(apache_site_config_for keystone)
}
@@ -167,12 +172,6 @@
local keystone_auth_port=$KEYSTONE_AUTH_PORT
local venv_path=""
- if is_ssl_enabled_service key; then
- keystone_ssl_listen=""
- keystone_ssl="SSLEngine On"
- keystone_certfile="SSLCertificateFile $KEYSTONE_SSL_CERT"
- keystone_keyfile="SSLCertificateKeyFile $KEYSTONE_SSL_KEY"
- fi
if is_service_enabled tls-proxy; then
keystone_service_port=$KEYSTONE_SERVICE_PORT_INT
keystone_auth_port=$KEYSTONE_AUTH_PORT_INT
@@ -202,7 +201,6 @@
if [[ "$KEYSTONE_CONF_DIR" != "$KEYSTONE_DIR/etc" ]]; then
install -m 600 $KEYSTONE_DIR/etc/keystone.conf.sample $KEYSTONE_CONF
- cp -p $KEYSTONE_DIR/etc/policy.json $KEYSTONE_CONF_DIR
if [[ -f "$KEYSTONE_DIR/etc/keystone-paste.ini" ]]; then
cp -p "$KEYSTONE_DIR/etc/keystone-paste.ini" "$KEYSTONE_PASTE_INI"
fi
@@ -233,6 +231,7 @@
fi
iniset $KEYSTONE_CONF identity driver "$KEYSTONE_IDENTITY_BACKEND"
+ iniset $KEYSTONE_CONF identity password_hash_rounds $KEYSTONE_PASSWORD_HASH_ROUNDS
iniset $KEYSTONE_CONF assignment driver "$KEYSTONE_ASSIGNMENT_BACKEND"
iniset $KEYSTONE_CONF role driver "$KEYSTONE_ROLE_BACKEND"
iniset $KEYSTONE_CONF resource driver "$KEYSTONE_RESOURCE_BACKEND"
@@ -244,11 +243,6 @@
iniset_rpc_backend keystone $KEYSTONE_CONF
- # Register SSL certificates if provided
- if is_ssl_enabled_service key; then
- ensure_certificates KEYSTONE
- fi
-
local service_port=$KEYSTONE_SERVICE_PORT
local auth_port=$KEYSTONE_AUTH_PORT
@@ -264,10 +258,8 @@
# work when you want to use a different port (in the case of proxy), or you
# don't want the port (in the case of putting keystone on a path in
# apache).
- if is_service_enabled tls-proxy || [ "$KEYSTONE_DEPLOY" == "mod_wsgi" ]; then
- iniset $KEYSTONE_CONF DEFAULT public_endpoint $KEYSTONE_SERVICE_URI
- iniset $KEYSTONE_CONF DEFAULT admin_endpoint $KEYSTONE_AUTH_URI
- fi
+ iniset $KEYSTONE_CONF DEFAULT public_endpoint $KEYSTONE_SERVICE_URI
+ iniset $KEYSTONE_CONF DEFAULT admin_endpoint $KEYSTONE_AUTH_URI
if [[ "$KEYSTONE_TOKEN_FORMAT" != "" ]]; then
iniset $KEYSTONE_CONF token provider $KEYSTONE_TOKEN_FORMAT
@@ -283,9 +275,7 @@
fi
# Format logging
- if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ] && [ "$KEYSTONE_DEPLOY" != "mod_wsgi" ] ; then
- setup_colorized_logging $KEYSTONE_CONF
- fi
+ setup_logging $KEYSTONE_CONF
iniset $KEYSTONE_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
@@ -293,45 +283,8 @@
iniset $KEYSTONE_CONF DEFAULT logging_exception_prefix "%(asctime)s.%(msecs)03d %(process)d TRACE %(name)s %(instance)s"
_config_keystone_apache_wsgi
else # uwsgi
- # iniset creates these files when it's called if they don't exist.
- KEYSTONE_PUBLIC_UWSGI_FILE=$KEYSTONE_CONF_DIR/keystone-uwsgi-public.ini
- KEYSTONE_ADMIN_UWSGI_FILE=$KEYSTONE_CONF_DIR/keystone-uwsgi-admin.ini
-
- rm -f "$KEYSTONE_PUBLIC_UWSGI_FILE"
- rm -f "$KEYSTONE_ADMIN_UWSGI_FILE"
-
- if is_ssl_enabled_service key; then
- iniset "$KEYSTONE_PUBLIC_UWSGI_FILE" uwsgi https $KEYSTONE_SERVICE_HOST:$service_port,$KEYSTONE_SSL_CERT,$KEYSTONE_SSL_KEY
- iniset "$KEYSTONE_ADMIN_UWSGI_FILE" uwsgi https $KEYSTONE_ADMIN_BIND_HOST:$auth_port,$KEYSTONE_SSL_CERT,$KEYSTONE_SSL_KEY
- else
- iniset "$KEYSTONE_PUBLIC_UWSGI_FILE" uwsgi http $KEYSTONE_SERVICE_HOST:$service_port
- iniset "$KEYSTONE_ADMIN_UWSGI_FILE" uwsgi http $KEYSTONE_ADMIN_BIND_HOST:$auth_port
- fi
-
- iniset "$KEYSTONE_PUBLIC_UWSGI_FILE" uwsgi wsgi-file "$KEYSTONE_BIN_DIR/keystone-wsgi-public"
- iniset "$KEYSTONE_PUBLIC_UWSGI_FILE" uwsgi processes $(nproc)
-
- iniset "$KEYSTONE_ADMIN_UWSGI_FILE" uwsgi wsgi-file "$KEYSTONE_BIN_DIR/keystone-wsgi-admin"
- iniset "$KEYSTONE_ADMIN_UWSGI_FILE" uwsgi processes $API_WORKERS
-
- # Common settings
- for file in "$KEYSTONE_PUBLIC_UWSGI_FILE" "$KEYSTONE_ADMIN_UWSGI_FILE"; do
- # This is running standalone
- iniset "$file" uwsgi master true
- # Set die-on-term & exit-on-reload so that uwsgi shuts down
- iniset "$file" uwsgi die-on-term true
- iniset "$file" uwsgi exit-on-reload true
- iniset "$file" uwsgi enable-threads true
- iniset "$file" uwsgi plugins python
- # uwsgi recommends this to prevent thundering herd on accept.
- iniset "$file" uwsgi thunder-lock true
- # Override the default size for headers from the 4k default.
- iniset "$file" uwsgi buffer-size 65535
- # Make sure the client doesn't try to re-use the connection.
- iniset "$file" uwsgi add-header "Connection: close"
- # This ensures that file descriptors aren't shared between processes.
- iniset "$file" uwsgi lazy-apps true
- done
+ write_uwsgi_config "$KEYSTONE_PUBLIC_UWSGI_CONF" "$KEYSTONE_PUBLIC_UWSGI" "/identity"
+ write_uwsgi_config "$KEYSTONE_ADMIN_UWSGI_CONF" "$KEYSTONE_ADMIN_UWSGI" "/identity_admin"
fi
iniset $KEYSTONE_CONF DEFAULT max_token_size 16384
@@ -489,14 +442,13 @@
local section=${4:-keystone_authtoken}
iniset $conf_file $section auth_type password
- iniset $conf_file $section auth_url $KEYSTONE_AUTH_URI
+ iniset $conf_file $section auth_url $KEYSTONE_SERVICE_URI
iniset $conf_file $section username $admin_user
iniset $conf_file $section password $SERVICE_PASSWORD
iniset $conf_file $section user_domain_name "$SERVICE_DOMAIN_NAME"
iniset $conf_file $section project_name $SERVICE_PROJECT_NAME
iniset $conf_file $section project_domain_name "$SERVICE_DOMAIN_NAME"
- iniset $conf_file $section auth_uri $KEYSTONE_SERVICE_URI
iniset $conf_file $section cafile $SSL_BUNDLE_FILE
iniset $conf_file $section signing_dir $signing_dir
iniset $conf_file $section memcached_servers $SERVICE_HOST:11211
@@ -513,8 +465,10 @@
recreate_database keystone
fi
+ time_start "dbsync"
# Initialize keystone database
$KEYSTONE_BIN_DIR/keystone-manage --config-file $KEYSTONE_CONF db_sync
+ time_stop "dbsync"
if [[ "$KEYSTONE_TOKEN_FORMAT" == "pki" || "$KEYSTONE_TOKEN_FORMAT" == "pkiz" ]]; then
# Set up certificates
@@ -578,9 +532,6 @@
if [ "$KEYSTONE_DEPLOY" == "mod_wsgi" ]; then
install_apache_wsgi
- if is_ssl_enabled_service "key"; then
- enable_mod_ssl
- fi
elif [ "$KEYSTONE_DEPLOY" == "uwsgi" ]; then
pip_install uwsgi
fi
@@ -602,11 +553,7 @@
tail_log key /var/log/$APACHE_NAME/keystone.log
tail_log key-access /var/log/$APACHE_NAME/keystone_access.log
else # uwsgi
- # TODO(sdague): we should really get down to a single keystone here
- enable_service key-p
- enable_service key-a
- run_process key-p "$KEYSTONE_BIN_DIR/uwsgi $KEYSTONE_PUBLIC_UWSGI_FILE" ""
- run_process key-a "$KEYSTONE_BIN_DIR/uwsgi $KEYSTONE_ADMIN_UWSGI_FILE" ""
+ run_process keystone "$KEYSTONE_BIN_DIR/uwsgi --ini $KEYSTONE_PUBLIC_UWSGI_CONF" ""
fi
echo "Waiting for keystone to start..."
@@ -615,10 +562,7 @@
# unencryted traffic at this point.
# If running in Apache, use the path rather than port.
- local service_uri=$auth_protocol://$KEYSTONE_SERVICE_HOST:$service_port/v$IDENTITY_API_VERSION/
- if [ "$KEYSTONE_DEPLOY" == "mod_wsgi" ]; then
- service_uri=$auth_protocol://$KEYSTONE_SERVICE_HOST/identity/v$IDENTITY_API_VERSION/
- fi
+ local service_uri=$auth_protocol://$KEYSTONE_SERVICE_HOST/identity/v$IDENTITY_API_VERSION/
if ! wait_for_service $SERVICE_TIMEOUT $service_uri; then
die $LINENO "keystone did not start"
@@ -639,6 +583,11 @@
if [ "$KEYSTONE_DEPLOY" == "mod_wsgi" ]; then
disable_apache_site keystone
restart_apache_server
+ else
+ stop_process keystone
+ remove_uwsgi_config "$KEYSTONE_PUBLIC_UWSGI_CONF" "$KEYSTONE_PUBLIC_UWSGI"
+ # TODO(remove in at pike-2)
+ remove_uwsgi_config "$KEYSTONE_ADMIN_UWSGI_CONF" "$KEYSTONE_ADMIN_UWSGI"
fi
# Kill the Keystone screen window
stop_process key
diff --git a/lib/neutron b/lib/neutron
index dd91466..2a660ec 100644
--- a/lib/neutron
+++ b/lib/neutron
@@ -61,7 +61,7 @@
NEUTRON_METERING_BINARY=${NEUTRON_METERING_BINARY:-neutron-metering-agent}
# Public facing bits
-if is_ssl_enabled_service "neutron" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
NEUTRON_SERVICE_PROTOCOL="https"
fi
NEUTRON_SERVICE_HOST=${NEUTRON_SERVICE_HOST:-$SERVICE_HOST}
@@ -91,6 +91,7 @@
# Test if any Neutron services are enabled
# is_neutron_enabled
function is_neutron_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"neutron" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"neutron-" || ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
return 1
}
@@ -98,6 +99,7 @@
# Test if any Neutron services are enabled
# is_neutron_enabled
function is_neutron_legacy_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"neutron" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
return 1
}
@@ -135,7 +137,11 @@
mkdir -p $NEUTRON_CORE_PLUGIN_CONF_PATH
- cp $NEUTRON_DIR/etc/neutron/plugins/$NEUTRON_CORE_PLUGIN/$NEUTRON_CORE_PLUGIN_CONF_FILENAME.sample $NEUTRON_CORE_PLUGIN_CONF
+ # NOTE(yamamoto): A decomposed plugin should prepare the config file in
+ # its devstack plugin.
+ if [ -f $NEUTRON_DIR/etc/neutron/plugins/$NEUTRON_CORE_PLUGIN/$NEUTRON_CORE_PLUGIN_CONF_FILENAME.sample ]; then
+ cp $NEUTRON_DIR/etc/neutron/plugins/$NEUTRON_CORE_PLUGIN/$NEUTRON_CORE_PLUGIN_CONF_FILENAME.sample $NEUTRON_CORE_PLUGIN_CONF
+ fi
iniset $NEUTRON_CONF database connection `database_connection_url neutron`
iniset $NEUTRON_CONF DEFAULT state_path $NEUTRON_STATE_PATH
@@ -226,31 +232,18 @@
iniset $NEUTRON_META_CONF agent root_helper_daemon "$NEUTRON_ROOTWRAP_DAEMON_CMD"
# TODO(dtroyer): remove the v2.0 hard code below
- iniset $NEUTRON_META_CONF DEFAULT auth_url $KEYSTONE_SERVICE_URI/v2.0
+ iniset $NEUTRON_META_CONF DEFAULT auth_url $KEYSTONE_SERVICE_URI
configure_auth_token_middleware $NEUTRON_META_CONF neutron $NEUTRON_AUTH_CACHE_DIR DEFAULT
fi
# Format logging
- if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ]; then
- setup_colorized_logging $NEUTRON_CONF DEFAULT project_id
- else
- # Show user_name and project_name by default
- iniset $NEUTRON_CONF DEFAULT logging_context_format_string "%(asctime)s.%(msecs)03d %(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s] %(instance)s%(message)s"
- fi
+ setup_logging $NEUTRON_CONF
if is_service_enabled tls-proxy; then
# Set the service port for a proxy to take the original
iniset $NEUTRON_CONF DEFAULT bind_port "$NEUTRON_SERVICE_PORT_INT"
fi
- if is_ssl_enabled_service "neutron"; then
- ensure_certificates NEUTRON
-
- iniset $NEUTRON_CONF DEFAULT use_ssl True
- iniset $NEUTRON_CONF DEFAULT ssl_cert_file "$NEUTRON_SSL_CERT"
- iniset $NEUTRON_CONF DEFAULT ssl_key_file "$NEUTRON_SSL_KEY"
- fi
-
# Metering
if is_service_enabled neutron-metering; then
cp $NEUTRON_DIR/etc/metering_agent.ini.sample $NEUTRON_METERING_AGENT_CONF
@@ -292,7 +285,7 @@
function configure_neutron_nova_new {
iniset $NOVA_CONF DEFAULT use_neutron True
iniset $NOVA_CONF neutron auth_type "password"
- iniset $NOVA_CONF neutron auth_url "$KEYSTONE_SERVICE_URI/v3"
+ iniset $NOVA_CONF neutron auth_url "$KEYSTONE_SERVICE_URI"
iniset $NOVA_CONF neutron username neutron
iniset $NOVA_CONF neutron password "$SERVICE_PASSWORD"
iniset $NOVA_CONF neutron user_domain_name "Default"
@@ -343,8 +336,10 @@
recreate_database neutron
+ time_start "dbsync"
# Run Neutron db migrations
$NEUTRON_BIN_DIR/neutron-db-manage upgrade heads
+ time_stop "dbsync"
create_neutron_cache_dir
}
@@ -404,17 +399,10 @@
# TODO(sc68cal) Stop hard coding this
run_process neutron-api "$NEUTRON_BIN_DIR/neutron-server $opts"
- if is_ssl_enabled_service "neutron"; then
- ssl_ca="--ca-certificate=${SSL_BUNDLE_FILE}"
- local testcmd="wget ${ssl_ca} --no-proxy -q -O- $service_protocol://$NEUTRON_SERVICE_HOST:$service_port"
- test_with_retry "$testcmd" "Neutron did not start" $SERVICE_TIMEOUT
- else
- if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$NEUTRON_SERVICE_HOST:$service_port; then
- die $LINENO "neutron-api did not start"
- fi
+ if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$NEUTRON_SERVICE_HOST:$service_port; then
+ die $LINENO "neutron-api did not start"
fi
-
# Start proxy if enabled
if is_service_enabled tls-proxy; then
start_tls_proxy neutron '*' $NEUTRON_SERVICE_PORT $NEUTRON_SERVICE_HOST $NEUTRON_SERVICE_PORT_INT
diff --git a/lib/neutron-legacy b/lib/neutron-legacy
index 86a2b1d..784f3a8 100644
--- a/lib/neutron-legacy
+++ b/lib/neutron-legacy
@@ -20,6 +20,7 @@
# - init_neutron_third_party
# - start_neutron_third_party
# - create_nova_conf_neutron
+# - configure_neutron_after_post_config
# - start_neutron_service_and_check
# - check_neutron_third_party_integration
# - start_neutron_agents
@@ -61,7 +62,7 @@
deprecated "Using lib/neutron-legacy is deprecated, and it will be removed in the future"
-if is_ssl_enabled_service "neutron" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
Q_PROTOCOL="https"
fi
@@ -331,7 +332,6 @@
_configure_neutron_common
iniset_rpc_backend neutron $NEUTRON_CONF
- # goes before q-svc to init Q_SERVICE_PLUGIN_CLASSES
if is_service_enabled q-metering; then
_configure_neutron_metering
fi
@@ -368,7 +368,7 @@
function create_nova_conf_neutron {
iniset $NOVA_CONF DEFAULT use_neutron True
iniset $NOVA_CONF neutron auth_type "password"
- iniset $NOVA_CONF neutron auth_url "$KEYSTONE_AUTH_URI/v3"
+ iniset $NOVA_CONF neutron auth_url "$KEYSTONE_AUTH_URI"
iniset $NOVA_CONF neutron username "$Q_ADMIN_USERNAME"
iniset $NOVA_CONF neutron password "$SERVICE_PASSWORD"
iniset $NOVA_CONF neutron user_domain_name "$SERVICE_DOMAIN_NAME"
@@ -417,8 +417,10 @@
# init_mutnauq() - Initialize databases, etc.
function init_mutnauq {
recreate_database $Q_DB_NAME
+ time_start "dbsync"
# Run Neutron db migrations
$NEUTRON_BIN_DIR/neutron-db-manage --config-file $NEUTRON_CONF --config-file /$Q_PLUGIN_CONF_FILE upgrade head
+ time_stop "dbsync"
}
# install_mutnauq() - Collect source and prepare
@@ -432,24 +434,6 @@
git_clone $NEUTRON_REPO $NEUTRON_DIR $NEUTRON_BRANCH
setup_develop $NEUTRON_DIR
-
- if [ "$VIRT_DRIVER" == 'xenserver' ]; then
- local dom0_ip
- dom0_ip=$(echo "$XENAPI_CONNECTION_URL" | cut -d "/" -f 3-)
-
- local ssh_dom0
- ssh_dom0="sudo -u $DOMZERO_USER ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@$dom0_ip"
-
- # Find where the plugins should go in dom0
- local xen_functions
- xen_functions=$(cat $TOP_DIR/tools/xen/functions)
- local plugin_dir
- plugin_dir=$($ssh_dom0 "$xen_functions; set -eux; xapi_plugin_location")
-
- # install neutron plugins to dom0
- tar -czf - -C $NEUTRON_DIR/neutron/plugins/ml2/drivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/ ./ |
- $ssh_dom0 "tar -xzf - -C $plugin_dir && chmod a+x $plugin_dir/*"
- fi
}
# install_neutron_agent_packages() - Collect source and prepare
@@ -464,6 +448,13 @@
fi
}
+# Finish neutron configuration
+function configure_neutron_after_post_config {
+ if [[ $Q_SERVICE_PLUGIN_CLASSES != '' ]]; then
+ iniset $NEUTRON_CONF DEFAULT service_plugins $Q_SERVICE_PLUGIN_CLASSES
+ fi
+}
+
# Start running processes, including screen
function start_neutron_service_and_check {
local service_port=$Q_PORT
@@ -479,9 +470,6 @@
# Start the Neutron service
run_process q-svc "$NEUTRON_BIN_DIR/neutron-server $cfg_file_options"
echo "Waiting for Neutron to start..."
- if is_ssl_enabled_service "neutron"; then
- ssl_ca="--ca-certificate=${SSL_BUNDLE_FILE}"
- fi
local testcmd="wget ${ssl_ca} --no-proxy -q -O- $service_protocol://$Q_HOST:$service_port"
test_with_retry "$testcmd" "Neutron did not start" $SERVICE_TIMEOUT
@@ -523,11 +511,6 @@
run_process q-meta "$AGENT_META_BINARY --config-file $NEUTRON_CONF --config-file $Q_META_CONF_FILE"
run_process q-metering "$AGENT_METERING_BINARY --config-file $NEUTRON_CONF --config-file $METERING_AGENT_CONF_FILENAME"
-
- if [ "$VIRT_DRIVER" = 'xenserver' ]; then
- # For XenServer, start an agent for the domU openvswitch
- run_process q-domua "$AGENT_BINARY --config-file $NEUTRON_CONF --config-file /$Q_PLUGIN_CONF_FILE.domU"
- fi
}
# Start running processes, including screen
@@ -539,10 +522,6 @@
function stop_mutnauq_l2_agent {
stop_process q-agt
-
- if [ "$VIRT_DRIVER" = 'xenserver' ]; then
- stop_process q-domua
- fi
}
# stop_mutnauq_other() - Stop running processes (non-screen)
@@ -600,7 +579,7 @@
local IP_DEL=""
local IP_UP=""
local DEFAULT_ROUTE_GW
- DEFAULT_ROUTE_GW=$(ip -f $af r | awk "/default.+$from_intf/ { print \$3; exit }")
+ DEFAULT_ROUTE_GW=$(ip -f $af r | awk "/default.+$from_intf\s/ { print \$3; exit }")
local ADD_OVS_PORT=""
local DEL_OVS_PORT=""
local ARP_CMD=""
@@ -741,18 +720,6 @@
iniset $NEUTRON_CONF DEFAULT bind_port "$Q_PORT_INT"
fi
- if is_ssl_enabled_service "nova"; then
- iniset $NEUTRON_CONF nova cafile $SSL_BUNDLE_FILE
- fi
-
- if is_ssl_enabled_service "neutron"; then
- ensure_certificates NEUTRON
-
- iniset $NEUTRON_CONF DEFAULT use_ssl True
- iniset $NEUTRON_CONF DEFAULT ssl_cert_file "$NEUTRON_SSL_CERT"
- iniset $NEUTRON_CONF DEFAULT ssl_key_file "$NEUTRON_SSL_KEY"
- fi
-
_neutron_setup_rootwrap
}
@@ -836,10 +803,6 @@
# Update either configuration file with plugin
iniset $NEUTRON_CONF DEFAULT core_plugin $Q_PLUGIN_CLASS
- if [[ $Q_SERVICE_PLUGIN_CLASSES != '' ]]; then
- iniset $NEUTRON_CONF DEFAULT service_plugins $Q_SERVICE_PLUGIN_CLASSES
- fi
-
iniset $NEUTRON_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
iniset $NEUTRON_CONF oslo_policy policy_file $Q_POLICY_FILE
iniset $NEUTRON_CONF DEFAULT allow_overlapping_ips $Q_ALLOW_OVERLAPPING_IP
diff --git a/lib/neutron_plugins/openvswitch_agent b/lib/neutron_plugins/openvswitch_agent
index acab582..b65a258 100644
--- a/lib/neutron_plugins/openvswitch_agent
+++ b/lib/neutron_plugins/openvswitch_agent
@@ -11,12 +11,6 @@
function neutron_plugin_create_nova_conf {
_neutron_ovs_base_configure_nova_vif_driver
- if [ "$VIRT_DRIVER" == 'xenserver' ]; then
- iniset $NOVA_CONF xenserver vif_driver nova.virt.xenapi.vif.XenAPIOpenVswitchDriver
- iniset $NOVA_CONF xenserver ovs_integration_bridge $XEN_INTEGRATION_BRIDGE
- # Disable nova's firewall so that it does not conflict with neutron
- iniset $NOVA_CONF DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
- fi
}
function neutron_plugin_install_agent_packages {
@@ -58,65 +52,6 @@
fi
AGENT_BINARY="$NEUTRON_BIN_DIR/neutron-openvswitch-agent"
- if [ "$VIRT_DRIVER" == 'xenserver' ]; then
- # Make a copy of our config for domU
- sudo cp /$Q_PLUGIN_CONF_FILE "/$Q_PLUGIN_CONF_FILE.domU"
-
- # change domU's config file to STACK_USER
- sudo chown $STACK_USER:$STACK_USER /$Q_PLUGIN_CONF_FILE.domU
-
- # Deal with Dom0's L2 Agent:
- Q_RR_DOM0_COMMAND="$NEUTRON_BIN_DIR/neutron-rootwrap-xen-dom0 $Q_RR_CONF_FILE"
-
- # For now, duplicate the xen configuration already found in nova.conf
- iniset $Q_RR_CONF_FILE xenapi xenapi_connection_url "$XENAPI_CONNECTION_URL"
- iniset $Q_RR_CONF_FILE xenapi xenapi_connection_username "$XENAPI_USER"
- iniset $Q_RR_CONF_FILE xenapi xenapi_connection_password "$XENAPI_PASSWORD"
-
- # Under XS/XCP, the ovs agent needs to target the dom0
- # integration bridge. This is enabled by using a root wrapper
- # that executes commands on dom0 via a XenAPI plugin.
- # XenAPI does not support daemon rootwrap now, so set root_helper_daemon empty
- iniset "/$Q_PLUGIN_CONF_FILE.domU" agent root_helper ""
- iniset "/$Q_PLUGIN_CONF_FILE.domU" agent root_helper_daemon "xenapi_root_helper"
- iniset "/$Q_PLUGIN_CONF_FILE.domU" xenapi connection_url "$XENAPI_CONNECTION_URL"
- iniset "/$Q_PLUGIN_CONF_FILE.domU" xenapi connection_username "$XENAPI_USER"
- iniset "/$Q_PLUGIN_CONF_FILE.domU" xenapi connection_password "$XENAPI_PASSWORD"
-
- # Disable minimize polling, so that it can always detect OVS and Port changes
- # This is a problem of xenserver + neutron, bug has been reported
- # https://bugs.launchpad.net/neutron/+bug/1495423
- iniset "/$Q_PLUGIN_CONF_FILE.domU" agent minimize_polling False
-
- # Set "physical" mapping
- iniset "/$Q_PLUGIN_CONF_FILE.domU" ovs bridge_mappings "physnet1:$FLAT_NETWORK_BRIDGE"
-
- # XEN_INTEGRATION_BRIDGE is the integration bridge in dom0
- iniset "/$Q_PLUGIN_CONF_FILE.domU" ovs integration_bridge $XEN_INTEGRATION_BRIDGE
-
- # Set OVS native interface for ovs-agent in compute node
- XEN_DOM0_IP=$(echo "$XENAPI_CONNECTION_URL" | cut -d "/" -f 3)
- iniset /$Q_PLUGIN_CONF_FILE.domU ovs ovsdb_connection tcp:$XEN_DOM0_IP:6640
- iniset /$Q_PLUGIN_CONF_FILE.domU ovs of_listen_address $HOST_IP
-
- # Set up domU's L2 agent:
-
- # Create a bridge "br-$VLAN_INTERFACE"
- _neutron_ovs_base_add_bridge "br-$VLAN_INTERFACE"
- # Add $VLAN_INTERFACE to that bridge
- sudo ovs-vsctl -- --may-exist add-port "br-$VLAN_INTERFACE" $VLAN_INTERFACE
-
- # Create external bridge and add port
- _neutron_ovs_base_add_public_bridge
- sudo ovs-vsctl -- --may-exist add-port $PUBLIC_BRIDGE $PUBLIC_INTERFACE
-
- # Set bridge mappings to "physnet1:br-$GUEST_INTERFACE_DEFAULT"
- iniset /$Q_PLUGIN_CONF_FILE ovs bridge_mappings "physnet1:br-$VLAN_INTERFACE,physnet-ex:$PUBLIC_BRIDGE"
- # Set integration bridge to domU's
- iniset /$Q_PLUGIN_CONF_FILE ovs integration_bridge $OVS_BRIDGE
- # Set root wrap
- iniset /$Q_PLUGIN_CONF_FILE agent root_helper "$Q_RR_COMMAND"
- fi
iniset /$Q_PLUGIN_CONF_FILE agent tunnel_types $Q_TUNNEL_TYPES
iniset /$Q_PLUGIN_CONF_FILE ovs datapath_type $OVS_DATAPATH_TYPE
}
diff --git a/lib/nova b/lib/nova
index a36a740..18715fc 100644
--- a/lib/nova
+++ b/lib/nova
@@ -68,7 +68,7 @@
# Toggle for deploying Nova-API under HTTPD + mod_wsgi
NOVA_USE_MOD_WSGI=${NOVA_USE_MOD_WSGI:-False}
-if is_ssl_enabled_service "nova" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
NOVA_SERVICE_PROTOCOL="https"
fi
@@ -175,6 +175,7 @@
# Test if any Nova services are enabled
# is_nova_enabled
function is_nova_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"nova" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"n-" ]] && return 0
return 1
}
@@ -262,11 +263,6 @@
local nova_metadata_port=$METADATA_SERVICE_PORT
local venv_path=""
- if is_ssl_enabled_service nova-api; then
- nova_ssl="SSLEngine On"
- nova_certfile="SSLCertificateFile $NOVA_SSL_CERT"
- nova_keyfile="SSLCertificateKeyFile $NOVA_SSL_KEY"
- fi
if [[ ${USE_VENV} = True ]]; then
venv_path="python-path=${PROJECT_VENV["nova"]}/lib/$(python_version)/site-packages"
fi
@@ -458,8 +454,8 @@
fi
iniset $NOVA_CONF wsgi api_paste_config "$NOVA_API_PASTE_INI"
iniset $NOVA_CONF DEFAULT rootwrap_config "$NOVA_CONF_DIR/rootwrap.conf"
- iniset $NOVA_CONF DEFAULT scheduler_driver "$SCHEDULER"
- iniset $NOVA_CONF DEFAULT scheduler_default_filters "$FILTERS"
+ iniset $NOVA_CONF scheduler driver "$SCHEDULER"
+ iniset $NOVA_CONF filter_scheduler enabled_filters "$FILTERS"
iniset $NOVA_CONF DEFAULT default_floating_pool "$PUBLIC_NETWORK_NAME"
if [[ $SERVICE_IP_VERSION == 6 ]]; then
iniset $NOVA_CONF DEFAULT my_ip "$HOST_IPV6"
@@ -471,6 +467,8 @@
iniset $NOVA_CONF DEFAULT osapi_compute_listen "$NOVA_SERVICE_LISTEN_ADDRESS"
iniset $NOVA_CONF DEFAULT metadata_listen "$NOVA_SERVICE_LISTEN_ADDRESS"
+ iniset $NOVA_CONF key_manager api_class nova.keymgr.conf_key_mgr.ConfKeyManager
+
if is_fedora || is_suse; then
# nova defaults to /usr/local/bin, but fedora and suse pip like to
# install things in /usr/bin
@@ -501,7 +499,7 @@
fi
if is_service_enabled cinder; then
- if is_ssl_enabled_service "cinder" || is_service_enabled tls-proxy; then
+ if is_service_enabled tls-proxy; then
CINDER_SERVICE_HOST=${CINDER_SERVICE_HOST:-$SERVICE_HOST}
CINDER_SERVICE_PORT=${CINDER_SERVICE_PORT:-8776}
iniset $NOVA_CONF cinder cafile $SSL_BUNDLE_FILE
@@ -586,20 +584,10 @@
iniset $NOVA_CONF cinder os_region_name "$REGION_NAME"
- if is_ssl_enabled_service glance || is_service_enabled tls-proxy; then
+ if is_service_enabled tls-proxy; then
iniset $NOVA_CONF DEFAULT glance_protocol https
fi
- # Register SSL certificates if provided
- if is_ssl_enabled_service nova; then
- ensure_certificates NOVA
-
- iniset $NOVA_CONF DEFAULT ssl_cert_file "$NOVA_SSL_CERT"
- iniset $NOVA_CONF DEFAULT ssl_key_file "$NOVA_SSL_KEY"
-
- iniset $NOVA_CONF DEFAULT enabled_ssl_apis "$NOVA_ENABLED_APIS"
- fi
-
if is_service_enabled n-sproxy; then
iniset $NOVA_CONF serial_console serialproxy_host "$NOVA_SERVICE_LISTEN_ADDRESS"
iniset $NOVA_CONF serial_console enabled True
@@ -661,7 +649,9 @@
iniset $NOVA_CELLS_CONF DEFAULT enabled_apis metadata
fi
+ time_start "dbsync"
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CELLS_CONF db sync
+ time_stop "dbsync"
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CELLS_CONF cell create --name=region --cell_type=parent --username=$RABBIT_USERID --hostname=$RABBIT_HOST --port=5672 --password=$RABBIT_PASSWORD --virtual_host=/ --woffset=0 --wscale=1
$NOVA_BIN_DIR/nova-manage cell create --name=child --cell_type=child --username=$RABBIT_USERID --hostname=$RABBIT_HOST --port=5672 --password=$RABBIT_PASSWORD --virtual_host=child_cell --woffset=0 --wscale=1
@@ -790,9 +780,6 @@
if [ "$NOVA_USE_MOD_WSGI" == "True" ]; then
install_apache_wsgi
- if is_ssl_enabled_service "nova-api"; then
- enable_mod_ssl
- fi
fi
}
@@ -894,7 +881,6 @@
run_process n-cond "$NOVA_BIN_DIR/nova-conductor --config-file $compute_cell_conf"
run_process n-cell-region "$NOVA_BIN_DIR/nova-cells --config-file $api_cell_conf"
run_process n-cell-child "$NOVA_BIN_DIR/nova-cells --config-file $compute_cell_conf"
- run_process n-crt "$NOVA_BIN_DIR/nova-cert --config-file $api_cell_conf"
if is_service_enabled n-net; then
if ! running_in_container; then
@@ -945,7 +931,7 @@
# Kill the nova screen windows
# Some services are listed here twice since more than one instance
# of a service may be running in certain configs.
- for serv in n-api n-crt n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cell n-cell n-api-meta n-sproxy; do
+ for serv in n-api n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cell n-cell n-api-meta n-sproxy; do
stop_process $serv
done
}
diff --git a/lib/nova_plugins/functions-libvirt b/lib/nova_plugins/functions-libvirt
index 56bb6bd..3e38b89 100644
--- a/lib/nova_plugins/functions-libvirt
+++ b/lib/nova_plugins/functions-libvirt
@@ -20,17 +20,56 @@
# extremely verbose.)
DEBUG_LIBVIRT=$(trueorfalse True DEBUG_LIBVIRT)
+# Try to enable coredumps for libvirt
+# Currently fairly specific to OpenStackCI hosts
+DEBUG_LIBVIRT_COREDUMPS=$(trueorfalse False DEBUG_LIBVIRT_COREDUMPS)
+
+# Only Xenial is left with libvirt-bin. Everywhere else is libvirtd
+if is_ubuntu && [ ! -f /etc/init.d/libvirtd ]; then
+ LIBVIRT_DAEMON=libvirt-bin
+else
+ LIBVIRT_DAEMON=libvirtd
+fi
+
+# Enable coredumps for libvirt
+# Bug: https://bugs.launchpad.net/nova/+bug/1643911
+function _enable_coredump {
+ local confdir=/etc/systemd/system/${LIBVIRT_DAEMON}.service.d
+ local conffile=${confdir}/coredump.conf
+
+ # Create a coredump directory, and instruct the kernel to save to
+ # here
+ sudo mkdir -p /var/core
+ sudo chmod a+wrx /var/core
+ echo '/var/core/core.%e.%p.%h.%t' | \
+ sudo tee /proc/sys/kernel/core_pattern
+
+ # Drop a config file to up the core ulimit
+ sudo mkdir -p ${confdir}
+ sudo tee ${conffile} <<EOF
+[Service]
+LimitCORE=infinity
+EOF
+
+ # Tell systemd to reload the unit (service restarts later after
+ # config anyway)
+ sudo systemctl daemon-reload
+}
+
+
# Installs required distro-specific libvirt packages.
function install_libvirt {
+
if is_ubuntu; then
install_package qemu-system
- install_package libvirt-bin libvirt-dev
- pip_install_gr libvirt-python
- if [[ ${DISTRO} == "trusty" && ${EBTABLES_RACE_FIX} == "True" ]]; then
- # Work around for bug #1501558. We can remove this once we
- # get to a version of Ubuntu that has new enough libvirt.
- TOP_DIR=$TOP_DIR $TOP_DIR/tools/install_ebtables_workaround.sh
+ if [[ ${DISTRO} == "xenial" ]]; then
+ install_package libvirt-bin libvirt-dev
+ else
+ install_package libvirt-clients libvirt-daemon-system libvirt-dev
fi
+ # uninstall in case the libvirt version changed
+ pip_uninstall libvirt-python
+ pip_install_gr libvirt-python
#pip_install_gr <there-si-no-guestfs-in-pypi>
elif is_fedora || is_suse; then
# On "KVM for IBM z Systems", kvm does not have its own package
@@ -47,15 +86,19 @@
fi
install_package libvirt libvirt-devel
+ pip_uninstall libvirt-python
pip_install_gr libvirt-python
+ fi
+ if [[ $DEBUG_LIBVIRT_COREDUMPS == True ]]; then
+ _enable_coredump
fi
}
# Configures the installed libvirt system so that is accessible by
# STACK_USER via qemu:///system with management capabilities.
function configure_libvirt {
- if is_service_enabled neutron && is_neutron_ovs_base_plugin && ! sudo grep -q '^cgroup_device_acl' $QEMU_CONF; then
+ if is_service_enabled neutron && ! sudo grep -q '^cgroup_device_acl' $QEMU_CONF; then
# Add /dev/net/tun to cgroup_device_acls, needed for type=ethernet interfaces
cat <<EOF | sudo tee -a $QEMU_CONF
cgroup_device_acl = [
@@ -68,14 +111,6 @@
EOF
fi
- # Since the release of Debian Wheezy the libvirt init script is libvirtd
- # and not libvirtd-bin anymore.
- if is_ubuntu && [ ! -f /etc/init.d/libvirtd ]; then
- LIBVIRT_DAEMON=libvirt-bin
- else
- LIBVIRT_DAEMON=libvirtd
- fi
-
if is_fedora || is_suse; then
# Starting with fedora 18 and opensuse-12.3 enable stack-user to
# virsh -c qemu:///system by creating a policy-kit rule for
diff --git a/lib/nova_plugins/hypervisor-fake b/lib/nova_plugins/hypervisor-fake
index f9b95c1..49c8dee 100644
--- a/lib/nova_plugins/hypervisor-fake
+++ b/lib/nova_plugins/hypervisor-fake
@@ -49,7 +49,7 @@
iniset $NOVA_CONF DEFAULT quota_security_groups -1
iniset $NOVA_CONF DEFAULT quota_security_group_rules -1
iniset $NOVA_CONF DEFAULT quota_key_pairs -1
- iniset $NOVA_CONF DEFAULT scheduler_default_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"
+ iniset $NOVA_CONF filter_scheduler enabled_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"
}
# install_nova_hypervisor() - Install external components
diff --git a/lib/nova_plugins/hypervisor-ironic b/lib/nova_plugins/hypervisor-ironic
index c9544fe..7d47ef0 100644
--- a/lib/nova_plugins/hypervisor-ironic
+++ b/lib/nova_plugins/hypervisor-ironic
@@ -49,7 +49,7 @@
iniset $NOVA_CONF ironic auth_type password
iniset $NOVA_CONF ironic username admin
iniset $NOVA_CONF ironic password $ADMIN_PASSWORD
- iniset $NOVA_CONF ironic auth_url $KEYSTONE_AUTH_URI/v3
+ iniset $NOVA_CONF ironic auth_url $KEYSTONE_AUTH_URI
iniset $NOVA_CONF ironic project_domain_id default
iniset $NOVA_CONF ironic user_domain_id default
iniset $NOVA_CONF ironic project_name demo
diff --git a/lib/nova_plugins/hypervisor-xenserver b/lib/nova_plugins/hypervisor-xenserver
index 0046a36..6f79e4f 100644
--- a/lib/nova_plugins/hypervisor-xenserver
+++ b/lib/nova_plugins/hypervisor-xenserver
@@ -26,10 +26,6 @@
# Allow ``build_domU.sh`` to specify the flat network bridge via kernel args
FLAT_NETWORK_BRIDGE_DEFAULT=$(sed -e 's/.* flat_network_bridge=\([[:alnum:]]*\).*$/\1/g' /proc/cmdline)
-if is_service_enabled neutron; then
- XEN_INTEGRATION_BRIDGE_DEFAULT=$(sed -e 's/.* xen_integration_bridge=\([[:alnum:]]*\).*$/\1/g' /proc/cmdline)
- XEN_INTEGRATION_BRIDGE=${XEN_INTEGRATION_BRIDGE:-$XEN_INTEGRATION_BRIDGE_DEFAULT}
-fi
VNCSERVER_PROXYCLIENT_ADDRESS=${VNCSERVER_PROXYCLIENT_ADDRESS=169.254.0.1}
@@ -88,28 +84,6 @@
* * * * * /root/rotate_xen_guest_logs.sh >/dev/null 2>&1
CRONTAB
- # Create directories for kernels and images
- {
- echo "set -eux"
- cat $TOP_DIR/tools/xen/functions
- echo "create_directory_for_images"
- echo "create_directory_for_kernels"
- echo "install_conntrack_tools"
- } | $ssh_dom0
-
- if is_service_enabled neutron; then
- # Remove restriction on linux bridge in Dom0 when neutron is enabled
- $ssh_dom0 "rm -f /etc/modprobe.d/blacklist-bridge*"
-
- count=`$ssh_dom0 "iptables -t filter -L XenServerDevstack |wc -l"`
- if [ "$count" = "0" ]; then
- {
- echo "iptables -t filter --new XenServerDevstack"
- echo "iptables -t filter -I INPUT -j XenServerDevstack"
- echo "iptables -t filter -I XenServerDevstack -p tcp --dport 6640 -j ACCEPT"
- } | $ssh_dom0
- fi
- fi
}
# install_nova_hypervisor() - Install external components
diff --git a/lib/oslo b/lib/oslo
index e34e48a..d15a303 100644
--- a/lib/oslo
+++ b/lib/oslo
@@ -23,7 +23,9 @@
# Defaults
# --------
GITDIR["automaton"]=$DEST/automaton
+GITDIR["castellan"]=$DEST/castellan
GITDIR["cliff"]=$DEST/cliff
+GITDIR["cursive"]=$DEST/cursive
GITDIR["debtcollector"]=$DEST/debtcollector
GITDIR["futurist"]=$DEST/futurist
GITDIR["os-client-config"]=$DEST/os-client-config
@@ -48,9 +50,15 @@
GITDIR["oslo.vmware"]=$DEST/oslo.vmware
GITDIR["osprofiler"]=$DEST/osprofiler
GITDIR["pycadf"]=$DEST/pycadf
+GITDIR["python-openstacksdk"]=$DEST/python-openstacksdk
GITDIR["stevedore"]=$DEST/stevedore
GITDIR["taskflow"]=$DEST/taskflow
GITDIR["tooz"]=$DEST/tooz
+# TODO(mriedem): This is a common pattern so even though os-traits isn't
+# officially an oslo library, it is nice to re-use this script for non-oslo
+# things like os-traits. We should rename this script to be more generic
+# and then fold os-brick into it also.
+GITDIR["os-traits"]=$DEST/os-traits
# Support entry points installation of console scripts
OSLO_BIN_DIR=$(get_python_exec_prefix)
@@ -70,7 +78,9 @@
# install_oslo() - Collect source and prepare
function install_oslo {
_do_install_oslo_lib "automaton"
+ _do_install_oslo_lib "castellan"
_do_install_oslo_lib "cliff"
+ _do_install_oslo_lib "cursive"
_do_install_oslo_lib "debtcollector"
_do_install_oslo_lib "futurist"
_do_install_oslo_lib "osc-lib"
@@ -95,9 +105,11 @@
_do_install_oslo_lib "oslo.vmware"
_do_install_oslo_lib "osprofiler"
_do_install_oslo_lib "pycadf"
+ _do_install_oslo_lib "python-openstacksdk"
_do_install_oslo_lib "stevedore"
_do_install_oslo_lib "taskflow"
_do_install_oslo_lib "tooz"
+ _do_install_oslo_lib "os-traits"
}
# Restore xtrace
diff --git a/lib/placement b/lib/placement
index 4cc5cd8..8adbbde 100644
--- a/lib/placement
+++ b/lib/placement
@@ -32,7 +32,15 @@
PLACEMENT_CONF_DIR=/etc/nova
PLACEMENT_CONF=$PLACEMENT_CONF_DIR/nova.conf
PLACEMENT_AUTH_STRATEGY=${PLACEMENT_AUTH_STRATEGY:-placement}
-
+# Nova virtual environment
+if [[ ${USE_VENV} = True ]]; then
+ PROJECT_VENV["nova"]=${NOVA_DIR}.venv
+ PLACEMENT_BIN_DIR=${PROJECT_VENV["nova"]}/bin
+else
+ PLACEMENT_BIN_DIR=$(get_python_exec_prefix)
+fi
+PLACEMENT_UWSGI=$PLACEMENT_BIN_DIR/nova-placement-api
+PLACEMENT_UWSGI_CONF=$PLACEMENT_CONF_DIR/placement-uwsgi.ini
# The placement service can optionally use a separate database
# connection. Set PLACEMENT_DB_ENABLED to True to use it.
@@ -40,7 +48,7 @@
# yet merged in nova but is coming soon.
PLACEMENT_DB_ENABLED=$(trueorfalse False PLACEMENT_DB_ENABLED)
-if is_ssl_enabled_service "placement-api" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
PLACEMENT_SERVICE_PROTOCOL="https"
fi
@@ -61,6 +69,7 @@
# cleanup_placement() - Remove residual data files, anything left over from previous
# runs that a clean run would need to clean up
function cleanup_placement {
+ sudo rm -f $(apache_site_config_for nova-placement-api)
sudo rm -f $(apache_site_config_for placement-api)
}
@@ -72,12 +81,6 @@
nova_bin_dir=$(get_python_exec_prefix)
placement_api_apache_conf=$(apache_site_config_for placement-api)
- # reuse nova's cert if a cert is being used
- if is_ssl_enabled_service "placement-api"; then
- placement_ssl="SSLEngine On"
- placement_certfile="SSLCertificateFile $NOVA_SSL_CERT"
- placement_keyfile="SSLCertificateKeyFile $NOVA_SSL_KEY"
- fi
# reuse nova's venv if there is one as placement code lives
# there
if [[ ${USE_VENV} = True ]]; then
@@ -100,7 +103,7 @@
function configure_placement_nova_compute {
iniset $NOVA_CONF placement auth_type "password"
- iniset $NOVA_CONF placement auth_url "$KEYSTONE_SERVICE_URI/v3"
+ iniset $NOVA_CONF placement auth_url "$KEYSTONE_SERVICE_URI"
iniset $NOVA_CONF placement username placement
iniset $NOVA_CONF placement password "$SERVICE_PASSWORD"
iniset $NOVA_CONF placement user_domain_name "$SERVICE_DOMAIN_NAME"
@@ -120,7 +123,12 @@
if [ "$PLACEMENT_DB_ENABLED" != False ]; then
iniset $PLACEMENT_CONF placement_database connection `database_connection_url placement`
fi
- _config_placement_apache_wsgi
+
+ if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+ write_uwsgi_config "$PLACEMENT_UWSGI_CONF" "$PLACEMENT_UWSGI" "/placement"
+ else
+ _config_placement_apache_wsgi
+ fi
}
# create_placement_accounts() - Set up required placement accounts
@@ -141,7 +149,9 @@
function init_placement {
if [ "$PLACEMENT_DB_ENABLED" != False ]; then
recreate_database placement
+ time_start "dbsync"
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CONF api_db sync
+ time_stop "dbsync"
fi
create_placement_accounts
}
@@ -149,16 +159,17 @@
# install_placement() - Collect source and prepare
function install_placement {
install_apache_wsgi
- if is_ssl_enabled_service "placement-api"; then
- enable_mod_ssl
- fi
}
# start_placement_api() - Start the API processes ahead of other things
function start_placement_api {
- enable_apache_site placement-api
- restart_apache_server
- tail_log placement-api /var/log/$APACHE_NAME/placement-api.log
+ if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+ run_process "placement-api" "$PLACEMENT_BIN_DIR/uwsgi --ini $PLACEMENT_UWSGI_CONF"
+ else
+ enable_apache_site placement-api
+ restart_apache_server
+ tail_log placement-api /var/log/$APACHE_NAME/placement-api.log
+ fi
echo "Waiting for placement-api to start..."
if ! wait_for_service $SERVICE_TIMEOUT $PLACEMENT_SERVICE_PROTOCOL://$PLACEMENT_SERVICE_HOST/placement; then
@@ -172,8 +183,13 @@
# stop_placement() - Disable the api service and stop it.
function stop_placement {
- disable_apache_site placement-api
- restart_apache_server
+ if [[ "$WSGI_MODE" == "uwsgi" ]]; then
+ stop_process "placement-api"
+ remove_uwsgi_config "$PLACEMENT_UWSGI_CONF" "$PLACEMENT_UWSGI"
+ else
+ disable_apache_site placement-api
+ restart_apache_server
+ fi
}
# Restore xtrace
diff --git a/lib/rpc_backend b/lib/rpc_backend
index 3c1404e..3177e88 100644
--- a/lib/rpc_backend
+++ b/lib/rpc_backend
@@ -122,6 +122,15 @@
fi
}
+# Repeat the definition, in case get_transport_url is overriden for RPC purpose.
+# get_notification_url can then be used to talk to rabbit for notifications.
+function get_notification_url {
+ local virtual_host=$1
+ if is_service_enabled rabbit || { [ -n "$RABBIT_HOST" ] && [ -n "$RABBIT_PASSWORD" ]; }; then
+ echo "rabbit://$RABBIT_USERID:$RABBIT_PASSWORD@$RABBIT_HOST:5672/$virtual_host"
+ fi
+}
+
# iniset configuration
function iniset_rpc_backend {
local package=$1
diff --git a/lib/swift b/lib/swift
index 6c2af61..e247f15 100644
--- a/lib/swift
+++ b/lib/swift
@@ -31,7 +31,7 @@
# Defaults
# --------
-if is_ssl_enabled_service "s-proxy" || is_service_enabled tls-proxy; then
+if is_service_enabled tls-proxy; then
SWIFT_SERVICE_PROTOCOL="https"
fi
@@ -128,6 +128,11 @@
SWIFT_REPLICAS=${SWIFT_REPLICAS:-1}
SWIFT_REPLICAS_SEQ=$(seq ${SWIFT_REPLICAS})
+# Set ``SWIFT_START_ALL_SERVICES`` to control whether all Swift
+# services (including the *-auditor, *-replicator, *-reconstructor, etc.
+# daemons) should be started.
+SWIFT_START_ALL_SERVICES=$(trueorfalse True SWIFT_START_ALL_SERVICES)
+
# Set ``SWIFT_LOG_TOKEN_LENGTH`` to configure how many characters of an auth
# token should be placed in the logs. When keystone is used with PKI tokens,
# the token values can be huge, seemingly larger the 2K, at the least. We
@@ -169,6 +174,7 @@
# Test if any Swift services are enabled
# is_swift_enabled
function is_swift_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"swift" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"s-" ]] && return 0
return 1
}
@@ -393,13 +399,6 @@
iniset ${SWIFT_CONFIG_PROXY_SERVER} DEFAULT bind_port ${SWIFT_DEFAULT_BIND_PORT}
fi
- if is_ssl_enabled_service s-proxy; then
- ensure_certificates SWIFT
-
- iniset ${SWIFT_CONFIG_PROXY_SERVER} DEFAULT cert_file "$SWIFT_SSL_CERT"
- iniset ${SWIFT_CONFIG_PROXY_SERVER} DEFAULT key_file "$SWIFT_SSL_KEY"
- fi
-
# DevStack is commonly run in a small slow environment, so bump the timeouts up.
# ``node_timeout`` is the node read operation response time to the proxy server
# ``conn_timeout`` is how long it takes a connect() system call to return
@@ -414,7 +413,7 @@
iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer "set log_level" "WARN"
iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer paste.filter_factory "ceilometermiddleware.swift:filter_factory"
iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer control_exchange "swift"
- iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer url $(get_transport_url)
+ iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer url $(get_notification_url)
iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer driver "messaging"
iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer topic "notifications"
SWIFT_EXTRAS_MIDDLEWARE_LAST="${SWIFT_EXTRAS_MIDDLEWARE_LAST} ceilometer"
@@ -532,11 +531,20 @@
local auth_vers
auth_vers=$(iniget ${testfile} func_test auth_version)
iniset ${testfile} func_test auth_host ${KEYSTONE_SERVICE_HOST}
- iniset ${testfile} func_test auth_port ${KEYSTONE_AUTH_PORT}
- if [[ $auth_vers == "3" ]]; then
- iniset ${testfile} func_test auth_prefix /v3/
+ if [[ "$KEYSTONE_AUTH_PROTOCOL" == "https" ]]; then
+ iniset ${testfile} func_test auth_port 443
else
- iniset ${testfile} func_test auth_prefix /v2.0/
+ iniset ${testfile} func_test auth_port 80
+ fi
+ iniset ${testfile} func_test auth_uri ${KEYSTONE_AUTH_URI}
+ if [[ "$auth_vers" == "3" ]]; then
+ iniset ${testfile} func_test auth_prefix /identity/v3/
+ else
+ iniset ${testfile} func_test auth_prefix /identity/v2.0/
+ fi
+ if is_service_enabled tls-proxy; then
+ iniset ${testfile} func_test cafile ${SSL_BUNDLE_FILE}
+ iniset ${testfile} func_test web_front_end apache2
fi
fi
@@ -551,6 +559,7 @@
if [[ $SYSLOG != "False" ]]; then
sed "s,%SWIFT_LOGDIR%,${swift_log_dir}," $FILES/swift/rsyslog.conf | sudo \
tee /etc/rsyslog.d/10-swift.conf
+ echo "MaxMessageSize 6k" | sudo tee /etc/rsyslog.d/99-maxsize.conf
# restart syslog to take the changes
sudo killall -HUP rsyslogd
fi
@@ -786,8 +795,11 @@
fi
if [ "$SWIFT_USE_MOD_WSGI" == "True" ]; then
+ # Apache should serve the "PACO" a.k.a "main" services
restart_apache_server
+ # The rest of the services should be started in backgroud
swift-init --run-dir=${SWIFT_DATA_DIR}/run rest start
+ # Be we still want the logs of Swift Proxy in our screen session
tail_log s-proxy /var/log/$APACHE_NAME/proxy-server
if [[ ${SWIFT_REPLICAS} == 1 ]]; then
for type in object container account; do
@@ -797,31 +809,42 @@
return 0
fi
- # By default with only one replica we are launching the proxy,
- # container, account and object server in screen in foreground and
- # other services in background. If we have ``SWIFT_REPLICAS`` set to something
- # greater than one we first spawn all the Swift services then kill the proxy
- # service so we can run it in foreground in screen. ``swift-init ...
- # {stop|restart}`` exits with '1' if no servers are running, ignore it just
- # in case
- local todo type
- swift-init --run-dir=${SWIFT_DATA_DIR}/run all restart || true
+
+ # By default with only one replica we are launching the proxy, container
+ # account and object server in screen in foreground. Then, the rest of
+ # the services is optionally started.
+ #
+ # If we have ``SWIFT_REPLICAS`` set to something greater than one
+ # we first spawn *all* the Swift services then kill the proxy service
+ # so we can run it in foreground in screen.
+ #
+ # ``swift-init ... {stop|restart}`` exits with '1' if no servers are
+ # running, ignore it just in case
if [[ ${SWIFT_REPLICAS} == 1 ]]; then
- todo="object container account"
+ local foreground_services type
+
+ foreground_services="object container account"
+ for type in ${foreground_services}; do
+ run_process s-${type} "$SWIFT_BIN_DIR/swift-${type}-server ${SWIFT_CONF_DIR}/${type}-server/1.conf -v"
+ done
+
+ if [[ "$SWIFT_START_ALL_SERVICES" == "True" ]]; then
+ swift-init --run-dir=${SWIFT_DATA_DIR}/run rest start
+ else
+ # The container-sync daemon is strictly needed to pass the container
+ # sync Tempest tests.
+ swift-init --run-dir=${SWIFT_DATA_DIR}/run container-sync start
+ fi
+ else
+ swift-init --run-dir=${SWIFT_DATA_DIR}/run all restart || true
+ swift-init --run-dir=${SWIFT_DATA_DIR}/run proxy stop || true
fi
- for type in proxy ${todo}; do
- swift-init --run-dir=${SWIFT_DATA_DIR}/run ${type} stop || true
- done
+
if is_service_enabled tls-proxy; then
local proxy_port=${SWIFT_DEFAULT_BIND_PORT}
start_tls_proxy swift '*' $proxy_port $SERVICE_HOST $SWIFT_DEFAULT_BIND_PORT_INT
fi
run_process s-proxy "$SWIFT_BIN_DIR/swift-proxy-server ${SWIFT_CONF_DIR}/proxy-server.conf -v"
- if [[ ${SWIFT_REPLICAS} == 1 ]]; then
- for type in object container account; do
- run_process s-${type} "$SWIFT_BIN_DIR/swift-${type}-server ${SWIFT_CONF_DIR}/${type}-server/1.conf -v"
- done
- fi
if [[ "$SWIFT_ENABLE_TEMPURLS" == "True" ]]; then
swift_configure_tempurls
diff --git a/lib/tempest b/lib/tempest
index ceba5e9..cc65ec7 100644
--- a/lib/tempest
+++ b/lib/tempest
@@ -11,6 +11,7 @@
# - ``DEST``, ``FILES``
# - ``ADMIN_PASSWORD``
# - ``DEFAULT_IMAGE_NAME``
+# - ``DEFAULT_IMAGE_FILE_NAME``
# - ``S3_SERVICE_PORT``
# - ``SERVICE_HOST``
# - ``BASE_SQL_CONN`` ``lib/database`` declares
@@ -278,8 +279,12 @@
iniset $TEMPEST_CONFIG identity-feature-enabled api_v2 False
fi
iniset $TEMPEST_CONFIG identity auth_version ${TEMPEST_AUTH_VERSION:-v3}
+ if [[ "$TEMPEST_AUTH_VERSION" != "v2.0" ]]; then
+ # we're going to disable v2 admin unless we're using v2.0 by default.
+ iniset $TEMPEST_CONFIG identity-feature-enabled api_v2_admin False
+ fi
- if is_ssl_enabled_service "key" || is_service_enabled tls-proxy; then
+ if is_service_enabled tls-proxy; then
iniset $TEMPEST_CONFIG identity ca_certificates_file $SSL_BUNDLE_FILE
fi
@@ -356,6 +361,7 @@
iniset $TEMPEST_CONFIG compute-feature-enabled live_migration ${LIVE_MIGRATION_AVAILABLE:-False}
iniset $TEMPEST_CONFIG compute-feature-enabled change_password False
iniset $TEMPEST_CONFIG compute-feature-enabled block_migration_for_live_migration ${USE_BLOCK_MIGRATION_FOR_LIVE_MIGRATION:-False}
+ iniset $TEMPEST_CONFIG compute-feature-enabled live_migrate_back_and_forth ${LIVE_MIGRATE_BACK_AND_FORTH:-False}
iniset $TEMPEST_CONFIG compute-feature-enabled attach_encrypted_volume ${ATTACH_ENCRYPTED_VOLUME_AVAILABLE:-True}
if is_service_enabled n-cell; then
# Cells doesn't support shelving/unshelving
@@ -388,24 +394,6 @@
iniset $TEMPEST_CONFIG network-feature-enabled ipv6_subnet_attributes "$IPV6_SUBNET_ATTRIBUTES_ENABLED"
iniset $TEMPEST_CONFIG network-feature-enabled port_security $NEUTRON_PORT_SECURITY
- # Orchestration Tests
- if is_service_enabled heat; then
- if [[ ! -z "$HEAT_CFN_IMAGE_URL" ]]; then
- iniset $TEMPEST_CONFIG orchestration image_ref $(basename "${HEAT_CFN_IMAGE_URL%.*}")
- fi
- # Nova might not be enabled, especially when we want to test tempest scenario/API that only create Neutron resources
- if is_service_enabled nova; then
- # build a specialized heat flavor
- available_flavors=$(nova flavor-list)
- if [[ ! ( $available_flavors =~ 'm1.heat' ) ]]; then
- openstack flavor create --id 451 --ram 512 --disk 0 --vcpus 1 m1.heat
- fi
- iniset $TEMPEST_CONFIG orchestration instance_type "m1.heat"
- fi
- iniset $TEMPEST_CONFIG orchestration build_timeout 900
- iniset $TEMPEST_CONFIG orchestration stack_owner_role Member
- fi
-
# Scenario
if [ "$VIRT_DRIVER" = "xenserver" ]; then
SCENARIO_IMAGE_DIR=${SCENARIO_IMAGE_DIR:-$FILES}
@@ -414,7 +402,7 @@
iniset $TEMPEST_CONFIG scenario img_container_format ovf
else
SCENARIO_IMAGE_DIR=${SCENARIO_IMAGE_DIR:-$FILES}
- SCENARIO_IMAGE_FILE=$DEFAULT_IMAGE_NAME
+ SCENARIO_IMAGE_FILE=$DEFAULT_IMAGE_FILE_NAME
fi
iniset $TEMPEST_CONFIG scenario img_dir $SCENARIO_IMAGE_DIR
iniset $TEMPEST_CONFIG scenario img_file $SCENARIO_IMAGE_FILE
@@ -425,7 +413,7 @@
TEMPEST_SSH_NETWORK_NAME=$PHYSICAL_NETWORK
fi
# Validation
- iniset $TEMPEST_CONFIG validation run_validation ${TEMPEST_RUN_VALIDATION:-False}
+ iniset $TEMPEST_CONFIG validation run_validation ${TEMPEST_RUN_VALIDATION:-True}
iniset $TEMPEST_CONFIG validation ip_version_for_ssh 4
iniset $TEMPEST_CONFIG validation ssh_timeout $BUILD_TIMEOUT
iniset $TEMPEST_CONFIG validation image_ssh_user ${DEFAULT_INSTANCE_USER:-cirros}
@@ -622,7 +610,7 @@
git_clone $TEMPEST_REPO $TEMPEST_DIR $TEMPEST_BRANCH
pip_install tox
pushd $TEMPEST_DIR
- tox --notest -efull
+ tox -r --notest -efull
# NOTE(mtreinish) Respect constraints in the tempest full venv, things that
# are using a tox job other than full will not be respecting constraints but
# running pip install -U on tempest requirements
diff --git a/lib/template b/lib/template
index b92fb40..25d653c 100644
--- a/lib/template
+++ b/lib/template
@@ -41,6 +41,7 @@
# Test if any XXXX services are enabled
# is_XXXX_enabled
function is_XXXX_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"XXXX" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"XX-" ]] && return 0
return 1
}
diff --git a/lib/tls b/lib/tls
index fb2fa3a..6d67c90 100644
--- a/lib/tls
+++ b/lib/tls
@@ -113,11 +113,11 @@
certificate = \$dir/cacert.pem
private_key = \$dir/private/cacert.key
RANDFILE = \$dir/private/.rand
-default_md = default
+default_md = sha256
[ req ]
-default_bits = 1024
-default_md = sha1
+default_bits = 2048
+default_md = sha256
prompt = no
distinguished_name = ca_distinguished_name
@@ -212,6 +212,9 @@
if is_fedora; then
sudo cp $INT_CA_DIR/ca-chain.pem /usr/share/pki/ca-trust-source/anchors/devstack-chain.pem
sudo update-ca-trust
+ elif is_suse; then
+ sudo cp $INT_CA_DIR/ca-chain.pem /usr/share/pki/trust/anchors/devstack-chain.pem
+ sudo update-ca-certificates
elif is_ubuntu; then
sudo cp $INT_CA_DIR/ca-chain.pem /usr/local/share/ca-certificates/devstack-int.crt
sudo cp $ROOT_CA_DIR/cacert.pem /usr/local/share/ca-certificates/devstack-root.crt
@@ -343,9 +346,10 @@
# one. If the value for the CA is not rooted in /etc then we know
# we need to change it.
function fix_system_ca_bundle_path {
- if is_service_enabled tls-proxy || [ "$USE_SSL" == "True" ]; then
+ if is_service_enabled tls-proxy; then
local capath
- capath=$(python -c $'try:\n from requests import certs\n print certs.where()\nexcept ImportError: pass')
+ local python_cmd=${1:-python}
+ capath=$($python_cmd -c $'try:\n from requests import certs\n print (certs.where())\nexcept ImportError: pass')
if [[ ! $capath == "" && ! $capath =~ ^/etc/.* && ! -L $capath ]]; then
if is_fedora; then
@@ -354,6 +358,9 @@
elif is_ubuntu; then
sudo rm -f $capath
sudo ln -s /etc/ssl/certs/ca-certificates.crt $capath
+ elif is_suse; then
+ sudo rm -f $capath
+ sudo ln -s /etc/ssl/ca-bundle.pem $capath
else
echo "Don't know how to set the CA bundle, expect the install to fail."
fi
@@ -362,27 +369,14 @@
}
+# Only for compatibility, return if the tls-proxy is enabled
+function is_ssl_enabled_service {
+ return is_service_enabled tls-proxy
+}
+
# Certificate Input Configuration
# ===============================
-# check to see if the service(s) specified are to be SSL enabled.
-#
-# Multiple services specified as arguments are ``OR``'ed together; the test
-# is a short-circuit boolean, i.e it returns on the first match.
-#
-# Uses global ``SSL_ENABLED_SERVICES``
-function is_ssl_enabled_service {
- local services=$@
- local service=""
- if [ "$USE_SSL" == "False" ]; then
- return 1
- fi
- for service in ${services}; do
- [[ ,${SSL_ENABLED_SERVICES}, =~ ,${service}, ]] && return 0
- done
- return 1
-}
-
# Ensure that the certificates for a service are in place. This function does
# not check that a service is SSL enabled, this should already have been
# completed.
@@ -429,6 +423,9 @@
if is_ubuntu; then
sudo a2enmod ssl
+ elif is_suse; then
+ sudo a2enmod ssl
+ sudo a2enflag SSL
elif is_fedora; then
# Fedora enables mod_ssl by default
:
@@ -525,7 +522,7 @@
KeepAlive Off
<Location />
- ProxyPass http://$b_host:$b_port/ retry=5 nocanon
+ ProxyPass http://$b_host:$b_port/ retry=0 nocanon
ProxyPassReverse http://$b_host:$b_port/
</Location>
ErrorLog $APACHE_LOG_DIR/tls-proxy_error.log
@@ -535,16 +532,14 @@
LogFormat "%v %h %l %u %t \"%r\" %>s %b"
</VirtualHost>
EOF
+ if is_suse ; then
+ sudo a2enflag SSL
+ fi
for mod in ssl proxy proxy_http; do
enable_apache_mod $mod
done
enable_apache_site $b_service
- # Only a reload is required to pull in new vhosts
- # Note that a restart reliably fails on centos7 and trusty
- # because apache can't open port 80 because the old apache
- # still has it open. Using reload fixes trusty but centos7
- # still doesn't work.
- reload_apache_server
+ restart_apache_server
}
# Follow TLS proxy
diff --git a/openrc b/openrc
index 483b5af..23c173c 100644
--- a/openrc
+++ b/openrc
@@ -72,19 +72,23 @@
GLANCE_HOST=${GLANCE_HOST:-$HOST_IP}
fi
-SERVICE_PROTOCOL=${SERVICE_PROTOCOL:-http}
-KEYSTONE_AUTH_PROTOCOL=${KEYSTONE_AUTH_PROTOCOL:-$SERVICE_PROTOCOL}
-KEYSTONE_AUTH_HOST=${KEYSTONE_AUTH_HOST:-$SERVICE_HOST}
-
# Identity API version
export OS_IDENTITY_API_VERSION=${IDENTITY_API_VERSION:-3}
+# Ask keystoneauth1 to use keystone
+export OS_AUTH_TYPE=password
+
# Authenticating against an OpenStack cloud using Keystone returns a **Token**
# and **Service Catalog**. The catalog contains the endpoints for all services
# the user/project has access to - including nova, glance, keystone, swift, ...
# We currently recommend using the version 3 *identity api*.
#
-export OS_AUTH_URL=$KEYSTONE_AUTH_PROTOCOL://$KEYSTONE_AUTH_HOST:5000/v${OS_IDENTITY_API_VERSION}
+
+# If you don't have a working .stackenv, this is the backup possition
+KEYSTONE_BACKUP=$SERVICE_PROTOCOL://$SERVICE_HOST:5000
+KEYSTONE_AUTH_URI=${KEYSTONE_AUTH_URI:-$KEYSTONE_BACKUP}
+
+export OS_AUTH_URL=${OS_AUTH_URL:-$KEYSTONE_AUTH_URI}
# Currently, in order to use openstackclient with Identity API v3,
# we need to set the domain which the user and project belong to.
diff --git a/setup.cfg b/setup.cfg
index e4b2888..73d22b5 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -2,7 +2,7 @@
name = DevStack
summary = OpenStack DevStack
description-file =
- README.md
+ README.rst
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = http://docs.openstack.org/developer/devstack
@@ -15,6 +15,7 @@
all_files = 1
build-dir = doc/build
source-dir = doc/source
+warning-is-error = 1
[pbr]
warnerrors = True
diff --git a/stack.sh b/stack.sh
index 89b61fc..cfdc2c4c 100755
--- a/stack.sh
+++ b/stack.sh
@@ -2,7 +2,7 @@
# ``stack.sh`` is an opinionated OpenStack developer installation. It
# installs and configures various combinations of **Cinder**, **Glance**,
-# **Heat**, **Horizon**, **Keystone**, **Nova**, **Neutron**, and **Swift**
+# **Horizon**, **Keystone**, **Nova**, **Neutron**, and **Swift**
# This script's options can be changed by setting appropriate environment
# variables. You can configure things like which git repositories to use,
@@ -27,11 +27,37 @@
# Make sure custom grep options don't get in the way
unset GREP_OPTIONS
-# Sanitize language settings to avoid commands bailing out
-# with "unsupported locale setting" errors.
+# NOTE(sdague): why do we explicitly set locale when running stack.sh?
+#
+# Devstack is written in bash, and many functions used throughout
+# devstack process text comming off a command (like the ip command)
+# and do transforms using grep, sed, cut, awk on the strings that are
+# returned. Many of these programs are interationalized, which is
+# great for end users, but means that the strings that devstack
+# functions depend upon might not be there in other locales. We thus
+# need to pin the world to an english basis during the runs.
+#
+# Previously we used the C locale for this, every system has it, and
+# it gives us a stable sort order. It does however mean that we
+# effectively drop unicode support.... boo! :(
+#
+# With python3 being more unicode aware by default, that's not the
+# right option. While there is a C.utf8 locale, some distros are
+# shipping it as C.UTF8 for extra confusingness. And it's support
+# isn't super clear across distros. This is made more challenging when
+# trying to support both out of the box distros, and the gate which
+# uses diskimage builder to build disk images in a different way than
+# the distros do.
+#
+# So... en_US.utf8 it is. That's existed for a very long time. It is a
+# compromise position, but it is the least worse idea at the time of
+# this comment.
+#
+# We also have to unset other variables that might impact LC_ALL
+# taking effect.
unset LANG
unset LANGUAGE
-LC_ALL=C
+LC_ALL=en_US.utf8
export LC_ALL
# Make sure umask is sane
@@ -161,16 +187,16 @@
extract_localrc_section $TOP_DIR/local.conf $TOP_DIR/localrc $TOP_DIR/.localrc.auto
# ``stack.sh`` is customizable by setting environment variables. Override a
-# default setting via export::
+# default setting via export:
#
# export DATABASE_PASSWORD=anothersecret
# ./stack.sh
#
-# or by setting the variable on the command line::
+# or by setting the variable on the command line:
#
# DATABASE_PASSWORD=simple ./stack.sh
#
-# Persistent variables can be placed in a ``local.conf`` file::
+# Persistent variables can be placed in a ``local.conf`` file:
#
# [[local|localrc]]
# DATABASE_PASSWORD=anothersecret
@@ -192,7 +218,7 @@
# Warn users who aren't on an explicitly supported distro, but allow them to
# override check and attempt installation with ``FORCE=yes ./stack``
-if [[ ! ${DISTRO} =~ (xenial|yakkety|zesty|stretch|jessie|f24|f25|rhel7|kvmibm1) ]]; then
+if [[ ! ${DISTRO} =~ (xenial|yakkety|zesty|stretch|jessie|f24|f25|opensuse-42.2|rhel7|kvmibm1) ]]; then
echo "WARNING: this script has not been tested on $DISTRO"
if [[ "$FORCE" != "yes" ]]; then
die $LINENO "If you wish to run this script anyway run with FORCE=yes"
@@ -328,6 +354,7 @@
DATA_DIR=${DATA_DIR:-${DEST}/data}
sudo mkdir -p $DATA_DIR
safe_chown -R $STACK_USER $DATA_DIR
+safe_chmod 0755 $DATA_DIR
# Configure proper hostname
# Certain services such as rabbitmq require that the local hostname resolves
@@ -347,6 +374,10 @@
# is pre-installed.
if [[ -f /etc/nodepool/provider ]]; then
SKIP_EPEL_INSTALL=True
+ if is_fedora; then
+ # However, EPEL is not enabled by default.
+ sudo yum-config-manager --enable epel
+ fi
fi
if is_fedora && [[ $DISTRO == "rhel7" ]] && \
@@ -543,13 +574,6 @@
source $TOP_DIR/lib/database
source $TOP_DIR/lib/rpc_backend
-# Service to enable with SSL if ``USE_SSL`` is True
-SSL_ENABLED_SERVICES="key,nova,cinder,glance,s-proxy,neutron"
-
-if is_service_enabled tls-proxy && [ "$USE_SSL" == "True" ]; then
- die $LINENO "tls-proxy and SSL are mutually exclusive"
-fi
-
# Configure Projects
# ==================
@@ -580,7 +604,7 @@
source $TOP_DIR/lib/neutron
source $TOP_DIR/lib/ldap
source $TOP_DIR/lib/dstat
-source $TOP_DIR/lib/dlm
+source $TOP_DIR/lib/etcd3
source $TOP_DIR/lib/os_brick
# Extras Source
@@ -753,6 +777,13 @@
# Do the ugly hacks for broken packages and distros
source $TOP_DIR/tools/fixup_stuff.sh
+if [[ "$USE_SYSTEMD" == "True" ]]; then
+ pip_install_gr systemd-python
+ # the default rate limit of 1000 messages / 30 seconds is not
+ # sufficient given how verbose our logging is.
+ iniset -sudo /etc/systemd/journald.conf "Journal" "RateLimitBurst" "0"
+ sudo systemctl restart systemd-journald
+fi
# Virtual Environment
# -------------------
@@ -765,13 +796,12 @@
# Phase: pre-install
run_phase stack pre-install
+# NOTE(danms): Set global limits before installing anything
+set_systemd_override DefaultLimitNOFILE ${ULIMIT_NOFILE}
+
install_rpc_backend
restart_rpc_backend
-# NOTE(sdague): dlm install is conditional on one being enabled by configuration
-install_dlm
-configure_dlm
-
if is_service_enabled $DATABASE_BACKENDS; then
install_database
fi
@@ -791,6 +821,9 @@
# Install Oslo libraries
install_oslo
+# Install uwsgi
+install_apache_uwsgi
+
# Install client libraries
install_keystoneauth
install_keystoneclient
@@ -810,7 +843,7 @@
fi
# Setup TLS certs
-if is_service_enabled tls-proxy || [ "$USE_SSL" == "True" ]; then
+if is_service_enabled tls-proxy; then
configure_CA
init_CA
init_cert
@@ -890,8 +923,11 @@
stack_install_service horizon
fi
-if is_service_enabled tls-proxy || [ "$USE_SSL" == "True" ]; then
+if is_service_enabled tls-proxy; then
fix_system_ca_bundle_path
+ if python3_enabled ; then
+ fix_system_ca_bundle_path python3
+ fi
fi
# Extras Install
@@ -1015,6 +1051,22 @@
# Save configuration values
save_stackenv $LINENO
+# Kernel Samepage Merging (KSM)
+# -----------------------------
+
+# Processes that mark their memory as mergeable can share identical memory
+# pages if KSM is enabled. This is particularly useful for nova + libvirt
+# backends but any other setup that marks its memory as mergeable can take
+# advantage. The drawback is there is higher cpu load; however, we tend to
+# be memory bound not cpu bound so enable KSM by default but allow people
+# to opt out if the CPU time is more important to them.
+
+if [[ "ENABLE_KSM" == "True" ]] ; then
+ if [[ -f /sys/kernel/mm/ksm/run ]] ; then
+ sudo sh -c "echo 1 > /sys/kernel/mm/ksm/run"
+ fi
+fi
+
# Start Services
# ==============
@@ -1025,6 +1077,13 @@
# A better kind of sysstat, with the top process per time slice
start_dstat
+# Etcd
+# -----
+
+# etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines
+if is_service_enabled etcd3; then
+ start_etcd3
+fi
# Keystone
# --------
@@ -1238,8 +1297,13 @@
fi
# Create a randomized default value for the key manager's fixed_key
+# NOTE(lyarwood): This is currently set to 36 as a workaround to the following
+# libvirt bug that incorrectly pads passphrases that are a multiple of 16 bytes
+# in length.
+# Unable to use LUKS passphrase that is exactly 16 bytes long
+# https://bugzilla.redhat.com/show_bug.cgi?id=1447297
if is_service_enabled nova; then
- iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 32)
+ iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 36)
fi
# Launch the nova-api and wait for it to answer before continuing
@@ -1253,6 +1317,7 @@
start_neutron_api
elif is_service_enabled q-svc; then
echo_summary "Starting Neutron"
+ configure_neutron_after_post_config
start_neutron_service_and_check
elif is_service_enabled $DATABASE_BACKENDS && is_service_enabled n-net; then
NM_CONF=${NOVA_CONF}
@@ -1270,6 +1335,13 @@
$NOVA_BIN_DIR/nova-manage --config-file $NM_CONF floating create --ip_range=$TEST_FLOATING_RANGE --pool=$TEST_FLOATING_POOL
fi
+# Start placement before any of the service that are likely to want
+# to use it to manage resource providers.
+if is_service_enabled placement; then
+ echo_summary "Starting Placement"
+ start_placement
+fi
+
if is_service_enabled neutron; then
start_neutron
fi
@@ -1284,10 +1356,6 @@
start_nova
create_flavors
fi
-if is_service_enabled placement; then
- echo_summary "Starting Placement"
- start_placement
-fi
if is_service_enabled cinder; then
echo_summary "Starting Cinder"
start_cinder
@@ -1316,10 +1384,6 @@
USERRC_PARAMS="$USERRC_PARAMS --os-cacert $SSL_BUNDLE_FILE"
fi
- if [[ "$HEAT_STANDALONE" = "True" ]]; then
- USERRC_PARAMS="$USERRC_PARAMS --heat-url http://$HEAT_API_HOST:$HEAT_API_PORT/v1"
- fi
-
$TOP_DIR/tools/create_userrc.sh $USERRC_PARAMS
fi
@@ -1462,6 +1526,13 @@
echo_summary "WARNING: $DEPRECATED_TEXT"
fi
+# If USE_SYSTEMD is enabled, tell the user about using it.
+if [[ "$USE_SYSTEMD" == "True" ]]; then
+ echo "Services are running under systemd unit files."
+ echo "For more information see: "
+ echo "https://docs.openstack.org/developer/devstack/systemd.html"
+fi
+
# Indicate how long this took to run (bash maintained variable ``SECONDS``)
echo_summary "stack.sh completed in $SECONDS seconds."
diff --git a/stackrc b/stackrc
index 61501b5..cfe2496 100644
--- a/stackrc
+++ b/stackrc
@@ -65,7 +65,7 @@
# Dashboard
ENABLED_SERVICES+=,horizon
# Additional services
- ENABLED_SERVICES+=,rabbit,tempest,mysql,dstat
+ ENABLED_SERVICES+=,rabbit,tempest,mysql,etcd3,dstat
fi
# Global toggle for enabling services under mod_wsgi. If this is set to
@@ -80,12 +80,19 @@
# Set the root URL for Horizon
HORIZON_APACHE_ROOT="/dashboard"
+# TODO(sdague): Queens
+#
+# All the non systemd paths should be removed in queens, they only
+# exist in Pike to support testing from grenade. Ensure that all this
+# is cleaned up and purged, which should dramatically simplify the
+# devstack codebase.
+
# Whether to use 'dev mode' for screen windows. Dev mode works by
# stuffing text into the screen windows so that a developer can use
# ctrl-c, up-arrow, enter to restart the service. Starting services
# this way is slightly unreliable, and a bit slower, so this can
# be disabled for automated testing by setting this value to False.
-USE_SCREEN=$(trueorfalse True USE_SCREEN)
+USE_SCREEN=$(trueorfalse False USE_SCREEN)
# Whether to use SYSTEMD to manage services
USE_SYSTEMD=$(trueorfalse False USE_SYSTEMD)
@@ -93,16 +100,19 @@
if [[ "$USER_UNITS" == "True" ]]; then
SYSTEMD_DIR="$HOME/.local/share/systemd/user"
SYSTEMCTL="systemctl --user"
- JOURNALCTL_F="journalctl -f -o short-precise --user-unit"
else
SYSTEMD_DIR="/etc/systemd/system"
SYSTEMCTL="sudo systemctl"
- JOURNALCTL_F="journalctl -f -o short-precise --unit"
fi
-if [[ "$USE_SYSTEMD" == "True" ]]; then
- USE_SCREEN=False
-fi
+
+# Whether or not to enable Kernel Samepage Merging (KSM) if available.
+# This allows programs that mark their memory as mergeable to share
+# memory pages if they are identical. This is particularly useful with
+# libvirt backends. This reduces memory useage at the cost of CPU overhead
+# to scan memory. We default to enabling it because we tend to be more
+# memory constrained than CPU bound.
+ENABLE_KSM=$(trueorfalse True ENABLE_KSM)
# When using screen, should we keep a log file on disk? You might
# want this False if you have a long-running setup where verbose logs
@@ -149,6 +159,19 @@
source $RC_DIR/.localrc.auto
fi
+# TODO(sdague): Delete all this in Queens.
+if [[ "$USE_SYSTEMD" == "True" ]]; then
+ USE_SCREEN=False
+fi
+# if we are forcing off USE_SCREEN (as we do in the gate), force on
+# systemd. This allows us to drop one of 3 paths through the code.
+if [[ "$USE_SCREEN" == "False" ]]; then
+ # Remove in Pike: this gets us through grenade upgrade
+ if [[ "$GRENADE_PHASE" != "target" ]]; then
+ USE_SYSTEMD="True"
+ fi
+fi
+
# Default for log coloring is based on interactive-or-not.
# Baseline assumption is that non-interactive invocations are for CI,
# where logs are to be presented as browsable text files; hence color
@@ -217,6 +240,12 @@
# Zero disables timeouts
GIT_TIMEOUT=${GIT_TIMEOUT:-0}
+# How should we be handling WSGI deployments. By default we're going
+# to allow for 2 modes, which is "uwsgi" which runs with an apache
+# proxy uwsgi in front of it, or "mod_wsgi", which runs in
+# apache. mod_wsgi is deprecated, don't use it.
+WSGI_MODE=${WSGI_MODE:-"uwsgi"}
+
# Repositories
# ------------
@@ -356,6 +385,10 @@
#
###################
+# castellan key manager interface
+GITREPO["castellan"]=${CASTELLAN_REPO:-${GIT_BASE}/openstack/castellan.git}
+GITBRANCH["castellan"]=${CASTELLAN_BRANCH:-master}
+
# cliff command line framework
GITREPO["cliff"]=${CLIFF_REPO:-${GIT_BASE}/openstack/cliff.git}
GITBRANCH["cliff"]=${CLIFF_BRANCH:-master}
@@ -475,18 +508,14 @@
#
##################
+# cursive library
+GITREPO["cursive"]=${CURSIVE_REPO:-${GIT_BASE}/openstack/cursive.git}
+GITBRANCH["cursive"]=${CURSIVE_BRANCH:-master}
+
# glance store library
GITREPO["glance_store"]=${GLANCE_STORE_REPO:-${GIT_BASE}/openstack/glance_store.git}
GITBRANCH["glance_store"]=${GLANCE_STORE_BRANCH:-master}
-# heat-cfntools server agent
-HEAT_CFNTOOLS_REPO=${HEAT_CFNTOOLS_REPO:-${GIT_BASE}/openstack/heat-cfntools.git}
-HEAT_CFNTOOLS_BRANCH=${HEAT_CFNTOOLS_BRANCH:-master}
-
-# heat example templates and elements
-HEAT_TEMPLATES_REPO=${HEAT_TEMPLATES_REPO:-${GIT_BASE}/openstack/heat-templates.git}
-HEAT_TEMPLATES_BRANCH=${HEAT_TEMPLATES_BRANCH:-master}
-
# django openstack_auth library
GITREPO["django_openstack_auth"]=${HORIZONAUTH_REPO:-${GIT_BASE}/openstack/django_openstack_auth.git}
GITBRANCH["django_openstack_auth"]=${HORIZONAUTH_BRANCH:-master}
@@ -521,6 +550,10 @@
GITREPO["osc-lib"]=${OSC_LIB_REPO:-${GIT_BASE}/openstack/osc-lib.git}
GITBRANCH["osc-lib"]=${OSC_LIB_BRANCH:-master}
+# python-openstacksdk OpenStack Python SDK
+GITREPO["python-openstacksdk"]=${OPENSTACKSDK_REPO:-${GIT_BASE}/openstack/python-openstacksdk.git}
+GITBRANCH["python-openstacksdk"]=${OPENSTACKSDK_BRANCH:-master}
+
# ironic common lib
GITREPO["ironic-lib"]=${IRONIC_LIB_REPO:-${GIT_BASE}/openstack/ironic-lib.git}
GITBRANCH["ironic-lib"]=${IRONIC_LIB_BRANCH:-master}
@@ -537,6 +570,10 @@
GITBRANCH["neutron-lib"]=${NEUTRON_LIB_BRANCH:-master}
GITDIR["neutron-lib"]=$DEST/neutron-lib
+# os-traits library for resource provider traits in the placement service
+GITREPO["os-traits"]=${OS_TRAITS_REPO:-${GIT_BASE}/openstack/os-traits.git}
+GITBRANCH["os-traits"]=${OS_TRAITS_BRANCH:-master}
+
##################
#
# TripleO / Heat Agent Components
@@ -574,7 +611,7 @@
# a websockets/html5 or flash powered VNC console for vm instances
NOVNC_REPO=${NOVNC_REPO:-https://github.com/kanaka/noVNC.git}
-NOVNC_BRANCH=${NOVNC_BRANCH:-master}
+NOVNC_BRANCH=${NOVNC_BRANCH:-stable/v0.6}
# a websockets/html5 or flash powered SPICE console for vm instances
SPICE_REPO=${SPICE_REPO:-http://anongit.freedesktop.org/git/spice/spice-html5.git}
@@ -591,8 +628,12 @@
case "$VIRT_DRIVER" in
ironic|libvirt)
LIBVIRT_TYPE=${LIBVIRT_TYPE:-kvm}
- if [[ "$os_VENDOR" =~ (Debian) ]]; then
- LIBVIRT_GROUP=libvirt
+ if [[ "$os_VENDOR" =~ (Debian|Ubuntu) ]]; then
+ # The groups change with newer libvirt. Older Ubuntu used
+ # 'libvirtd', but now uses libvirt like Debian. Do a quick check
+ # to see if libvirtd group already exists to handle grenade's case.
+ LIBVIRT_GROUP=$(cut -d ':' -f 1 /etc/group | grep 'libvirtd$' || true)
+ LIBVIRT_GROUP=${LIBVIRT_GROUP:-libvirt}
else
LIBVIRT_GROUP=libvirtd
fi
@@ -618,6 +659,8 @@
;;
esac
+# By default, devstack will use Ubuntu Cloud Archive.
+ENABLE_UBUNTU_CLOUD_ARCHIVE=$(trueorfalse True ENABLE_UBUNTU_CLOUD_ARCHIVE)
# Images
# ------
@@ -657,17 +700,21 @@
case "$LIBVIRT_TYPE" in
lxc) # the cirros root disk in the uec tarball is empty, so it will not work for lxc
DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs}
- IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img.gz";;
+ DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_FILE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img.gz}
+ IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/${DEFAULT_IMAGE_FILE_NAME}";;
*) # otherwise, use the qcow image
- DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img}
- IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img";;
+ DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk}
+ DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_FILE_NAME:-cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img}
+ IMAGE_URLS+="http://download.cirros-cloud.net/${CIRROS_VERSION}/${DEFAULT_IMAGE_FILE_NAME}";;
esac
;;
vsphere)
DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-0.3.2-i386-disk.vmdk}
- IMAGE_URLS+="http://partnerweb.vmware.com/programs/vmdkimage/cirros-0.3.2-i386-disk.vmdk";;
+ DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_FILE_NAME:-$DEFAULT_IMAGE_NAME}
+ IMAGE_URLS+="http://partnerweb.vmware.com/programs/vmdkimage/${DEFAULT_IMAGE_FILE_NAME}";;
xenserver)
DEFAULT_IMAGE_NAME=${DEFAULT_IMAGE_NAME:-cirros-0.3.4-x86_64-disk}
+ DEFAULT_IMAGE_FILE_NAME=${DEFAULT_IMAGE_NAME:-cirros-0.3.4-x86_64-disk.vhd.tgz}
IMAGE_URLS+="http://ca.downloads.xensource.com/OpenStack/cirros-0.3.4-x86_64-disk.vhd.tgz"
IMAGE_URLS+=",http://download.cirros-cloud.net/${CIRROS_VERSION}/cirros-${CIRROS_VERSION}-x86_64-uec.tar.gz";;
esac
@@ -830,24 +877,10 @@
# Set to 0 to disable shallow cloning
GIT_DEPTH=${GIT_DEPTH:-0}
-# Use native SSL for servers in ``SSL_ENABLED_SERVICES``
-USE_SSL=$(trueorfalse False USE_SSL)
-
# We may not need to recreate database in case 2 Keystone services
# sharing the same database. It would be useful for multinode Grenade tests.
RECREATE_KEYSTONE_DB=$(trueorfalse True RECREATE_KEYSTONE_DB)
-# ebtables is inherently racey. If you run it by two or more processes
-# simultaneously it will collide, badly, in the kernel and produce
-# failures or corruption of ebtables. The only way around it is for
-# all tools running ebtables to only ever do so with the --concurrent
-# flag. This requires libvirt >= 1.2.11.
-#
-# If you don't have this then the following work around will replace
-# ebtables with a wrapper script so that it is safe to run without
-# that flag.
-EBTABLES_RACE_FIX=$(trueorfalse False EBTABLES_RACE_FIX)
-
# Following entries need to be last items in file
# Compatibility bits required by other callers like Grenade
@@ -896,6 +929,9 @@
# ``LOGDIR`` is always set at this point so it is not useful as a 'enable' for service logs
# ``SCREEN_LOGDIR`` may be set, it is useful to enable the compat symlinks
+# System-wide ulimit file descriptors override
+ULIMIT_NOFILE=${ULIMIT_NOFILE:-2048}
+
# Local variables:
# mode: shell-script
# End:
diff --git a/tests/test_functions.sh b/tests/test_functions.sh
index 8aae23d..adf20cd 100755
--- a/tests/test_functions.sh
+++ b/tests/test_functions.sh
@@ -224,7 +224,7 @@
# test against removed package...was a bug on Ubuntu
if is_ubuntu; then
- PKG=cowsay
+ PKG=cowsay-off
if ! (dpkg -s $PKG >/dev/null 2>&1); then
# it was never installed...set up the condition
sudo apt-get install -y cowsay >/dev/null 2>&1
diff --git a/tests/test_libs_from_pypi.sh b/tests/test_libs_from_pypi.sh
index 415fec5..1f2d3c2 100755
--- a/tests/test_libs_from_pypi.sh
+++ b/tests/test_libs_from_pypi.sh
@@ -37,12 +37,13 @@
ALL_LIBS+=" oslo.versionedobjects oslo.vmware keystonemiddleware"
ALL_LIBS+=" oslo.serialization django_openstack_auth"
ALL_LIBS+=" python-openstackclient osc-lib os-client-config oslo.rootwrap"
-ALL_LIBS+=" oslo.i18n oslo.utils python-swiftclient"
+ALL_LIBS+=" oslo.i18n oslo.utils python-openstacksdk python-swiftclient"
ALL_LIBS+=" python-neutronclient tooz ceilometermiddleware oslo.policy"
-ALL_LIBS+=" debtcollector os-brick automaton futurist oslo.service"
-ALL_LIBS+=" oslo.cache oslo.reports osprofiler"
+ALL_LIBS+=" debtcollector os-brick os-traits automaton futurist oslo.service"
+ALL_LIBS+=" oslo.cache oslo.reports osprofiler cursive"
ALL_LIBS+=" keystoneauth ironic-lib neutron-lib oslo.privsep"
ALL_LIBS+=" diskimage-builder os-vif python-brick-cinderclient-ext"
+ALL_LIBS+=" castellan"
# Generate the above list with
# echo ${!GITREPO[@]}
diff --git a/tests/test_refs.sh b/tests/test_refs.sh
index bccca5d..65848cd 100755
--- a/tests/test_refs.sh
+++ b/tests/test_refs.sh
@@ -15,7 +15,7 @@
echo "Ensuring we don't have crazy refs"
-REFS=`grep BRANCH stackrc | grep -v -- '-master'`
+REFS=`grep BRANCH stackrc | grep -v -- '-master' | grep -v 'NOVNC_BRANCH'`
rc=$?
if [[ $rc -eq 0 ]]; then
echo "Branch defaults must be master. Found:"
diff --git a/tools/dstat.sh b/tools/dstat.sh
index 1c80fb7..01c6d9b 100755
--- a/tools/dstat.sh
+++ b/tools/dstat.sh
@@ -9,14 +9,14 @@
# Assumes:
# - dstat command is installed
-# Retreive log directory as argument from calling script.
+# Retrieve log directory as argument from calling script.
LOGDIR=$1
# Command line arguments for primary DStat process.
-DSTAT_OPTS="-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap"
+DSTAT_OPTS="-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap --tcp"
# Command-line arguments for secondary background DStat process.
-DSTAT_CSV_OPTS="-tcmndrylpg --output $LOGDIR/dstat-csv.log"
+DSTAT_CSV_OPTS="-tcmndrylpg --tcp --output $LOGDIR/dstat-csv.log"
# Execute and background the secondary dstat process and discard its output.
dstat $DSTAT_CSV_OPTS >& /dev/null &
diff --git a/tools/fixup_stuff.sh b/tools/fixup_stuff.sh
index 4dec95e..0b78bde 100755
--- a/tools/fixup_stuff.sh
+++ b/tools/fixup_stuff.sh
@@ -67,6 +67,40 @@
echo_summary "WARNING: unable to reserve keystone ports"
fi
+# Ubuntu Cloud Archive
+#---------------------
+# We've found that Libvirt on Xenial is flaky and crashes enough to be
+# a regular top e-r bug. Opt into Ubuntu Cloud Archive if on Xenial to
+# get newer Libvirt.
+# Make it possible to switch this based on an environment variable as
+# libvirt 2.5.0 doesn't handle nested virtualization quite well and this
+# is required for the trove development environment.
+if [[ "${ENABLE_UBUNTU_CLOUD_ARCHIVE}" == "True" && "$DISTRO" = "xenial" ]]; then
+ # This pulls in apt-add-repository
+ install_package "software-properties-common"
+ # Use UCA for newer libvirt. Should give us libvirt 2.5.0.
+ if [[ -f /etc/ci/mirror_info.sh ]] ; then
+ # If we are on a nodepool provided host and it has told us about where
+ # we can find local mirrors then use that mirror.
+ source /etc/ci/mirror_info.sh
+
+ sudo apt-add-repository -y "deb $NODEPOOL_UCA_MIRROR xenial-updates/ocata main"
+ else
+ # Otherwise use upstream UCA
+ sudo add-apt-repository -y cloud-archive:ocata
+ fi
+
+ # Disable use of libvirt wheel since a cached wheel build might be
+ # against older libvirt binary. Particularly a problem if using
+ # the openstack wheel mirrors, but can hit locally too.
+ # TODO(clarkb) figure out how to use upstream wheel again.
+ iniset -sudo /etc/pip.conf "global" "no-binary" "libvirt-python"
+
+ # Force update our APT repos, since we added UCA above.
+ REPOS_UPDATED=False
+ apt_get_update
+fi
+
# Python Packages
# ---------------
diff --git a/tools/install_ebtables_workaround.sh b/tools/install_ebtables_workaround.sh
deleted file mode 100755
index 45ced87..0000000
--- a/tools/install_ebtables_workaround.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/bash -eu
-#
-# Copyright 2015 Hewlett-Packard Development Company, L.P.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License. You may obtain
-# a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations
-# under the License.
-#
-#
-# This replaces the ebtables on your system with a wrapper script that
-# does implicit locking. This is needed if libvirt < 1.2.11 on your platform.
-
-EBTABLES=/sbin/ebtables
-EBTABLESREAL=/sbin/ebtables.real
-FILES=$TOP_DIR/files
-
-if [[ -f "$EBTABLES" ]]; then
- if file $EBTABLES | grep ELF; then
- sudo mv $EBTABLES $EBTABLESREAL
- sudo install -m 0755 $FILES/ebtables.workaround $EBTABLES
- echo "Replaced ebtables with locking workaround"
- fi
-fi
diff --git a/tools/memory_tracker.sh b/tools/memory_tracker.sh
index 7397c03..cbdeb8f 100755
--- a/tools/memory_tracker.sh
+++ b/tools/memory_tracker.sh
@@ -14,6 +14,8 @@
set -o errexit
+PYTHON=${PYTHON:-python}
+
# time to sleep between checks
SLEEP_TIME=20
@@ -86,7 +88,7 @@
# list processes that lock memory from swap
if [[ $unevictable -ne $unevictable_point ]]; then
unevictable_point=$unevictable
- ./tools/mlock_report.py
+ ${PYTHON} ./tools/mlock_report.py
fi
echo "]]]"
diff --git a/tools/mlock_report.py b/tools/mlock_report.py
index 1d23af9..2169cc2 100755
--- a/tools/mlock_report.py
+++ b/tools/mlock_report.py
@@ -8,14 +8,15 @@
import psutil
-SUMMARY_REGEX = re.compile(r".*\s+(?P<locked>[\d]+)\s+KB")
+SUMMARY_REGEX = re.compile(b".*\s+(?P<locked>[\d]+)\s+KB")
def main():
try:
- print _get_report()
+ print(_get_report())
except Exception as e:
- print "Failure listing processes locking memory: %s" % str(e)
+ print("Failure listing processes locking memory: %s" % str(e))
+ raise
def _get_report():
diff --git a/tools/worlddump.py b/tools/worlddump.py
index eb109b9..6fff149 100755
--- a/tools/worlddump.py
+++ b/tools/worlddump.py
@@ -223,6 +223,14 @@
print("guru meditation report in %s log" % service)
+def var_core():
+ if os.path.exists('/var/core'):
+ _header("/var/core dumps")
+ # NOTE(ianw) : see DEBUG_LIBVIRT_COREDUMPS. We could think
+ # about getting backtraces out of these. There are other
+ # tools out there that can do that sort of thing though.
+ _dump_cmd("ls -ltrah /var/core")
+
def main():
opts = get_options()
fname = filename(opts.dir, opts.name)
@@ -238,6 +246,7 @@
ebtables_dump()
compute_consoles()
guru_meditation_reports()
+ var_core()
if __name__ == '__main__':
diff --git a/tools/xen/README.md b/tools/xen/README.md
index 7062ecb..9559e77 100644
--- a/tools/xen/README.md
+++ b/tools/xen/README.md
@@ -171,8 +171,3 @@
umount "$mountdir"
rm -rf "$mountdir"
-### Migrate OpenStack DomU to another host
-
-Given you need to migrate your DomU with OpenStack installed to another host,
-you need to set `XEN_INTEGRATION_BRIDGE` in localrc if neutron network is used.
-It is the bridge for `XEN_INT_BRIDGE_OR_NET_NAME` network created in Dom0
diff --git a/tools/xen/install_os_domU.sh b/tools/xen/install_os_domU.sh
index d2e2c57..f4ca71a 100755
--- a/tools/xen/install_os_domU.sh
+++ b/tools/xen/install_os_domU.sh
@@ -66,10 +66,6 @@
setup_network "$MGT_BRIDGE_OR_NET_NAME"
setup_network "$PUB_BRIDGE_OR_NET_NAME"
-# With neutron, one more network is required, which is internal to the
-# hypervisor, and used by the VMs
-setup_network "$XEN_INT_BRIDGE_OR_NET_NAME"
-
if parameter_is_specified "FLAT_NETWORK_BRIDGE"; then
if [ "$(bridge_for "$VM_BRIDGE_OR_NET_NAME")" != "$(bridge_for "$FLAT_NETWORK_BRIDGE")" ]; then
cat >&2 << EOF
@@ -292,16 +288,6 @@
#
$THIS_DIR/build_xva.sh "$GUEST_NAME"
-# Attach a network interface for the integration network (so that the bridge
-# is created by XenServer). This is required for Neutron. Also pass that as a
-# kernel parameter for DomU
-attach_network "$XEN_INT_BRIDGE_OR_NET_NAME"
-
-XEN_INTEGRATION_BRIDGE_DEFAULT=$(bridge_for "$XEN_INT_BRIDGE_OR_NET_NAME")
-append_kernel_cmdline \
- "$GUEST_NAME" \
- "xen_integration_bridge=${XEN_INTEGRATION_BRIDGE_DEFAULT}"
-
FLAT_NETWORK_BRIDGE="${FLAT_NETWORK_BRIDGE:-$(bridge_for "$VM_BRIDGE_OR_NET_NAME")}"
append_kernel_cmdline "$GUEST_NAME" "flat_network_bridge=${FLAT_NETWORK_BRIDGE}"
diff --git a/tools/xen/xenrc b/tools/xen/xenrc
index 60be02f..169e042 100644
--- a/tools/xen/xenrc
+++ b/tools/xen/xenrc
@@ -29,7 +29,6 @@
# Get the management network from the XS installation
VM_BRIDGE_OR_NET_NAME="OpenStack VM Network"
PUB_BRIDGE_OR_NET_NAME="OpenStack Public Network"
-XEN_INT_BRIDGE_OR_NET_NAME="OpenStack VM Integration Network"
# VM Password
GUEST_PASSWORD=${GUEST_PASSWORD:-secret}
diff --git a/tox.ini b/tox.ini
index 55a06d0..cc7c544 100644
--- a/tox.ini
+++ b/tox.ini
@@ -37,8 +37,8 @@
deps =
Pygments
docutils
- sphinx>=1.1.2,<1.2
- pbr>=0.6,!=0.7,<1.0
+ sphinx>=1.5.1,<1.6.1
+ pbr>=2.0.0,!=2.1.0
oslosphinx
nwdiag
blockdiag
@@ -52,8 +52,8 @@
[testenv:venv]
deps =
- pbr>=0.6,!=0.7,<1.0
- sphinx>=1.1.2,<1.2
+ pbr>=2.0.0,!=2.1.0
+ sphinx>=1.5.1,<1.6.1
oslosphinx
blockdiag
sphinxcontrib-blockdiag
diff --git a/unstack.sh b/unstack.sh
index b0ebaf7..77a151f 100755
--- a/unstack.sh
+++ b/unstack.sh
@@ -69,7 +69,7 @@
source $TOP_DIR/lib/neutron
source $TOP_DIR/lib/ldap
source $TOP_DIR/lib/dstat
-source $TOP_DIR/lib/dlm
+source $TOP_DIR/lib/etcd3
# Extras Source
# --------------
@@ -129,9 +129,6 @@
stop_tls_proxy
cleanup_CA
fi
-if [ "$USE_SSL" == "True" ]; then
- cleanup_CA
-fi
SCSI_PERSIST_DIR=$CINDER_STATE_PATH/volumes/*
@@ -165,6 +162,11 @@
cleanup_neutron
fi
+if is_service_enabled etcd3; then
+ stop_etcd3
+ cleanup_etcd3
+fi
+
if is_service_enabled dstat; then
stop_dstat
fi