Merge "update sphinx"
diff --git a/.gitignore b/.gitignore
index d1781bc..7967e14 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,7 @@
files/pip-*
files/get-pip.py*
files/ir-deploy*
+files/ironic-inspector*
local.conf
local.sh
localrc
diff --git a/HACKING.rst b/HACKING.rst
index b76cb6c..fc67f09 100644
--- a/HACKING.rst
+++ b/HACKING.rst
@@ -322,7 +322,7 @@
Review Criteria
-===============
+---------------
There are some broad criteria that will be followed when reviewing
your change
@@ -364,3 +364,26 @@
* **Reviewers** -- please see ``MAINTAINERS.rst`` for a list of people
that should be added to reviews of various sub-systems.
+
+
+Making Changes, Testing, and CI
+-------------------------------
+
+Changes to Devstack are tested by automated continuous integration jobs
+that run on a variety of Linux Distros using a handful of common
+configurations. What this means is that every change to Devstack is
+self testing. One major benefit of this is that developers do not
+typically need to add new non voting test jobs to add features to
+Devstack. Instead the features can be added, then if testing passes
+with the feature enabled the change is ready to merge (pending code
+review).
+
+A concrete example of this was the switch from screen based service
+management to systemd based service management. No new jobs were
+created for this. Instead the features were added to devstack, tested
+locally and in CI using a change that enabled the feature, then once
+the enabling change was passing and the new behavior communicated and
+documented it was merged.
+
+Using this process has been proven to be effective and leads to
+quicker implementation of desired features.
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 318e044..66b8702 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -609,29 +609,6 @@
Only Swift proxy server is launched in the systemd system all other
services are started in background and managed by ``swift-init`` tool.
-Heat
-~~~~
-
-Heat is disabled by default (see ``stackrc`` file). To enable it
-explicitly you'll need the following settings in your ``localrc``
-section
-
-::
-
- enable_service heat h-api h-api-cfn h-api-cw h-eng
-
-Heat can also run in standalone mode, and be configured to orchestrate
-on an external OpenStack cloud. To launch only Heat in standalone mode
-you'll need the following settings in your ``localrc`` section
-
-::
-
- disable_all_services
- enable_service rabbit mysql heat h-api h-api-cfn h-api-cw h-eng
- HEAT_STANDALONE=True
- KEYSTONE_SERVICE_HOST=...
- KEYSTONE_AUTH_HOST=...
-
Tempest
~~~~~~~
diff --git a/doc/source/faq.rst b/doc/source/faq.rst
index cb2f328..a186336 100644
--- a/doc/source/faq.rst
+++ b/doc/source/faq.rst
@@ -159,16 +159,6 @@
often good enough for a single-node installation. And in an extreme
case, use ``clean.sh`` to eradicate it and try again.
-Configure ``local.conf`` thusly:
-
- ::
-
- [[local|localrc]]
- HEAT_STANDALONE=True
- ENABLED_SERVICES=rabbit,mysql,heat,h-api,h-api-cfn,h-api-cw,h-eng
- KEYSTONE_SERVICE_HOST=<keystone-host>
- KEYSTONE_AUTH_HOST=<keystone-host>
-
Why are my configuration changes ignored?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/doc/source/overview.rst b/doc/source/overview.rst
index e648193..c07a8e6 100644
--- a/doc/source/overview.rst
+++ b/doc/source/overview.rst
@@ -64,8 +64,7 @@
The default services configured by DevStack are Identity (keystone),
Object Storage (swift), Image Service (glance), Block Storage
-(cinder), Compute (nova), Networking (neutron), Dashboard (horizon),
-Orchestration (heat)
+(cinder), Compute (nova), Networking (neutron), Dashboard (horizon)
Additional services not included directly in DevStack can be tied in to
``stack.sh`` using the :doc:`plugin mechanism <plugins>` to call
diff --git a/doc/source/systemd.rst b/doc/source/systemd.rst
index efe79e4..60a7719 100644
--- a/doc/source/systemd.rst
+++ b/doc/source/systemd.rst
@@ -109,6 +109,17 @@
journalctl -f -o short-precise --unit devstack@n-cpu.service
+By default, journalctl strips out "unprintable" characters, including
+ASCII color codes. To keep the color codes (which can be interpreted by
+an appropriate terminal/pager - e.g. ``less``, the default)::
+
+ journalctl -a --unit devstack@n-cpu.service
+
+When outputting to the terminal using the default pager, long lines
+appear to be truncated, but horizontal scrolling is supported via the
+left/right arrow keys.
+
+See ``man 1 journalctl`` for more.
Known Issues
============
@@ -137,13 +148,6 @@
Future Work
===========
-log colorizing
---------------
-
-We lose log colorization through this process. We might want to build
-a custom colorizer that we could run journalctl output through
-optionally for people.
-
user units
----------
diff --git a/files/debs/n-api b/files/debs/n-api
deleted file mode 100644
index 0928cd5..0000000
--- a/files/debs/n-api
+++ /dev/null
@@ -1 +0,0 @@
-fping
diff --git a/files/debs/zookeeper b/files/debs/zookeeper
deleted file mode 100644
index f41b559..0000000
--- a/files/debs/zookeeper
+++ /dev/null
@@ -1 +0,0 @@
-zookeeperd
diff --git a/files/rpms-suse/n-api b/files/rpms-suse/n-api
index af5ac2f..0f08daa 100644
--- a/files/rpms-suse/n-api
+++ b/files/rpms-suse/n-api
@@ -1,2 +1 @@
-fping
python-dateutil
diff --git a/files/rpms/n-api b/files/rpms/n-api
deleted file mode 100644
index 0928cd5..0000000
--- a/files/rpms/n-api
+++ /dev/null
@@ -1 +0,0 @@
-fping
diff --git a/files/rpms/zookeeper b/files/rpms/zookeeper
deleted file mode 100644
index 1bfac53..0000000
--- a/files/rpms/zookeeper
+++ /dev/null
@@ -1 +0,0 @@
-zookeeper
diff --git a/files/zookeeper/environment b/files/zookeeper/environment
deleted file mode 100644
index afa2d2f..0000000
--- a/files/zookeeper/environment
+++ /dev/null
@@ -1,36 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Modified from http://packages.ubuntu.com/saucy/zookeeperd
-NAME=zookeeper
-ZOOCFGDIR=/etc/zookeeper/conf
-
-# seems, that log4j requires the log4j.properties file to be in the classpath
-CLASSPATH="$ZOOCFGDIR:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar"
-
-ZOOCFG="$ZOOCFGDIR/zoo.cfg"
-ZOO_LOG_DIR=/var/log/zookeeper
-USER=$NAME
-GROUP=$NAME
-PIDDIR=/var/run/$NAME
-PIDFILE=$PIDDIR/$NAME.pid
-SCRIPTNAME=/etc/init.d/$NAME
-JAVA=/usr/bin/java
-ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
-ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
-JMXLOCALONLY=false
-JAVA_OPTS=""
diff --git a/files/zookeeper/log4j.properties b/files/zookeeper/log4j.properties
deleted file mode 100644
index 6c45a4a..0000000
--- a/files/zookeeper/log4j.properties
+++ /dev/null
@@ -1,69 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# From http://packages.ubuntu.com/saucy/zookeeperd
-
-# ZooKeeper Logging Configuration
-#
-
-# Format is "<default threshold> (, <appender>)+
-
-log4j.rootLogger=${zookeeper.root.logger}
-
-# Example: console appender only
-# log4j.rootLogger=INFO, CONSOLE
-
-# Example with rolling log file
-#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE
-
-# Example with rolling log file and tracing
-#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE
-
-#
-# Log INFO level and above messages to the console
-#
-log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
-log4j.appender.CONSOLE.Threshold=INFO
-log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
-log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
-
-#
-# Add ROLLINGFILE to rootLogger to get log file output
-# Log DEBUG level and above messages to a log file
-log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
-log4j.appender.ROLLINGFILE.Threshold=WARN
-log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log
-
-# Max log file size of 10MB
-log4j.appender.ROLLINGFILE.MaxFileSize=10MB
-# uncomment the next line to limit number of backup files
-#log4j.appender.ROLLINGFILE.MaxBackupIndex=10
-
-log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
-log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
-
-
-#
-# Add TRACEFILE to rootLogger to get log file output
-# Log DEBUG level and above messages to a log file
-log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
-log4j.appender.TRACEFILE.Threshold=TRACE
-log4j.appender.TRACEFILE.File=${zookeeper.log.dir}/zookeeper_trace.log
-
-log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
-### Notice we are including log4j's NDC here (%x)
-log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n
diff --git a/files/zookeeper/myid b/files/zookeeper/myid
deleted file mode 100644
index c227083..0000000
--- a/files/zookeeper/myid
+++ /dev/null
@@ -1 +0,0 @@
-0
\ No newline at end of file
diff --git a/files/zookeeper/zoo.cfg b/files/zookeeper/zoo.cfg
deleted file mode 100644
index b8f5582..0000000
--- a/files/zookeeper/zoo.cfg
+++ /dev/null
@@ -1,74 +0,0 @@
-#
-# (C) Copyright 2015 Hewlett Packard Enterprise Development Company LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
-
-# The number of milliseconds of each tick
-tickTime=2000
-# The number of ticks that the initial
-# synchronization phase can take
-initLimit=10
-# The number of ticks that can pass between
-# sending a request and getting an acknowledgement
-syncLimit=5
-# the directory where the snapshot is stored.
-dataDir=/var/lib/zookeeper
-# Place the dataLogDir to a separate physical disc for better performance
-# dataLogDir=/disk2/zookeeper
-
-# the port at which the clients will connect
-clientPort=2181
-
-# Maximum number of clients that can connect from one client
-maxClientCnxns=60
-
-# specify all zookeeper servers
-# The fist port is used by followers to connect to the leader
-# The second one is used for leader election
-
-server.0=127.0.0.1:2888:3888
-
-# To avoid seeks ZooKeeper allocates space in the transaction log file in
-# blocks of preAllocSize kilobytes. The default block size is 64M. One reason
-# for changing the size of the blocks is to reduce the block size if snapshots
-# are taken more often. (Also, see snapCount).
-#preAllocSize=65536
-
-# Clients can submit requests faster than ZooKeeper can process them,
-# especially if there are a lot of clients. To prevent ZooKeeper from running
-# out of memory due to queued requests, ZooKeeper will throttle clients so that
-# there is no more than globalOutstandingLimit outstanding requests in the
-# system. The default limit is 1,000.ZooKeeper logs transactions to a
-# transaction log. After snapCount transactions are written to a log file a
-# snapshot is started and a new transaction log file is started. The default
-# snapCount is 10,000.
-#snapCount=1000
-
-# If this option is defined, requests will be will logged to a trace file named
-# traceFile.year.month.day.
-#traceFile=
-
-# Leader accepts client connections. Default value is "yes". The leader machine
-# coordinates updates. For higher update throughput at thes slight expense of
-# read throughput the leader can be configured to not accept clients and focus
-# on coordination.
-#leaderServes=yes
-
-# Autopurge every hour to avoid using lots of disk in bursts
-# Order of the next 2 properties matters.
-# autopurge.snapRetainCount must be before autopurge.purgeInterval.
-autopurge.snapRetainCount=3
-autopurge.purgeInterval=1
\ No newline at end of file
diff --git a/functions b/functions
index 8968a18..52a82fa 100644
--- a/functions
+++ b/functions
@@ -310,6 +310,11 @@
disk_format=qcow2
container_format=bare
;;
+ *.raw)
+ image_name=$(basename "$image" ".raw")
+ disk_format=raw
+ container_format=bare
+ ;;
*.iso)
image_name=$(basename "$image" ".iso")
disk_format=iso
@@ -611,21 +616,18 @@
# request-id. However, there may be an eventlet interaction here,
# so going off for now.
USE_JOURNAL=$(trueorfalse USE_JOURNAL False)
+ local pidstr=""
if [[ "$USE_JOURNAL" == "True" ]]; then
iniset $conf_file $conf_section use_journal "True"
# if we are using the journal directly, our process id is already correct
- iniset $conf_file $conf_section logging_debug_format_suffix \
- "{{%(funcName)s %(pathname)s:%(lineno)d}}"
else
- iniset $conf_file $conf_section logging_debug_format_suffix \
- "{{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}"
+ pidstr="(pid=%(process)d) "
fi
+ iniset $conf_file $conf_section logging_debug_format_suffix "[00;33m{{${pidstr}%(funcName)s %(pathname)s:%(lineno)d}}[00m"
- iniset $conf_file $conf_section logging_context_format_string \
- "%(levelname)s %(name)s [%(request_id)s %(project_name)s %(user_name)s] %(instance)s%(message)s"
- iniset $conf_file $conf_section logging_default_format_string \
- "%(levelname)s %(name)s [-] %(instance)s%(color)s%(message)s"
- iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s %(instance)s"
+ iniset $conf_file $conf_section logging_context_format_string "%(color)s%(levelname)s %(name)s [[01;36m%(request_id)s [00;36m%(project_name)s %(user_name)s%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m"
+ iniset $conf_file $conf_section logging_default_format_string "%(color)s%(levelname)s %(name)s [[00;36m-%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m"
+ iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s [01;35m%(instance)s[00m"
}
function setup_standard_logging_identity {
@@ -715,6 +717,22 @@
}
+# Set a systemd system override
+#
+# This sets a system-side override in system.conf. A per-service
+# override would be /etc/systemd/system/${service}.service/override.conf
+function set_systemd_override {
+ local key="$1"
+ local value="$2"
+
+ local sysconf="/etc/systemd/system.conf"
+ iniset -sudo "${sysconf}" "Manager" "$key" "$value"
+ echo "Set systemd system override for ${key}=${value}"
+
+ sudo systemctl daemon-reload
+}
+
+
# Restore xtrace
$_XTRACE_FUNCTIONS
diff --git a/functions-common b/functions-common
index 965f051..13559da 100644
--- a/functions-common
+++ b/functions-common
@@ -48,8 +48,8 @@
# Save these variables to .stackenv
STACK_ENV_VARS="BASE_SQL_CONN DATA_DIR DEST ENABLED_SERVICES HOST_IP \
- KEYSTONE_AUTH_PROTOCOL KEYSTONE_AUTH_URI KEYSTONE_SERVICE_URI \
- LOGFILE OS_CACERT SERVICE_HOST SERVICE_PROTOCOL STACK_USER TLS_IP \
+ KEYSTONE_AUTH_URI KEYSTONE_SERVICE_URI \
+ LOGFILE OS_CACERT SERVICE_HOST STACK_USER TLS_IP \
HOST_IPV6 SERVICE_IP_VERSION"
diff --git a/inc/python b/inc/python
index 2443c4d..4c443d6 100644
--- a/inc/python
+++ b/inc/python
@@ -351,6 +351,20 @@
return $result
}
+function pip_uninstall {
+ local name=$1
+ if [[ -n ${PIP_VIRTUAL_ENV:=} && -d ${PIP_VIRTUAL_ENV} ]]; then
+ local cmd_pip=$PIP_VIRTUAL_ENV/bin/pip
+ local sudo_pip="env"
+ else
+ local cmd_pip
+ cmd_pip=$(get_pip_command $PYTHON2_VERSION)
+ local sudo_pip="sudo -H"
+ fi
+ # don't error if we can't uninstall, it might not be there
+ $sudo_pip $cmd_pip uninstall -y $name || /bin/true
+}
+
# get version of a package from global requirements file
# get_from_global_requirements <package>
function get_from_global_requirements {
diff --git a/lib/cinder b/lib/cinder
index 9fc25c7..1b4f4e6 100644
--- a/lib/cinder
+++ b/lib/cinder
@@ -244,6 +244,10 @@
configure_rootwrap cinder
+ if [[ -f "$CINDER_DIR/etc/cinder/resource_filters.json" ]]; then
+ cp -p "$CINDER_DIR/etc/cinder/resource_filters.json" "$CINDER_CONF_DIR/resource_filters.json"
+ fi
+
cp $CINDER_DIR/etc/cinder/api-paste.ini $CINDER_API_PASTE_INI
inicomment $CINDER_API_PASTE_INI filter:authtoken auth_host
@@ -359,11 +363,10 @@
iniset $CINDER_CONF DEFAULT os_privileged_user_tenant "$SERVICE_PROJECT_NAME"
iniset $CINDER_CONF DEFAULT graceful_shutdown_timeout "$SERVICE_GRACEFUL_SHUTDOWN_TIMEOUT"
- # Set the backend url according to the configured dlm backend
- if is_dlm_enabled; then
- if [[ "$(dlm_backend)" == "zookeeper" ]]; then
- iniset $CINDER_CONF coordination backend_url "zookeeper://${SERVICE_HOST}:2181"
- fi
+ if [[ ! -z "$CINDER_COORDINATION_URL" ]]; then
+ iniset $CINDER_CONF coordination backend_url "$CINDER_COORDINATION_URL"
+ elif is_service_enabled etcd3; then
+ iniset $CINDER_CONF coordination backend_url "etcd3+http://${SERVICE_HOST}:2379"
fi
}
diff --git a/lib/dlm b/lib/dlm
deleted file mode 100644
index b5ac0f5..0000000
--- a/lib/dlm
+++ /dev/null
@@ -1,111 +0,0 @@
-#!/bin/bash
-#
-# lib/dlm
-#
-# Functions to control the installation and configuration of software
-# that provides a dlm (and possibly other functions). The default is
-# **zookeeper**, and is going to be the only backend supported in the
-# devstack tree.
-
-# Dependencies:
-#
-# - ``functions`` file
-
-# ``stack.sh`` calls the entry points in this order:
-#
-# - is_dlm_enabled
-# - install_dlm
-# - configure_dlm
-# - cleanup_dlm
-
-# Save trace setting
-_XTRACE_DLM=$(set +o | grep xtrace)
-set +o xtrace
-
-
-# Defaults
-# --------
-
-# <define global variables here that belong to this project>
-
-# Set up default directories
-ZOOKEEPER_DATA_DIR=$DEST/data/zookeeper
-ZOOKEEPER_CONF_DIR=/etc/zookeeper
-
-
-# Entry Points
-# ------------
-#
-# NOTE(sdague): it is expected that when someone wants to implement
-# another one of these out of tree, they'll implement the following
-# functions:
-#
-# - dlm_backend
-# - install_dlm
-# - configure_dlm
-# - cleanup_dlm
-
-# This should be declared in the settings file of any plugin or
-# service that needs to have a dlm in their environment.
-function use_dlm {
- enable_service $(dlm_backend)
-}
-
-# A function to return the name of the backend in question, some users
-# are going to need to know this.
-function dlm_backend {
- echo "zookeeper"
-}
-
-# Test if a dlm is enabled (defaults to a zookeeper specific check)
-function is_dlm_enabled {
- [[ ,${ENABLED_SERVICES}, =~ ,"$(dlm_backend)", ]] && return 0
- return 1
-}
-
-# cleanup_dlm() - Remove residual data files, anything left over from previous
-# runs that a clean run would need to clean up
-function cleanup_dlm {
- # NOTE(sdague): we don't check for is_enabled here because we
- # should just delete this regardless. Some times users updated
- # their service list before they run cleanup.
- sudo rm -rf $ZOOKEEPER_DATA_DIR
-}
-
-# configure_dlm() - Set config files, create data dirs, etc
-function configure_dlm {
- if is_dlm_enabled; then
- sudo cp $FILES/zookeeper/* $ZOOKEEPER_CONF_DIR
- sudo sed -i -e 's|.*dataDir.*|dataDir='$ZOOKEEPER_DATA_DIR'|' $ZOOKEEPER_CONF_DIR/zoo.cfg
- # clean up from previous (possibly aborted) runs
- # create required data files
- sudo rm -rf $ZOOKEEPER_DATA_DIR
- sudo mkdir -p $ZOOKEEPER_DATA_DIR
- # restart after configuration, there is no reason to make this
- # another step, because having data files that don't match the
- # zookeeper running is just going to cause tears.
- restart_service zookeeper
- fi
-}
-
-# install_dlm() - Collect source and prepare
-function install_dlm {
- if is_dlm_enabled; then
- pip_install_gr_extras tooz zookeeper
- if is_ubuntu; then
- install_package zookeeperd
- elif is_fedora; then
- install_package zookeeper
- else
- die $LINENO "Don't know how to install zookeeper on this platform"
- fi
- fi
-}
-
-# Restore xtrace
-$_XTRACE_DLM
-
-# Tell emacs to use shell-script-mode
-## Local variables:
-## mode: shell-script
-## End:
diff --git a/lib/etcd3 b/lib/etcd3
new file mode 100644
index 0000000..5cab3f5
--- /dev/null
+++ b/lib/etcd3
@@ -0,0 +1,147 @@
+#!/bin/bash
+#
+# lib/etcd3
+#
+# Functions to control the installation and configuration of etcd 3.x
+# that provides a key-value store (and possibly other functions).
+
+# Dependencies:
+#
+# - ``functions`` file
+
+# ``stack.sh`` calls the entry points in this order:
+#
+# - start_etcd3
+# - stop_etcd3
+# - cleanup_etcd3
+
+# Save trace setting
+_XTRACE_ETCD3=$(set +o | grep xtrace)
+set +o xtrace
+
+
+# Defaults
+# --------
+
+# Set up default values for etcd
+ETCD_DOWNLOAD_URL=${ETCD_DOWNLOAD_URL:-https://github.com/coreos/etcd/releases/download}
+ETCD_VERSION=${ETCD_VERSION:-v3.1.7}
+ETCD_DATA_DIR="$DEST/data/etcd"
+ETCD_SYSTEMD_SERVICE="devstack@etcd.service"
+ETCD_BIN_DIR="$DEST/bin"
+ETCD_SHA256_AMD64="4fde194bbcd259401e2b5c462dfa579ee7f6af539f13f130b8f5b4f52e3b3c52"
+# NOTE(sdague): etcd v3.1.7 doesn't have anything for these architectures, though 3.2.0 does.
+ETCD_SHA256_ARM64=""
+ETCD_SHA256_PPC64=""
+
+if is_ubuntu ; then
+ UBUNTU_RELEASE_BASE_NUM=`lsb_release -r | awk '{print $2}' | cut -d '.' -f 1`
+fi
+
+# start_etcd3() - Starts to run the etcd process
+function start_etcd3 {
+ # Don't install in sub nodes (multinode scenario)
+ if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+ return
+ fi
+
+ _install_etcd
+
+ local cmd="$ETCD_BIN_DIR/etcd"
+ cmd+=" --name $HOSTNAME --data-dir $ETCD_DATA_DIR"
+ cmd+=" --initial-cluster-state new --initial-cluster-token etcd-cluster-01"
+ cmd+=" --initial-cluster $HOSTNAME=http://$SERVICE_HOST:2380"
+ cmd+=" --initial-advertise-peer-urls http://$SERVICE_HOST:2380"
+ cmd+=" --advertise-client-urls http://$SERVICE_HOST:2379"
+ cmd+=" --listen-peer-urls http://0.0.0.0:2380 "
+ cmd+=" --listen-client-urls http://$SERVICE_HOST:2379"
+
+ local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
+ write_user_unit_file $ETCD_SYSTEMD_SERVICE "$cmd" "" "root"
+
+ iniset -sudo $unitfile "Unit" "After" "network.target"
+ iniset -sudo $unitfile "Service" "Type" "notify"
+ iniset -sudo $unitfile "Service" "Restart" "on-failure"
+ iniset -sudo $unitfile "Service" "LimitNOFILE" "65536"
+
+ $SYSTEMCTL daemon-reload
+ $SYSTEMCTL enable $ETCD_SYSTEMD_SERVICE
+ $SYSTEMCTL start $ETCD_SYSTEMD_SERVICE
+}
+
+# stop_etcd3() stops the etcd3 process
+function stop_etcd3 {
+ # Don't install in sub nodes (multinode scenario)
+ if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+ return
+ fi
+
+ $SYSTEMCTL stop $ETCD_SYSTEMD_SERVICE
+}
+
+function cleanup_etcd3 {
+ # Don't install in sub nodes (multinode scenario)
+ if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
+ return
+ fi
+
+ $SYSTEMCTL disable $ETCD_SYSTEMD_SERVICE
+
+ local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
+ sudo rm -f $unitfile
+
+ $SYSTEMCTL daemon-reload
+
+ sudo rm -rf $ETCD_DATA_DIR
+}
+
+function _install_etcd {
+ echo "Installing etcd"
+
+ # Make sure etcd3 downloads the correct architecture
+ if is_arch "x86_64"; then
+ ETCD_ARCH="amd64"
+ ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_AMD64}
+ elif is_arch "aarch64"; then
+ ETCD_ARCH="arm64"
+ ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_ARM64}
+ elif is_arch "ppc64le"; then
+ ETCD_ARCH="ppc64le"
+ ETCD_SHA256=${ETCD_SHA256:-$ETCD_SHA256_PPC64}
+ else
+ exit_distro_not_supported "invalid hardware type - $ETCD_ARCH"
+ fi
+
+ ETCD_NAME=etcd-$ETCD_VERSION-linux-$ETCD_ARCH
+
+ # Install the libraries needed. Note: tooz for example does not have a hard dependency on these libraries
+ pip_install etcd3
+ pip_install etcd3gw
+
+ # Create the necessary directories
+ sudo mkdir -p $ETCD_BIN_DIR
+ sudo mkdir -p $ETCD_DATA_DIR
+
+ # Download and cache the etcd tgz for subsequent use
+ if [ ! -f "files/etcd-$ETCD_VERSION-linux-$ETCD_ARCH/etcd" ]; then
+ ETCD_DOWNLOAD_FILE=$ETCD_NAME.tar.gz
+ wget $ETCD_DOWNLOAD_URL/$ETCD_VERSION/$ETCD_DOWNLOAD_FILE -O files/$ETCD_DOWNLOAD_FILE
+ echo "${ETCD_SHA256} files/${ETCD_DOWNLOAD_FILE}" > files/etcd.sha256sum
+ # NOTE(sdague): this should go fatal if this fails
+ sha256sum -c files/etcd.sha256sum
+
+ tar xzvf files/$ETCD_DOWNLOAD_FILE -C files
+ sudo cp files/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+ fi
+ if [ ! -f "$ETCD_BIN_DIR/etcd" ]; then
+ sudo cp files/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+ fi
+}
+
+# Restore xtrace
+$_XTRACE_ETCD3
+
+# Tell emacs to use shell-script-mode
+## Local variables:
+## mode: shell-script
+## End:
diff --git a/lib/nova_plugins/functions-libvirt b/lib/nova_plugins/functions-libvirt
index 47605af..3e38b89 100644
--- a/lib/nova_plugins/functions-libvirt
+++ b/lib/nova_plugins/functions-libvirt
@@ -67,6 +67,8 @@
else
install_package libvirt-clients libvirt-daemon-system libvirt-dev
fi
+ # uninstall in case the libvirt version changed
+ pip_uninstall libvirt-python
pip_install_gr libvirt-python
#pip_install_gr <there-si-no-guestfs-in-pypi>
elif is_fedora || is_suse; then
@@ -84,6 +86,7 @@
fi
install_package libvirt libvirt-devel
+ pip_uninstall libvirt-python
pip_install_gr libvirt-python
fi
@@ -95,7 +98,7 @@
# Configures the installed libvirt system so that is accessible by
# STACK_USER via qemu:///system with management capabilities.
function configure_libvirt {
- if is_service_enabled neutron && is_neutron_ovs_base_plugin && ! sudo grep -q '^cgroup_device_acl' $QEMU_CONF; then
+ if is_service_enabled neutron && ! sudo grep -q '^cgroup_device_acl' $QEMU_CONF; then
# Add /dev/net/tun to cgroup_device_acls, needed for type=ethernet interfaces
cat <<EOF | sudo tee -a $QEMU_CONF
cgroup_device_acl = [
diff --git a/lib/tempest b/lib/tempest
index fd8e3e7..cc65ec7 100644
--- a/lib/tempest
+++ b/lib/tempest
@@ -394,24 +394,6 @@
iniset $TEMPEST_CONFIG network-feature-enabled ipv6_subnet_attributes "$IPV6_SUBNET_ATTRIBUTES_ENABLED"
iniset $TEMPEST_CONFIG network-feature-enabled port_security $NEUTRON_PORT_SECURITY
- # Orchestration Tests
- if is_service_enabled heat; then
- if [[ ! -z "$HEAT_CFN_IMAGE_URL" ]]; then
- iniset $TEMPEST_CONFIG orchestration image_ref $(basename "${HEAT_CFN_IMAGE_URL%.*}")
- fi
- # Nova might not be enabled, especially when we want to test tempest scenario/API that only create Neutron resources
- if is_service_enabled nova; then
- # build a specialized heat flavor
- available_flavors=$(nova flavor-list)
- if [[ ! ( $available_flavors =~ 'm1.heat' ) ]]; then
- openstack flavor create --id 451 --ram 512 --disk 0 --vcpus 1 m1.heat
- fi
- iniset $TEMPEST_CONFIG orchestration instance_type "m1.heat"
- fi
- iniset $TEMPEST_CONFIG orchestration build_timeout 900
- iniset $TEMPEST_CONFIG orchestration stack_owner_role Member
- fi
-
# Scenario
if [ "$VIRT_DRIVER" = "xenserver" ]; then
SCENARIO_IMAGE_DIR=${SCENARIO_IMAGE_DIR:-$FILES}
@@ -431,7 +413,7 @@
TEMPEST_SSH_NETWORK_NAME=$PHYSICAL_NETWORK
fi
# Validation
- iniset $TEMPEST_CONFIG validation run_validation ${TEMPEST_RUN_VALIDATION:-False}
+ iniset $TEMPEST_CONFIG validation run_validation ${TEMPEST_RUN_VALIDATION:-True}
iniset $TEMPEST_CONFIG validation ip_version_for_ssh 4
iniset $TEMPEST_CONFIG validation ssh_timeout $BUILD_TIMEOUT
iniset $TEMPEST_CONFIG validation image_ssh_user ${DEFAULT_INSTANCE_USER:-cirros}
diff --git a/openrc b/openrc
index 38f5c47..23c173c 100644
--- a/openrc
+++ b/openrc
@@ -72,8 +72,6 @@
GLANCE_HOST=${GLANCE_HOST:-$HOST_IP}
fi
-SERVICE_PROTOCOL=${SERVICE_PROTOCOL:-http}
-
# Identity API version
export OS_IDENTITY_API_VERSION=${IDENTITY_API_VERSION:-3}
@@ -85,7 +83,12 @@
# the user/project has access to - including nova, glance, keystone, swift, ...
# We currently recommend using the version 3 *identity api*.
#
-export OS_AUTH_URL=$KEYSTONE_AUTH_URI
+
+# If you don't have a working .stackenv, this is the backup possition
+KEYSTONE_BACKUP=$SERVICE_PROTOCOL://$SERVICE_HOST:5000
+KEYSTONE_AUTH_URI=${KEYSTONE_AUTH_URI:-$KEYSTONE_BACKUP}
+
+export OS_AUTH_URL=${OS_AUTH_URL:-$KEYSTONE_AUTH_URI}
# Currently, in order to use openstackclient with Identity API v3,
# we need to set the domain which the user and project belong to.
diff --git a/stack.sh b/stack.sh
index 9fdb47e..a016370 100755
--- a/stack.sh
+++ b/stack.sh
@@ -2,7 +2,7 @@
# ``stack.sh`` is an opinionated OpenStack developer installation. It
# installs and configures various combinations of **Cinder**, **Glance**,
-# **Heat**, **Horizon**, **Keystone**, **Nova**, **Neutron**, and **Swift**
+# **Horizon**, **Keystone**, **Nova**, **Neutron**, and **Swift**
# This script's options can be changed by setting appropriate environment
# variables. You can configure things like which git repositories to use,
@@ -573,7 +573,7 @@
source $TOP_DIR/lib/neutron
source $TOP_DIR/lib/ldap
source $TOP_DIR/lib/dstat
-source $TOP_DIR/lib/dlm
+source $TOP_DIR/lib/etcd3
source $TOP_DIR/lib/os_brick
# Extras Source
@@ -765,13 +765,12 @@
# Phase: pre-install
run_phase stack pre-install
+# NOTE(danms): Set global limits before installing anything
+set_systemd_override DefaultLimitNOFILE ${ULIMIT_NOFILE}
+
install_rpc_backend
restart_rpc_backend
-# NOTE(sdague): dlm install is conditional on one being enabled by configuration
-install_dlm
-configure_dlm
-
if is_service_enabled $DATABASE_BACKENDS; then
install_database
fi
@@ -1040,6 +1039,13 @@
# A better kind of sysstat, with the top process per time slice
start_dstat
+# Etcd
+# -----
+
+# etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines
+if is_service_enabled etcd3; then
+ start_etcd3
+fi
# Keystone
# --------
@@ -1253,8 +1259,13 @@
fi
# Create a randomized default value for the key manager's fixed_key
+# NOTE(lyarwood): This is currently set to 36 as a workaround to the following
+# libvirt bug that incorrectly pads passphrases that are a multiple of 16 bytes
+# in length.
+# Unable to use LUKS passphrase that is exactly 16 bytes long
+# https://bugzilla.redhat.com/show_bug.cgi?id=1447297
if is_service_enabled nova; then
- iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 32)
+ iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 36)
fi
# Launch the nova-api and wait for it to answer before continuing
@@ -1332,10 +1343,6 @@
USERRC_PARAMS="$USERRC_PARAMS --os-cacert $SSL_BUNDLE_FILE"
fi
- if [[ "$HEAT_STANDALONE" = "True" ]]; then
- USERRC_PARAMS="$USERRC_PARAMS --heat-url http://$HEAT_API_HOST:$HEAT_API_PORT/v1"
- fi
-
$TOP_DIR/tools/create_userrc.sh $USERRC_PARAMS
fi
diff --git a/stackrc b/stackrc
index 1300f45..9203f8b 100644
--- a/stackrc
+++ b/stackrc
@@ -65,7 +65,7 @@
# Dashboard
ENABLED_SERVICES+=,horizon
# Additional services
- ENABLED_SERVICES+=,rabbit,tempest,mysql,dstat
+ ENABLED_SERVICES+=,rabbit,tempest,mysql,etcd3,dstat
fi
# Global toggle for enabling services under mod_wsgi. If this is set to
@@ -100,11 +100,9 @@
if [[ "$USER_UNITS" == "True" ]]; then
SYSTEMD_DIR="$HOME/.local/share/systemd/user"
SYSTEMCTL="systemctl --user"
- JOURNALCTL_F="journalctl -f -o short-precise --user-unit"
else
SYSTEMD_DIR="/etc/systemd/system"
SYSTEMCTL="sudo systemctl"
- JOURNALCTL_F="journalctl -f -o short-precise --unit"
fi
@@ -518,14 +516,6 @@
GITREPO["glance_store"]=${GLANCE_STORE_REPO:-${GIT_BASE}/openstack/glance_store.git}
GITBRANCH["glance_store"]=${GLANCE_STORE_BRANCH:-master}
-# heat-cfntools server agent
-HEAT_CFNTOOLS_REPO=${HEAT_CFNTOOLS_REPO:-${GIT_BASE}/openstack/heat-cfntools.git}
-HEAT_CFNTOOLS_BRANCH=${HEAT_CFNTOOLS_BRANCH:-master}
-
-# heat example templates and elements
-HEAT_TEMPLATES_REPO=${HEAT_TEMPLATES_REPO:-${GIT_BASE}/openstack/heat-templates.git}
-HEAT_TEMPLATES_BRANCH=${HEAT_TEMPLATES_BRANCH:-master}
-
# django openstack_auth library
GITREPO["django_openstack_auth"]=${HORIZONAUTH_REPO:-${GIT_BASE}/openstack/django_openstack_auth.git}
GITBRANCH["django_openstack_auth"]=${HORIZONAUTH_BRANCH:-master}
@@ -933,6 +923,9 @@
# ``LOGDIR`` is always set at this point so it is not useful as a 'enable' for service logs
# ``SCREEN_LOGDIR`` may be set, it is useful to enable the compat symlinks
+# System-wide ulimit file descriptors override
+ULIMIT_NOFILE=${ULIMIT_NOFILE:-2048}
+
# Local variables:
# mode: shell-script
# End:
diff --git a/tools/dstat.sh b/tools/dstat.sh
index ae7306e..01c6d9b 100755
--- a/tools/dstat.sh
+++ b/tools/dstat.sh
@@ -13,10 +13,10 @@
LOGDIR=$1
# Command line arguments for primary DStat process.
-DSTAT_OPTS="-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap"
+DSTAT_OPTS="-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap --tcp"
# Command-line arguments for secondary background DStat process.
-DSTAT_CSV_OPTS="-tcmndrylpg --output $LOGDIR/dstat-csv.log"
+DSTAT_CSV_OPTS="-tcmndrylpg --tcp --output $LOGDIR/dstat-csv.log"
# Execute and background the secondary dstat process and discard its output.
dstat $DSTAT_CSV_OPTS >& /dev/null &
diff --git a/tools/fixup_stuff.sh b/tools/fixup_stuff.sh
index f3ba702..d07d267 100755
--- a/tools/fixup_stuff.sh
+++ b/tools/fixup_stuff.sh
@@ -82,15 +82,17 @@
source /etc/ci/mirror_info.sh
sudo apt-add-repository -y "deb $NODEPOOL_UCA_MIRROR xenial-updates/ocata main"
-
- # Disable use of libvirt wheel here as presence of mirror implies
- # presence of cached wheel build against older libvirt binary.
- # TODO(clarkb) figure out how to use wheel again.
- sudo bash -c 'echo "no-binary = libvirt-python" >> /etc/pip.conf'
else
# Otherwise use upstream UCA
sudo add-apt-repository -y cloud-archive:ocata
fi
+
+ # Disable use of libvirt wheel since a cached wheel build might be
+ # against older libvirt binary. Particularly a problem if using
+ # the openstack wheel mirrors, but can hit locally too.
+ # TODO(clarkb) figure out how to use upstream wheel again.
+ iniset -sudo /etc/pip.conf "global" "no-binary" "libvirt-python"
+
# Force update our APT repos, since we added UCA above.
REPOS_UPDATED=False
apt_get_update
diff --git a/unstack.sh b/unstack.sh
index 485fed7..77a151f 100755
--- a/unstack.sh
+++ b/unstack.sh
@@ -69,7 +69,7 @@
source $TOP_DIR/lib/neutron
source $TOP_DIR/lib/ldap
source $TOP_DIR/lib/dstat
-source $TOP_DIR/lib/dlm
+source $TOP_DIR/lib/etcd3
# Extras Source
# --------------
@@ -162,6 +162,11 @@
cleanup_neutron
fi
+if is_service_enabled etcd3; then
+ stop_etcd3
+ cleanup_etcd3
+fi
+
if is_service_enabled dstat; then
stop_dstat
fi