Merge "Change restart Always to always"
diff --git a/.gitignore b/.gitignore
index 7967e14..d2c127d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -24,6 +24,7 @@
files/get-pip.py*
files/ir-deploy*
files/ironic-inspector*
+files/etcd*
local.conf
local.sh
localrc
diff --git a/README.rst b/README.rst
index dfa68b9..b4240bd 100644
--- a/README.rst
+++ b/README.rst
@@ -92,5 +92,5 @@
`local.conf`. It is likely that you will need to provide and modify
this file if you want anything other than the most basic setup. Start
by reading the `configuration guide
-<https://docs.openstack.org/developer/devstack/configuration.html>_`
+<https://docs.openstack.org/developer/devstack/configuration.html>`_
for details of the configuration file and the many available options.
diff --git a/clean.sh b/clean.sh
index ef38fbf..9ffe3be 100755
--- a/clean.sh
+++ b/clean.sh
@@ -125,6 +125,13 @@
sudo rm -rf $SCREEN_LOGDIR
fi
+# Clean out the sytemd user unit files if systemd was used.
+if [[ "$USE_SYSTEMD" = "True" ]]; then
+ sudo find $SYSTEMD_DIR -type f -name '*devstack@*service' -delete
+ # Make systemd aware of the deletion.
+ $SYSTEMCTL daemon-reload
+fi
+
# Clean up venvs
DIRS_TO_CLEAN="$WHEELHOUSE ${PROJECT_VENV[@]} .config/openstack"
rm -rf $DIRS_TO_CLEAN
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 66b8702..064bf51 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -195,6 +195,9 @@
Setting it here also makes it available for ``openrc`` to set ``OS_AUTH_URL``.
``HOST_IPV6`` is not set by default.
+For architecture specific configurations which differ from the x86 default
+here, see `arch-configuration`_.
+
Historical Notes
================
@@ -749,3 +752,60 @@
::
TERMINATE_TIMEOUT=30
+
+
+.. _arch-configuration:
+
+Architectures
+-------------
+
+The upstream CI runs exclusively on nodes with x86 architectures, but
+OpenStack supports even more architectures. Some of them need to configure
+Devstack in a certain way.
+
+KVM on s390x (IBM z Systems)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+KVM on s390x (IBM z Systems) is supported since the *Kilo* release. For
+an all-in-one setup, these minimal settings in the ``local.conf`` file
+are needed::
+
+ [[local|localrc]]
+ ADMIN_PASSWORD=secret
+ DATABASE_PASSWORD=$ADMIN_PASSWORD
+ RABBIT_PASSWORD=$ADMIN_PASSWORD
+ SERVICE_PASSWORD=$ADMIN_PASSWORD
+
+ DOWNLOAD_DEFAULT_IMAGES=False
+ IMAGE_URLS="https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-s390x-disk1.img"
+
+ enable_service n-sproxy
+ disable_service n-novnc
+ disable_service etcd3 # https://bugs.launchpad.net/devstack/+bug/1693192
+
+ [[post-config|$NOVA_CONF]]
+
+ [serial_console]
+ base_url=ws://$HOST_IP:6083/ # optional
+
+Reasoning:
+
+* The default image of Devstack is x86 only, so we deactivate the download
+ with ``DOWNLOAD_DEFAULT_IMAGES``. The referenced guest image
+ in the code above (``IMAGE_URLS``) serves as an example. The list of
+ possible s390x guest images is not limited to that.
+
+* This platform doesn't support a graphical console like VNC or SPICE.
+ The technical reason is the missing framebuffer on the platform. This
+ means we rely on the substitute feature *serial console* which needs the
+ proxy service ``n-sproxy``. We also disable VNC's proxy ``n-novnc`` for
+ that reason . The configuration in the ``post-config`` section is only
+ needed if you want to use the *serial console* outside of the all-in-one
+ setup.
+
+* The service ``etcd3`` needs to be disabled as long as bug report
+ https://bugs.launchpad.net/devstack/+bug/1693192 is not resolved.
+
+.. note:: To run *Tempest* against this *Devstack* all-in-one, you'll need
+ to use a guest image which is smaller than 1GB when uncompressed.
+ The example image from above is bigger than that!
diff --git a/doc/source/plugin-registry.rst b/doc/source/plugin-registry.rst
index 96a2733..92e5ecd 100644
--- a/doc/source/plugin-registry.rst
+++ b/doc/source/plugin-registry.rst
@@ -48,10 +48,12 @@
devstack-plugin-glusterfs `git://git.openstack.org/openstack/devstack-plugin-glusterfs <https://git.openstack.org/cgit/openstack/devstack-plugin-glusterfs>`__
devstack-plugin-hdfs `git://git.openstack.org/openstack/devstack-plugin-hdfs <https://git.openstack.org/cgit/openstack/devstack-plugin-hdfs>`__
devstack-plugin-kafka `git://git.openstack.org/openstack/devstack-plugin-kafka <https://git.openstack.org/cgit/openstack/devstack-plugin-kafka>`__
+devstack-plugin-libvirt-qemu `git://git.openstack.org/openstack/devstack-plugin-libvirt-qemu <https://git.openstack.org/cgit/openstack/devstack-plugin-libvirt-qemu>`__
devstack-plugin-mariadb `git://git.openstack.org/openstack/devstack-plugin-mariadb <https://git.openstack.org/cgit/openstack/devstack-plugin-mariadb>`__
devstack-plugin-nfs `git://git.openstack.org/openstack/devstack-plugin-nfs <https://git.openstack.org/cgit/openstack/devstack-plugin-nfs>`__
devstack-plugin-pika `git://git.openstack.org/openstack/devstack-plugin-pika <https://git.openstack.org/cgit/openstack/devstack-plugin-pika>`__
devstack-plugin-sheepdog `git://git.openstack.org/openstack/devstack-plugin-sheepdog <https://git.openstack.org/cgit/openstack/devstack-plugin-sheepdog>`__
+devstack-plugin-vmax `git://git.openstack.org/openstack/devstack-plugin-vmax <https://git.openstack.org/cgit/openstack/devstack-plugin-vmax>`__
devstack-plugin-zmq `git://git.openstack.org/openstack/devstack-plugin-zmq <https://git.openstack.org/cgit/openstack/devstack-plugin-zmq>`__
dragonflow `git://git.openstack.org/openstack/dragonflow <https://git.openstack.org/cgit/openstack/dragonflow>`__
drbd-devstack `git://git.openstack.org/openstack/drbd-devstack <https://git.openstack.org/cgit/openstack/drbd-devstack>`__
@@ -62,7 +64,6 @@
fuxi `git://git.openstack.org/openstack/fuxi <https://git.openstack.org/cgit/openstack/fuxi>`__
gce-api `git://git.openstack.org/openstack/gce-api <https://git.openstack.org/cgit/openstack/gce-api>`__
glare `git://git.openstack.org/openstack/glare <https://git.openstack.org/cgit/openstack/glare>`__
-gnocchi `git://git.openstack.org/openstack/gnocchi <https://git.openstack.org/cgit/openstack/gnocchi>`__
group-based-policy `git://git.openstack.org/openstack/group-based-policy <https://git.openstack.org/cgit/openstack/group-based-policy>`__
heat `git://git.openstack.org/openstack/heat <https://git.openstack.org/cgit/openstack/heat>`__
horizon-mellanox `git://git.openstack.org/openstack/horizon-mellanox <https://git.openstack.org/cgit/openstack/horizon-mellanox>`__
@@ -123,9 +124,11 @@
neutron `git://git.openstack.org/openstack/neutron <https://git.openstack.org/cgit/openstack/neutron>`__
neutron-dynamic-routing `git://git.openstack.org/openstack/neutron-dynamic-routing <https://git.openstack.org/cgit/openstack/neutron-dynamic-routing>`__
neutron-fwaas `git://git.openstack.org/openstack/neutron-fwaas <https://git.openstack.org/cgit/openstack/neutron-fwaas>`__
+neutron-fwaas-dashboard `git://git.openstack.org/openstack/neutron-fwaas-dashboard <https://git.openstack.org/cgit/openstack/neutron-fwaas-dashboard>`__
neutron-lbaas `git://git.openstack.org/openstack/neutron-lbaas <https://git.openstack.org/cgit/openstack/neutron-lbaas>`__
neutron-lbaas-dashboard `git://git.openstack.org/openstack/neutron-lbaas-dashboard <https://git.openstack.org/cgit/openstack/neutron-lbaas-dashboard>`__
neutron-vpnaas `git://git.openstack.org/openstack/neutron-vpnaas <https://git.openstack.org/cgit/openstack/neutron-vpnaas>`__
+neutron-vpnaas-dashboard `git://git.openstack.org/openstack/neutron-vpnaas-dashboard <https://git.openstack.org/cgit/openstack/neutron-vpnaas-dashboard>`__
nova-dpm `git://git.openstack.org/openstack/nova-dpm <https://git.openstack.org/cgit/openstack/nova-dpm>`__
nova-lxd `git://git.openstack.org/openstack/nova-lxd <https://git.openstack.org/cgit/openstack/nova-lxd>`__
nova-mksproxy `git://git.openstack.org/openstack/nova-mksproxy <https://git.openstack.org/cgit/openstack/nova-mksproxy>`__
@@ -136,6 +139,7 @@
os-xenapi `git://git.openstack.org/openstack/os-xenapi <https://git.openstack.org/cgit/openstack/os-xenapi>`__
osprofiler `git://git.openstack.org/openstack/osprofiler <https://git.openstack.org/cgit/openstack/osprofiler>`__
panko `git://git.openstack.org/openstack/panko <https://git.openstack.org/cgit/openstack/panko>`__
+patrole `git://git.openstack.org/openstack/patrole <https://git.openstack.org/cgit/openstack/patrole>`__
picasso `git://git.openstack.org/openstack/picasso <https://git.openstack.org/cgit/openstack/picasso>`__
rally `git://git.openstack.org/openstack/rally <https://git.openstack.org/cgit/openstack/rally>`__
sahara `git://git.openstack.org/openstack/sahara <https://git.openstack.org/cgit/openstack/sahara>`__
@@ -145,12 +149,15 @@
searchlight-ui `git://git.openstack.org/openstack/searchlight-ui <https://git.openstack.org/cgit/openstack/searchlight-ui>`__
senlin `git://git.openstack.org/openstack/senlin <https://git.openstack.org/cgit/openstack/senlin>`__
solum `git://git.openstack.org/openstack/solum <https://git.openstack.org/cgit/openstack/solum>`__
+stackube `git://git.openstack.org/openstack/stackube <https://git.openstack.org/cgit/openstack/stackube>`__
tacker `git://git.openstack.org/openstack/tacker <https://git.openstack.org/cgit/openstack/tacker>`__
tap-as-a-service `git://git.openstack.org/openstack/tap-as-a-service <https://git.openstack.org/cgit/openstack/tap-as-a-service>`__
+tap-as-a-service-dashboard `git://git.openstack.org/openstack/tap-as-a-service-dashboard <https://git.openstack.org/cgit/openstack/tap-as-a-service-dashboard>`__
tricircle `git://git.openstack.org/openstack/tricircle <https://git.openstack.org/cgit/openstack/tricircle>`__
trio2o `git://git.openstack.org/openstack/trio2o <https://git.openstack.org/cgit/openstack/trio2o>`__
trove `git://git.openstack.org/openstack/trove <https://git.openstack.org/cgit/openstack/trove>`__
trove-dashboard `git://git.openstack.org/openstack/trove-dashboard <https://git.openstack.org/cgit/openstack/trove-dashboard>`__
+valet `git://git.openstack.org/openstack/valet <https://git.openstack.org/cgit/openstack/valet>`__
vitrage `git://git.openstack.org/openstack/vitrage <https://git.openstack.org/cgit/openstack/vitrage>`__
vitrage-dashboard `git://git.openstack.org/openstack/vitrage-dashboard <https://git.openstack.org/cgit/openstack/vitrage-dashboard>`__
vmware-nsx `git://git.openstack.org/openstack/vmware-nsx <https://git.openstack.org/cgit/openstack/vmware-nsx>`__
diff --git a/functions b/functions
index e497e45..e056c3f 100644
--- a/functions
+++ b/functions
@@ -615,7 +615,7 @@
# native systemd path, which provides for things like search on
# request-id. However, there may be an eventlet interaction here,
# so going off for now.
- USE_JOURNAL=$(trueorfalse USE_JOURNAL False)
+ USE_JOURNAL=$(trueorfalse False USE_JOURNAL)
local pidstr=""
if [[ "$USE_JOURNAL" == "True" ]]; then
iniset $conf_file $conf_section use_journal "True"
@@ -732,6 +732,24 @@
sudo systemctl daemon-reload
}
+# Get a random port from the local port range
+#
+# This function returns an available port in the local port range. The search
+# order is not truly random, but should be considered a random value by the
+# user because it depends on the state of your local system.
+function get_random_port {
+ read lower_port upper_port < /proc/sys/net/ipv4/ip_local_port_range
+ while true; do
+ for (( port = upper_port ; port >= lower_port ; port-- )); do
+ sudo lsof -i ":$port" &> /dev/null
+ if [[ $? > 0 ]] ; then
+ break 2
+ fi
+ done
+ done
+ echo $port
+}
+
# Restore xtrace
$_XTRACE_FUNCTIONS
diff --git a/functions-common b/functions-common
index 26d1c28..4f336ea 100644
--- a/functions-common
+++ b/functions-common
@@ -2491,7 +2491,7 @@
if [[ -n "$start_time" ]]; then
die $LINENO "Trying to start the clock on $name, but it's already been started"
fi
- _TIME_START[$name]=$(date +%s)
+ _TIME_START[$name]=$(date +%s%3N)
}
# time_stop $name
@@ -2512,7 +2512,7 @@
if [[ -z "$start_time" ]]; then
die $LINENO "Trying to stop the clock on $name, but it was never started"
fi
- end_time=$(date +%s)
+ end_time=$(date +%s%3N)
elapsed_time=$(($end_time - $start_time))
total=${_TIME_TOTAL[$name]:-0}
# reset the clock so we can start it in the future
@@ -2520,6 +2520,49 @@
_TIME_TOTAL[$name]=$(($total + $elapsed_time))
}
+function oscwrap {
+ local out
+ local rc
+ local start
+ local end
+ # Cannot use timer_start and timer_stop as we run in subshells
+ # and those rely on modifying vars in the same process (which cannot
+ # happen from a subshell.
+ start=$(date +%s%3N)
+ out=$(command openstack "$@")
+ rc=$?
+ end=$(date +%s%3N)
+ echo $((end - start)) >> $OSCWRAP_TIMER_FILE
+
+ echo "$out"
+ return $rc
+}
+
+function install_oscwrap {
+ # File to accumulate our timing data
+ OSCWRAP_TIMER_FILE=$(mktemp)
+ # Bash by default doesn't expand aliases, allow it for the aliases
+ # we want to whitelist.
+ shopt -s expand_aliases
+ # Remove all aliases that might be expanded to preserve old unexpanded
+ # behavior
+ unalias -a
+ # Add only the alias we want for openstack
+ alias openstack=oscwrap
+}
+
+function cleanup_oscwrap {
+ local total=0
+ if python3_enabled ; then
+ local python=python3
+ else
+ local python=python
+ fi
+ total=$(cat $OSCWRAP_TIMER_FILE | $python -c "import sys; print(sum(int(l) for l in sys.stdin))")
+ _TIME_TOTAL["osc"]=$total
+ rm $OSCWRAP_TIMER_FILE
+}
+
# time_totals
# Print out total time summary
function time_totals {
@@ -2538,6 +2581,8 @@
fi
done
+ cleanup_oscwrap
+
xtrace=$(set +o | grep xtrace)
set +o xtrace
@@ -2549,6 +2594,8 @@
echo
for t in ${!_TIME_TOTAL[*]}; do
local v=${_TIME_TOTAL[$t]}
+ # because we're recording in milliseconds
+ v=$(($v / 1000))
printf "%-${len}s %3d\n" "$t" "$v"
done
echo "========================="
diff --git a/inc/python b/inc/python
index 4c443d6..718cbb2 100644
--- a/inc/python
+++ b/inc/python
@@ -320,6 +320,14 @@
fi
$xtrace
+
+ # Also install test requirements
+ local install_test_reqs=""
+ local test_req="${!#}/test-requirements.txt"
+ if [[ -e "$test_req" ]]; then
+ install_test_reqs="-r $test_req"
+ fi
+
# adding SETUPTOOLS_SYS_PATH_TECHNIQUE is a workaround to keep
# the same behaviour of setuptools before version 25.0.0.
# related issue: https://github.com/pypa/pip/issues/3874
@@ -329,24 +337,10 @@
no_proxy="${no_proxy:-}" \
PIP_FIND_LINKS=$PIP_FIND_LINKS \
SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite \
- $cmd_pip $upgrade \
+ $cmd_pip $upgrade $install_test_reqs \
$@
result=$?
- # Also install test requirements
- local test_req="${!#}/test-requirements.txt"
- if [[ $result == 0 ]] && [[ -e "$test_req" ]]; then
- echo "Installing test-requirements for $test_req"
- $sudo_pip \
- http_proxy=${http_proxy:-} \
- https_proxy=${https_proxy:-} \
- no_proxy=${no_proxy:-} \
- PIP_FIND_LINKS=$PIP_FIND_LINKS \
- $cmd_pip $upgrade \
- -r $test_req
- result=$?
- fi
-
time_stop "pip_install"
return $result
}
diff --git a/lib/apache b/lib/apache
index 43d5000..c1b6bf8 100644
--- a/lib/apache
+++ b/lib/apache
@@ -238,7 +238,13 @@
# create a home for the sockets; note don't use /tmp -- apache has
# a private view of it on some platforms.
local socket_dir='/var/run/uwsgi'
- sudo install -d -o $STACK_USER -m 755 $socket_dir
+
+ # /var/run will be empty on ubuntu after reboot, so we can use systemd-temptiles
+ # to automatically create $socket_dir.
+ sudo mkdir -p /etc/tmpfiles.d/
+ echo "d $socket_dir 0755 $STACK_USER root" | sudo tee /etc/tmpfiles.d/uwsgi.conf
+ sudo systemd-tmpfiles --create /etc/tmpfiles.d/uwsgi.conf
+
local socket="$socket_dir/${name}.socket"
# always cleanup given that we are using iniset here
diff --git a/lib/cinder b/lib/cinder
index 762edc4..2068812 100644
--- a/lib/cinder
+++ b/lib/cinder
@@ -129,6 +129,7 @@
# Test if any Cinder services are enabled
# is_cinder_enabled
function is_cinder_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"cinder" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"c-" ]] && return 0
return 1
}
@@ -406,8 +407,10 @@
# (Re)create cinder database
recreate_database cinder
+ time_start "dbsync"
# Migrate cinder database
$CINDER_BIN_DIR/cinder-manage --config-file $CINDER_CONF db sync
+ time_stop "dbsync"
fi
if is_service_enabled c-vol && [[ -n "$CINDER_ENABLED_BACKENDS" ]]; then
@@ -498,17 +501,24 @@
fi
fi
- if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
- enable_apache_site osapi-volume
- restart_apache_server
- tail_log c-api /var/log/$APACHE_NAME/c-api.log
- else
- run_process c-api "$CINDER_BIN_DIR/cinder-api --config-file $CINDER_CONF"
- fi
+ if is_service_enabled c-api ; then
+ if [ "$CINDER_USE_MOD_WSGI" == "True" ]; then
+ enable_apache_site osapi-volume
+ restart_apache_server
+ tail_log c-api /var/log/$APACHE_NAME/c-api.log
+ else
+ run_process c-api "$CINDER_BIN_DIR/cinder-api --config-file $CINDER_CONF"
+ fi
- echo "Waiting for Cinder API to start..."
- if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$CINDER_SERVICE_HOST:$service_port; then
- die $LINENO "c-api did not start"
+ echo "Waiting for Cinder API to start..."
+ if ! wait_for_service $SERVICE_TIMEOUT $service_protocol://$CINDER_SERVICE_HOST:$service_port; then
+ die $LINENO "c-api did not start"
+ fi
+
+ # Start proxies if enabled
+ if is_service_enabled tls-proxy; then
+ start_tls_proxy cinder '*' $CINDER_SERVICE_PORT $CINDER_SERVICE_HOST $CINDER_SERVICE_PORT_INT
+ fi
fi
run_process c-sch "$CINDER_BIN_DIR/cinder-scheduler --config-file $CINDER_CONF"
@@ -518,11 +528,6 @@
# NOTE(jdg): For cinder, startup order matters. To ensure that repor_capabilities is received
# by the scheduler start the cinder-volume service last (or restart it) after the scheduler
# has started. This is a quick fix for lp bug/1189595
-
- # Start proxies if enabled
- if is_service_enabled c-api && is_service_enabled tls-proxy; then
- start_tls_proxy cinder '*' $CINDER_SERVICE_PORT $CINDER_SERVICE_HOST $CINDER_SERVICE_PORT_INT
- fi
}
# stop_cinder() - Stop running processes
diff --git a/lib/databases/mysql b/lib/databases/mysql
index 7bbcace..a0cf7a4 100644
--- a/lib/databases/mysql
+++ b/lib/databases/mysql
@@ -71,6 +71,10 @@
elif is_fedora; then
mysql=mariadb
my_conf=/etc/my.cnf
+ local cracklib_conf=/etc/my.cnf.d/cracklib_password_check.cnf
+ if [ -f "$cracklib_conf" ]; then
+ inicomment -sudo "$cracklib_conf" "mariadb" "plugin-load-add"
+ fi
else
exit_distro_not_supported "mysql configuration"
fi
diff --git a/lib/etcd3 b/lib/etcd3
index 5cab3f5..0e1fbd5 100644
--- a/lib/etcd3
+++ b/lib/etcd3
@@ -33,6 +33,7 @@
# NOTE(sdague): etcd v3.1.7 doesn't have anything for these architectures, though 3.2.0 does.
ETCD_SHA256_ARM64=""
ETCD_SHA256_PPC64=""
+ETCD_PORT=2379
if is_ubuntu ; then
UBUNTU_RELEASE_BASE_NUM=`lsb_release -r | awk '{print $2}' | cut -d '.' -f 1`
@@ -40,21 +41,14 @@
# start_etcd3() - Starts to run the etcd process
function start_etcd3 {
- # Don't install in sub nodes (multinode scenario)
- if [ "$SERVICE_HOST" != "$HOST_IP" ]; then
- return
- fi
-
- _install_etcd
-
local cmd="$ETCD_BIN_DIR/etcd"
cmd+=" --name $HOSTNAME --data-dir $ETCD_DATA_DIR"
cmd+=" --initial-cluster-state new --initial-cluster-token etcd-cluster-01"
cmd+=" --initial-cluster $HOSTNAME=http://$SERVICE_HOST:2380"
cmd+=" --initial-advertise-peer-urls http://$SERVICE_HOST:2380"
- cmd+=" --advertise-client-urls http://$SERVICE_HOST:2379"
+ cmd+=" --advertise-client-urls http://${HOST_IP}:$ETCD_PORT"
cmd+=" --listen-peer-urls http://0.0.0.0:2380 "
- cmd+=" --listen-client-urls http://$SERVICE_HOST:2379"
+ cmd+=" --listen-client-urls http://${HOST_IP}:$ETCD_PORT"
local unitfile="$SYSTEMD_DIR/$ETCD_SYSTEMD_SERVICE"
write_user_unit_file $ETCD_SYSTEMD_SERVICE "$cmd" "" "root"
@@ -95,7 +89,7 @@
sudo rm -rf $ETCD_DATA_DIR
}
-function _install_etcd {
+function install_etcd3 {
echo "Installing etcd"
# Make sure etcd3 downloads the correct architecture
@@ -114,27 +108,25 @@
ETCD_NAME=etcd-$ETCD_VERSION-linux-$ETCD_ARCH
- # Install the libraries needed. Note: tooz for example does not have a hard dependency on these libraries
- pip_install etcd3
- pip_install etcd3gw
-
# Create the necessary directories
sudo mkdir -p $ETCD_BIN_DIR
sudo mkdir -p $ETCD_DATA_DIR
# Download and cache the etcd tgz for subsequent use
- if [ ! -f "files/etcd-$ETCD_VERSION-linux-$ETCD_ARCH/etcd" ]; then
+ if [ ! -f "$FILES/etcd-$ETCD_VERSION-linux-$ETCD_ARCH/etcd" ]; then
ETCD_DOWNLOAD_FILE=$ETCD_NAME.tar.gz
- wget $ETCD_DOWNLOAD_URL/$ETCD_VERSION/$ETCD_DOWNLOAD_FILE -O files/$ETCD_DOWNLOAD_FILE
- echo "${ETCD_SHA256} files/${ETCD_DOWNLOAD_FILE}" > files/etcd.sha256sum
+ if [ ! -f "$FILES/$ETCD_DOWNLOAD_FILE" ]; then
+ wget $ETCD_DOWNLOAD_URL/$ETCD_VERSION/$ETCD_DOWNLOAD_FILE -O $FILES/$ETCD_DOWNLOAD_FILE
+ fi
+ echo "${ETCD_SHA256} $FILES/${ETCD_DOWNLOAD_FILE}" > $FILES/etcd.sha256sum
# NOTE(sdague): this should go fatal if this fails
- sha256sum -c files/etcd.sha256sum
+ sha256sum -c $FILES/etcd.sha256sum
- tar xzvf files/$ETCD_DOWNLOAD_FILE -C files
- sudo cp files/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+ tar xzvf $FILES/$ETCD_DOWNLOAD_FILE -C $FILES
+ sudo cp $FILES/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
fi
if [ ! -f "$ETCD_BIN_DIR/etcd" ]; then
- sudo cp files/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
+ sudo cp $FILES/$ETCD_NAME/etcd $ETCD_BIN_DIR/etcd
fi
}
diff --git a/lib/glance b/lib/glance
index d6438a6..baf8c61 100644
--- a/lib/glance
+++ b/lib/glance
@@ -78,6 +78,7 @@
# Test if any Glance services are enabled
# is_glance_enabled
function is_glance_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"glance" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"g-" ]] && return 0
return 1
}
@@ -284,11 +285,13 @@
# (Re)create glance database
recreate_database glance
+ time_start "dbsync"
# Migrate glance database
$GLANCE_BIN_DIR/glance-manage --config-file $GLANCE_CONF_DIR/glance-api.conf db_sync
# Load metadata definitions
$GLANCE_BIN_DIR/glance-manage --config-file $GLANCE_CONF_DIR/glance-api.conf db_load_metadefs
+ time_stop "dbsync"
create_glance_cache_dir
}
diff --git a/lib/keystone b/lib/keystone
index 4bb6893..eb46526 100644
--- a/lib/keystone
+++ b/lib/keystone
@@ -124,9 +124,15 @@
# Security compliance
KEYSTONE_SECURITY_COMPLIANCE_ENABLED=${KEYSTONE_SECURITY_COMPLIANCE_ENABLED:-True}
KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS=${KEYSTONE_LOCKOUT_FAILURE_ATTEMPTS:-2}
-KEYSTONE_LOCKOUT_DURATION=${KEYSTONE_LOCKOUT_DURATION:-5}
+KEYSTONE_LOCKOUT_DURATION=${KEYSTONE_LOCKOUT_DURATION:-10}
KEYSTONE_UNIQUE_LAST_PASSWORD_COUNT=${KEYSTONE_UNIQUE_LAST_PASSWORD_COUNT:-2}
+# Number of bcrypt hashing rounds, increasing number exponentially increases required
+# resources to generate password hash. This is very effective way to protect from
+# bruteforce attacks. 4 is minimal value that can be specified for bcrypt and
+# it works way faster than default 12. Minimal value is great for CI and development
+# however may not be suitable for real production.
+KEYSTONE_PASSWORD_HASH_ROUNDS=${KEYSTONE_PASSWORD_HASH_ROUNDS:-4}
# Functions
# ---------
@@ -134,6 +140,7 @@
# Test if Keystone is enabled
# is_keystone_enabled
function is_keystone_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"keystone" ]] && return 1
[[ ,${ENABLED_SERVICES}, =~ ,"key", ]] && return 0
return 1
}
@@ -224,6 +231,7 @@
fi
iniset $KEYSTONE_CONF identity driver "$KEYSTONE_IDENTITY_BACKEND"
+ iniset $KEYSTONE_CONF identity password_hash_rounds $KEYSTONE_PASSWORD_HASH_ROUNDS
iniset $KEYSTONE_CONF assignment driver "$KEYSTONE_ASSIGNMENT_BACKEND"
iniset $KEYSTONE_CONF role driver "$KEYSTONE_ROLE_BACKEND"
iniset $KEYSTONE_CONF resource driver "$KEYSTONE_RESOURCE_BACKEND"
@@ -457,8 +465,10 @@
recreate_database keystone
fi
+ time_start "dbsync"
# Initialize keystone database
$KEYSTONE_BIN_DIR/keystone-manage --config-file $KEYSTONE_CONF db_sync
+ time_stop "dbsync"
if [[ "$KEYSTONE_TOKEN_FORMAT" == "pki" || "$KEYSTONE_TOKEN_FORMAT" == "pkiz" ]]; then
# Set up certificates
diff --git a/lib/libraries b/lib/libraries
new file mode 100644
index 0000000..4ceb804
--- /dev/null
+++ b/lib/libraries
@@ -0,0 +1,143 @@
+#!/bin/bash
+#
+# lib/oslo
+#
+# Functions to install libraries from git
+#
+# We need this to handle the fact that projects would like to use
+# pre-released versions of oslo libraries.
+
+# Dependencies:
+#
+# - ``functions`` file
+
+# ``stack.sh`` calls the entry points in this order:
+#
+# - install_libraries
+
+# Save trace setting
+_XTRACE_LIB_LIBRARIES=$(set +o | grep xtrace)
+set +o xtrace
+
+
+# Defaults
+# --------
+GITDIR["automaton"]=$DEST/automaton
+GITDIR["castellan"]=$DEST/castellan
+GITDIR["cliff"]=$DEST/cliff
+GITDIR["cursive"]=$DEST/cursive
+GITDIR["debtcollector"]=$DEST/debtcollector
+GITDIR["futurist"]=$DEST/futurist
+GITDIR["os-client-config"]=$DEST/os-client-config
+GITDIR["osc-lib"]=$DEST/osc-lib
+GITDIR["oslo.cache"]=$DEST/oslo.cache
+GITDIR["oslo.concurrency"]=$DEST/oslo.concurrency
+GITDIR["oslo.config"]=$DEST/oslo.config
+GITDIR["oslo.context"]=$DEST/oslo.context
+GITDIR["oslo.db"]=$DEST/oslo.db
+GITDIR["oslo.i18n"]=$DEST/oslo.i18n
+GITDIR["oslo.log"]=$DEST/oslo.log
+GITDIR["oslo.messaging"]=$DEST/oslo.messaging
+GITDIR["oslo.middleware"]=$DEST/oslo.middleware
+GITDIR["oslo.policy"]=$DEST/oslo.policy
+GITDIR["oslo.privsep"]=$DEST/oslo.privsep
+GITDIR["oslo.reports"]=$DEST/oslo.reports
+GITDIR["oslo.rootwrap"]=$DEST/oslo.rootwrap
+GITDIR["oslo.serialization"]=$DEST/oslo.serialization
+GITDIR["oslo.service"]=$DEST/oslo.service
+GITDIR["oslo.utils"]=$DEST/oslo.utils
+GITDIR["oslo.versionedobjects"]=$DEST/oslo.versionedobjects
+GITDIR["oslo.vmware"]=$DEST/oslo.vmware
+GITDIR["osprofiler"]=$DEST/osprofiler
+GITDIR["pycadf"]=$DEST/pycadf
+GITDIR["python-openstacksdk"]=$DEST/python-openstacksdk
+GITDIR["stevedore"]=$DEST/stevedore
+GITDIR["taskflow"]=$DEST/taskflow
+GITDIR["tooz"]=$DEST/tooz
+
+# Non oslo libraries are welcomed below as well, this prevents
+# duplication of this code.
+GITDIR["os-brick"]=$DEST/os-brick
+GITDIR["os-traits"]=$DEST/os-traits
+
+# Support entry points installation of console scripts
+OSLO_BIN_DIR=$(get_python_exec_prefix)
+
+
+# Functions
+# ---------
+
+function _install_lib_from_source {
+ local name=$1
+ if use_library_from_git "$name"; then
+ git_clone_by_name "$name"
+ setup_dev_lib "$name"
+ fi
+}
+
+# install_oslo - install libraries that oslo needs
+function install_oslo {
+ install_libs
+}
+
+# install_libs() - Install additional libraries that we need and want
+# on all environments. Some will only install here if from source,
+# others will always install.
+function install_libs {
+ _install_lib_from_source "automaton"
+ _install_lib_from_source "castellan"
+ _install_lib_from_source "cliff"
+ _install_lib_from_source "cursive"
+ _install_lib_from_source "debtcollector"
+ _install_lib_from_source "futurist"
+ _install_lib_from_source "osc-lib"
+ _install_lib_from_source "os-client-config"
+ _install_lib_from_source "oslo.cache"
+ _install_lib_from_source "oslo.concurrency"
+ _install_lib_from_source "oslo.config"
+ _install_lib_from_source "oslo.context"
+ _install_lib_from_source "oslo.db"
+ _install_lib_from_source "oslo.i18n"
+ _install_lib_from_source "oslo.log"
+ _install_lib_from_source "oslo.messaging"
+ _install_lib_from_source "oslo.middleware"
+ _install_lib_from_source "oslo.policy"
+ _install_lib_from_source "oslo.privsep"
+ _install_lib_from_source "oslo.reports"
+ _install_lib_from_source "oslo.rootwrap"
+ _install_lib_from_source "oslo.serialization"
+ _install_lib_from_source "oslo.service"
+ _install_lib_from_source "oslo.utils"
+ _install_lib_from_source "oslo.versionedobjects"
+ _install_lib_from_source "oslo.vmware"
+ _install_lib_from_source "osprofiler"
+ _install_lib_from_source "pycadf"
+ _install_lib_from_source "python-openstacksdk"
+ _install_lib_from_source "stevedore"
+ _install_lib_from_source "taskflow"
+ _install_lib_from_source "tooz"
+ # installation of additional libraries
+ #
+ # os-traits for nova
+ _install_lib_from_source "os-brick"
+ _install_lib_from_source "os-traits"
+ #
+ # python client libraries we might need from git can go here
+ _install_lib_from_source "python-barbicanclient"
+
+
+ # etcd (because tooz does not have a hard dependency on these)
+ #
+ # NOTE(sdague): this is currently a work around because tooz
+ # doesn't pull in etcd3.
+ pip_install etcd3
+ pip_install etcd3gw
+}
+
+# Restore xtrace
+$_XTRACE_LIB_LIBRARIES
+
+# Tell emacs to use shell-script-mode
+## Local variables:
+## mode: shell-script
+## End:
diff --git a/lib/neutron b/lib/neutron
index efca880..2a660ec 100644
--- a/lib/neutron
+++ b/lib/neutron
@@ -91,6 +91,7 @@
# Test if any Neutron services are enabled
# is_neutron_enabled
function is_neutron_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"neutron" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"neutron-" || ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
return 1
}
@@ -98,6 +99,7 @@
# Test if any Neutron services are enabled
# is_neutron_enabled
function is_neutron_legacy_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"neutron" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
return 1
}
@@ -334,8 +336,10 @@
recreate_database neutron
+ time_start "dbsync"
# Run Neutron db migrations
$NEUTRON_BIN_DIR/neutron-db-manage upgrade heads
+ time_stop "dbsync"
create_neutron_cache_dir
}
diff --git a/lib/neutron-legacy b/lib/neutron-legacy
index 67cf110..784f3a8 100644
--- a/lib/neutron-legacy
+++ b/lib/neutron-legacy
@@ -417,8 +417,10 @@
# init_mutnauq() - Initialize databases, etc.
function init_mutnauq {
recreate_database $Q_DB_NAME
+ time_start "dbsync"
# Run Neutron db migrations
$NEUTRON_BIN_DIR/neutron-db-manage --config-file $NEUTRON_CONF --config-file /$Q_PLUGIN_CONF_FILE upgrade head
+ time_stop "dbsync"
}
# install_mutnauq() - Collect source and prepare
diff --git a/lib/nova b/lib/nova
index e580abb..788588f 100644
--- a/lib/nova
+++ b/lib/nova
@@ -53,10 +53,18 @@
NOVA_CONF_DIR=/etc/nova
NOVA_CONF=$NOVA_CONF_DIR/nova.conf
NOVA_CELLS_CONF=$NOVA_CONF_DIR/nova-cells.conf
+NOVA_CPU_CONF=$NOVA_CONF_DIR/nova-cpu.conf
NOVA_FAKE_CONF=$NOVA_CONF_DIR/nova-fake.conf
NOVA_CELLS_DB=${NOVA_CELLS_DB:-nova_cell}
NOVA_API_DB=${NOVA_API_DB:-nova_api}
+# The total number of cells we expect. Must be greater than one and doesn't
+# count cell0.
+NOVA_NUM_CELLS=${NOVA_NUM_CELLS:-1}
+# Our cell index, so we know what rabbit vhost to connect to.
+# This should be in the range of 1-$NOVA_NUM_CELLS
+NOVA_CPU_CELL=${NOVA_CPU_CELL:-1}
+
NOVA_API_PASTE_INI=${NOVA_API_PASTE_INI:-$NOVA_CONF_DIR/api-paste.ini}
if is_suse; then
@@ -175,6 +183,7 @@
# Test if any Nova services are enabled
# is_nova_enabled
function is_nova_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"nova" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"n-" ]] && return 0
return 1
}
@@ -453,8 +462,8 @@
fi
iniset $NOVA_CONF wsgi api_paste_config "$NOVA_API_PASTE_INI"
iniset $NOVA_CONF DEFAULT rootwrap_config "$NOVA_CONF_DIR/rootwrap.conf"
- iniset $NOVA_CONF DEFAULT scheduler_driver "$SCHEDULER"
- iniset $NOVA_CONF DEFAULT scheduler_default_filters "$FILTERS"
+ iniset $NOVA_CONF scheduler driver "$SCHEDULER"
+ iniset $NOVA_CONF filter_scheduler enabled_filters "$FILTERS"
iniset $NOVA_CONF DEFAULT default_floating_pool "$PUBLIC_NETWORK_NAME"
if [[ $SERVICE_IP_VERSION == 6 ]]; then
iniset $NOVA_CONF DEFAULT my_ip "$HOST_IPV6"
@@ -478,7 +487,7 @@
# require them running on the host. The ensures that n-cpu doesn't
# leak a need to use the db in a multinode scenario.
if is_service_enabled n-api n-cond n-sched; then
- iniset $NOVA_CONF database connection `database_connection_url nova`
+ iniset $NOVA_CONF database connection `database_connection_url nova_cell0`
iniset $NOVA_CONF api_database connection `database_connection_url nova_api`
fi
@@ -613,6 +622,20 @@
if [ "$NOVA_USE_SERVICE_TOKEN" == "True" ]; then
init_nova_service_user_conf
fi
+
+ if is_service_enabled n-cond; then
+ for i in $(seq 1 $NOVA_NUM_CELLS); do
+ local conf
+ local vhost
+ conf=$(conductor_conf $i)
+ vhost="nova_cell${i}"
+ iniset $conf database connection `database_connection_url nova_cell${i}`
+ iniset $conf conductor workers "$API_WORKERS"
+ iniset $conf DEFAULT debug "$ENABLE_DEBUG_LOG_LEVEL"
+ rpc_backend_add_vhost $vhost
+ iniset_rpc_backend nova $conf DEFAULT $vhost
+ done
+ fi
}
function init_nova_service_user_conf {
@@ -627,6 +650,11 @@
iniset $NOVA_CONF service_user auth_strategy keystone
}
+function conductor_conf {
+ local cell="$1"
+ echo "${NOVA_CONF_DIR}/nova_cell${cell}.conf"
+}
+
function init_nova_cells {
if is_service_enabled n-cell; then
cp $NOVA_CONF $NOVA_CELLS_CONF
@@ -648,7 +676,9 @@
iniset $NOVA_CELLS_CONF DEFAULT enabled_apis metadata
fi
+ time_start "dbsync"
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CELLS_CONF db sync
+ time_stop "dbsync"
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CELLS_CONF cell create --name=region --cell_type=parent --username=$RABBIT_USERID --hostname=$RABBIT_HOST --port=5672 --password=$RABBIT_PASSWORD --virtual_host=/ --woffset=0 --wscale=1
$NOVA_BIN_DIR/nova-manage cell create --name=child --cell_type=child --username=$RABBIT_USERID --hostname=$RABBIT_HOST --port=5672 --password=$RABBIT_PASSWORD --virtual_host=child_cell --woffset=0 --wscale=1
@@ -691,8 +721,6 @@
recreate_database $NOVA_API_DB
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CONF api_db sync
- # (Re)create nova databases
- recreate_database nova
recreate_database nova_cell0
# map_cell0 will create the cell mapping record in the nova_api DB so
@@ -704,6 +732,12 @@
# Migrate nova and nova_cell0 databases.
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CONF db sync
+ # (Re)create nova databases
+ for i in $(seq 1 $NOVA_NUM_CELLS); do
+ recreate_database nova_cell${i}
+ $NOVA_BIN_DIR/nova-manage --config-file $(conductor_conf $i) db sync
+ done
+
if is_service_enabled n-cell; then
recreate_database $NOVA_CELLS_DB
fi
@@ -712,9 +746,13 @@
# Needed for flavor conversion
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CONF db online_data_migrations
+ # FIXME(danms): Should this be configurable?
+ iniset $NOVA_CONF workarounds disable_group_policy_check_upcall True
+
# create the cell1 cell for the main nova db where the hosts live
- nova-manage cell_v2 create_cell --transport-url $(get_transport_url) \
- --name 'cell1'
+ for i in $(seq 1 $NOVA_NUM_CELLS); do
+ nova-manage --config-file $NOVA_CONF --config-file $(conductor_conf $i) cell_v2 create_cell --name "cell$i"
+ done
fi
create_nova_cache_dir
@@ -822,25 +860,38 @@
# start_nova_compute() - Start the compute process
function start_nova_compute {
+ local nomulticellflag="$1"
# Hack to set the path for rootwrap
local old_path=$PATH
export PATH=$NOVA_BIN_DIR:$PATH
if is_service_enabled n-cell; then
local compute_cell_conf=$NOVA_CELLS_CONF
+ # NOTE(danms): Don't setup conductor fleet for cellsv1
+ nomulticellflag='nomulticell'
else
local compute_cell_conf=$NOVA_CONF
fi
+ if [ "$nomulticellflag" = 'nomulticell' ]; then
+ # NOTE(danms): Grenade doesn't setup multi-cell rabbit, so
+ # skip these bits and use the normal config.
+ NOVA_CPU_CONF=$compute_cell_conf
+ echo "Skipping multi-cell conductor fleet setup"
+ else
+ cp $compute_cell_conf $NOVA_CPU_CONF
+ iniset_rpc_backend nova $NOVA_CPU_CONF DEFAULT "nova_cell${NOVA_CPU_CELL}"
+ fi
+
if [[ "$VIRT_DRIVER" = 'libvirt' ]]; then
# The group **$LIBVIRT_GROUP** is added to the current user in this script.
# ``sg`` is used in run_process to execute nova-compute as a member of the
# **$LIBVIRT_GROUP** group.
- run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $compute_cell_conf" $LIBVIRT_GROUP
+ run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $NOVA_CPU_CONF" $LIBVIRT_GROUP
elif [[ "$VIRT_DRIVER" = 'lxd' ]]; then
- run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $compute_cell_conf" $LXD_GROUP
+ run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $NOVA_CPU_CONF" $LXD_GROUP
elif [[ "$VIRT_DRIVER" = 'docker' || "$VIRT_DRIVER" = 'zun' ]]; then
- run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $compute_cell_conf" $DOCKER_GROUP
+ run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $NOVA_CPU_CONF" $DOCKER_GROUP
elif [[ "$VIRT_DRIVER" = 'fake' ]]; then
local i
for i in `seq 1 $NUMBER_FAKE_NOVA_COMPUTE`; do
@@ -849,13 +900,13 @@
# gets its own configuration and own log file.
local fake_conf="${NOVA_FAKE_CONF}-${i}"
iniset $fake_conf DEFAULT nhost "${HOSTNAME}${i}"
- run_process "n-cpu-${i}" "$NOVA_BIN_DIR/nova-compute --config-file $compute_cell_conf --config-file $fake_conf"
+ run_process "n-cpu-${i}" "$NOVA_BIN_DIR/nova-compute --config-file $NOVA_CPU_CONF --config-file $fake_conf"
done
else
if is_service_enabled n-cpu && [[ -r $NOVA_PLUGINS/hypervisor-$VIRT_DRIVER ]]; then
start_nova_hypervisor
fi
- run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $compute_cell_conf"
+ run_process n-cpu "$NOVA_BIN_DIR/nova-compute --config-file $NOVA_CPU_CONF"
fi
export PATH=$old_path
@@ -875,7 +926,6 @@
fi
# ``run_process`` checks ``is_service_enabled``, it is not needed here
- run_process n-cond "$NOVA_BIN_DIR/nova-conductor --config-file $compute_cell_conf"
run_process n-cell-region "$NOVA_BIN_DIR/nova-cells --config-file $api_cell_conf"
run_process n-cell-child "$NOVA_BIN_DIR/nova-cells --config-file $compute_cell_conf"
@@ -898,8 +948,38 @@
export PATH=$old_path
}
+function enable_nova_fleet {
+ if is_service_enabled n-cond; then
+ enable_service n-super-cond
+ for i in $(seq 1 $NOVA_NUM_CELLS); do
+ enable_service n-cond-cell${i}
+ done
+ fi
+}
+
+function start_nova_conductor {
+ if is_service_enabled n-cell; then
+ echo "Starting nova-conductor in a cellsv1-compatible way"
+ run_process n-cond "$NOVA_BIN_DIR/nova-conductor --config-file $NOVA_CELLS_CONF"
+ return
+ fi
+
+ enable_nova_fleet
+ if is_service_enabled n-super-cond; then
+ run_process n-super-cond "$NOVA_BIN_DIR/nova-conductor --config-file $NOVA_CONF"
+ fi
+ for i in $(seq 1 $NOVA_NUM_CELLS); do
+ if is_service_enabled n-cond-cell${i}; then
+ local conf
+ conf=$(conductor_conf $i)
+ run_process n-cond-cell${i} "$NOVA_BIN_DIR/nova-conductor --config-file $conf"
+ fi
+ done
+}
+
function start_nova {
start_nova_rest
+ start_nova_conductor
start_nova_compute
}
@@ -928,14 +1008,24 @@
# Kill the nova screen windows
# Some services are listed here twice since more than one instance
# of a service may be running in certain configs.
- for serv in n-api n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cell n-cell n-api-meta n-sproxy; do
+ for serv in n-api n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cell n-cell n-api-meta n-sproxy; do
stop_process $serv
done
}
+function stop_nova_conductor {
+ enable_nova_fleet
+ for srv in n-super-cond $(seq -f n-cond-cell%0.f 1 $NOVA_NUM_CELLS); do
+ if is_service_enabled $srv; then
+ stop_process $srv
+ fi
+ done
+}
+
# stop_nova() - Stop running processes (non-screen)
function stop_nova {
stop_nova_rest
+ stop_nova_conductor
stop_nova_compute
}
diff --git a/lib/nova_plugins/hypervisor-fake b/lib/nova_plugins/hypervisor-fake
index f9b95c1..49c8dee 100644
--- a/lib/nova_plugins/hypervisor-fake
+++ b/lib/nova_plugins/hypervisor-fake
@@ -49,7 +49,7 @@
iniset $NOVA_CONF DEFAULT quota_security_groups -1
iniset $NOVA_CONF DEFAULT quota_security_group_rules -1
iniset $NOVA_CONF DEFAULT quota_key_pairs -1
- iniset $NOVA_CONF DEFAULT scheduler_default_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"
+ iniset $NOVA_CONF filter_scheduler enabled_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"
}
# install_nova_hypervisor() - Install external components
diff --git a/lib/nova_plugins/hypervisor-xenserver b/lib/nova_plugins/hypervisor-xenserver
index 880b87f..6f79e4f 100644
--- a/lib/nova_plugins/hypervisor-xenserver
+++ b/lib/nova_plugins/hypervisor-xenserver
@@ -84,14 +84,6 @@
* * * * * /root/rotate_xen_guest_logs.sh >/dev/null 2>&1
CRONTAB
- # Create directories for kernels and images
- {
- echo "set -eux"
- cat $TOP_DIR/tools/xen/functions
- echo "create_directory_for_images"
- echo "create_directory_for_kernels"
- echo "install_conntrack_tools"
- } | $ssh_dom0
}
# install_nova_hypervisor() - Install external components
diff --git a/lib/os_brick b/lib/os_brick
deleted file mode 100644
index d1cca4a..0000000
--- a/lib/os_brick
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash
-#
-# lib/os_brick
-# Install **os-brick** python module from source
-
-# Dependencies:
-#
-# - functions
-# - DEST, DATA_DIR must be defined
-
-# stack.sh
-# ---------
-# - install_os_brick
-
-# Save trace setting
-_XTRACE_OS_BRICK=$(set +o | grep xtrace)
-set +o xtrace
-
-
-GITDIR["os-brick"]=$DEST/os-brick
-
-# Install os_brick from git only if requested, otherwise it will be pulled from
-# pip repositories by requirements of projects that need it.
-function install_os_brick {
- if use_library_from_git "os-brick"; then
- git_clone_by_name "os-brick"
- setup_dev_lib "os-brick"
- fi
-}
-
-# Restore xtrace
-$_XTRACE_OS_BRICK
\ No newline at end of file
diff --git a/lib/oslo b/lib/oslo
index 2895503..3ae64c8 100644
--- a/lib/oslo
+++ b/lib/oslo
@@ -6,110 +6,6 @@
#
# We need this to handle the fact that projects would like to use
# pre-released versions of oslo libraries.
-
-# Dependencies:
#
-# - ``functions`` file
-
-# ``stack.sh`` calls the entry points in this order:
-#
-# - install_oslo
-
-# Save trace setting
-_XTRACE_LIB_OSLO=$(set +o | grep xtrace)
-set +o xtrace
-
-
-# Defaults
-# --------
-GITDIR["automaton"]=$DEST/automaton
-GITDIR["castellan"]=$DEST/castellan
-GITDIR["cliff"]=$DEST/cliff
-GITDIR["cursive"]=$DEST/cursive
-GITDIR["debtcollector"]=$DEST/debtcollector
-GITDIR["futurist"]=$DEST/futurist
-GITDIR["os-client-config"]=$DEST/os-client-config
-GITDIR["osc-lib"]=$DEST/osc-lib
-GITDIR["oslo.cache"]=$DEST/oslo.cache
-GITDIR["oslo.concurrency"]=$DEST/oslo.concurrency
-GITDIR["oslo.config"]=$DEST/oslo.config
-GITDIR["oslo.context"]=$DEST/oslo.context
-GITDIR["oslo.db"]=$DEST/oslo.db
-GITDIR["oslo.i18n"]=$DEST/oslo.i18n
-GITDIR["oslo.log"]=$DEST/oslo.log
-GITDIR["oslo.messaging"]=$DEST/oslo.messaging
-GITDIR["oslo.middleware"]=$DEST/oslo.middleware
-GITDIR["oslo.policy"]=$DEST/oslo.policy
-GITDIR["oslo.privsep"]=$DEST/oslo.privsep
-GITDIR["oslo.reports"]=$DEST/oslo.reports
-GITDIR["oslo.rootwrap"]=$DEST/oslo.rootwrap
-GITDIR["oslo.serialization"]=$DEST/oslo.serialization
-GITDIR["oslo.service"]=$DEST/oslo.service
-GITDIR["oslo.utils"]=$DEST/oslo.utils
-GITDIR["oslo.versionedobjects"]=$DEST/oslo.versionedobjects
-GITDIR["oslo.vmware"]=$DEST/oslo.vmware
-GITDIR["osprofiler"]=$DEST/osprofiler
-GITDIR["pycadf"]=$DEST/pycadf
-GITDIR["python-openstacksdk"]=$DEST/python-openstacksdk
-GITDIR["stevedore"]=$DEST/stevedore
-GITDIR["taskflow"]=$DEST/taskflow
-GITDIR["tooz"]=$DEST/tooz
-
-# Support entry points installation of console scripts
-OSLO_BIN_DIR=$(get_python_exec_prefix)
-
-
-# Functions
-# ---------
-
-function _do_install_oslo_lib {
- local name=$1
- if use_library_from_git "$name"; then
- git_clone_by_name "$name"
- setup_dev_lib "$name"
- fi
-}
-
-# install_oslo() - Collect source and prepare
-function install_oslo {
- _do_install_oslo_lib "automaton"
- _do_install_oslo_lib "castellan"
- _do_install_oslo_lib "cliff"
- _do_install_oslo_lib "cursive"
- _do_install_oslo_lib "debtcollector"
- _do_install_oslo_lib "futurist"
- _do_install_oslo_lib "osc-lib"
- _do_install_oslo_lib "os-client-config"
- _do_install_oslo_lib "oslo.cache"
- _do_install_oslo_lib "oslo.concurrency"
- _do_install_oslo_lib "oslo.config"
- _do_install_oslo_lib "oslo.context"
- _do_install_oslo_lib "oslo.db"
- _do_install_oslo_lib "oslo.i18n"
- _do_install_oslo_lib "oslo.log"
- _do_install_oslo_lib "oslo.messaging"
- _do_install_oslo_lib "oslo.middleware"
- _do_install_oslo_lib "oslo.policy"
- _do_install_oslo_lib "oslo.privsep"
- _do_install_oslo_lib "oslo.reports"
- _do_install_oslo_lib "oslo.rootwrap"
- _do_install_oslo_lib "oslo.serialization"
- _do_install_oslo_lib "oslo.service"
- _do_install_oslo_lib "oslo.utils"
- _do_install_oslo_lib "oslo.versionedobjects"
- _do_install_oslo_lib "oslo.vmware"
- _do_install_oslo_lib "osprofiler"
- _do_install_oslo_lib "pycadf"
- _do_install_oslo_lib "python-openstacksdk"
- _do_install_oslo_lib "stevedore"
- _do_install_oslo_lib "taskflow"
- _do_install_oslo_lib "tooz"
-}
-
-# Restore xtrace
-$_XTRACE_LIB_OSLO
-
-# Tell emacs to use shell-script-mode
-## Local variables:
-## mode: shell-script
-## End:
+# Included for compatibility with grenade, remove in Queens
+source $TOP_DIR/lib/libraries
diff --git a/lib/placement b/lib/placement
index ad12824..8adbbde 100644
--- a/lib/placement
+++ b/lib/placement
@@ -149,7 +149,9 @@
function init_placement {
if [ "$PLACEMENT_DB_ENABLED" != False ]; then
recreate_database placement
+ time_start "dbsync"
$NOVA_BIN_DIR/nova-manage --config-file $NOVA_CONF api_db sync
+ time_stop "dbsync"
fi
create_placement_accounts
}
diff --git a/lib/swift b/lib/swift
index 8fad6b8..e247f15 100644
--- a/lib/swift
+++ b/lib/swift
@@ -174,6 +174,7 @@
# Test if any Swift services are enabled
# is_swift_enabled
function is_swift_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"swift" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"s-" ]] && return 0
return 1
}
@@ -541,6 +542,10 @@
else
iniset ${testfile} func_test auth_prefix /identity/v2.0/
fi
+ if is_service_enabled tls-proxy; then
+ iniset ${testfile} func_test cafile ${SSL_BUNDLE_FILE}
+ iniset ${testfile} func_test web_front_end apache2
+ fi
fi
local user_group
diff --git a/lib/template b/lib/template
index b92fb40..25d653c 100644
--- a/lib/template
+++ b/lib/template
@@ -41,6 +41,7 @@
# Test if any XXXX services are enabled
# is_XXXX_enabled
function is_XXXX_enabled {
+ [[ ,${DISABLED_SERVICES} =~ ,"XXXX" ]] && return 1
[[ ,${ENABLED_SERVICES} =~ ,"XX-" ]] && return 0
return 1
}
diff --git a/stack.sh b/stack.sh
index 048acf8..326d186 100755
--- a/stack.sh
+++ b/stack.sh
@@ -27,11 +27,37 @@
# Make sure custom grep options don't get in the way
unset GREP_OPTIONS
-# Sanitize language settings to avoid commands bailing out
-# with "unsupported locale setting" errors.
+# NOTE(sdague): why do we explicitly set locale when running stack.sh?
+#
+# Devstack is written in bash, and many functions used throughout
+# devstack process text comming off a command (like the ip command)
+# and do transforms using grep, sed, cut, awk on the strings that are
+# returned. Many of these programs are interationalized, which is
+# great for end users, but means that the strings that devstack
+# functions depend upon might not be there in other locales. We thus
+# need to pin the world to an english basis during the runs.
+#
+# Previously we used the C locale for this, every system has it, and
+# it gives us a stable sort order. It does however mean that we
+# effectively drop unicode support.... boo! :(
+#
+# With python3 being more unicode aware by default, that's not the
+# right option. While there is a C.utf8 locale, some distros are
+# shipping it as C.UTF8 for extra confusingness. And it's support
+# isn't super clear across distros. This is made more challenging when
+# trying to support both out of the box distros, and the gate which
+# uses diskimage builder to build disk images in a different way than
+# the distros do.
+#
+# So... en_US.utf8 it is. That's existed for a very long time. It is a
+# compromise position, but it is the least worse idea at the time of
+# this comment.
+#
+# We also have to unset other variables that might impact LC_ALL
+# taking effect.
unset LANG
unset LANGUAGE
-LC_ALL=C
+LC_ALL=en_US.utf8
export LC_ALL
# Make sure umask is sane
@@ -192,7 +218,7 @@
# Warn users who aren't on an explicitly supported distro, but allow them to
# override check and attempt installation with ``FORCE=yes ./stack``
-if [[ ! ${DISTRO} =~ (xenial|yakkety|zesty|stretch|jessie|f24|f25|rhel7|kvmibm1) ]]; then
+if [[ ! ${DISTRO} =~ (xenial|yakkety|zesty|stretch|jessie|f24|f25|opensuse-42.2|rhel7|kvmibm1) ]]; then
echo "WARNING: this script has not been tested on $DISTRO"
if [[ "$FORCE" != "yes" ]]; then
die $LINENO "If you wish to run this script anyway run with FORCE=yes"
@@ -498,6 +524,11 @@
kill 2>&1 $jobs
fi
+ #Remove timing data file
+ if [ -f "$OSCWRAP_TIMER_FILE" ] ; then
+ rm "$OSCWRAP_TIMER_FILE"
+ fi
+
# Kill the last spinner process
kill_spinner
@@ -561,7 +592,7 @@
# Source project function libraries
source $TOP_DIR/lib/infra
-source $TOP_DIR/lib/oslo
+source $TOP_DIR/lib/libraries
source $TOP_DIR/lib/lvm
source $TOP_DIR/lib/horizon
source $TOP_DIR/lib/keystone
@@ -574,7 +605,6 @@
source $TOP_DIR/lib/ldap
source $TOP_DIR/lib/dstat
source $TOP_DIR/lib/etcd3
-source $TOP_DIR/lib/os_brick
# Extras Source
# --------------
@@ -782,13 +812,17 @@
install_neutron_agent_packages
fi
+if is_service_enabled etcd3; then
+ install_etcd3
+fi
+
# Check Out and Install Source
# ----------------------------
echo_summary "Installing OpenStack project source"
-# Install Oslo libraries
-install_oslo
+# Install additional libraries
+install_libs
# Install uwsgi
install_apache_uwsgi
@@ -806,11 +840,6 @@
install_neutronclient
fi
-# Install shared libraries
-if is_service_enabled cinder nova; then
- install_os_brick
-fi
-
# Setup TLS certs
if is_service_enabled tls-proxy; then
configure_CA
@@ -913,6 +942,10 @@
pip_install_gr python-openstackclient
fi
+# Installs alias for osc so that we can collect timing for all
+# osc commands. Alias dies with stack.sh.
+install_oscwrap
+
if [[ $TRACK_DEPENDS = True ]]; then
$DEST/.venv/bin/pip freeze > $DEST/requires-post-pip
if ! diff -Nru $DEST/requires-pre-pip $DEST/requires-post-pip > $DEST/requires.diff; then
@@ -1268,7 +1301,9 @@
# Unable to use LUKS passphrase that is exactly 16 bytes long
# https://bugzilla.redhat.com/show_bug.cgi?id=1447297
if is_service_enabled nova; then
- iniset $NOVA_CONF key_manager fixed_key $(generate_hex_string 36)
+ key=$(generate_hex_string 36)
+ iniset $NOVA_CONF key_manager fixed_key "$key"
+ iniset $NOVA_CPU_CONF key_manager fixed_key "$key"
fi
# Launch the nova-api and wait for it to answer before continuing
@@ -1300,6 +1335,13 @@
$NOVA_BIN_DIR/nova-manage --config-file $NM_CONF floating create --ip_range=$TEST_FLOATING_RANGE --pool=$TEST_FLOATING_POOL
fi
+# Start placement before any of the service that are likely to want
+# to use it to manage resource providers.
+if is_service_enabled placement; then
+ echo_summary "Starting Placement"
+ start_placement
+fi
+
if is_service_enabled neutron; then
start_neutron
fi
@@ -1314,10 +1356,6 @@
start_nova
create_flavors
fi
-if is_service_enabled placement; then
- echo_summary "Starting Placement"
- start_placement
-fi
if is_service_enabled cinder; then
echo_summary "Starting Cinder"
start_cinder
diff --git a/stackrc b/stackrc
index e9b8df2..50f7c89 100644
--- a/stackrc
+++ b/stackrc
@@ -341,6 +341,11 @@
GITREPO["python-brick-cinderclient-ext"]=${BRICK_CINDERCLIENT_REPO:-${GIT_BASE}/openstack/python-brick-cinderclient-ext.git}
GITBRANCH["python-brick-cinderclient-ext"]=${BRICK_CINDERCLIENT_BRANCH:-master}
+# python barbican client library
+GITREPO["python-barbicanclient"]=${BARBICANCLIENT_REPO:-${GIT_BASE}/openstack/python-barbicanclient.git}
+GITBRANCH["python-barbicanclient"]=${BARBICANCLIENT_BRANCH:-master}
+GITDIR["python-barbicanclient"]=$DEST/python-barbicanclient
+
# python glance client library
GITREPO["python-glanceclient"]=${GLANCECLIENT_REPO:-${GIT_BASE}/openstack/python-glanceclient.git}
GITBRANCH["python-glanceclient"]=${GLANCECLIENT_BRANCH:-master}
@@ -570,6 +575,10 @@
GITBRANCH["neutron-lib"]=${NEUTRON_LIB_BRANCH:-master}
GITDIR["neutron-lib"]=$DEST/neutron-lib
+# os-traits library for resource provider traits in the placement service
+GITREPO["os-traits"]=${OS_TRAITS_REPO:-${GIT_BASE}/openstack/os-traits.git}
+GITBRANCH["os-traits"]=${OS_TRAITS_BRANCH:-master}
+
##################
#
# TripleO / Heat Agent Components
@@ -655,6 +664,8 @@
;;
esac
+# By default, devstack will use Ubuntu Cloud Archive.
+ENABLE_UBUNTU_CLOUD_ARCHIVE=$(trueorfalse True ENABLE_UBUNTU_CLOUD_ARCHIVE)
# Images
# ------
diff --git a/tests/test_libs_from_pypi.sh b/tests/test_libs_from_pypi.sh
index 608ef6a..5b4ff32 100755
--- a/tests/test_libs_from_pypi.sh
+++ b/tests/test_libs_from_pypi.sh
@@ -39,11 +39,11 @@
ALL_LIBS+=" python-openstackclient osc-lib os-client-config oslo.rootwrap"
ALL_LIBS+=" oslo.i18n oslo.utils python-openstacksdk python-swiftclient"
ALL_LIBS+=" python-neutronclient tooz ceilometermiddleware oslo.policy"
-ALL_LIBS+=" debtcollector os-brick automaton futurist oslo.service"
+ALL_LIBS+=" debtcollector os-brick os-traits automaton futurist oslo.service"
ALL_LIBS+=" oslo.cache oslo.reports osprofiler cursive"
ALL_LIBS+=" keystoneauth ironic-lib neutron-lib oslo.privsep"
ALL_LIBS+=" diskimage-builder os-vif python-brick-cinderclient-ext"
-ALL_LIBS+=" castellan"
+ALL_LIBS+=" castellan python-barbicanclient"
# Generate the above list with
# echo ${!GITREPO[@]}
diff --git a/tools/fixup_stuff.sh b/tools/fixup_stuff.sh
index d07d267..0b78bde 100755
--- a/tools/fixup_stuff.sh
+++ b/tools/fixup_stuff.sh
@@ -72,7 +72,10 @@
# We've found that Libvirt on Xenial is flaky and crashes enough to be
# a regular top e-r bug. Opt into Ubuntu Cloud Archive if on Xenial to
# get newer Libvirt.
-if [[ "$DISTRO" = "xenial" ]]; then
+# Make it possible to switch this based on an environment variable as
+# libvirt 2.5.0 doesn't handle nested virtualization quite well and this
+# is required for the trove development environment.
+if [[ "${ENABLE_UBUNTU_CLOUD_ARCHIVE}" == "True" && "$DISTRO" = "xenial" ]]; then
# This pulls in apt-add-repository
install_package "software-properties-common"
# Use UCA for newer libvirt. Should give us libvirt 2.5.0.