Add option to install everything in global venvs
Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.
There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.
Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.
Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).
Some distros still have issues, keep them using the old setup for now.
Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25
diff --git a/inc/python b/inc/python
index a24f4e9..cc6e01f 100644
--- a/inc/python
+++ b/inc/python
@@ -32,6 +32,23 @@
# Python Functions
# ================
+# Setup the global devstack virtualenvs and the associated environment
+# updates.
+function setup_devstack_virtualenv {
+ # We run devstack out of a global virtualenv.
+ if [[ ! -d $DEVSTACK_VENV ]] ; then
+ # Using system site packages to enable nova to use libguestfs.
+ # This package is currently installed via the distro and not
+ # available on pypi.
+ python$PYTHON3_VERSION -m venv --system-site-packages $DEVSTACK_VENV
+ pip_install -U pip
+ fi
+ if [[ ":$PATH:" != *":$DEVSTACK_VENV/bin:"* ]] ; then
+ export PATH="$DEVSTACK_VENV/bin:$PATH"
+ export PYTHON="$DEVSTACK_VENV/bin/python3"
+ fi
+}
+
# Get the path to the pip command.
# get_pip_command
function get_pip_command {
@@ -60,8 +77,11 @@
fi
$xtrace
- local PYTHON_PATH=/usr/local/bin
- echo $PYTHON_PATH
+ if [[ "$GLOBAL_VENV" == "True" ]] ; then
+ echo "$DEVSTACK_VENV/bin"
+ else
+ echo "/usr/local/bin"
+ fi
}
# Wrapper for ``pip install`` that only installs versions of libraries
@@ -166,6 +186,14 @@
if [[ -n ${PIP_VIRTUAL_ENV:=} && -d ${PIP_VIRTUAL_ENV} ]]; then
local cmd_pip=$PIP_VIRTUAL_ENV/bin/pip
local sudo_pip="env"
+ elif [[ "${GLOBAL_VENV}" == "True" && -d ${DEVSTACK_VENV} ]] ; then
+ # We have to check that the DEVSTACK_VENV exists because early
+ # devstack boostrapping needs to operate in a system context
+ # too bootstrap pip. Once pip is bootstrapped we create the
+ # global venv and can start to use it.
+ local cmd_pip=$DEVSTACK_VENV/bin/pip
+ local sudo_pip="env"
+ echo "Using python $PYTHON3_VERSION to install $package_dir"
else
local cmd_pip="python$PYTHON3_VERSION -m pip"
# See
@@ -439,7 +467,7 @@
pip_install $flags "$project_dir$extras"
# ensure that further actions can do things like setup.py sdist
- if [[ "$flags" == "-e" ]]; then
+ if [[ "$flags" == "-e" && "$GLOBAL_VENV" == "False" ]]; then
safe_chown -R $STACK_USER $1/*.egg-info
fi
}
diff --git a/inc/rootwrap b/inc/rootwrap
index 2a6e4b6..4c65440 100644
--- a/inc/rootwrap
+++ b/inc/rootwrap
@@ -60,6 +60,11 @@
sudo install -o root -g root -m 644 $rootwrap_conf_src_dir/rootwrap.conf /etc/${project}/rootwrap.conf
sudo sed -e "s:^filters_path=.*$:filters_path=/etc/${project}/rootwrap.d:" -i /etc/${project}/rootwrap.conf
+ # Rely on $PATH set by devstack to determine what is safe to execute
+ # by rootwrap rather than use explicit whitelist of paths in
+ # rootwrap.conf
+ sudo sed -e 's/^exec_dirs=.*/#&/' -i /etc/${project}/rootwrap.conf
+
# Set up the rootwrap sudoers
local tempfile
tempfile=$(mktemp)