Merge "Add bionic as supported distro"
diff --git a/.gitignore b/.gitignore
index d2c127d..8553b3f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@
*.log
*.log.[1-9]
*.pem
+*.pyc
.localrc.auto
.localrc.password
.prereqs
diff --git a/.zuul.yaml b/.zuul.yaml
index 5fda6b1..b772481 100644
--- a/.zuul.yaml
+++ b/.zuul.yaml
@@ -249,7 +249,7 @@
SERVICE_HOST: "{{ hostvars['controller']['nodepool']['private_ipv4'] }}"
HOST_IP: "{{ hostvars['controller']['nodepool']['private_ipv4'] }}"
PUBLIC_BRIDGE_MTU: "{{ external_bridge_mtu }}"
- devstack_localconf:
+ devstack_local_conf:
post-config:
$NEUTRON_CONF:
DEFAULT:
@@ -280,12 +280,20 @@
n-sch: true
placement-api: true
# Neutron services
- neutron-api: true
- neutron-agent: true
- neutron-dhcp: true
- neutron-l3: true
- neutron-metadata-agent: true
- neutron-metering: true
+ # We need to keep using the neutron-legacy based services for
+ # now until all issues with the new lib/neutron code are solved
+ q-agt: true
+ q-dhcp: true
+ q-l3: true
+ q-meta: true
+ q-metering: true
+ q-svc: true
+ # neutron-api: true
+ # neutron-agent: true
+ # neutron-dhcp: true
+ # neutron-l3: true
+ # neutron-metadata-agent: true
+ # neutron-metering: true
# Swift services
s-account: true
s-container: true
@@ -463,8 +471,15 @@
# being experimental any more, so we can keep this list somewhat
# pruned.
#
+ # * nova-cells-v1: maintained by nova for cells v1 (nova-cells service);
+ # nova gates on this job, it's in experimental for testing cells v1
+ # changes to devstack w/o gating on it for all devstack changes.
# * nova-next: maintained by nova for unreleased/undefaulted
# things like cellsv2 and placement-api
experimental:
jobs:
+ - nova-cells-v1:
+ irrelevant-files:
+ - ^.*\.rst$
+ - ^doc/.*$
- nova-next
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
index 1d02395..7efe4d6 100644
--- a/doc/source/configuration.rst
+++ b/doc/source/configuration.rst
@@ -41,6 +41,7 @@
- **extra** - runs after services are started and before any files in
``extra.d`` are executed
- **post-extra** - runs after files in ``extra.d`` are executed
+- **test-config** - runs after tempest (and plugins) are configured
The file is processed strictly in sequence; meta-sections may be
specified more than once but if any settings are duplicated the last to
@@ -655,7 +656,7 @@
Cells
~~~~~
-`Cells <http://wiki.openstack.org/blueprint-nova-compute-cells>`__ is
+`Cells <https://wiki.openstack.org/wiki/Blueprint-nova-compute-cells>`__ is
an alternative scaling option. To setup a cells environment add the
following to your ``localrc`` section:
diff --git a/doc/source/guides/devstack-with-lbaas-v2.rst b/doc/source/guides/devstack-with-lbaas-v2.rst
index 3592844..7dee520 100644
--- a/doc/source/guides/devstack-with-lbaas-v2.rst
+++ b/doc/source/guides/devstack-with-lbaas-v2.rst
@@ -2,7 +2,7 @@
=================================
Starting in the OpenStack Liberty release, the
-`neutron LBaaS v2 API <http://developer.openstack.org/api-ref-networking-v2-ext.html>`_
+`neutron LBaaS v2 API <https://developer.openstack.org/api-ref/network/v2/index.html>`_
is now stable while the LBaaS v1 API has been deprecated. The LBaaS v2 reference
driver is based on Octavia.
diff --git a/doc/source/guides/neutron.rst b/doc/source/guides/neutron.rst
index 092809a..1b8dccd 100644
--- a/doc/source/guides/neutron.rst
+++ b/doc/source/guides/neutron.rst
@@ -396,7 +396,7 @@
In this configuration we are defining IPV4_ADDRS_SAFE_TO_USE to be a
publicly routed IPv4 subnet. In this specific instance we are using
-the special TEST-NET-3 subnet defined in `RFC 5737 <http://tools.ietf.org/html/rfc5737>`_,
+the special TEST-NET-3 subnet defined in `RFC 5737 <https://tools.ietf.org/html/rfc5737>`_,
which is used for documentation. In your DevStack setup, IPV4_ADDRS_SAFE_TO_USE
would be a public IP address range that you or your organization has
allocated to you, so that you could access your instances from the
diff --git a/doc/source/plugin-registry.rst b/doc/source/plugin-registry.rst
index 04b7698..c21e0ef 100644
--- a/doc/source/plugin-registry.rst
+++ b/doc/source/plugin-registry.rst
@@ -35,7 +35,7 @@
ceilometer `git://git.openstack.org/openstack/ceilometer <https://git.openstack.org/cgit/openstack/ceilometer>`__
ceilometer-powervm `git://git.openstack.org/openstack/ceilometer-powervm <https://git.openstack.org/cgit/openstack/ceilometer-powervm>`__
cloudkitty `git://git.openstack.org/openstack/cloudkitty <https://git.openstack.org/cgit/openstack/cloudkitty>`__
-collectd-ceilometer-plugin `git://git.openstack.org/openstack/collectd-ceilometer-plugin <https://git.openstack.org/cgit/openstack/collectd-ceilometer-plugin>`__
+collectd-openstack-plugins `git://git.openstack.org/openstack/collectd-openstack-plugins <https://git.openstack.org/cgit/openstack/collectd-openstack-plugins>`__
congress `git://git.openstack.org/openstack/congress <https://git.openstack.org/cgit/openstack/congress>`__
cyborg `git://git.openstack.org/openstack/cyborg <https://git.openstack.org/cgit/openstack/cyborg>`__
designate `git://git.openstack.org/openstack/designate <https://git.openstack.org/cgit/openstack/designate>`__
@@ -147,13 +147,13 @@
octavia `git://git.openstack.org/openstack/octavia <https://git.openstack.org/cgit/openstack/octavia>`__
octavia-dashboard `git://git.openstack.org/openstack/octavia-dashboard <https://git.openstack.org/cgit/openstack/octavia-dashboard>`__
omni `git://git.openstack.org/openstack/omni <https://git.openstack.org/cgit/openstack/omni>`__
+openstacksdk `git://git.openstack.org/openstack/openstacksdk <https://git.openstack.org/cgit/openstack/openstacksdk>`__
os-xenapi `git://git.openstack.org/openstack/os-xenapi <https://git.openstack.org/cgit/openstack/os-xenapi>`__
osprofiler `git://git.openstack.org/openstack/osprofiler <https://git.openstack.org/cgit/openstack/osprofiler>`__
oswin-tempest-plugin `git://git.openstack.org/openstack/oswin-tempest-plugin <https://git.openstack.org/cgit/openstack/oswin-tempest-plugin>`__
panko `git://git.openstack.org/openstack/panko <https://git.openstack.org/cgit/openstack/panko>`__
patrole `git://git.openstack.org/openstack/patrole <https://git.openstack.org/cgit/openstack/patrole>`__
picasso `git://git.openstack.org/openstack/picasso <https://git.openstack.org/cgit/openstack/picasso>`__
-python-openstacksdk `git://git.openstack.org/openstack/python-openstacksdk <https://git.openstack.org/cgit/openstack/python-openstacksdk>`__
qinling `git://git.openstack.org/openstack/qinling <https://git.openstack.org/cgit/openstack/qinling>`__
rally `git://git.openstack.org/openstack/rally <https://git.openstack.org/cgit/openstack/rally>`__
rally-openstack `git://git.openstack.org/openstack/rally-openstack <https://git.openstack.org/cgit/openstack/rally-openstack>`__
diff --git a/functions-common b/functions-common
index 279cfcf..b1b0995 100644
--- a/functions-common
+++ b/functions-common
@@ -2304,12 +2304,7 @@
function cleanup_oscwrap {
local total=0
- if python3_enabled ; then
- local python=python3
- else
- local python=python
- fi
- total=$(cat $OSCWRAP_TIMER_FILE | $python -c "import sys; print(sum(int(l) for l in sys.stdin))")
+ total=$(cat $OSCWRAP_TIMER_FILE | $PYTHON -c "import sys; print(sum(int(l) for l in sys.stdin))")
_TIME_TOTAL["osc"]=$total
rm $OSCWRAP_TIMER_FILE
}
diff --git a/inc/python b/inc/python
index e074ea4..ec4233b 100644
--- a/inc/python
+++ b/inc/python
@@ -411,12 +411,6 @@
function lib_installed_from_git {
local name=$1
local safe_name
- # TODO(mordred) This is a special case for python-openstacksdk, where the
- # repo name and the pip name do not match. We should either add systemic
- # support for providing aliases, or we should rename the git repo.
- if [[ $name == 'python-openstacksdk' ]] ; then
- name=openstacksdk
- fi
safe_name=$(python -c "from pkg_resources import safe_name; \
print(safe_name('${name}'))")
# Note "pip freeze" doesn't always work here, because it tries to
diff --git a/lib/cinder b/lib/cinder
index c0356fe..3a8097f 100644
--- a/lib/cinder
+++ b/lib/cinder
@@ -227,7 +227,6 @@
configure_auth_token_middleware $CINDER_CONF cinder $CINDER_AUTH_CACHE_DIR
- iniset $CINDER_CONF DEFAULT auth_strategy keystone
iniset $CINDER_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
iniset $CINDER_CONF DEFAULT target_helper "$CINDER_ISCSI_HELPER"
@@ -541,7 +540,17 @@
local be be_name
for be in ${CINDER_ENABLED_BACKENDS//,/ }; do
be_name=${be##*:}
- openstack --os-region-name="$REGION_NAME" volume type create --property volume_backend_name="${be_name}" ${be_name}
+ # NOTE (e0ne): openstack client doesn't work with cinder in noauth mode
+ if is_service_enabled keystone; then
+ openstack --os-region-name="$REGION_NAME" volume type create --property volume_backend_name="${be_name}" ${be_name}
+ else
+ # TODO (e0ne): use openstack client once it will support cinder in noauth mode:
+ # https://bugs.launchpad.net/python-cinderclient/+bug/1755279
+ local cinder_url
+ cinder_url=$CINDER_SERVICE_PROTOCOL://$SERVICE_HOST:$CINDER_SERVICE_PORT/v3
+ OS_USER_ID=$OS_USERNAME OS_PROJECT_ID=$OS_PROJECT_NAME cinder --os-auth-type noauth --os-endpoint=$cinder_url type-create ${be_name}
+ OS_USER_ID=$OS_USERNAME OS_PROJECT_ID=$OS_PROJECT_NAME cinder --os-auth-type noauth --os-endpoint=$cinder_url type-key ${be_name} set volume_backend_name=${be_name}
+ fi
done
fi
}
diff --git a/lib/libraries b/lib/libraries
index 6d52f64..52ec784 100644
--- a/lib/libraries
+++ b/lib/libraries
@@ -28,6 +28,7 @@
GITDIR["cursive"]=$DEST/cursive
GITDIR["debtcollector"]=$DEST/debtcollector
GITDIR["futurist"]=$DEST/futurist
+GITDIR["openstacksdk"]=$DEST/openstacksdk
GITDIR["os-client-config"]=$DEST/os-client-config
GITDIR["osc-lib"]=$DEST/osc-lib
GITDIR["osc-placement"]=$DEST/osc-placement
@@ -51,7 +52,6 @@
GITDIR["oslo.vmware"]=$DEST/oslo.vmware
GITDIR["osprofiler"]=$DEST/osprofiler
GITDIR["pycadf"]=$DEST/pycadf
-GITDIR["python-openstacksdk"]=$DEST/python-openstacksdk
GITDIR["stevedore"]=$DEST/stevedore
GITDIR["taskflow"]=$DEST/taskflow
GITDIR["tooz"]=$DEST/tooz
diff --git a/lib/neutron b/lib/neutron
index 0834792..cef8d1f 100644
--- a/lib/neutron
+++ b/lib/neutron
@@ -32,6 +32,17 @@
NEUTRON_DIR=$DEST/neutron
NEUTRON_AUTH_CACHE_DIR=${NEUTRON_AUTH_CACHE_DIR:-/var/cache/neutron}
+NEUTRON_DISTRIBUTED_ROUTING=$(trueorfalse False NEUTRON_DISTRIBUTED_ROUTING)
+# Distributed Virtual Router (DVR) configuration
+# Can be:
+# - ``legacy`` - No DVR functionality
+# - ``dvr_snat`` - Controller or single node DVR
+# - ``dvr`` - Compute node in multi-node DVR
+# - ``dvr_no_external`` - Compute node in multi-node DVR, no external network
+#
+# Default is 'dvr_snat' since it can handle both DVR and legacy routers
+NEUTRON_DVR_MODE=${NEUTRON_DVR_MODE:-dvr_snat}
+
NEUTRON_BIN_DIR=$(get_python_exec_prefix)
NEUTRON_DHCP_BINARY="neutron-dhcp-agent"
@@ -174,6 +185,7 @@
iniset $NEUTRON_CONF DEFAULT policy_file $policy_file
iniset $NEUTRON_CONF DEFAULT allow_overlapping_ips True
+ iniset $NEUTRON_CONF DEFAULT router_distributed $NEUTRON_DISTRIBUTED_ROUTING
iniset $NEUTRON_CONF DEFAULT auth_strategy $NEUTRON_AUTH_STRATEGY
configure_auth_token_middleware $NEUTRON_CONF neutron $NEUTRON_AUTH_CACHE_DIR keystone_authtoken
@@ -182,7 +194,15 @@
# Configure VXLAN
# TODO(sc68cal) not hardcode?
iniset $NEUTRON_CORE_PLUGIN_CONF ml2 tenant_network_types vxlan
- iniset $NEUTRON_CORE_PLUGIN_CONF ml2 mechanism_drivers openvswitch,linuxbridge
+
+ local mech_drivers="openvswitch"
+ if [[ "$NEUTRON_DISTRIBUTED_ROUTING" = "True" ]]; then
+ mech_drivers+=",l2population"
+ else
+ mech_drivers+=",linuxbridge"
+ fi
+ iniset $NEUTRON_CORE_PLUGIN_CONF ml2 mechanism_drivers $mech_drivers
+
iniset $NEUTRON_CORE_PLUGIN_CONF ml2_type_vxlan vni_ranges 1001:2000
iniset $NEUTRON_CORE_PLUGIN_CONF ml2_type_flat flat_networks public
if [[ "$NEUTRON_PORT_SECURITY" = "True" ]]; then
@@ -203,6 +223,11 @@
else
iniset $NEUTRON_CORE_PLUGIN_CONF securitygroup firewall_driver iptables_hybrid
iniset $NEUTRON_CORE_PLUGIN_CONF ovs local_ip $HOST_IP
+
+ if [[ "$NEUTRON_DISTRIBUTED_ROUTING" = "True" ]]; then
+ iniset $NEUTRON_CORE_PLUGIN_CONF agent l2_population True
+ iniset $NEUTRON_CORE_PLUGIN_CONF agent enable_distributed_routing True
+ fi
fi
if ! running_in_container; then
@@ -237,6 +262,10 @@
else
iniset $NEUTRON_CORE_PLUGIN_CONF ovs bridge_mappings "$PUBLIC_NETWORK_NAME:$PUBLIC_BRIDGE"
fi
+
+ if [[ "$NEUTRON_DISTRIBUTED_ROUTING" = "True" ]]; then
+ iniset $NEUTRON_L3_CONF DEFAULT agent_mode $NEUTRON_DVR_MODE
+ fi
fi
# Metadata
@@ -307,7 +336,6 @@
iniset $NOVA_CONF neutron project_domain_name "Default"
iniset $NOVA_CONF neutron auth_strategy $NEUTRON_AUTH_STRATEGY
iniset $NOVA_CONF neutron region_name "$REGION_NAME"
- iniset $NOVA_CONF neutron url $NEUTRON_SERVICE_PROTOCOL://$NEUTRON_SERVICE_HOST:$NEUTRON_SERVICE_PORT
iniset $NOVA_CONF DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
diff --git a/lib/neutron-legacy b/lib/neutron-legacy
index 9701ee7..0cd7e31 100644
--- a/lib/neutron-legacy
+++ b/lib/neutron-legacy
@@ -376,7 +376,6 @@
iniset $NOVA_CONF neutron project_domain_name "$SERVICE_DOMAIN_NAME"
iniset $NOVA_CONF neutron auth_strategy "$Q_AUTH_STRATEGY"
iniset $NOVA_CONF neutron region_name "$REGION_NAME"
- iniset $NOVA_CONF neutron url "${Q_PROTOCOL}://$Q_HOST:$Q_PORT"
if [[ "$Q_USE_SECGROUP" == "True" ]]; then
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
diff --git a/lib/neutron_plugins/services/l3 b/lib/neutron_plugins/services/l3
index 41a467d..9be32b7 100644
--- a/lib/neutron_plugins/services/l3
+++ b/lib/neutron_plugins/services/l3
@@ -39,9 +39,9 @@
Q_L3_ROUTER_PER_TENANT=${Q_L3_ROUTER_PER_TENANT:-True}
-# Use flat providernet for public network
+# Use providernet for public network
#
-# If Q_USE_PROVIDERNET_FOR_PUBLIC=True, use a flat provider network
+# If Q_USE_PROVIDERNET_FOR_PUBLIC=True, use a provider network
# for external interface of neutron l3-agent. In that case,
# PUBLIC_PHYSICAL_NETWORK specifies provider:physical_network value
# used for the network. In case of ofagent, you should add the
@@ -59,6 +59,10 @@
# Q_USE_PROVIDERNET_FOR_PUBLIC=True
# PUBLIC_PHYSICAL_NETWORK=public
# OVS_BRIDGE_MAPPINGS=public:br-ex
+#
+# The provider-network-type defaults to flat, however, the values
+# PUBLIC_PROVIDERNET_TYPE and PUBLIC_PROVIDERNET_SEGMENTATION_ID could
+# be set to specify the parameters for an alternate network type.
Q_USE_PROVIDERNET_FOR_PUBLIC=${Q_USE_PROVIDERNET_FOR_PUBLIC:-True}
PUBLIC_PHYSICAL_NETWORK=${PUBLIC_PHYSICAL_NETWORK:-public}
@@ -240,7 +244,7 @@
fi
# Create an external network, and a subnet. Configure the external network as router gw
if [ "$Q_USE_PROVIDERNET_FOR_PUBLIC" = "True" ]; then
- EXT_NET_ID=$(openstack --os-cloud devstack-admin --os-region "$REGION_NAME" network create "$PUBLIC_NETWORK_NAME" $EXTERNAL_NETWORK_FLAGS --provider-network-type flat --provider-physical-network ${PUBLIC_PHYSICAL_NETWORK} | grep ' id ' | get_field 2)
+ EXT_NET_ID=$(openstack --os-cloud devstack-admin --os-region "$REGION_NAME" network create "$PUBLIC_NETWORK_NAME" $EXTERNAL_NETWORK_FLAGS --provider-network-type ${PUBLIC_PROVIDERNET_TYPE:-flat} ${PUBLIC_PROVIDERNET_SEGMENTATION_ID:+--provider-segment $PUBLIC_PROVIDERNET_SEGMENTATION_ID} --provider-physical-network ${PUBLIC_PHYSICAL_NETWORK} | grep ' id ' | get_field 2)
else
EXT_NET_ID=$(openstack --os-cloud devstack-admin --os-region "$REGION_NAME" network create "$PUBLIC_NETWORK_NAME" $EXTERNAL_NETWORK_FLAGS | grep ' id ' | get_field 2)
fi
diff --git a/lib/nova b/lib/nova
index 580f87f..56e3093 100644
--- a/lib/nova
+++ b/lib/nova
@@ -424,6 +424,9 @@
iniset $NOVA_CONF DEFAULT rootwrap_config "$NOVA_CONF_DIR/rootwrap.conf"
iniset $NOVA_CONF scheduler driver "$SCHEDULER"
iniset $NOVA_CONF filter_scheduler enabled_filters "$FILTERS"
+ if [[ $SCHEDULER == "filter_scheduler" ]]; then
+ iniset $NOVA_CONF scheduler workers "$API_WORKERS"
+ fi
iniset $NOVA_CONF DEFAULT default_floating_pool "$PUBLIC_NETWORK_NAME"
if [[ $SERVICE_IP_VERSION == 6 ]]; then
iniset $NOVA_CONF DEFAULT my_ip "$HOST_IPV6"
diff --git a/lib/tempest b/lib/tempest
index f60b477..0605ffb 100644
--- a/lib/tempest
+++ b/lib/tempest
@@ -299,6 +299,10 @@
iniset $TEMPEST_CONFIG identity-feature-enabled domain_specific_drivers True
fi
+ # TODO(felipemonteiro): Remove this once Tempest no longer supports Pike
+ # as this is supported in Queens and beyond.
+ iniset $TEMPEST_CONFIG identity-feature-enabled project_tags True
+
# Image
# We want to be able to override this variable in the gate to avoid
# doing an external HTTP fetch for this test.
diff --git a/roles/write-devstack-local-conf/library/devstack_local_conf.py b/roles/write-devstack-local-conf/library/devstack_local_conf.py
index 55ba4af..746f54f 100644
--- a/roles/write-devstack-local-conf/library/devstack_local_conf.py
+++ b/roles/write-devstack-local-conf/library/devstack_local_conf.py
@@ -14,16 +14,69 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import os
import re
-class VarGraph(object):
+class DependencyGraph(object):
# This is based on the JobGraph from Zuul.
+ def __init__(self):
+ self._names = set()
+ self._dependencies = {} # dependent_name -> set(parent_names)
+
+ def add(self, name, dependencies):
+ # Append the dependency information
+ self._dependencies.setdefault(name, set())
+ try:
+ for dependency in dependencies:
+ # Make sure a circular dependency is never created
+ ancestors = self._getParentNamesRecursively(
+ dependency, soft=True)
+ ancestors.add(dependency)
+ if name in ancestors:
+ raise Exception("Dependency cycle detected in {}".
+ format(name))
+ self._dependencies[name].add(dependency)
+ except Exception:
+ del self._dependencies[name]
+ raise
+
+ def getDependenciesRecursively(self, parent):
+ dependencies = []
+
+ current_dependencies = self._dependencies[parent]
+ for current in current_dependencies:
+ if current not in dependencies:
+ dependencies.append(current)
+ for dep in self.getDependenciesRecursively(current):
+ if dep not in dependencies:
+ dependencies.append(dep)
+ return dependencies
+
+ def _getParentNamesRecursively(self, dependent, soft=False):
+ all_parent_items = set()
+ items_to_iterate = set([dependent])
+ while len(items_to_iterate) > 0:
+ current_item = items_to_iterate.pop()
+ current_parent_items = self._dependencies.get(current_item)
+ if current_parent_items is None:
+ if soft:
+ current_parent_items = set()
+ else:
+ raise Exception("Dependent item {} not found: ".format(
+ dependent))
+ new_parent_items = current_parent_items - all_parent_items
+ items_to_iterate |= new_parent_items
+ all_parent_items |= new_parent_items
+ return all_parent_items
+
+
+class VarGraph(DependencyGraph):
def __init__(self, vars):
+ super(VarGraph, self).__init__()
self.vars = {}
self._varnames = set()
- self._dependencies = {} # dependent_var_name -> set(parent_var_names)
for k, v in vars.items():
self._varnames.add(k)
for k, v in vars.items():
@@ -38,28 +91,21 @@
raise Exception("Variable {} already added".format(key))
self.vars[key] = value
# Append the dependency information
- self._dependencies.setdefault(key, set())
+ dependencies = set()
+ for dependency in self.getDependencies(value):
+ if dependency == key:
+ # A variable is allowed to reference itself; no
+ # dependency link needed in that case.
+ continue
+ if dependency not in self._varnames:
+ # It's not necessary to create a link for an
+ # external variable.
+ continue
+ dependencies.add(dependency)
try:
- for dependency in self.getDependencies(value):
- if dependency == key:
- # A variable is allowed to reference itself; no
- # dependency link needed in that case.
- continue
- if dependency not in self._varnames:
- # It's not necessary to create a link for an
- # external variable.
- continue
- # Make sure a circular dependency is never created
- ancestor_vars = self._getParentVarNamesRecursively(
- dependency, soft=True)
- ancestor_vars.add(dependency)
- if any((key == anc_var) for anc_var in ancestor_vars):
- raise Exception("Dependency cycle detected in var {}".
- format(key))
- self._dependencies[key].add(dependency)
+ self.add(key, dependencies)
except Exception:
del self.vars[key]
- del self._dependencies[key]
raise
def getVars(self):
@@ -67,48 +113,105 @@
keys = sorted(self.vars.keys())
seen = set()
for key in keys:
- dependencies = self.getDependentVarsRecursively(key)
+ dependencies = self.getDependenciesRecursively(key)
for var in dependencies + [key]:
if var not in seen:
ret.append((var, self.vars[var]))
seen.add(var)
return ret
- def getDependentVarsRecursively(self, parent_var):
- dependent_vars = []
- current_dependent_vars = self._dependencies[parent_var]
- for current_var in current_dependent_vars:
- if current_var not in dependent_vars:
- dependent_vars.append(current_var)
- for dep in self.getDependentVarsRecursively(current_var):
- if dep not in dependent_vars:
- dependent_vars.append(dep)
- return dependent_vars
+class PluginGraph(DependencyGraph):
+ def __init__(self, base_dir, plugins):
+ super(PluginGraph, self).__init__()
+ # The dependency trees expressed by all the plugins we found
+ # (which may be more than those the job is using).
+ self._plugin_dependencies = {}
+ self.loadPluginNames(base_dir)
- def _getParentVarNamesRecursively(self, dependent_var, soft=False):
- all_parent_vars = set()
- vars_to_iterate = set([dependent_var])
- while len(vars_to_iterate) > 0:
- current_var = vars_to_iterate.pop()
- current_parent_vars = self._dependencies.get(current_var)
- if current_parent_vars is None:
- if soft:
- current_parent_vars = set()
- else:
- raise Exception("Dependent var {} not found: ".format(
- dependent_var))
- new_parent_vars = current_parent_vars - all_parent_vars
- vars_to_iterate |= new_parent_vars
- all_parent_vars |= new_parent_vars
- return all_parent_vars
+ self.plugins = {}
+ self._pluginnames = set()
+ for k, v in plugins.items():
+ self._pluginnames.add(k)
+ for k, v in plugins.items():
+ self._addPlugin(k, str(v))
+
+ def loadPluginNames(self, base_dir):
+ if base_dir is None:
+ return
+ git_roots = []
+ for root, dirs, files in os.walk(base_dir):
+ if '.git' not in dirs:
+ continue
+ # Don't go deeper than git roots
+ dirs[:] = []
+ git_roots.append(root)
+ for root in git_roots:
+ devstack = os.path.join(root, 'devstack')
+ if not (os.path.exists(devstack) and os.path.isdir(devstack)):
+ continue
+ settings = os.path.join(devstack, 'settings')
+ if not (os.path.exists(settings) and os.path.isfile(settings)):
+ continue
+ self.loadDevstackPluginInfo(settings)
+
+ define_re = re.compile(r'^define_plugin\s+(\w+).*')
+ require_re = re.compile(r'^plugin_requires\s+(\w+)\s+(\w+).*')
+ def loadDevstackPluginInfo(self, fn):
+ name = None
+ reqs = set()
+ with open(fn) as f:
+ for line in f:
+ m = self.define_re.match(line)
+ if m:
+ name = m.group(1)
+ m = self.require_re.match(line)
+ if m:
+ if name == m.group(1):
+ reqs.add(m.group(2))
+ if name and reqs:
+ self._plugin_dependencies[name] = reqs
+
+ def getDependencies(self, value):
+ return self._plugin_dependencies.get(value, [])
+
+ def _addPlugin(self, key, value):
+ if key in self.plugins:
+ raise Exception("Plugin {} already added".format(key))
+ self.plugins[key] = value
+ # Append the dependency information
+ dependencies = set()
+ for dependency in self.getDependencies(key):
+ if dependency == key:
+ continue
+ dependencies.add(dependency)
+ try:
+ self.add(key, dependencies)
+ except Exception:
+ del self.plugins[key]
+ raise
+
+ def getPlugins(self):
+ ret = []
+ keys = sorted(self.plugins.keys())
+ seen = set()
+ for key in keys:
+ dependencies = self.getDependenciesRecursively(key)
+ for plugin in dependencies + [key]:
+ if plugin not in seen:
+ ret.append((plugin, self.plugins[plugin]))
+ seen.add(plugin)
+ return ret
class LocalConf(object):
- def __init__(self, localrc, localconf, base_services, services, plugins):
+ def __init__(self, localrc, localconf, base_services, services, plugins,
+ base_dir):
self.localrc = []
self.meta_sections = {}
+ self.plugin_deps = {}
+ self.base_dir = base_dir
if plugins:
self.handle_plugins(plugins)
if services or base_services:
@@ -119,7 +222,8 @@
self.handle_localconf(localconf)
def handle_plugins(self, plugins):
- for k, v in plugins.items():
+ pg = PluginGraph(self.base_dir, plugins)
+ for k, v in pg.getPlugins():
if v:
self.localrc.append('enable_plugin {} {}'.format(k, v))
@@ -171,6 +275,7 @@
services=dict(type='dict'),
localrc=dict(type='dict'),
local_conf=dict(type='dict'),
+ base_dir=dict(type='path'),
path=dict(type='str'),
)
)
@@ -180,14 +285,18 @@
p.get('local_conf'),
p.get('base_services'),
p.get('services'),
- p.get('plugins'))
+ p.get('plugins'),
+ p.get('base_dir'))
lc.write(p['path'])
module.exit_json()
-from ansible.module_utils.basic import * # noqa
-from ansible.module_utils.basic import AnsibleModule
+try:
+ from ansible.module_utils.basic import * # noqa
+ from ansible.module_utils.basic import AnsibleModule
+except ImportError:
+ pass
if __name__ == '__main__':
main()
diff --git a/roles/write-devstack-local-conf/library/test.py b/roles/write-devstack-local-conf/library/test.py
new file mode 100644
index 0000000..843ca6e
--- /dev/null
+++ b/roles/write-devstack-local-conf/library/test.py
@@ -0,0 +1,166 @@
+# Copyright (C) 2017 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+#
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import shutil
+import tempfile
+import unittest
+
+from devstack_local_conf import LocalConf
+from collections import OrderedDict
+
+class TestDevstackLocalConf(unittest.TestCase):
+ def setUp(self):
+ self.tmpdir = tempfile.mkdtemp()
+
+ def tearDown(self):
+ shutil.rmtree(self.tmpdir)
+
+ def test_plugins(self):
+ "Test that plugins without dependencies work"
+ localrc = {'test_localrc': '1'}
+ local_conf = {'install':
+ {'nova.conf':
+ {'main':
+ {'test_conf': '2'}}}}
+ services = {'cinder': True}
+ # We use ordereddict here to make sure the plugins are in the
+ # *wrong* order for testing.
+ plugins = OrderedDict([
+ ('bar', 'git://git.openstack.org/openstack/bar-plugin'),
+ ('foo', 'git://git.openstack.org/openstack/foo-plugin'),
+ ('baz', 'git://git.openstack.org/openstack/baz-plugin'),
+ ])
+ p = dict(localrc=localrc,
+ local_conf=local_conf,
+ base_services=[],
+ services=services,
+ plugins=plugins,
+ base_dir='./test',
+ path=os.path.join(self.tmpdir, 'test.local.conf'))
+ lc = LocalConf(p.get('localrc'),
+ p.get('local_conf'),
+ p.get('base_services'),
+ p.get('services'),
+ p.get('plugins'),
+ p.get('base_dir'))
+ lc.write(p['path'])
+
+ plugins = []
+ with open(p['path']) as f:
+ for line in f:
+ if line.startswith('enable_plugin'):
+ plugins.append(line.split()[1])
+ self.assertEqual(['bar', 'baz', 'foo'], plugins)
+
+ def test_plugin_deps(self):
+ "Test that plugins with dependencies work"
+ os.makedirs(os.path.join(self.tmpdir, 'foo-plugin', 'devstack'))
+ os.makedirs(os.path.join(self.tmpdir, 'foo-plugin', '.git'))
+ os.makedirs(os.path.join(self.tmpdir, 'bar-plugin', 'devstack'))
+ os.makedirs(os.path.join(self.tmpdir, 'bar-plugin', '.git'))
+ with open(os.path.join(
+ self.tmpdir,
+ 'foo-plugin', 'devstack', 'settings'), 'w') as f:
+ f.write('define_plugin foo\n')
+ with open(os.path.join(
+ self.tmpdir,
+ 'bar-plugin', 'devstack', 'settings'), 'w') as f:
+ f.write('define_plugin bar\n')
+ f.write('plugin_requires bar foo\n')
+
+ localrc = {'test_localrc': '1'}
+ local_conf = {'install':
+ {'nova.conf':
+ {'main':
+ {'test_conf': '2'}}}}
+ services = {'cinder': True}
+ # We use ordereddict here to make sure the plugins are in the
+ # *wrong* order for testing.
+ plugins = OrderedDict([
+ ('bar', 'git://git.openstack.org/openstack/bar-plugin'),
+ ('foo', 'git://git.openstack.org/openstack/foo-plugin'),
+ ])
+ p = dict(localrc=localrc,
+ local_conf=local_conf,
+ base_services=[],
+ services=services,
+ plugins=plugins,
+ base_dir=self.tmpdir,
+ path=os.path.join(self.tmpdir, 'test.local.conf'))
+ lc = LocalConf(p.get('localrc'),
+ p.get('local_conf'),
+ p.get('base_services'),
+ p.get('services'),
+ p.get('plugins'),
+ p.get('base_dir'))
+ lc.write(p['path'])
+
+ plugins = []
+ with open(p['path']) as f:
+ for line in f:
+ if line.startswith('enable_plugin'):
+ plugins.append(line.split()[1])
+ self.assertEqual(['foo', 'bar'], plugins)
+
+ def test_plugin_circular_deps(self):
+ "Test that plugins with circular dependencies fail"
+ os.makedirs(os.path.join(self.tmpdir, 'foo-plugin', 'devstack'))
+ os.makedirs(os.path.join(self.tmpdir, 'foo-plugin', '.git'))
+ os.makedirs(os.path.join(self.tmpdir, 'bar-plugin', 'devstack'))
+ os.makedirs(os.path.join(self.tmpdir, 'bar-plugin', '.git'))
+ with open(os.path.join(
+ self.tmpdir,
+ 'foo-plugin', 'devstack', 'settings'), 'w') as f:
+ f.write('define_plugin foo\n')
+ f.write('plugin_requires foo bar\n')
+ with open(os.path.join(
+ self.tmpdir,
+ 'bar-plugin', 'devstack', 'settings'), 'w') as f:
+ f.write('define_plugin bar\n')
+ f.write('plugin_requires bar foo\n')
+
+ localrc = {'test_localrc': '1'}
+ local_conf = {'install':
+ {'nova.conf':
+ {'main':
+ {'test_conf': '2'}}}}
+ services = {'cinder': True}
+ # We use ordereddict here to make sure the plugins are in the
+ # *wrong* order for testing.
+ plugins = OrderedDict([
+ ('bar', 'git://git.openstack.org/openstack/bar-plugin'),
+ ('foo', 'git://git.openstack.org/openstack/foo-plugin'),
+ ])
+ p = dict(localrc=localrc,
+ local_conf=local_conf,
+ base_services=[],
+ services=services,
+ plugins=plugins,
+ base_dir=self.tmpdir,
+ path=os.path.join(self.tmpdir, 'test.local.conf'))
+ with self.assertRaises(Exception):
+ lc = LocalConf(p.get('localrc'),
+ p.get('local_conf'),
+ p.get('base_services'),
+ p.get('services'),
+ p.get('plugins'),
+ p.get('base_dir'))
+ lc.write(p['path'])
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/roles/write-devstack-local-conf/tasks/main.yaml b/roles/write-devstack-local-conf/tasks/main.yaml
index cc21426..2a9f898 100644
--- a/roles/write-devstack-local-conf/tasks/main.yaml
+++ b/roles/write-devstack-local-conf/tasks/main.yaml
@@ -8,3 +8,4 @@
services: "{{ devstack_services|default(omit) }}"
localrc: "{{ devstack_localrc|default(omit) }}"
local_conf: "{{ devstack_local_conf|default(omit) }}"
+ base_dir: "{{ devstack_base_dir|default(omit) }}"
diff --git a/stackrc b/stackrc
index e8c35f4..f05bc6e 100644
--- a/stackrc
+++ b/stackrc
@@ -133,7 +133,7 @@
# base name of the directory from which they are installed. See
# enable_python3_package to edit this variable and use_python3_for to
# test membership.
-export ENABLED_PYTHON3_PACKAGES="nova,glance,cinder,uwsgi,python-openstackclient,python-openstacksdk"
+export ENABLED_PYTHON3_PACKAGES="nova,glance,cinder,uwsgi,python-openstackclient,openstacksdk"
# Explicitly list services not to run under Python 3. See
# disable_python3_package to edit this variable.
@@ -525,6 +525,10 @@
GITBRANCH["ceilometermiddleware"]=${CEILOMETERMIDDLEWARE_BRANCH:-$TARGET_BRANCH}
GITDIR["ceilometermiddleware"]=$DEST/ceilometermiddleware
+# openstacksdk OpenStack Python SDK
+GITREPO["openstacksdk"]=${OPENSTACKSDK_REPO:-${GIT_BASE}/openstack/openstacksdk.git}
+GITBRANCH["openstacksdk"]=${OPENSTACKSDK_BRANCH:-$TARGET_BRANCH}
+
# os-brick library to manage local volume attaches
GITREPO["os-brick"]=${OS_BRICK_REPO:-${GIT_BASE}/openstack/os-brick.git}
GITBRANCH["os-brick"]=${OS_BRICK_BRANCH:-$TARGET_BRANCH}
@@ -542,10 +546,6 @@
GITREPO["osc-lib"]=${OSC_LIB_REPO:-${GIT_BASE}/openstack/osc-lib.git}
GITBRANCH["osc-lib"]=${OSC_LIB_BRANCH:-$TARGET_BRANCH}
-# python-openstacksdk OpenStack Python SDK
-GITREPO["python-openstacksdk"]=${OPENSTACKSDK_REPO:-${GIT_BASE}/openstack/python-openstacksdk.git}
-GITBRANCH["python-openstacksdk"]=${OPENSTACKSDK_BRANCH:-$TARGET_BRANCH}
-
# ironic common lib
GITREPO["ironic-lib"]=${IRONIC_LIB_REPO:-${GIT_BASE}/openstack/ironic-lib.git}
GITBRANCH["ironic-lib"]=${IRONIC_LIB_BRANCH:-$TARGET_BRANCH}
diff --git a/tests/test_libs_from_pypi.sh b/tests/test_libs_from_pypi.sh
index a544b56..c3b4457 100755
--- a/tests/test_libs_from_pypi.sh
+++ b/tests/test_libs_from_pypi.sh
@@ -38,7 +38,7 @@
ALL_LIBS+=" oslo.serialization"
ALL_LIBS+=" python-openstackclient osc-lib osc-placement"
ALL_LIBS+=" os-client-config oslo.rootwrap"
-ALL_LIBS+=" oslo.i18n oslo.utils python-openstacksdk python-swiftclient"
+ALL_LIBS+=" oslo.i18n oslo.utils openstacksdk python-swiftclient"
ALL_LIBS+=" python-neutronclient tooz ceilometermiddleware oslo.policy"
ALL_LIBS+=" debtcollector os-brick os-traits automaton futurist oslo.service"
ALL_LIBS+=" oslo.cache oslo.reports osprofiler cursive"
diff --git a/tests/test_write_devstack_local_conf_role.sh b/tests/test_write_devstack_local_conf_role.sh
new file mode 100755
index 0000000..b2bc0a2
--- /dev/null
+++ b/tests/test_write_devstack_local_conf_role.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+TOP=$(cd $(dirname "$0")/.. && pwd)
+
+# Import common functions
+source $TOP/functions
+source $TOP/tests/unittest.sh
+
+python ./roles/write-devstack-local-conf/library/test.py