| =============================== |
| Migrating Zuul V2 CI jobs to V3 |
| =============================== |
| |
| The OpenStack CI system moved from Zuul v2 to Zuul v3, and all CI jobs moved to |
| the new CI system. All jobs have been migrated automatically to a format |
| compatible with Zuul v3; the jobs produced in this way however are suboptimal |
| and do not use the capabilities introduced by Zuul v3, which allow for re-use of |
| job parts, in the form of Ansible roles, as well as inheritance between jobs. |
| |
| DevStack hosts a set of roles, plays and jobs that can be used by other |
| repositories to define their DevStack based jobs. To benefit from them, jobs |
| must be migrated from the legacy v2 ones into v3 native format. |
| |
| This document provides guidance and examples to make the migration process as |
| painless and smooth as possible. |
| |
| Where to host the job definitions. |
| ================================== |
| |
| In Zuul V3 jobs can be defined in the repository that contains the code they |
| excercise. If you are writing CI jobs for an OpenStack service you can define |
| your DevStack based CI jobs in one of the repositories that host the code for |
| your service. If you have a branchless repo, like a Tempest plugin, that is |
| a convenient choice to host the job definitions since job changes do not have |
| to be backported. For example, see the beginning of the ``.zuul.yaml`` from the |
| sahara Tempest plugin repo: |
| |
| .. code:: yaml |
| |
| # In http://git.openstack.org/cgit/openstack/sahara-tests/tree/.zuul.yaml: |
| - job: |
| name: sahara-tests-tempest |
| description: | |
| Run Tempest tests from the Sahara plugin. |
| parent: devstack-tempest |
| |
| Which base job to start from |
| ============================ |
| |
| If your job needs an OpenStack cloud deployed via DevStack, but you don't plan |
| on running Tempest tests, you can start from one of the base |
| :doc:`jobs <zuul_jobs>` defined in the DevStack repo. |
| |
| The ``devstack`` job can be used for both single-node jobs and multi-node jobs, |
| and it includes the list of services used in the integrated gate (keystone, |
| glance, nova, cinder, neutron and swift). Different topologies can be achieved |
| by switching the nodeset used in the child job. |
| |
| The ``devstack-base`` job is similar to ``devstack`` but it does not specify any |
| required repo or service to be run in DevStack. It can be useful to setup |
| children jobs that use a very narrow DevStack setup. |
| |
| If your job needs an OpenStack cloud deployed via DevStack, and you do plan |
| on running Tempest tests, you can start from one of the base jobs defined in the |
| Tempest repo. |
| |
| The ``devstack-tempest`` job can be used for both single-node jobs and |
| multi-node jobs. Different topologies can be achieved by switching the nodeset |
| used in the child job. |
| |
| Jobs can be customized as follows without writing any Ansible code: |
| |
| - add and/or remove DevStack services |
| - add or modify DevStack and services configuration |
| - install DevStack plugins |
| - extend the number of sub-nodes (multinode only) |
| - define extra log files and/or directories to be uploaded on logs.o.o |
| - define extra log file extensions to be rewritten to .txt for ease of access |
| |
| Tempest jobs can be further customized as follows: |
| |
| - define the Tempest tox environment to be used |
| - define the test concurrency |
| - define the test regular expression |
| |
| Writing Ansible code, or importing existing custom roles, jobs can be further |
| extended by: |
| |
| - adding pre and/or post playbooks |
| - overriding the run playbook, add custom roles |
| |
| The (partial) example below extends a Tempest single node base job |
| "devstack-tempest" in the Kuryr repository. The parent job name is defined in |
| job.parent. |
| |
| .. code:: yaml |
| |
| # https://git.openstack.org/cgit/openstack/kuryr-kubernetes/tree/.zuul.yaml: |
| - job: |
| name: kuryr-kubernetes-tempest-base |
| parent: devstack-tempest |
| description: Base kuryr-kubernetes-job |
| required-projects: |
| - openstack/devstack-plugin-container |
| - openstack/kuryr |
| - openstack/kuryr-kubernetes |
| - openstack/kuryr-tempest-plugin |
| - openstack/neutron-lbaas |
| vars: |
| tempest_test_regex: '^(kuryr_tempest_plugin.tests.)' |
| tox_envlist: 'all' |
| devstack_localrc: |
| KURYR_K8S_API_PORT: 8080 |
| devstack_services: |
| kubernetes-api: true |
| kubernetes-controller-manager: true |
| kubernetes-scheduler: true |
| kubelet: true |
| kuryr-kubernetes: true |
| (...) |
| devstack_plugins: |
| kuryr-kubernetes: https://git.openstack.org/openstack/kuryr |
| devstack-plugin-container: https://git.openstack.org/openstack/devstack-plugin-container |
| neutron-lbaas: https://git.openstack.org/openstack/neutron-lbaas |
| tempest_plugins: |
| - kuryr-tempest-plugin |
| (...) |
| |
| Job variables |
| ============= |
| |
| Variables can be added to the job in three different places: |
| |
| - job.vars: these are global variables available to all node in the nodeset |
| - job.host-vars.[HOST]: these are variables available only to the specified HOST |
| - job.group-vars.[GROUP]: these are variables available only to the specified |
| GROUP |
| |
| Zuul merges dict variables through job inheritance. Host and group variables |
| override variables with the same name defined as global variables. |
| |
| In the example below, for the sundaes job, hosts that are not part of the |
| subnode group will run vanilla and chocolate. Hosts in the subnode group will |
| run stracciatella and strawberry. |
| |
| .. code:: yaml |
| |
| - job: |
| name: ice-creams |
| vars: |
| devstack_service: |
| vanilla: true |
| chocolate: false |
| group-vars: |
| subnode: |
| devstack_service: |
| pistacchio: true |
| stracciatella: true |
| |
| - job: |
| name: sundaes |
| parent: ice-creams |
| vars: |
| devstack_service: |
| chocolate: true |
| group-vars: |
| subnode: |
| devstack_service: |
| strawberry: true |
| pistacchio: false |
| |
| |
| DevStack Gate Flags |
| =================== |
| |
| The old CI system worked using a combination of DevStack, Tempest and |
| devstack-gate to setup a test environment and run tests against it. With Zuul |
| V3, the logic that used to live in devstack-gate is moved into different repos, |
| including DevStack, Tempest and grenade. |
| |
| DevStack-gate exposes an interface for job definition based on a number of |
| DEVSTACK_GATE_* environment variables, or flags. This guide shows how to map |
| DEVSTACK_GATE flags into the new |
| system. |
| |
| The repo column indicates in which repository is hosted the code that replaces |
| the devstack-gate flag. The new implementation column explains how to reproduce |
| the same or a similar behaviour in Zuul v3 jobs. For localrc settings, |
| devstack-gate defined a default value. In ansible jobs the default is either the |
| value defined in the parent job, or the default from DevStack, if any. |
| |
| ============================================== ============= ================== |
| DevStack gate flag Repo New implementation |
| ============================================== ============= ================== |
| OVERRIDE_ZUUL_BRANCH zuul override-checkout: |
| [branch] |
| in the job definition. |
| DEVSTACK_GATE_NET_OVERLAY zuul-jobs A bridge called |
| br-infra is set up for |
| all jobs that inherit |
| from multinode with |
| a dedicated `bridge role <https://docs.openstack.org/infra/zuul-jobs/roles.html#role-multi-node-bridge>`_. |
| DEVSTACK_GATE_FEATURE_MATRIX devstack-gate ``test_matrix_features`` |
| variable of the |
| test-matrix role in |
| devstack-gate. This |
| is a temporary |
| solution, feature |
| matrix will go away. |
| In the future services |
| will be defined in |
| jobs only. |
| DEVSTACK_CINDER_VOLUME_CLEAR devstack *CINDER_VOLUME_CLEAR: true/false* |
| in devstack_localrc |
| in the job vars. |
| DEVSTACK_GATE_NEUTRON devstack True by default. To |
| disable, disable all |
| neutron services in |
| devstack_services in |
| the job definition. |
| DEVSTACK_GATE_CONFIGDRIVE devstack *FORCE_CONFIG_DRIVE: true/false* |
| in devstack_localrc |
| in the job vars. |
| DEVSTACK_GATE_INSTALL_TESTONLY devstack *INSTALL_TESTONLY_PACKAGES: true/false* |
| in devstack_localrc |
| in the job vars. |
| DEVSTACK_GATE_VIRT_DRIVER devstack *VIRT_DRIVER: [virt driver]* |
| in devstack_localrc |
| in the job vars. |
| DEVSTACK_GATE_LIBVIRT_TYPE devstack *LIBVIRT_TYPE: [libvirt type]* |
| in devstack_localrc |
| in the job vars. |
| DEVSTACK_GATE_TEMPEST devstack Defined by the job |
| tempest that is used. The |
| ``devstack`` job only |
| runs devstack. |
| The ``devstack-tempest`` |
| one triggers a Tempest |
| run as well. |
| DEVSTACK_GATE_TEMPEST_FULL tempest *tox_envlist: full* |
| in the job vars. |
| DEVSTACK_GATE_TEMPEST_ALL tempest *tox_envlist: all* |
| in the job vars. |
| DEVSTACK_GATE_TEMPEST_ALL_PLUGINS tempest *tox_envlist: all-plugin* |
| in the job vars. |
| DEVSTACK_GATE_TEMPEST_SCENARIOS tempest *tox_envlist: scenario* |
| in the job vars. |
| TEMPEST_CONCURRENCY tempest *tempest_concurrency: [value]* |
| in the job vars. This |
| is available only on |
| jobs that inherit from |
| ``devstack-tempest`` |
| down. |
| DEVSTACK_GATE_TEMPEST_NOTESTS tempest *tox_envlist: venv-tempest* |
| in the job vars. This |
| will create Tempest |
| virtual environment |
| but run no tests. |
| DEVSTACK_GATE_SMOKE_SERIAL tempest *tox_envlist: smoke-serial* |
| in the job vars. |
| DEVSTACK_GATE_TEMPEST_DISABLE_TENANT_ISOLATION tempest *tox_envlist: full-serial* |
| in the job vars. |
| *TEMPEST_ALLOW_TENANT_ISOLATION: false* |
| in devstack_localrc in |
| the job vars. |
| ============================================== ============= ================== |
| |
| The following flags have not been migrated yet or are legacy and won't be |
| migrated at all. |
| |
| ===================================== ====== ========================== |
| DevStack gate flag Status Details |
| ===================================== ====== ========================== |
| DEVSTACK_GATE_TOPOLOGY WIP The topology depends on the base |
| job that is used and more |
| specifically on the nodeset |
| attached to it. The new job |
| format allows project to define |
| the variables to be passed to |
| every node/node-group that exists |
| in the topology. Named topologies |
| that include the nodeset and the |
| matching variables can be defined |
| in the form of base jobs. |
| DEVSTACK_GATE_GRENADE TBD Grenade Zuul V3 jobs will be |
| hosted in the grenade repo. |
| GRENADE_BASE_BRANCH TBD Grenade Zuul V3 jobs will be |
| hosted in the grenade repo. |
| DEVSTACK_GATE_NEUTRON_DVR TBD Depends on multinode support. |
| DEVSTACK_GATE_EXERCISES TBD Can be done on request. |
| DEVSTACK_GATE_IRONIC TBD This will probably be implemented |
| on ironic side. |
| DEVSTACK_GATE_IRONIC_DRIVER TBD This will probably be implemented |
| on ironic side. |
| DEVSTACK_GATE_IRONIC_BUILD_RAMDISK TBD This will probably be implemented |
| on ironic side. |
| DEVSTACK_GATE_POSTGRES Legacy This flag exists in d-g but the |
| only thing that it does is |
| capture postgres logs. This is |
| already supported by the roles in |
| post, so the flag is useless in |
| the new jobs. postgres itself can |
| be enabled via the |
| devstack_service job variable. |
| DEVSTACK_GATE_ZEROMQ Legacy This has no effect in d-g. |
| DEVSTACK_GATE_MQ_DRIVER Legacy This has no effect in d-g. |
| DEVSTACK_GATE_TEMPEST_STRESS_ARGS Legacy Stress is not in Tempest anymore. |
| DEVSTACK_GATE_TEMPEST_HEAT_SLOW Legacy This is not used anywhere. |
| DEVSTACK_GATE_CELLS Legacy This has no effect in d-g. |
| DEVSTACK_GATE_NOVA_API_METADATA_SPLIT Legacy This has no effect in d-g. |
| ===================================== ====== ========================== |