blob: b4fb5a5572a73866ff9db3daf995e6af8ef72deb [file] [log] [blame]
David Patersonce781492014-09-18 01:07:01 -04001# Copyright 2014 Dell Inc.
2# Licensed under the Apache License, Version 2.0 (the "License"); you may
3# not use this file except in compliance with the License. You may obtain
4# a copy of the License at
5#
6# http://www.apache.org/licenses/LICENSE-2.0
7#
8# Unless required by applicable law or agreed to in writing, software
9# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
David Patersond6babc52014-10-14 00:11:56 -040010# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
David Patersonce781492014-09-18 01:07:01 -040011# License for the specific language governing permissions and limitations
12# under the License.
David Patersonce781492014-09-18 01:07:01 -040013
14"""
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050015Utility for cleaning up environment after Tempest test run
16
17**Usage:** ``tempest cleanup [--help] [OPTIONS]``
18
19If run with no arguments, ``tempest cleanup`` will query your OpenStack
20deployment and build a list of resources to delete and destroy them. This list
21will exclude the resources from ``saved_state.json`` and will include the
22configured admin account if the ``--delete-tempest-conf-objects`` flag is
23specified. By default the admin project is not deleted and the admin user
24specified in ``tempest.conf`` is never deleted.
25
26Example Run
27-----------
28
Masayuki Igawabbbaad62017-11-21 16:04:03 +090029.. warning::
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050030
Masayuki Igawabbbaad62017-11-21 16:04:03 +090031 If step 1 is skipped in the example below, the cleanup procedure
32 may delete resources that existed in the cloud before the test run. This
33 may cause an unwanted destruction of cloud resources, so use caution with
34 this command.
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050035
Masayuki Igawabbbaad62017-11-21 16:04:03 +090036 Examples::
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050037
Masayuki Igawabbbaad62017-11-21 16:04:03 +090038 $ tempest cleanup --init-saved-state
39 $ # Actual running of Tempest tests
40 $ tempest cleanup
David Patersonce781492014-09-18 01:07:01 -040041
42Runtime Arguments
43-----------------
44
Masayuki Igawabbbaad62017-11-21 16:04:03 +090045* ``--init-saved-state``: Initializes the saved state of the OpenStack
46 deployment and will output a ``saved_state.json`` file containing resources
47 from your deployment that will be preserved from the cleanup command. This
48 should be done prior to running Tempest tests.
David Patersonce781492014-09-18 01:07:01 -040049
Masayuki Igawabbbaad62017-11-21 16:04:03 +090050* ``--delete-tempest-conf-objects``: If option is present, then the command
51 will delete the admin project in addition to the resources associated with
52 them on clean up. If option is not present, the command will delete the
53 resources associated with the Tempest and alternate Tempest users and
54 projects but will not delete the projects themselves.
David Patersonce781492014-09-18 01:07:01 -040055
Masayuki Igawabbbaad62017-11-21 16:04:03 +090056* ``--dry-run``: Creates a report (``./dry_run.json``) of the projects that
57 will be cleaned up (in the ``_projects_to_clean`` dictionary [1]_) and the
58 global objects that will be removed (domains, flavors, images, roles,
59 projects, and users). Once the cleanup command is executed (e.g. run without
60 parameters), running it again with ``--dry-run`` should yield an empty
61 report.
David Patersonce781492014-09-18 01:07:01 -040062
Masayuki Igawabbbaad62017-11-21 16:04:03 +090063* ``--help``: Print the help text for the command and parameters.
David Patersonce781492014-09-18 01:07:01 -040064
Arx Cruz05fe4bc2017-10-20 10:48:28 +020065.. [1] The ``_projects_to_clean`` dictionary in ``dry_run.json`` lists the
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050066 projects that ``tempest cleanup`` will loop through to delete child
67 objects, but the command will, by default, not delete the projects
Arx Cruz05fe4bc2017-10-20 10:48:28 +020068 themselves. This may differ from the ``projects`` list as you can clean
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050069 the Tempest and alternate Tempest users and projects but they will not be
Masayuki Igawabbbaad62017-11-21 16:04:03 +090070 deleted unless the ``--delete-tempest-conf-objects`` flag is used to
Dustin Schoenbrund4d83462017-03-09 18:54:03 -050071 force their deletion.
David Patersonce781492014-09-18 01:07:01 -040072
Martin Kopec219e3222019-08-05 20:02:20 +000073.. note::
74
75 If during execution of ``tempest cleanup`` NotImplemented exception
76 occurres, ``tempest cleanup`` won't fail on that, it will be logged only.
77 NotImplemented errors are ignored because they are an outcome of some
78 extensions being disabled and ``tempest cleanup`` is not checking their
79 availability as it tries to clean up as much as possible without any
80 complicated logic.
81
David Patersonce781492014-09-18 01:07:01 -040082"""
David Patersonce781492014-09-18 01:07:01 -040083import sys
Marc Kodererf3397942015-12-21 11:17:09 +010084import traceback
David Patersonce781492014-09-18 01:07:01 -040085
David Paterson07661de2015-10-29 20:15:04 -070086from cliff import command
Doug Hellmann583ce2c2015-03-11 14:55:46 +000087from oslo_log import log as logging
Matthew Treinish21905512015-07-13 10:33:35 -040088from oslo_serialization import jsonutils as json
Doug Hellmann583ce2c2015-03-11 14:55:46 +000089
David Patersonce781492014-09-18 01:07:01 -040090from tempest import clients
91from tempest.cmd import cleanup_service
Andrea Frittoli (andreaf)290b3e12015-10-08 10:25:02 +010092from tempest.common import credentials_factory as credentials
Ken'ichi Ohmichi6ea3f982015-11-09 12:41:13 +000093from tempest.common import identity
David Patersonce781492014-09-18 01:07:01 -040094from tempest import config
Martin Kopec219e3222019-08-05 20:02:20 +000095from tempest.lib import exceptions
David Patersonce781492014-09-18 01:07:01 -040096
97SAVED_STATE_JSON = "saved_state.json"
98DRY_RUN_JSON = "dry_run.json"
99LOG = logging.getLogger(__name__)
100CONF = config.CONF
101
102
David Paterson07661de2015-10-29 20:15:04 -0700103class TempestCleanup(command.Command):
David Patersonce781492014-09-18 01:07:01 -0400104
Martin Kopec97857942019-06-12 15:23:21 +0000105 GOT_EXCEPTIONS = []
106
David Paterson07661de2015-10-29 20:15:04 -0700107 def take_action(self, parsed_args):
Marc Kodererf3397942015-12-21 11:17:09 +0100108 try:
109 self.init(parsed_args)
Ghanshyam41711d32016-02-17 17:11:22 +0900110 if not parsed_args.init_saved_state:
111 self._cleanup()
Marc Kodererf3397942015-12-21 11:17:09 +0100112 except Exception:
113 LOG.exception("Failure during cleanup")
114 traceback.print_exc()
115 raise
Martin Kopec219e3222019-08-05 20:02:20 +0000116 # ignore NotImplemented errors as those are an outcome of some
117 # extensions being disabled and cleanup is not checking their
118 # availability as it tries to clean up as much as possible without
119 # any complicated logic
120 critical_exceptions = [ex for ex in self.GOT_EXCEPTIONS if
121 not isinstance(ex, exceptions.NotImplemented)]
122 if critical_exceptions:
Martin Kopec97857942019-06-12 15:23:21 +0000123 raise Exception(self.GOT_EXCEPTIONS)
Marc Kodererf3397942015-12-21 11:17:09 +0100124
125 def init(self, parsed_args):
David Paterson07661de2015-10-29 20:15:04 -0700126 cleanup_service.init_conf()
127 self.options = parsed_args
ghanshyam009a1f62017-08-08 10:22:57 +0300128 self.admin_mgr = clients.Manager(
129 credentials.get_configured_admin_credentials())
David Patersonce781492014-09-18 01:07:01 -0400130 self.dry_run_data = {}
131 self.json_data = {}
David Patersonce781492014-09-18 01:07:01 -0400132
133 self.admin_id = ""
134 self.admin_role_id = ""
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200135 self.admin_project_id = ""
David Patersonce781492014-09-18 01:07:01 -0400136 self._init_admin_ids()
137
David Patersonce781492014-09-18 01:07:01 -0400138 # available services
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200139 self.project_services = cleanup_service.get_project_cleanup_services()
David Patersonce781492014-09-18 01:07:01 -0400140 self.global_services = cleanup_service.get_global_cleanup_services()
David Patersonce781492014-09-18 01:07:01 -0400141
David Paterson07661de2015-10-29 20:15:04 -0700142 if parsed_args.init_saved_state:
David Patersonce781492014-09-18 01:07:01 -0400143 self._init_state()
144 return
145
146 self._load_json()
David Patersonce781492014-09-18 01:07:01 -0400147
148 def _cleanup(self):
yuyafeie2dbc1f2016-07-06 16:09:19 +0800149 print("Begin cleanup")
David Patersonce781492014-09-18 01:07:01 -0400150 is_dry_run = self.options.dry_run
David Patersond6babc52014-10-14 00:11:56 -0400151 is_preserve = not self.options.delete_tempest_conf_objects
David Patersonce781492014-09-18 01:07:01 -0400152 is_save_state = False
153
154 if is_dry_run:
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200155 self.dry_run_data["_projects_to_clean"] = {}
David Patersonce781492014-09-18 01:07:01 -0400156
157 admin_mgr = self.admin_mgr
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200158 # Always cleanup tempest and alt tempest projects unless
David Patersonce781492014-09-18 01:07:01 -0400159 # they are in saved state json. Therefore is_preserve is False
160 kwargs = {'data': self.dry_run_data,
161 'is_dry_run': is_dry_run,
162 'saved_state_json': self.json_data,
163 'is_preserve': False,
164 'is_save_state': is_save_state}
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200165 project_service = cleanup_service.ProjectService(admin_mgr, **kwargs)
166 projects = project_service.list()
167 print("Process %s projects" % len(projects))
David Patersonce781492014-09-18 01:07:01 -0400168
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200169 # Loop through list of projects and clean them up.
170 for project in projects:
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200171 self._clean_project(project)
David Patersonce781492014-09-18 01:07:01 -0400172
173 kwargs = {'data': self.dry_run_data,
174 'is_dry_run': is_dry_run,
175 'saved_state_json': self.json_data,
176 'is_preserve': is_preserve,
Martin Kopec97857942019-06-12 15:23:21 +0000177 'is_save_state': is_save_state,
178 'got_exceptions': self.GOT_EXCEPTIONS}
David Patersonce781492014-09-18 01:07:01 -0400179 for service in self.global_services:
180 svc = service(admin_mgr, **kwargs)
181 svc.run()
182
183 if is_dry_run:
zhang.leia4b1cef2016-03-01 10:50:01 +0800184 with open(DRY_RUN_JSON, 'w+') as f:
185 f.write(json.dumps(self.dry_run_data, sort_keys=True,
186 indent=2, separators=(',', ': ')))
David Patersonce781492014-09-18 01:07:01 -0400187
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200188 def _clean_project(self, project):
189 print("Cleaning project: %s " % project['name'])
David Patersonce781492014-09-18 01:07:01 -0400190 is_dry_run = self.options.dry_run
191 dry_run_data = self.dry_run_data
David Patersond6babc52014-10-14 00:11:56 -0400192 is_preserve = not self.options.delete_tempest_conf_objects
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200193 project_id = project['id']
194 project_name = project['name']
195 project_data = None
David Patersonce781492014-09-18 01:07:01 -0400196 if is_dry_run:
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200197 project_data = dry_run_data["_projects_to_clean"][project_id] = {}
198 project_data['name'] = project_name
David Patersonce781492014-09-18 01:07:01 -0400199
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200200 kwargs = {'data': project_data,
David Patersonce781492014-09-18 01:07:01 -0400201 'is_dry_run': is_dry_run,
Martin Kopec5a884bf2019-02-11 18:10:55 +0000202 'saved_state_json': self.json_data,
David Patersonce781492014-09-18 01:07:01 -0400203 'is_preserve': is_preserve,
204 'is_save_state': False,
Martin Kopec97857942019-06-12 15:23:21 +0000205 'project_id': project_id,
206 'got_exceptions': self.GOT_EXCEPTIONS}
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200207 for service in self.project_services:
Martin Kopec7ca86022019-10-04 14:13:59 +0000208 svc = service(self.admin_mgr, **kwargs)
David Patersonce781492014-09-18 01:07:01 -0400209 svc.run()
210
211 def _init_admin_ids(self):
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200212 pr_cl = self.admin_mgr.projects_client
213 rl_cl = self.admin_mgr.roles_v3_client
214 rla_cl = self.admin_mgr.role_assignments_client
215 us_cl = self.admin_mgr.users_v3_client
David Patersonce781492014-09-18 01:07:01 -0400216
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200217 project = identity.get_project_by_name(pr_cl,
218 CONF.auth.admin_project_name)
219 self.admin_project_id = project['id']
220 user = identity.get_user_by_project(us_cl, rla_cl,
221 self.admin_project_id,
222 CONF.auth.admin_username)
David Patersonce781492014-09-18 01:07:01 -0400223 self.admin_id = user['id']
224
Daniel Mellado6b16b922015-12-07 12:43:08 +0000225 roles = rl_cl.list_roles()['roles']
David Patersonce781492014-09-18 01:07:01 -0400226 for role in roles:
227 if role['name'] == CONF.identity.admin_role:
228 self.admin_role_id = role['id']
229 break
230
David Paterson07661de2015-10-29 20:15:04 -0700231 def get_parser(self, prog_name):
232 parser = super(TempestCleanup, self).get_parser(prog_name)
David Patersonce781492014-09-18 01:07:01 -0400233 parser.add_argument('--init-saved-state', action="store_true",
234 dest='init_saved_state', default=False,
235 help="Creates JSON file: " + SAVED_STATE_JSON +
236 ", representing the current state of your "
David Patersond6babc52014-10-14 00:11:56 -0400237 "deployment, specifically object types "
238 "tempest creates and destroys during a run. "
David Patersonce781492014-09-18 01:07:01 -0400239 "You must run with this flag prior to "
David Patersond6babc52014-10-14 00:11:56 -0400240 "executing cleanup in normal mode, which is with "
241 "no arguments.")
David Patersonce781492014-09-18 01:07:01 -0400242 parser.add_argument('--delete-tempest-conf-objects',
David Patersond6babc52014-10-14 00:11:56 -0400243 action="store_true",
244 dest='delete_tempest_conf_objects',
David Patersonce781492014-09-18 01:07:01 -0400245 default=False,
David Patersond6babc52014-10-14 00:11:56 -0400246 help="Force deletion of the tempest and "
Arx Cruz05fe4bc2017-10-20 10:48:28 +0200247 "alternate tempest users and projects.")
David Patersonce781492014-09-18 01:07:01 -0400248 parser.add_argument('--dry-run', action="store_true",
249 dest='dry_run', default=False,
250 help="Generate JSON file:" + DRY_RUN_JSON +
251 ", that reports the objects that would have "
252 "been deleted had a full cleanup been run.")
David Paterson07661de2015-10-29 20:15:04 -0700253 return parser
David Patersonce781492014-09-18 01:07:01 -0400254
David Paterson07661de2015-10-29 20:15:04 -0700255 def get_description(self):
256 return 'Cleanup after tempest run'
David Patersonce781492014-09-18 01:07:01 -0400257
David Patersonce781492014-09-18 01:07:01 -0400258 def _init_state(self):
yuyafeie2dbc1f2016-07-06 16:09:19 +0800259 print("Initializing saved state.")
David Patersonce781492014-09-18 01:07:01 -0400260 data = {}
261 admin_mgr = self.admin_mgr
262 kwargs = {'data': data,
263 'is_dry_run': False,
264 'saved_state_json': data,
265 'is_preserve': False,
Martin Kopec97857942019-06-12 15:23:21 +0000266 'is_save_state': True,
267 'got_exceptions': self.GOT_EXCEPTIONS}
David Patersonce781492014-09-18 01:07:01 -0400268 for service in self.global_services:
269 svc = service(admin_mgr, **kwargs)
270 svc.run()
271
Martin Kopec5a884bf2019-02-11 18:10:55 +0000272 for service in self.project_services:
273 svc = service(admin_mgr, **kwargs)
274 svc.run()
275
zhang.leia4b1cef2016-03-01 10:50:01 +0800276 with open(SAVED_STATE_JSON, 'w+') as f:
afazekas40fcb9b2019-03-08 11:25:11 +0100277 f.write(json.dumps(data, sort_keys=True,
278 indent=2, separators=(',', ': ')))
David Patersonce781492014-09-18 01:07:01 -0400279
Martin Kopec6caf3fa2019-01-20 15:24:10 +0000280 def _load_json(self, saved_state_json=SAVED_STATE_JSON):
David Patersonce781492014-09-18 01:07:01 -0400281 try:
Martin Kopec6caf3fa2019-01-20 15:24:10 +0000282 with open(saved_state_json, 'rb') as json_file:
zhang.leia4b1cef2016-03-01 10:50:01 +0800283 self.json_data = json.load(json_file)
284
David Patersonce781492014-09-18 01:07:01 -0400285 except IOError as ex:
286 LOG.exception("Failed loading saved state, please be sure you"
287 " have first run cleanup with --init-saved-state "
Jordan Pittier525ec712016-12-07 17:51:26 +0100288 "flag prior to running tempest. Exception: %s", ex)
David Patersonce781492014-09-18 01:07:01 -0400289 sys.exit(ex)
290 except Exception as ex:
Jordan Pittier525ec712016-12-07 17:51:26 +0100291 LOG.exception("Exception parsing saved state json : %s", ex)
David Patersonce781492014-09-18 01:07:01 -0400292 sys.exit(ex)