| Sean Dague | 31a57ad | 2013-05-07 15:24:05 -0400 | [diff] [blame^] | 1 | Tempest Field Guide |
| 2 | ----------- |
| 3 | |
| 4 | Tempest is designed to be useful for a large number of different |
| 5 | environments. This includes being useful for gating commits to |
| 6 | OpenStack core projects, being used to validate OpenStack cloud |
| 7 | implementations for both correctness, as well as a burn in tool for |
| 8 | OpenStack clouds. |
| 9 | |
| 10 | As such Tempest tests come in many flavors, each with their own rules |
| 11 | and guidelines. Below is the proposed Havana restructuring for Tempest |
| 12 | to make this clear. |
| 13 | |
| 14 | tempest/ |
| 15 | 3rdparty/ - 3rd party api tests |
| 16 | api/ - API tests |
| 17 | cli/ - CLI tests |
| 18 | scenario/ - complex scenario tests |
| 19 | stress/ - stress tests |
| 20 | whitebox/ - white box testing |
| 21 | |
| 22 | Each of these directories contains different types of tests. What |
| 23 | belongs in each directory, the rules and examples for good tests, are |
| 24 | documented in a README.rst file in the directory. |
| 25 | |
| 26 | |
| 27 | 3rdparty |
| 28 | ------------ |
| 29 | |
| 30 | Many openstack components include 3rdparty API support. It is |
| 31 | completely legitmate for Tempest to include tests of 3rdparty APIs, |
| 32 | but those should be kept seperate from the normal OpenStack |
| 33 | validation. |
| 34 | |
| 35 | TODO: tempest/tests/boto should become tempest/3rdparty/boto |
| 36 | |
| 37 | |
| 38 | api |
| 39 | ------------ |
| 40 | |
| 41 | API tests are validation tests for the OpenStack API. They should not |
| 42 | use the existing python clients for OpenStack, but should instead use |
| 43 | the tempest implementations of clients. This allows us to test both |
| 44 | XML and JSON. Having raw clients also lets us pass invalid JSON and |
| 45 | XML to the APIs and see the results, something we could not get with |
| 46 | the native clients. |
| 47 | |
| 48 | When it makes sense, API testing should be moved closer to the |
| 49 | projects themselves, possibly as functional tests in their unit test |
| 50 | frameworks. |
| 51 | |
| 52 | TODO: The bulk of tempest/tests should move to tempest/api |
| 53 | |
| 54 | |
| 55 | cli |
| 56 | ------------ |
| 57 | |
| 58 | CLI tests use the openstack CLI to interact with the OpenStack |
| 59 | cloud. CLI testing in unit tests is somewhat difficult because unlike |
| 60 | server testing, there is no access to server code to |
| 61 | instantiate. Tempest seems like a logical place for this, as it |
| 62 | prereqs having a running OpenStack cloud. |
| 63 | |
| 64 | TODO: the top level cli directory moves to tempest/cli |
| 65 | |
| 66 | |
| 67 | scenario |
| 68 | ------------ |
| 69 | |
| 70 | Scenario tests are complex "through path" tests for OpenStack |
| 71 | functionality. They are typically a series of steps where complicated |
| 72 | state requiring multiple services is set up exercised, and torn down. |
| 73 | |
| 74 | Scenario tests can and should use the OpenStack python clients. |
| 75 | |
| 76 | TODO: tests/network/test_network_basic_ops.py, |
| 77 | tests/compute/servers/*_ops.py should move to tempest/scenario (others) |
| 78 | |
| 79 | |
| 80 | stress |
| 81 | ----------- |
| 82 | |
| 83 | Stress tests are designed to stress an OpenStack environment by |
| 84 | running a high workload against it and seeing what breaks. Tools may |
| 85 | be provided to help detect breaks (stack traces in the logs). |
| 86 | |
| 87 | TODO: old stress tests deleted, new_stress that david is working on |
| 88 | moves into here. |
| 89 | |
| 90 | |
| 91 | whitebox |
| 92 | ---------- |
| 93 | |
| 94 | Whitebox tests are tests which require access to the database of the |
| 95 | target OpenStack machine to verify internal state after opperations |
| 96 | are made. White box tests are allowed to use the python clients. |
| 97 | |
| 98 | TODO: collect out whitebox tests to this location. |