Modular Grenade Architecture¶
Grenade was originally created to demonstrate some level of upgrade capacity for OpenStack projects. Originally this just included a small number of services.
Proposed new basic flow:
setup_grenade - all the magic setup involved around err traps and filehandle redirects - setup devstack trees
setup_base - run stack.sh to build the correct base environment
verify_base - for project in projects; do verify_project; done
resources.sh verify pre-upgrade
shutdown - for project in projects; do shutdown; done
snapshot.sh pre_upgrade (NOT YET IMPLEMENTED)
resources.sh verify_noapi pre-upgrade
resources.sh verify post-upgrade
Assuming the following tree in target projects:
devstack/ - devstack plugin directory
upgrade/ - upgrade scripts
settings - adds settings for the upgrade path
snapshot.sh - snapshots the state of the service, typically a
database dump (NOT YET IMPLEMENTED)
from-juno/ - per release
- This same modular structure exists in the grenade tree with::
resources.sh is a per-service resource create / verify / destroy interface. What a service does inside a script is up to them.
You can assume your resource script will only be called if your service is running in an upgrade environment. The script should return zero on success for actions, and nonzero on failure.
The following is the supported calling interface
creates a set of sample resources that should survive very early in the upgrade process. This should only be used for horizontal resources that impact other services, that have to be available before they do any of their setup. For instance setup of neutron networks.
Do not use the phase unless you really know why
createwill not work for you.
creates a set of sample resources that should survive upgrade. Script should exit with a nonzero exit code if any resources could not be created.
Example: create an instance in nova or a volume in cinder
resources.sh verify (pre-upgrade|post-upgrade)
verify that the resources were created. Services are running at this point, and the APIs may be expected to work. The second argument indicates whether we are pre-upgrade or post-upgrade.
Example: use the nova command to verify that the test instance is still ACTIVE, or the cinder command to verify that the volume is still available.
verify that the resources are still present. This is called in the phase where services are stopped, and APIs are expected to not be accessible. Resource verification at this phase my require probing underlying components to make sure nothing has gone awry during service shutdown. The second argument indicates whether we are pre-upgrade or post-upgrade.
Example: check with libvirt to make sure the instance is actually created and running. Bonus points for being able to ping the instance, or otherwise check its live-ness. With cinder, checking that the LVM volume exists and looks reasonable.
Resource scripts should be responsible and cleanup all their resources when asked to destroy.
The calling sequence during a grenade run looks as follows:
# start old side
create (create will be called during the working old side)
# shutdown all services
# upgrade and start all services
The important thing to remember is verify/verify_noapi will be called multiple times, with multiple different versions of OpenStack. Those phases of the script must not be rerunnable multiple times.
While create / destroy are only going to be called once in the current interface, bonus points for also making those idempotent for resiliancy in testing.
Per-release upgrade scripts
There are times when exceptional manual upgrade steps must be performed to get from one release to the next, or even within the same release. Grenade supports this with per-release scripts found in each project, e.g.:
Regarding the sequence of when these per-release scripts are called, any
within-$base script should be run before installing new code, and any
from-$base script should be run after installing new code but before
starting the services with the new code. This is because configuration or
database changes may be needed before the upgraded code is started.
In order to assist with the checks listed the following functions exist:
resource_save project key value
resource_get project key
This allow resource scripts to have memory, and keep track of things like the allocated IP addresses, IDs, and other non deterministic data that is returned from OpenStack API calls.
Resource scripts get called in a specific environment already set:
TOP_DIR - will be set to the root of the devstack directory for the BASE version of devstack incase this is needed to find files like a working
GRENADE_DIR - the root directory of the grenade directory.
The following snippet will give you access to both the grenade and TARGET devstack functions:
Do as many actions as non admin as possible. As early as you can in your resource script it’s worth allocating a user/project for the script to run as. This ensures isolation against other scripts, and ensures that actions don’t only work because admin gets to bypass safeties.
Test side effects, not just API actions. The point of these resource survival scripts is to test that things created beyond the API / DB interaction still work later. Just testing that data can be stored / retrieved from the database isn’t very interesting, and should be covered other places. The value in the resource scripts is these side effects. Actual VMs running, actual iscsi targets running, etc. And ensuring these things are not disrupted when the control plane is shifted out from under them.
Out of Tree Plugins¶
A grenade plugin can be hosted out of tree in a project tree, similar to external devstack plugins. There are a few subtle differences when this happens.
The plugin structure will live under
The plugin is enabled by adding:
enable_grenade_plugin <$project> <giturl> [branch]
pluginrc in the
GRENADE_DIR. An additional rc file was
required due to sequencing of when plugin functions become available.
Note: when running a job based on the
for each devstack plugin defined using the
the corresponding grenade plugin is enabled automatically.
Changing Devstack Localrc¶
There is also a mechanism that allows a
settings file change the
devstack localrc files with the
devstack_localrc <base|target> arbitrary stuff to add
Which will take all the rest of the stuff on that line and add it to the localrc for either the base or target devstack.
Please note that
devstack_localrc only works when grenade
performs the configuration of the devstack settings and runs devstack
against the base target. When GRENADE_USE_EXTERNAL_DEVSTACK is set
to True, as it happens on the Zuul grenade jobs where devstack is
configured and executed before grenade, the function has no effect.
The following is a reasonable example
settings for out of tree
devstack_localrc base enable_service h-api h-api-cfn h-api-cw h-eng heat
devstack_localrc target enable_service h-api h-api-cfn h-api-cw h-eng heat
This registers the project for upgrade, symbolicly enables the heat database for dump during upgrade, and adds the heat services into the service list for base and target.
It’s expected that most
settings files for out of tree plugins
will need equivalent lines.