Upgrades¶

Database schema and data migrations¶

In general incompatible database schema migrations can be tracked to ALTER and DROP SQL commands instruction issued either against a column or table. This is why a unit test that blocks such migrations was introduced. We should try to keep our DB modifications additive. Moreover we should aim not to introduce migrations that cause the database tables to lock for a long period. Long lock on whole table can block other queries and may make real requests to fail.

RPC API changes¶

It can obviously break service communication if RPC interface changes. In particular this applies to changes of the RPC method definitions. To avoid that we assume N’s RPC API compatibility with N-1 version (both ways - rpcapi module should be able to downgrade the message if needed and manager module should be able to tolerate receiving messages in older version.

Below is an example RPC compatibility shim from Mitaka’s cinder.volume.manager. This code allows us to tolerate older versions of the messages:

def create_volume(self, context, volume_id, request_spec=None,
                  filter_properties=None, allow_reschedule=True,
                  volume=None):

    """Creates the volume."""
    # FIXME(thangp): Remove this in v2.0 of RPC API.
    if volume is None:
        # For older clients, mimic the old behavior and look up the volume
        # by its volume_id.
        volume = objects.Volume.get_by_id(context, volume_id)

And here’s a contrary shim in cinder.volume.rpcapi (RPC client) that downgrades the message to make sure it will be understood by older instances of the service:

def create_volume(self, ctxt, volume, host, request_spec,
                  filter_properties, allow_reschedule=True):
    request_spec_p = jsonutils.to_primitive(request_spec)
    msg_args = {'volume_id': volume.id, 'request_spec': request_spec_p,
                'filter_properties': filter_properties,
                'allow_reschedule': allow_reschedule}
    if self.client.can_send_version('1.32'):
        version = '1.32'
        msg_args['volume'] = volume
    else:
        version = '1.24'

    new_host = utils.extract_host(host)
    cctxt = self.client.prepare(server=new_host, version=version)
    request_spec_p = jsonutils.to_primitive(request_spec)
    cctxt.cast(ctxt, 'create_volume', **msg_args)

As can be seen there’s this magic self.client.can_send_version() method which detects if we’re running in a version-heterogeneous environment and need to downgrade the message. Detection is based on dynamic RPC version pinning. In general all the services (managers) report supported RPC API version. RPC API client gets all the versions from the DB, chooses the lowest one and starts to downgrade messages to it.

To limit impact on the DB the pinned version of certain RPC API is cached. After all the services in the deployment are updated, operator should restart all the services or send them a SIGHUP signal to force reload of version pins.

As we need to support only N RPC API in N+1 release, we should be able to drop all the compatibility shims in N+1. To be technically correct when doing so we should also bump the major RPC API version. We do not need to do that in every release (it may happen that through the release nothing will change in RPC API or cost of technical debt of compatibility code is lower than the cost of complicated procedure of increasing major version of RPC APIs).

The process of increasing the major version is explained in details in Nova’s documentation. Please note that in case of Cinder we’re accessing the DB from all of the services, so we should follow the more complicated “Mixed version environments” process for every of our services.

In case of removing whole RPC method we need to leave it there in N’s manager and can remove it in N+1 (because N-1 will be talking with N). When adding a new one we need to make sure that when the RPC client is pinned to a too low version any attempt to send new message should fail (because client will not know if manager receiving the message will understand it) or ensure the manager will get updated before clients by stating the recommended order of upgrades for that release.

RPC payload changes (oslo.versionedobjects)¶

oslo.versionedobjects is a library that helps us to maintain compatibility of the payload sent over RPC. As during the process of upgrades it is possible that a newer version of the service will send an object to an older one, it may happen that newer object is incompatible with older service.

Version of an object should be bumped every time we make a change that will result in an incompatible change of the serialized object. Tests will inform you when you need to bump the version of a versioned object, but rule of thumb is that we should never bump the version when we modify/adding/removing a method to the versioned object (unlike Nova we don’t use remotable methods), and should always bump it when we modify the fields dictionary.

There are exceptions to this rule, for example when we change a fields.StringField by a custom fields.BaseEnumField. The reason why a version bump is not required in this case it’s because the actual data doesn’t change, we are just removing magic string by an enumerate, but the strings used are exactly the same.

As mentioned before, you don’t have to know all the rules, as we have a test that calculates the hash of all objects taking all these rules into consideration and will tell you exactly when you need to bump the version of a versioned object.

You can run this test with tox -epy35 -- --path cinder/tests/unit/objects/test_objects.py. But you may need to run it multiple times until it passes since it may not detect all required bumps at once.

Then you’ll see which versioned object requires a bump and you need to bump that version and update the object_data dictionary in the test file to reflect the new version as well as the new hash.

There is a very common false positive on the version bump test, and that is when we have modified a versioned object that is being used by other objects using the fields.ObjectField class. Due to the backporting mechanism implemented in Cinder we don’t require bumping the version for these cases and we’ll just need to update the hash used in the test.

For example if we were to add a new field to the Volume object and then run the test we may think that we need to bump Volume, Snapshot, Backup, RequestSpec, and VolumeAttachment objects, but we really only need to bump the version of the Volume object and update the hash for all the other objects.

Imagine that we (finally!) decide that request_spec sent in create_volume RPC cast is duplicating data and we want to start to remove redundant occurrences. When running in version-mixed environment older services will still expect this redundant data. We need a way to somehow downgrade the request_spec before sending it over RPC. And this is were o.vo come in handy. o.vo provide us the infrastructure to keep the changes in object versioned and to be able to downgrade them to a particular version.

Let’s take a step back - similarly to the RPC API situation we need a way to tell if we need to send a backward-compatible version of the message. In this case we need to know to what version to downgrade the object. We’re using a similar solution to the one used for RPC API for that. A problem here is that we need a single identifier (that we will be reported to services DB table) to denote whole set of versions of all the objects. To do that we’ve introduced a concept of CinderObjectVersionHistory object, where we keep sets of individual object versions aggregated into a single version string. When making an incompatible change in a single object you need to bump its version (we have a unit test enforcing that) and add a new version to cinder.objects.base.CinderObjectVersionsHistory (there’s a unit test as well). Example code doing that is below:

OBJ_VERSIONS.add('1.1', {'Service': '1.2', 'ServiceList': '1.1'})

This line adds a new 1.1 aggregated object version that is different from 1.0 by two objects - Service in 1.2 and ServiceList in 1.1. This means that the commit which added this line bumped versions of these two objects.

Now if we know that a service we’re talking to is running 1.1 aggregated version - we need to downgrade Service and ServiceList to 1.2 and 1.1 respectively before sending. Please note that of course other objects are included in the 1.1 aggregated version, but you just need to specify what changed (all the other versions of individual objects will be taken from the last version - 1.0 in this case).

Getting back to request_spec example. So let’s assume we want to remove volume_properties from there (most of data in there is already somewhere else inside the request_spec object). We’ve made a change in the object fields, we’ve bumped it’s version (from 1.0 to 1.1), we’ve updated hash in the cinder.tests.unit.test_objects to synchronize it with the current state of the object, making the unit test pass and we’ve added a new aggregated object history version in cinder.objects.base.

What else is required? We need to provide code that actually downgrades RequestSpec object from 1.1 to 1.0 - to be used when sending the object to older services. This is done by implementing obj_make_compatible method in the object:

from oslo_utils import versionutils

def obj_make_compatible(self, primitive, target_version):
    super(RequestSpec, self).obj_make_compatible(primitive, target_version)
    target_version = versionutils.convert_version_to_tuple(target_version)
    if target_version < (1, 1) and not 'volume_properties' in primitive:
        volume_properties = {}
        # TODO: Aggregate all the required information from primitive.
        primitive['volume_properties'] = volume_properties

Please note that primitive is a dictionary representation of the object and not an object itself. This is because o.vo are of course sent over RPC as dicts.

With these pieces in place Cinder will take care of sending request_spec with volume_properties when running in mixed environment and without when all services are upgraded and will understand request_spec without volume_properties element.

Note that o.vo layer is able to recursively downgrade all of its fields, so when request_spec will be used as a field in other object, it will be correctly downgraded.

A more common case where we need backporting code is when we add new fields. In such case the backporting consist on removing the newly added fields. For example if we add 3 new fields to the Group object in version 1.1, then we need to remove them if backporting to earlier versions:

from oslo_utils import versionutils

def obj_make_compatible(self, primitive, target_version):
    super(Group, self).obj_make_compatible(primitive, target_version)
    target_version = versionutils.convert_version_to_tuple(target_version)
    if target_version < (1, 1):
        for key in ('group_snapshot_id', 'source_group_id',
                    'group_snapshots'):
            primitive.pop(key, None)

As time goes on we will be adding more and more new fields to our objects, so we may end up with a long series of if and for statements like in the Volume object:

from oslo_utils import versionutils

def obj_make_compatible(self, primitive, target_version):
    super(Volume, self).obj_make_compatible(primitive, target_version)
    target_version = versionutils.convert_version_to_tuple(target_version)
    if target_version < (1, 4):
        for key in ('cluster', 'cluster_name'):
            primitive.pop(key, None)
    if target_version < (1, 5):
        for key in ('group', 'group_id'):
            primitive.pop(key, None)

So a different pattern would be preferable as it will make the backporting easier for future additions:

from oslo_utils import versionutils

def obj_make_compatible(self, primitive, target_version):
    added_fields = (((1, 4), ('cluster', 'cluster_name')),
                    ((1, 5), ('group', 'group_id')))
    super(Volume, self).obj_make_compatible(primitive, target_version)
    target_version = versionutils.convert_version_to_tuple(target_version)
    for version, remove_fields in added_fields:
        if target_version < version:
            for obj_field in remove_fields:
                primitive.pop(obj_field, None)

Upgrade Checks¶

Starting with the Stein release of OpenStack, Cinder has added support for Upgrade Checks. Upgrade checks provide a release-specific readiness check before restarting services with new code. Details on how to run an Upgrade Check can be seen in the CLI interface for :doc:`cinder status commands </cli/cinder-status> page.

Upgrade checks are intended to help identify changes between releases that may impact the deployment environment. As a result, developers should take time to consider if the operator would benefit from having an Upgrade Check added along with changes they are proposing. The following are a few examples of changes that would require an Upgrade Check:

• Changes to Configuration Options * Removal * Change in Behavior

• Driver Removal

• Changes to Configuration File Locations

• Deprecations

To add an Upgrade Check edit the cinder/cmd/status.py file. Add a new function that contains the check you wish to implement. Functions need to return either a uc.Result where the result can be one of:

• SUCCESS

• FAILURE, <Failure explanation>

• WARNING, <Warning explanation>

Your new function should then be added to the _upgrade_checks tuple. For your check give the name of the Upgrade Check to be displayed to end users upon success or failure as well as the name of the function used to implement your check. Upgrade Checks should be submitted with Unit Tests.

The doc/source/cli/cinder-status.rst documentation should be updated to indicate the release for which your Upgrade Check was released and to explain the reason or limitations of your check, if appropriate. A release note should also be created with an explanation of the Upgrade Check in the upgrade section.

It is preferable to have Upgrade Checks submitted as part of the patch that is making the change in question. The checks, however, can be submitted as a separate patch and are appropriate for backport if they are being created after a release has been cut.

For additional details on Upgrade Checks please see Nova’s Upgrade Checks Documentation .