HTools support for multiple storage units per node

Created:

2015-Jan-21

Status:

Draft

This design document describes changes to hbal and related components (first and foremost LUXI), that will allow it to handle nodes that can’t be considered monolithic in regard to disk layout, for example because they have multiple different storage units available.

Current state and shortcomings

Currently the htools assume that there is one storage unit per node and that it can be arbitrarily split among instances. This leads to problems in clusters where multiple storage units are present: There might be 10GB DRBD and 10GB plain storage available on a node, for a total of 20GB. If an instance that uses 15GB of a single type of storage is requested, it can’t actually fit on the node, but the current implementation of hail doesn’t notice this.

This behaviour is clearly wrong, but the problem doesn’t arise often in current setup, due to the fact that instances currently only have a single storage type and that users typically use node groups to differentiate between different node storage layouts.

For the node show action, RAPI only returns

  • dfree: The total amount of free disk space

  • dtotal: The total amount of disk space

which is insufficient for the same reasons.

Proposed changes

Definitions

  • All disks have exactly one desired storage unit, which determines where and how the disk can be stored. If the disk is transfered, the desired storage unit remains unchanged. The desired storage unit includes specifics like the volume group in the case of LVM based storage.

  • A storage unit is a specific storage location on a specific node. Storage units have exactly one desired storage unit they can contain. A storage unit further has an identifier (containing the storage type, a key and possibly parameters), a total capacity, and a free capacity. A node cannot contain multiple storage units of the same desired storage unit.

  • For the purposes of this document a disk has a desired storage unit and a size.

  • A disk can be moved to a node, if there is at least one storage unit on that node which can contain the desired storage unit of the disk and if the free capacity is at least the size of the disk.

  • An instance can be moved to a node, if all its disks can be moved there one-by-one.

LUXI and IAllocator protocol extension

The LUXI and IAllocator protocols are extended to include in the node:

  • storage: a list of objects (storage units) with #. Storage unit, containing in order:

    1. storage type

    2. storage key (e.g. volume group name)

    3. extra parameters (e.g. flag for exclusive storage) as a list.

    1. Amount free in MiB

    2. Amount total in MiB

{
  "storage": [
    { "sunit": ["drbd8", "xenvg", []]
    , "free": 2000,
    , "total": 4000
    },
    { "sunit": ["file", "/path/to/storage1", []]
    , "free": 5000,
    , "total": 10000
    },
    { "sunit": ["file", "/path/to/storage2", []]
    , "free": 1000,
    , "total": 20000
    },
    { "sunit": ["lvm-vg", "xenssdvg", [false]]
    , "free": 1024,
    , "total": 1024
    }
  ]
}

is a node with an LVM volume group mirrored over DRBD, two file storage directories, one half full, one mostly full, and a non-mirrored volume group.

The storage type drbd8 needs to be added in order to differentiate between mirrored storage and non-mirrored storage. The storage key signals the volume group used and the storage unit takes no additional parameters.

Text protocol extension

The same field is optionally present in the HTools text protocol:

  • a new “storage” column is added to the node section, which is a semicolon separated list of comma separated fields in the order #. free #. total #. sunit, which in itself contains

    1. the storage type

    2. the storage key

    3. extra arguments

For example:

2000,4000,drbd,xenvg;5000,10000,file,/path/to/storage1;1000,20000; […]

Interpretation

hbal and hail will use this information only if available, if the data file doesn’t contain the storage field the old algorithm is used.

If the node information contains the storage field, hbal and hail will assume that only the space compatible with the disk’s requirements is available. For an instance to fit a node, all it’s disks need to fit there separately. For a disk to fit a node, a storage unit of the type of the disk needs to have enough free space to contain it. The total free storage is not taken into consideration.

Ignoring the old information will in theory introduce a backwards incompatibility: If the total free storage is smaller than to the sum of the free storage reported in the storage field a previously illegal move will become legal.

Balancing

In order to determine a storage location for an instance, we collect analogous metrics to the current total node free space metric – namely the standard deviation statistic of the free space per storage unit.

The standard deviation metric of a desired storage unit is the sample standard deviation of the percentage of free space of storage units compatible.

The full storage metric is a average of the standard deviation metrics of the desired storage units.

This is backwards compatible in-so-far as that

  1. For a single storage unit per node it will have the same value.

  2. The weight of the storage versus the other metrics remains unchanged.

Further this retains the property that scarce resources with low total will tend to have bigger impact on the metric than those with large totals, because in latter case the relative differences will not make for a large standard deviation.

Ignoring nodes that do not contain the desired storage unit additionally boosts the importance of the scarce desired storage units, because having more storage units of a desired storage unit will tend to make the standard deviation metric smaller.