System tests

System tests test bootstrap-vz in its entirety. This testing includes building images from manifests and creating/booting said images.

Since hardcoding manifests for each test, bootstrapping them and booting the resulting images is too much code for a single test, a testing harness has been developed that reduces each test to it’s bare essentials:

• Combine available manifest partials into a single manifest
• Boot an instance from a manifest
• Run tests on the booted instance

In order for the system testing harness to be able to bootstrap it must know about your build-servers. Depending on the manifest that is bootstrapped, the harness chooses a fitting build-server, connects to it and starts the bootstrapping process.

When running system tests, the framework will look for build-servers.yml at the root of the repo and raise an error if it is not found.

Manifest combinations

The tests mainly focus on varying key parts of an image (e.g. partitioning, Debian release, bootloader, ec2 backing, ec2 virtualization method) that have been problem areas. Essentially the tests are the cartesian product of these key parts.

Aborting a test

You can press Ctrl+C at any time during the testing to abort - the harness will automatically clean up any temporary resources and shut down running instances. Pressing Ctrl+C a second time stops the cleanup and quits immediately.

Manifest partials

Instead of creating manifests from scratch for each single test, reusable parts are factored out into partials in the manifest folder. This allows code like this:

partials = {'vdi': '{provider: {name: virtualbox}, volume: {backing: vdi}}',
            'vmdk': '{provider: {name: virtualbox}, volume: {backing: vmdk}}',
            }

def test_unpartitioned_extlinux_oldstable():
    std_partials = ['base', 'stable64', 'extlinux', 'unpartitioned', 'root_password']
    custom_partials = [partials['vmdk']]
    manifest_data = merge_manifest_data(std_partials, custom_partials)

The code above produces a manifest for Debian stable 64-bit unpartitioned virtualbox VMDK image. root_password is a special partial in that the actual password is randomly generated on load.

Missing parts

The system testing harness is in no way complete.

• It still has no support for providers other than Virtualbox, EC2 and Docker.
• Creating an SSH connection to a booted instance is cumbersome and does not happen in any of the tests - this would be particularly useful when manifests are to be tested beyond whether they boot up.