Bcfg2 unit testing

You will first need to install the Python Mock Module and Python Nose modules. You can then run the existing tests with the following:

You should see output something like the following:

..................................................
----------------------------------------------------------------------
Ran 50 tests in 0.121s

OK

Unit tests are also run by Travis-CI, a free continuous integration service, at http://travis-ci.org/#!/Bcfg2/bcfg2/

Testing in a virtualenv

Travis-CI runs the unit tests in a virtual environment, so to emulate that testing environment as closely as possible you can also use a virtual environment. To do so, you must have virtualenv installed.

There are two ways to test: Either with just the bare essential packages installed, or with optional packages installed as well. (Optional packages are things like Genshi; you can run Bcfg2 with them or without them.) For completeness, the tests should be run in both manners. (On Python 3, almost none of the optional packages are available, so it can only be run with just the required packages.) To install the optional packages, set:

export WITH_OPTIONAL_DEPS=yes

This flag tells the install script to install optional dependencies as well as requirements.

This assumes that you will create a virtual environment in ~/venvs/, and that the Bcfg2 source tree is cloned into ~/bcfg2.

First, create a new virtual environment and activate it:

cd ~/venvs
virtualenv travis
source travis/bin/activate

Get the test suite from bcfg2:

cp -R ~/bcfg2/* ~/venvs/travis/

Next, you must install prerequisite packages that are required to build some of the required Python packages, and some optional packages that are much easier to install from binary (rather than from source). If you are running on Ubuntu (the platform Travis-CI runs on) and have sudo, you can simply run:

testsuite/before_install.sh

If not, you will need to examine testsuite/before_install.sh and install the packages manually. The equivalent for Fedora, for instance, would be:

sudo yum -y update
sudo yum -y install swig pylint libxml2
if [[ "$WITH_OPTIONAL_DEPS" == "yes" ]]; then
    sudo yum -y install libselinux-python pylibacl python-inotify \
        PyYAML
fi

You could install these requirements using pip, but you’ll likely need to install a great many development packages required to compile them.

Next, install required Python packages:

testsuite/install.sh

Install Bcfg2 itself to the virtualenv:

pip install -e .

Now you can run tests:

nosetests testsuite

Writing Unit Tests

Bcfg2 makes extremely heavy use of object inheritance, which can make it challenging at times to write reusable tests. For instance, when writing tests for the base Bcfg2.Server.Plugin.base.Plugin class, which all Bcfg2 Plugins inherit from via the Plugin interfaces, yielding several levels of often-multiple inheritance. To make this easier, our unit tests adhere to several design considerations.

Inherit Tests

Our test objects should have inheritance trees that mirror the inheritance trees of their tested objects. For instance, the Bcfg2.Server.Plugins.Metadata.Metadata class definition is:

class Metadata(Bcfg2.Server.Plugin.Metadata,
               Bcfg2.Server.Plugin.Statistics,
               Bcfg2.Server.Plugin.DatabaseBacked):

Consequently, the TestMetadata class definition is:

class TestMetadata(TestPlugin.TestMetadata,
                   TestPlugin.TestStatistics,
                   TestPlugin.TestDatabaseBacked):

Note

The test object names are abbreviated because of the system of relative imports in the testsuite tree, described below.

This gives us a large number of tests basically “for free”: all core Bcfg2.Server.Plugin.interfaces.Metadata, Bcfg2.Server.Plugin.interfaces.Statistics, and Bcfg2.Server.Plugin.helpers.DatabaseBacked functionality is automatically tested on the Metadata class, which gives the test writer a lot of free functionality and also an easy list of which tests must be overridden to provide tests appropriate for the Metadata class implementation.

Additionally, a test class should have a class variable that describes the class that is being tested, and tests in that class should use that class variable to instantate the tested object. For instance, the test for Bcfg2.Server.Plugin.helpers.DirectoryBacked looks like this:

class TestDirectoryBacked(Bcfg2TestCase):
    test_obj = DirectoryBacked
    ...


    def test_child_interface(self):
        """ ensure that the child object has the correct interface """
        self.assertTrue(hasattr(self.test_obj.__child__, "HandleEvent"))

Then test objects that inherit from TestDirectoryBacked can override that object, and the test_child_interface test (e.g.) will still work. For example:

class TestPropDirectoryBacked(TestDirectoryBacked):
    test_obj = PropDirectoryBacked

Finally, each test class must also provide a get_obj method that takes no required arguments and produces an instance of test_obj. All test methods must use self.get_obj() to instantiate an object to be tested.

An object that does not inherit from any other tested Bcfg2 objects should inherit from testsuite.common.Bcfg2TestCase, described below.

Relative Imports

In order to reuse test code and allow for test inheritance, each test module should add all parent module paths to its sys.path. For instance, assuming a test in testsuite/Testsrc/Testlib/TestServer/TestPlugins/TestMetadata.py, the following paths should be added to sys.path:

testsuite
testsuite/Testsrc
testsuite/Testsrc/Testlib
testsuite/Testsrc/Testlib/TestServer
testsuite/Testsrc/Testlib/TestServer/TestPlugins

This must be done because Python 2.4, one of our target platforms, does not support relative imports. An easy way to do this is to add the following snippet to the top of each test file:

import os
import sys

# add all parent testsuite directories to sys.path to allow (most)
# relative imports in python 2.4
path = os.path.dirname(__file__)
while path != "/":
    if os.path.basename(path).lower().startswith("test"):
        sys.path.append(path)
    if os.path.basename(path) == "testsuite":
        break
    path = os.path.dirname(path)

In addition, each new directory created in testsuite must contain an empty __init__.py.

This will allow you, within TestMetadata.py, to import common test code and the parent objects the TestMetadata class will inherit from:

from common import inPy3k, call, builtins, u, can_skip, \
    skip, skipIf, skipUnless, Bcfg2TestCase, DBModelTestCase, syncdb, \
    patchIf, datastore
from TestPlugin import TestXMLFileBacked, TestMetadata as _TestMetadata, \
    TestStatistics, TestDatabaseBacked

Avoid Patching Where Possible

The Python Mock Module provides a patch decorator that can be used to replace tested objects with Mock objects. This is wonderful and necessary, but due to differences in the way various versions of Python and Python Mock handle object scope, it’s not always reliable when combined with our system of test object inheritance. Consequently, you should follow these rules when considering whether to use patch:

• If you need to mock an object that is not part of Bcfg2 (e.g., a builtin or an object in another Python library), use patch.
• If you need to patch an object being tested in order to instantiate it, use patch, but see below.
• If you need to patch a function (not a method) that is part of Bcfg2, use patch.
• If you need to mock an object that is part of the object being tested, do not use patch.

As an example of the last rule, assume you are writing tests for Bcfg2.Server.Plugin.helpers.FileBacked. Bcfg2.Server.Plugin.helpers.FileBacked.HandleEvent() calls Bcfg2.Server.Plugin.helpers.FileBacked.Index(), so we need to mock the Index function. This is the wrong way to do that:

class TestFileBacked(Bcfg2TestCase):
    @patch("%s.open" % builtins)
    @patch("Bcfg2.Server.Plugin.helpers.FileBacked.Index")
    def test_HandleEvent(self, mock_Index, mock_open):
        ...

Tests that inherit from TestFileBacked will not reliably patch the correct Index function. Instead, assign the object to be mocked directly:

class TestFileBacked(Bcfg2TestCase):
    @patch("%s.open" % builtins)
    def test_HandleEvent(self, mock_open):
        fb = self.get_obj()
        fb.Index = Mock()

Note

@patch decorations are evaluated at compile-time, so a workaround like this does not work:

class TestFileBacked(Bcfg2TestCase):
    @patch("%s.open" % builtins)
    @patch("%s.%s.Index" % (self.test_obj.__module__,
                            self.test_obj.__name))
    def test_HandleEvent(self, mock_Index, mock_open):
        ...

But see below about patching objects before instantiation.

In some cases, you will need to patch an object in order to instantiate it. For instance, consider Bcfg2.Server.Plugin.helpers.DirectoryBacked, which attempts to set a file access monitor watch when it is instantiated. This won’t work during unit testing, so we have to patch Bcfg2.Server.Plugin.helpers.DirectoryBacked.add_directory_monitor() in order to successfully instantiate a DirectoryBacked object. In order to do that, we need to patch the object being tested, which is a variable, but we need to evaluate the patch at run-time, not at compile time, in order to deal with inheritance. This can be done with a @patch decorator on an inner function, e.g.:

class TestDirectoryBacked(Bcfg2TestCase):
    test_obj = DirectoryBacked

    def test__init(self):
        @patch("%s.%s.add_directory_monitor" % (self.test_obj.__module__,
                                                self.test_obj.__name__))
        def inner(mock_add_monitor):
            db = self.test_obj(datastore, Mock())
            mock_add_monitor.assert_called_with('')

        inner()

inner() is patched when test__init() is called, and so @patch() is called with the module and the name of the object being tested as defined by the test object (i.e., not as defined by the parent object). If this is not done, then the patch will be applied at compile-time and add_directory_monitor will be patched on the DirectoryBacked class instead of on the class to be tested.

Some of our older unit tests do not follow these rules religiously, so as more tests are written that inherit from larger portions of the testsuite tree they may need to be refactored.

Naming

In order to make the system of inheritance we implement possible, we must follow these naming conventions fairly religiously.

• Test classes are given the name of the object to be tested with Test prepended. E.g., the test for the Bcfg2.Server.Plugins.Metadata.Metadata is named TestMetadata.
• Test classes that test miscellaneous functions in a module are named TestFunctions.
• Test modules are given the name of the module to be tested with Test prepended. Tests for __init__.py are named Test_init.py (one underscore).
• Tests for methods or functions are given the name of the method or function to be tested with test_ prepended. E.g., the test for Bcfg2.Server.Plugin.helpers.StructFile.Match is called test_Match; the test for Bcfg2.Server.Plugin.helpers.StructFile._match is called test__match.
• Tests for magic methods – those that start and end with double underscores – are named test__<name>, where name is the name of the magic method without underscores. E.g., a test for __init__ is called test__init, and a test for __getitem__ is called test__getitem. If this causes a collision with a non-magic function (e.g., if a class also has a function called _getitem(), the test for which would also be called test__getitem, seriously consider refactoring the code for the class.

Common Test Code

In order to make testing easier and more consistent, we provide a number of convenience functions, variables, and classes, for a wide variety of reasons. To import this module, first set up Relative Imports and then simply do:

from common import *

class testsuite.common.Bcfg2TestCase(methodName='runTest')

Bases: unittest.case.TestCase

Base TestCase class that inherits from unittest.TestCase. This class adds assertXMLEqual(), a useful assertion method given all the XML used by Bcfg2.

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

assertXMLEqual(el1, el2, msg=None)

Test that the two XML trees given are equal.

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

class testsuite.common.DBModelTestCase(methodName='runTest')

Bases: testsuite.common.Bcfg2TestCase

Test case class for Django database models

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

test_cleandb(**kwargs)

Ensure that we a) can connect to the database; b) start with a clean database

test_syncdb(**kwargs)

Create the test database and sync the schema

class testsuite.common.MockExecutor(timeout=None)

Bases: object

mock object for Bcfg2.Utils.Executor objects.

testsuite.common.XI = '{http://www.w3.org/2001/XInclude}'

The XInclude namespace in a format suitable for use in XPath expressions

testsuite.common.XI_NAMESPACE = 'http://www.w3.org/2001/XInclude'

The XInclude namespace name

testsuite.common.builtins = '__builtin__'

The name of the builtin module, for mocking Python builtins. In Python 2, this is __builtin__, in Python 3 builtins. To patch a builtin, you must do something like:

@patch("%s.open" % open)
def test_something(self, mock_open):
    ...

testsuite.common.datastore = '/'

The path to the Bcfg2 specification root for the tests. Using the root directory exposes a lot of potential problems with building paths.

testsuite.common.inPy3k = False

Whether or not the tests are being run on Python 3.

class testsuite.common.patchIf(condition, target, new=sentinel.DEFAULT, spec=None, create=False, spec_set=None)

Bases: object

Decorator class to perform conditional patching. This is necessary because some libraries might not be installed (e.g., selinux, pylibacl), and patching will barf on that. Other workarounds are not available to us; e.g., context managers aren’t in python 2.4, and using inner functions doesn’t work because python 2.6 parses all decorators at compile-time, not at run-time, so decorating inner functions does not prevent the decorators from being run.

Parameters:

• condition (bool) – The condition to evaluate to decide if the patch will be applied.
• target (str) – The name of the target object to patch
• new (any) – The new object to replace the target with. If this is omitted, a new mock.MagicMock is created and passed as an extra argument to the decorated function.
• spec (List of strings or existing object) – Spec passed to the MagicMock object if patchIf is creating one for you.
• create (bool) – Tell patch to create attributes on the fly. See the documentation for mock.patch() for more details on this.
• spec_set (List of strings or existing object) – Spec set passed to the MagicMock object if patchIf is creating one for you.

testsuite.common.re_type

alias of _sre.SRE_Pattern

testsuite.common.u(s)

Get a unicode string, whatever that means. In Python 2, returns a unicode object; in Python 3, returns a str object.

Parameters:s (str) – The string to unicode-ify. Returns:str or unicode