Bcfg2 Option Parsing

Bcfg2 uses an option parsing mechanism based on the Python argparse module. It does several very useful things that argparse does not:

  • Collects options from various places, which lets us easily specify per-plugin options, for example;
  • Automatically loads components (such as plugins);
  • Synthesizes option values from the command line, config files, and environment variables;
  • Can dynamically create commands with many subcommands (e.g., bcfg2-info and bcfg2-admin); and
  • Supports keeping documentation inline with the option declaration, which will make it easier to generate man pages.

Collecting Options

One of the more important features of the option parser is its ability to automatically collect options from loaded components (e.g., Bcfg2 server plugins). Given the highly pluggable architecture of Bcfg2, this helps ensure two things:

  1. We do not have to specify all options in all places, or even in most places. Options are specified alongside the class(es) that use them.
  2. All options needed for a given script to run are guaranteed to be loaded, without the need to specify all components that script uses manually.

For instance, assume a few plugins:

  • The Foo plugin takes one option, --foo
  • The Bar plugin takes two options, --bar and --force

The plugins are used by the bcfg2-quux command, which itself takes two options: --plugins (which selects the plugins) and --test. The options would be selected at runtime, so for instance these would be valid:

bcfg2-quux --plugins Foo --foo --test
bcfg2-quux --plugins Foo,Bar --foo --bar --force
bcfg2-quux --plugins Bar --force

But this would not:

bcfg2-quux –plugins Foo –bar

The help message would reflect the options that are available to the default set of plugins. (For this reason, allowing component lists to be set in the config file is very useful; that way, usage messages reflect the components in the config file.)

Components (in this example, the plugins) can be classes or modules. There is no required interface for an option component. They may optionally have:

  • An options attribute that is a list of Bcfg2.Options.Options.Option objects or option groups.
  • A boolean parse_first attribute; if set to True, the options for the component are parsed before all other options. This is useful for, e.g., Django database settings, which must be parsed before plugins that use Django can be loaded.
  • A function or static method, options_parsed_hook, that is called when all options have been parsed. (This will be called again if Bcfg2.Options.Parser.Parser.reparse() is called.)
  • A function or static method, component_parsed_hook, that is called when early option parsing for a given component has completed. This is only called for components with parse_first set to True. It is passed a single argument: a argparse.Namespace object containing the complete set of early options.

Options are collected through two primary mechanisms:

  1. The Bcfg2.Options.Actions.ComponentAction class. When a ComponentAction subclass is used as the action of an option, then options contained in the classes (or modules) given in the option value will be added to the parser.
  2. Modules that are not loaded via a Bcfg2.Options.Actions.ComponentAction option may load options at runtime.

Since it is preferred to add components instead of just options, loading options at runtime is generally best accomplished by creating a container object whose only purpose is to hold options. For instance:

def foo():
    # do stuff

class _OptionContainer(object):
    options = [
        Bcfg2.Options.BooleanOption("--foo", help="Enable foo")]

    @staticmethod
    def options_parsed_hook():
        if Bcfg2.Options.setup.foo:
            foo()

Bcfg2.Options.get_parser().add_component(_OptionContainer)

The Bcfg2.Options module

Bcfg2.Options.setup = Namespace(name='Bcfg2', uri='http://trac.mcs.anl.gov/projects/bcfg2', version='1.4.0pre2')

Simple object for storing attributes.

Implements equality by attribute names and values, and provides a simple string representation.

Options

The base Bcfg2.Options.Option object represents an option. Unlike options in argparse, an Option object does not need to be associated with an option parser; it exists on its own.

class Bcfg2.Options.Option(*args, **kwargs)[source]

Bases: object

Representation of an option that can be specified on the command line, as an environment variable, or in a config file. Precedence is in that order; that is, an option specified on the command line takes precendence over an option given by the environment, which takes precedence over an option specified in the config file.

See argparse.ArgumentParser.add_argument() for a full list of accepted parameters.

In addition to supporting all arguments and keyword arguments from argparse.ArgumentParser.add_argument(), several additional keyword arguments are allowed.

Parameters:
  • cf (tuple) – A tuple giving the section and option name that this argument can be referenced as in the config file. The option name may contain the wildcard ‘*’, in which case the value will be a dict of all options matching the glob. (To use a wildcard in the section, use a Bcfg2.Options.WildcardSectionGroup.)
  • env (string) – An environment variable that the value of this option can be taken from.
  • man (string) – A detailed description of the option that will be used to populate automatically-generated manpages.
actions = None

A dict of Bcfg2.Options.Parser -> argparse.Action that gives the actions that resulted from adding this option to each parser that it was added to. If this option cannot be specified on the command line (i.e., it only takes its value from the config file), then this will be empty.

add_to_parser(parser)[source]

Add this option to the given parser.

Parameters:parser (Bcfg2.Options.Parser) – The parser to add the option to.
Returns:argparse.Action
args = None

The options by which this option can be called. (Coincidentally, this is also the list of arguments that will be passed to argparse.ArgumentParser.add_argument() when this option is added to a parser.) As a result, args can be tested to see if this argument can be given on the command line at all, or if it is purely a config file option.

cf = None

The tuple giving the section and option name for this option in the config file

default

The current default value of this option

default_from_config(cfp)[source]

Set the default value of this option from the config file or from the environment.

Parameters:cfp (ConfigParser.ConfigParser) – The config parser to get the option value from
dest

The namespace destination of this option (see dest)

early_parsing_hook(early_opts)[source]

Hook called at the end of early option parsing.

This can be used to save option values for macro fixup.

env = None

The environment variable that this option can take its value from

finalize(namespace)[source]

Finalize the default value for this option. This is used with actions (such as Bcfg2.Options.ComponentAction) that allow you to specify a default in a different format than its final storage format; this can be called after it has been determined that the default will be used (i.e., the option is not given on the command line or in the config file) to store the appropriate default value in the appropriate format.

from_config(cfp)[source]

Get the value of this option from the given ConfigParser.ConfigParser. If it is not found in the config file, the default is returned. (If there is no default, None is returned.)

Parameters:cfp (ConfigParser.ConfigParser) – The config parser to get the option value from
Returns:The default value
get_config_value(cfp)[source]

fetch a value from the config file.

This is passed the config parser. Its result is passed to the type function for this option. It can be overridden to, e.g., handle boolean options.

get_environ_value(value)[source]

fetch a value from the environment.

This is passed the raw value from the environment variable, and its result is passed to the type function for this option. It can be overridden to, e.g., handle boolean options.

list_options()[source]

List options contained in this option. This exists to provide a consistent interface with Bcfg2.Options.OptionGroup

man = None

A detailed description of this option that will be used in man pages.

parsers = None

A list of Bcfg2.Options.Parser objects to which this option has been added. (There will be more than one parser if this option is added to a subparser, for instance.)

class Bcfg2.Options.PathOption(*args, **kwargs)[source]

Bases: Bcfg2.Options.Options.RepositoryMacroOption

Shortcut for options that expect a path argument.

Uses Bcfg2.Options.Types.path() to transform the argument into a canonical path. The type of a path option can also be overridden to return a file-like object. For example:

options = [
    Bcfg2.Options.PathOption(
        "--input", type=argparse.FileType('r'),
        help="The input file")]

PathOptions also do translation of <repository> macros.

transform_value(value)[source]

transform the value after macro expansion.

this can be overridden to further transform the value set by

the user after macros are expanded, but before the user’s type function is applied. principally exists for PathOption to canonicalize the path.

class Bcfg2.Options.BooleanOption(*args, **kwargs)[source]

Bases: Bcfg2.Options.Options.Option

Shortcut for boolean options. The default is False, but this can easily be overridden:

options = [
    Bcfg2.Options.PathOption(
        "--dwim", default=True, help="Do What I Mean")]
get_config_value(cfp)[source]

fetch a value from the config file.

This is passed the config parser. Its result is passed to the type function for this option. It can be overridden to, e.g., handle boolean options.

get_environ_value(value)[source]

fetch a value from the environment.

This is passed the raw value from the environment variable, and its result is passed to the type function for this option. It can be overridden to, e.g., handle boolean options.

class Bcfg2.Options.PositionalArgument(*args, **kwargs)[source]

Bases: Bcfg2.Options.Options.Option

Shortcut for positional arguments.

The Parser

class Bcfg2.Options.Parser(**kwargs)[source]

Bases: argparse.ArgumentParser

The Bcfg2 option parser. Most interfaces should not need to instantiate a parser, but should instead use Bcfg2.Options.get_parser() to get the parser that already exists.

See argparse.ArgumentParser for a full list of accepted parameters.

In addition to supporting all arguments and keyword arguments from argparse.ArgumentParser, several additional keyword arguments are allowed.

Parameters:
add_component(component)[source]

Add a component (and all of its options) to the parser.

add_config_file(dest, cfile, reparse=True)[source]

Add a config file, which triggers a full reparse of all options.

add_options(options)[source]

Add an explicit list of options to the parser. When possible, prefer Bcfg2.Options.Parser.add_component() to add a whole component instead.

argv = None

The argument list that was parsed.

components = None

Components that have been added to the parser

configfile = PathOption(config: sources=['-C', '--config', '$BCFG2_CONFIG_FILE'], default=/etc/bcfg2.conf, 1 parsers)

Option for specifying the path to the Bcfg2 config file

namespace = None

The namespace options will be stored in.

option_list = None

Options that have been added to the parser

options = [PathOption(config: sources=['-C', '--config', '$BCFG2_CONFIG_FILE'], default=/etc/bcfg2.conf, 1 parsers), Option(version: sources=['--version'], default=None, 1 parsers), Option(encoding: sources=['-E', '--encoding', 'components.encoding'], default=UTF-8, 1 parsers)]

Builtin options that apply to all commands

parse(argv=None)[source]

Parse options.

Parameters:argv (list) – The argument list to parse. By default, sys.argv[1:] is used. This is stored in Bcfg2.Options.Parser.argv for reuse by Bcfg2.Options.Parser.reparse().
parsed = None

Whether or not parsing has completed on all current options.

reparse(argv=None)[source]

Reparse options after they have already been parsed.

Parameters:argv (list) – The argument list to parse. By default, Bcfg2.Options.Parser.argv is reused. (I.e., the argument list that was initially parsed.)
unit_test = False

Flag used in unit tests to disable actual config file reads

Bcfg2.Options.get_parser(description=None, components=None, namespace=None)[source]

Get an existing Bcfg2.Options.Parser object.

A Parser is created at the module level when Bcfg2.Options is imported. If any arguments are given, then the existing parser is modified before being returned.

Parameters:
  • description (string) – Set the parser description
  • components (list) – Load the given components in the parser
  • namespace (argparse.Namespace) – Use the given namespace instead of Bcfg2.Options.setup
Returns:

Bcfg2.Options.Parser object

exception Bcfg2.Options.OptionParserException[source]

Bases: exceptions.Exception

Base exception raised for generic option parser errors

Option Groups

Options can be grouped in various meaningful ways. This uses a variety of argparse functionality behind the scenes.

In all cases, options can be added to groups in-line by simply specifying them in the object group constructor:

options = [
    Bcfg2.Options.ExclusiveOptionGroup(
        Bcfg2.Options.Option(...),
        Bcfg2.Options.Option(...),
        required=True),
    ....]

Nesting object groups is supported in theory, but barely tested.

class Bcfg2.Options.OptionGroup(*items, **kwargs)[source]

Bases: Bcfg2.Options.OptionGroups._OptionContainer

Generic option group that is used only to organize options. This uses argparse.ArgumentParser.add_argument_group() behind the scenes.

Parameters:
  • *args (Bcfg2.Options.Option) – Child options
  • title (string) – The title of the option group
  • description – A longer description of the option group
  • description – string
add_to_parser(parser)[source]

Add this option group to a Bcfg2.Options.Parser object.

class Bcfg2.Options.ExclusiveOptionGroup(*items, **kwargs)[source]

Bases: Bcfg2.Options.OptionGroups._OptionContainer

Option group that ensures that only one argument in the group is present. This uses argparse.ArgumentParser.add_mutually_exclusive_group() behind the scenes.

Parameters:
  • *args (Bcfg2.Options.Option) – Child options
  • required (boolean) – Exactly one argument in the group must be specified.
add_to_parser(parser)[source]

Add this option group to a Bcfg2.Options.Parser object.

class Bcfg2.Options.Subparser(*items, **kwargs)[source]

Bases: Bcfg2.Options.OptionGroups._OptionContainer

Option group that adds options in it to a subparser. This uses a lot of functionality tied to argparse Sub-commands.

The subcommand string itself is stored in the Bcfg2.Options.setup namespace as subcommand.

This is commonly used with Bcfg2.Options.Subcommand groups.

Parameters:
  • *args (Bcfg2.Options.Option) – Child options
  • name (string) – The name of the subparser. Required.
  • help – A help message for the subparser
  • help – string
add_to_parser(parser)[source]

Add this option group to a Bcfg2.Options.Parser object.

class Bcfg2.Options.WildcardSectionGroup(*items, **kwargs)[source]

Bases: Bcfg2.Options.OptionGroups._OptionContainer, Bcfg2.Options.Options.Option

WildcardSectionGroups contain options that may exist in several different sections of the config that match a glob. It works by creating options on the fly to match the sections described in the glob. For example, consider:

options = [
    Bcfg2.Options.WildcardSectionGroup(
        Bcfg2.Options.Option(cf=("myplugin:*", "number"), type=int),
        Bcfg2.Options.Option(cf=("myplugin:*", "description"))]

If the config file contained [myplugin:foo] and [myplugin:bar] sections, then this would automagically create options for each of those. The end result would be:

>>> Bcfg2.Options.setup
Namespace(myplugin_bar_description='Bar description', myplugin_myplugin_bar_number=2, myplugin_myplugin_foo_description='Foo description', myplugin_myplugin_foo_number=1, myplugin_sections=['myplugin:foo', 'myplugin:bar'])

All options must have the same section glob.

The options are stored in an automatically-generated destination given by:

<prefix><section>_<destination>

<destination> is the original dest of the option. <section> is the section that it’s found in. <prefix> is automatically generated from the section glob. (This can be overridden with the constructor.) Both <section> and <prefix> have had all consecutive characters disallowed in Python variable names replaced with underscores.

This group stores an additional option, the sections themselves, in an option given by <prefix>sections.

Parameters:
  • *args (Bcfg2.Options.Option) – Child options
  • prefix (string) – The prefix to use for options generated by this option group. By default this is generated automatically from the config glob; see above for details.
  • dest – The destination for the list of known sections that match the glob.
  • dest – string
add_to_parser(parser)[source]

Add this option group to a Bcfg2.Options.Parser object.

from_config(cfp)[source]

Get the value of this option from the given ConfigParser.ConfigParser. If it is not found in the config file, the default is returned. (If there is no default, None is returned.)

Parameters:cfp (ConfigParser.ConfigParser) – The config parser to get the option value from
Returns:The default value
list_options()[source]

Get a list of all options contained in this group, including options contained in option groups in this group, and so on.

Subcommands

This library makes it easier to work with programs that have a large number of subcommands (e.g., bcfg2-info and bcfg2-admin).

The normal implementation pattern is this:

  1. Define all of your subcommands as children of Bcfg2.Options.Subcommand.
  2. Create a Bcfg2.Options.CommandRegistry object that will be used to register all of the commands. Registering a command collect its options and adds it as a Bcfg2.Options.Subparser option group to the main option parser.
  3. Register your commands with the Bcfg2.Options.CommandRegistry.register_commands() method of your CommandRegistry object.
  4. Add options from the Bcfg2.Options.CommandRegistry.command_options attribute to the option parser.
  5. Parse options, and run.

Bcfg2.Server.Admin provides a fairly simple implementation, where the CLI class subclasses the command registry:

class CLI(Bcfg2.Options.CommandRegistry):
    def __init__(self):
        Bcfg2.Options.CommandRegistry.__init__(self)
        self.register_commands(globals().values(), parent=AdminCmd)
        parser = Bcfg2.Options.get_parser(
            description="Manage a running Bcfg2 server",
            components=[self])
        parser.add_options(self.subcommand_options)
        parser.parse()

In this case, commands are collected from amongst all global variables (the most likely scenario), and they must be children of Bcfg2.Server.Admin.AdminCmd, which itself subclasses Bcfg2.Options.Subcommand.

Commands are defined by subclassing Bcfg2.Options.Subcommand. At a minimum, the Bcfg2.Options.Subcommand.run() method must be overridden, and a docstring written.

class Bcfg2.Options.Subcommand[source]

Bases: object

Base class for subcommands. This must be subclassed to create commands.

Specifically, you must override Bcfg2.Options.Subcommand.run(). You may want to override:

You should not need to override Bcfg2.Options.Subcommand.__call__() or Bcfg2.Options.Subcommand.usage().

A Subcommand subclass constructor must not take any arguments.

aliases = []

Additional aliases for the command. The contents of the list gets added to the default command name (the lowercased class name)

help = None

Longer help message

interactive = True

Whether or not to expose this command in an interactive cmd.Cmd shell, if one is used. (bcfg2-info uses one, bcfg2-admin does not.)

logger = None

A logging.Logger that can be used to produce logging output for this command.

only_interactive = False

Whether or not to expose this command as command line parameter or only in an interactive cmd.Cmd shell.

options = []

Options this command takes

parser = None

The Bcfg2.Options.Parser that will be used to parse options if this subcommand is called from an interactive cmd.Cmd shell.

run(setup)[source]

Run the command.

Parameters:setup (argparse.Namespace) – A namespace giving the options for this command. This must be used instead of Bcfg2.Options.setup because this command may have been called from an interactive cmd.Cmd shell, and thus has its own option parser and its own (private) namespace. setup is guaranteed to contain all of the options in the global Bcfg2.Options.setup namespace, in addition to any local options given to this command from the interactive shell.
shutdown()[source]

Perform any necessary shutdown tasks for this command This is called to when the program exits (not when this command is finished executing).

usage()[source]

Get the short usage message.

class Bcfg2.Options.CommandRegistry[source]

Bases: object

A CommandRegistry is used to register subcommands and provides a single interface to run them. It’s also used by Bcfg2.Options.Subcommands.Help to produce help messages for all available commands.

commands = None

A dict of registered commands. Keys are the class names, lowercased (i.e., the command names), and values are instances of the command objects.

help = None

the help command

register_command(cls_or_obj)[source]

Register a single command.

Parameters:cls_or_obj (type or Subcommand) – The command class or object to register
Returns:An instance of cmdcls
register_commands(candidates, parent=<class 'Bcfg2.Options.Subcommands.Subcommand'>)[source]

Register all subcommands in candidates against the Bcfg2.Options.CommandRegistry subclass given in registry. A command is registered if and only if:

  • It is a subclass of the given parent (by default, Bcfg2.Options.Subcommand);
  • It is not the parent class itself; and
  • Its name does not start with an underscore.
Parameters:
runcommand()[source]

Run the single command named in Bcfg2.Options.setup.subcommand, which is where Bcfg2.Options.Subparser groups store the subcommand.

shutdown()[source]

Perform shutdown tasks.

This calls the shutdown method of the subcommand that was run.

subcommand_options = None

A list of options that should be added to the option parser in order to handle registered subcommands.

Actions

Several custom argparse actions provide some of the option collection magic of Bcfg2.Options.

class Bcfg2.Options.ConfigFileAction(*args, **kwargs)[source]

Bases: Bcfg2.Options.Actions.FinalizableAction

ConfigFileAction automatically loads and parses a supplementary config file (e.g., bcfg2-web.conf or bcfg2-lint.conf).

class Bcfg2.Options.ComponentAction(*args, **kwargs)[source]

Bases: Bcfg2.Options.Actions.FinalizableAction

ComponentAction automatically imports classes and modules based on the value of the option, and automatically collects options from the loaded classes and modules. It cannot be used by itself, but must be subclassed, with either mapping or bases overridden. See Bcfg2.Options.PluginsAction for an example.

ComponentActions expect to be given a list of class names. If bases is overridden, then it will attempt to import those classes from identically named modules within the given bases. For instance:

class FooComponentAction(Bcfg2.Options.ComponentAction):
    bases = ["Bcfg2.Server.Foo"]


class FooLoader(object):
    options = [
        Bcfg2.Options.Option(
            "--foo",
            type=Bcfg2.Options.Types.comma_list,
            default=["One"],
            action=FooComponentAction)]

If “–foo One,Two,Three” were given on the command line, then FooComponentAction would attempt to import Bcfg2.Server.Foo.One.One, Bcfg2.Server.Foo.Two.Two, and Bcfg2.Server.Foo.Three.Three. (It would also call Bcfg2.Options.Parser.add_component() with each of those classes as arguments.)

Note that, although ComponentActions expect lists of components (by default; this can be overridden by setting islist), you must still explicitly specify a type argument to the Bcfg2.Options.Option constructor to split the value into a list.

Note also that, unlike other actions, the default value of a ComponentAction option does not need to be the actual literal final value. (I.e., you don’t have to import Bcfg2.Server.Foo.One.One and set it as the default in the example above; the string “One” suffices.)

bases = []

A list of parent modules where modules or classes should be imported from.

fail_silently = False

If fail_silently is True, then failures to import modules or classes will not be logged. This is useful when the default is to import everything, some of which are expected to fail.

islist = True

By default, ComponentActions expect a list of components to load. If islist is False, then it will only expect a single component.

mapping = {}

A mapping of <name> => <object> that components will be loaded from. This can be used to permit much more complex behavior than just a list of bases.

module = False

If module is True, then only the module will be loaded, not a class from the module. For instance, in the example above, FooComponentAction would attempt instead to import Bcfg2.Server.Foo.One, Bcfg2.Server.Foo.Two, and Bcfg2.Server.Foo.Three.

class Bcfg2.Options.PluginsAction(*args, **kwargs)[source]

Bases: Bcfg2.Options.Actions.ComponentAction

Bcfg2.Options.ComponentAction subclass for loading Bcfg2 server plugins.

Option Types

Bcfg2.Options provides a number of useful types for use as the type keyword argument to the Bcfg2.Options.Option constructor.

Bcfg2.Options.Types.path(value)[source]

A generic path. ~ will be expanded with os.path.expanduser() and the absolute resulting path will be used. This does not ensure that the path exists.

Bcfg2.Options.Types.comma_list(value)[source]

Split a comma-delimited list, with optional whitespace around the commas.

Bcfg2.Options.Types.colon_list(value)[source]

Split a colon-delimited list. Whitespace is not allowed around the colons.

Bcfg2.Options.Types.octal(value)[source]

Given an octal string, get an integer representation.

Bcfg2.Options.Types.username(value)[source]

Given a username or numeric UID, get a numeric UID. The user must exist.

Bcfg2.Options.Types.groupname(value)[source]

Given a group name or numeric GID, get a numeric GID. The user must exist.

Bcfg2.Options.Types.timeout(value)[source]

Convert the value into a float or None.

Bcfg2.Options.Types.size(value)[source]

Given a number of bytes in a human-readable format (e.g., ‘512m’, ‘2g’), get the absolute number of bytes as an integer.

Common Options

class Bcfg2.Options.Common[source]

Bases: object

Common options used in multiple different contexts.

client_timeout = Option(None: sources=['-t', '--timeout', 'communication.timeout'], default=90.0, 0 parsers)

Client timeout

daemon = PathOption(None: sources=['-D', '--daemon'], default=None, 0 parsers)

Daemonize process, storing PID

default_paranoid = Option(default_paranoid: sources=['mdata.paranoid'], default=true, 1 parsers)

Default Path paranoid setting

interactive = BooleanOption(None: sources=['-I', '--interactive'], default=False, 0 parsers)

Run interactively, prompting the user for each change

location = Option(None: sources=['-S', '--server', 'components.bcfg2'], default=https://localhost:6789, 0 parsers)

Server location

password = Option(None: sources=['-x', '--password', 'communication.password'], default=None, 0 parsers)

Communication password

protocol = Option(protocol: sources=['communication.protocol'], default=xmlrpc/tlsv1, 0 parsers)

Communication protocol

repository = PathOption(repository: sources=['-Q', '--repository', 'server.repository'], default=/var/lib/bcfg2, 1 parsers)

Set the path to the Bcfg2 repository

ssl_ca = PathOption(ca: sources=['communication.ca'], default=None, 0 parsers)

Path to SSL CA certificate

syslog = BooleanOption(syslog: sources=['logging.syslog'], default=True, 0 parsers)

Log to syslog