File Monitor Development

Bcfg2 depends heavily on file activity monitoring (FAM) to reload data from disk when it changes. A number of FAM backends are supported (documented thoroughly below), but you may wish to develop additional backends. For instance, the current best FAM backend on Linux is INotify, but if you are running a non-Linux system that lacks INotify support you may wish to write a backend for your OS (e.g., a kqueue backend for BSD-based Bcfg2 servers). This page documents the FAM API and the existing FAM backends.

Event Codes

Five event codes are generally understood:

Event	Description
exists	Produced when a monitor is added to a file or directory that exists, and produced for all files or directories inside a directory that is monitored (non-recursively).
endExist	Produced immediately after exists. No plugins should process this event meaningfully, so FAM backends do not need to produce it.
created	Produced when a file is created inside a monitored directory.
changed	Produced when a monitored file, or a file inside a monitored directory, is changed.
deleted	Produced when a monitored file, or a file inside a monitored directory, is deleted.

Basics

Bcfg2.Server.FileMonitor provides the support for monitoring files. The FAM acts as a dispatcher for events: An event is detected on a file (e.g., the file content is changed), and then that event is dispatched to the HandleEvent method of an object that knows how to handle the event. Consequently, Bcfg2.Server.FileMonitor.FileMonitor.AddMonitor() takes two arguments: the path to monitor, and the object that handles events detected on that event.

HandleEvent is called with a single argument, the Bcfg2.Server.FileMonitor.Event object to be handled.

Assumptions

The FAM API Bcfg2 uses is based on the API of SGI’s File Alteration Monitor (also called “FAM”). Consequently, a few assumptions apply:

• When a file or directory is monitored for changes, we call that a “monitor”; other backends my use the term “watch,” but for consistency we will use “monitor.”
• Monitors can be set on files or directories.
• A monitor set on a directory monitors all files within that directory, non-recursively. If the object that requested the monitor wishes to monitor recursively, it must implement that itself.
• Setting a monitor immediately produces “exists” and “endExist” events for the monitored file or directory and all files or directories contained within it (non-recursively).
• An event on a file or directory that is monitored directly yields the full path to the file or directory.
• An event on a file or directory that is only contained within a monitored directory yields the relative path to the file or directory within the monitored parent. It is the responsibility of the handler to reconstruct full paths as necessary.
• Each monitor that is set must have a unique ID that identifies it, in order to make it possible to reconstruct full paths as necessary. This ID will be stored in Bcfg2.Server.FileMonitor.FileMonitor.handles. It may be any hashable value; some FAM backends use monotonically increasing integers, while others use the path to the monitor.

Base Classes

class Bcfg2.Server.FileMonitor.Event(request_id, filename, code)

Bases: object

Base class for all FAM events.

Parameters:

• request_id (Varies) – The handler ID of the monitor that produced this event
• filename (string) – The file or directory on which the event was detected. An event on a file or directory that is monitored directly yields the full path to the file or directory; an event on a file or directory that is only contained within a monitored directory yields the relative path to the file or directory within the monitored parent.
• code (string) – The event code produced. I.e., the type of event.

action = None

The event code produced. I.e., the type of event.

code2str()

Return the event code for this event. This is just an alias for action.

filename = None

The file or directory on which the event was detected. An event on a file or directory that is monitored directly yields the full path to the file or directory; an event on a file or directory that is only contained within a monitored directory yields the relative path to the file or directory within the monitored parent.

requestID = None

The handler ID of the monitor that produced this event

class Bcfg2.Server.FileMonitor.FileMonitor

Bases: Bcfg2.Logger.Debuggable

The base class that all FAM implementions must inherit.

The simplest instance of a FileMonitor subclass needs only to add monitor objects to handles and received events to events; the basic interface will handle the rest.

Parameters:

• ignore (list of strings (filename globs)) – A list of filename globs describing events that should be ignored (i.e., not processed by any object)
• debug (bool) – Produce debugging information about the events received and handled.

__priority__ = -1

The relative priority of this FAM backend. Better backends should have higher priorities.

AddMonitor(path, obj, handleID=None)

Monitor the specified path, alerting obj to events. This method must be overridden by a subclass of Bcfg2.Server.FileMonitor.FileMonitor.

Parameters:

• path (string) – The path to monitor
• obj (Varies) – The object whose HandleEvent method will be called when an event is produced.
• handleID (Varies) – The handle ID to use for the monitor. This is useful when requests to add a monitor must be enqueued and the actual monitors added after start() is called.

Returns:

Varies - The handler ID for the newly created monitor

events = None

Queue of events to handle

fileno()

Get the file descriptor of the file monitor thread.

Returns:int - The FD number

get_event()

Get the oldest pending event in events.

Returns:Bcfg2.Server.FileMonitor.Event

handle_event_set(lock=None)

Handle all pending events.

Parameters:lock (threading.Lock) – A thread lock to use while handling events. If None, then no thread locking will be performed. This can possibly lead to race conditions in event handling, although it’s unlikely to cause any real problems. Returns:None

handle_events_in_interval(interval)

Handle events for the specified period of time (in seconds). This call will block for interval seconds and handle all events received during that period by calling handle_event_set().

Parameters:interval (int) – The interval, in seconds, during which events should be handled. Any events that are already pending when handle_events_in_interval() is called will also be handled. Returns:None

handle_one_event(event)

Handle the given event by dispatching it to the object that handles it. This is only called by handle_event_set(), so if a backend overrides that method it does not necessarily need to implement this function.

Parameters:event (Bcfg2.Server.FileMonitor.Event) – The event to handle. Returns:None

handles = None

A dict that records which objects handle which events. Keys are monitor handle IDs and values are objects whose HandleEvent method will be called to handle an event

ignore = None

List of filename globs to ignore events for. For events that include the full path, both the full path and the bare filename will be checked against ignore.

list_event_handlers()

XML-RPC that returns Bcfg2.Server.FileMonitor.FileMonitor.handles for debugging purposes.

pending()

Returns True if there are pending events (i.e., events in events that have not been processed), False otherwise.

should_ignore(event)

Returns True if an event should be ignored, False otherwise. For events that include the full path, both the full path and the bare filename will be checked against ignore. If the event is ignored, a debug message will be logged with debug_log().

Parameters:event (Bcfg2.Server.FileMonitor.Event) – Check if this event matches ignore Returns:bool - Whether not to ignore the event

shutdown()

Handle any tasks required to shut down the monitor.

start()

Start threads or anything else that needs to be done after the server forks and daemonizes. Note that monitors may (and almost certainly will) be added before start() is called, so if a backend depends on being started to add monitors, those requests will need to be enqueued and added after start(). See Bcfg2.Server.FileMonitor.Inotify.Inotify for an example of this.

started = None

Whether or not the FAM has been started. See start().

Bcfg2.Server.FileMonitor.available = {'default': <class 'Bcfg2.Server.FileMonitor.Inotify.Inotify'>, 'gamin': <class 'Bcfg2.Server.FileMonitor.Gamin.Gamin'>, 'inotify': <class 'Bcfg2.Server.FileMonitor.Inotify.Inotify'>, 'pseudo': <class 'Bcfg2.Server.FileMonitor.Pseudo.Pseudo'>}

A dict of all available FAM backends. Keys are the human-readable names of the backends, which are used in bcfg2.conf to select a backend; values are the backend classes. In addition, the default key will be set to the best FAM backend as determined by Bcfg2.Server.FileMonitor.FileMonitor.__priority__

Bcfg2.Server.FileMonitor.get_fam()

Get a Bcfg2.Server.FileMonitor.FileMonitor object. If _FAM has not been populated, then a new default FileMonitor will be created.

Returns:Bcfg2.Server.FileMonitor.FileMonitor

Existing FAM Backends

Pseudo

Pseudo provides static monitor support for file alteration events. That is, it only produces “exists” and “endExist” events and does not monitor for ongoing changes.

class Bcfg2.Server.FileMonitor.Pseudo.Pseudo

Bases: Bcfg2.Server.FileMonitor.FileMonitor

File monitor that only produces events on server startup and doesn’t actually monitor for ongoing changes at all.

Parameters:

• ignore (list of strings (filename globs)) – A list of filename globs describing events that should be ignored (i.e., not processed by any object)
• debug (bool) – Produce debugging information about the events received and handled.

__priority__ = 1

The Pseudo monitor should only be used if no other FAM backends are available.

AddMonitor(path, obj, handleID=None)

Monitor the specified path, alerting obj to events. This method must be overridden by a subclass of Bcfg2.Server.FileMonitor.FileMonitor.

Parameters:

• path (string) – The path to monitor
• obj (Varies) – The object whose HandleEvent method will be called when an event is produced.
• handleID (Varies) – The handle ID to use for the monitor. This is useful when requests to add a monitor must be enqueued and the actual monitors added after start() is called.

Returns:

Varies - The handler ID for the newly created monitor

Gamin

File monitor backend with Gamin support.

class Bcfg2.Server.FileMonitor.Gamin.Gamin

Bases: Bcfg2.Server.FileMonitor.FileMonitor

File monitor backend with Gamin support. Deprecated.

Parameters:

• ignore (list of strings (filename globs)) – A list of filename globs describing events that should be ignored (i.e., not processed by any object)
• debug (bool) – Produce debugging information about the events received and handled.

__priority__ = 50

The Gamin backend is deprecated, but better than pseudo, so it has a medium priority.

AddMonitor(path, obj, handle=None)

Monitor the specified path, alerting obj to events. This method must be overridden by a subclass of Bcfg2.Server.FileMonitor.FileMonitor.

Parameters:

• path (string) – The path to monitor
• obj (Varies) – The object whose HandleEvent method will be called when an event is produced.
• handleID (Varies) – The handle ID to use for the monitor. This is useful when requests to add a monitor must be enqueued and the actual monitors added after start() is called.

Returns:

Varies - The handler ID for the newly created monitor

add_q = None

The queue used to record monitors that are added before start() has been called and mon is created.

counter = None

The counter used to produce monotonically increasing monitor handle IDs

fileno()

Get the file descriptor of the file monitor thread.

Returns:int - The FD number

get_event()

Get the oldest pending event in events.

Returns:Bcfg2.Server.FileMonitor.Event

mon = None

The Gamin.WatchMonitor object for this monitor.

pending()

Returns True if there are pending events (i.e., events in events that have not been processed), False otherwise.

queue(path, action, request_id)

Create a new GaminEvent and add it to the events queue for later handling.

start()

The Gamin watch monitor in mon must be created by the daemonized process, so is created in start(). Before the Gamin.WatchMonitor object is created, monitors are added to add_q, and are created once the watch monitor is created.

class Bcfg2.Server.FileMonitor.Gamin.GaminEvent(request_id, filename, code)

Bases: Bcfg2.Server.FileMonitor.Event

This class maps Gamin event constants to FAM event codes.

Parameters:

• request_id (Varies) – The handler ID of the monitor that produced this event
• filename (string) – The file or directory on which the event was detected. An event on a file or directory that is monitored directly yields the full path to the file or directory; an event on a file or directory that is only contained within a monitored directory yields the relative path to the file or directory within the monitored parent.
• code (string) – The event code produced. I.e., the type of event.

action_map = {1: 'changed', 2: 'deleted', 5: 'created', 8: 'exists', 9: 'endExist'}

The map of gamin event constants (which mirror FAM event names closely) to event codes

Inotify

File monitor backend with inotify support.

class Bcfg2.Server.FileMonitor.Inotify.Inotify

Bases: Bcfg2.Server.FileMonitor.Pseudo.Pseudo, pyinotify.ProcessEvent

File monitor backend with inotify support.

AddMonitor(path, obj, handleID=None)

Monitor the specified path, alerting obj to events. This method must be overridden by a subclass of Bcfg2.Server.FileMonitor.FileMonitor.

Parameters:

• path (string) – The path to monitor
• obj (Varies) – The object whose HandleEvent method will be called when an event is produced.
• handleID (Varies) – The handle ID to use for the monitor. This is useful when requests to add a monitor must be enqueued and the actual monitors added after start() is called.

Returns:

Varies - The handler ID for the newly created monitor

action_map = {2: 'changed', 64: 'deleted', 128: 'created', 256: 'created', 512: 'deleted'}

Map pyinotify event constants to FAM event codes. The mapping is not terrifically exact.

add_q = None

The queue used to record monitors that are added before start() has been called and notifier and watchmgr are created.

event_filter = None

inotify can’t set useful monitors directly on files, only on directories, so when a monitor is added on a file we add its parent directory to event_filter and then only produce events on a file in that directory if the file is listed in event_filter. Keys are directories – the parent directories of individual files that are monitored – and values are lists of full paths to files in each directory that events should be produced for. An event on a file whose parent directory is in event_filter but which is not itself listed will be silently suppressed.

fileno()

Get the file descriptor of the file monitor thread.

Returns:int - The FD number

list_paths()

XML-RPC that returns a list of paths that are handled for debugging purposes. Because inotify doesn’t like watching files, but prefers to watch directories, this will be different from Bcfg2.Server.FileMonitor.Inotify.Inotify.ListWatches(). For instance, if a plugin adds a monitor to /var/lib/bcfg2/Plugin/foo.xml, ListPaths() will return /var/lib/bcfg2/Plugin/foo.xml, while ListWatches() will return /var/lib/bcfg2/Plugin.

list_watches()

XML-RPC that returns a list of current inotify watches for debugging purposes.

mask = 962

The pyinotify event mask. We only ask for events that are listed in action_map

notifier = None

The pyinotify.ThreadedNotifier object. This is created in start() after the server is done daemonizing.

process_default(ievent)

Process all inotify events received. This process a pyinotify._Event object, creates a Bcfg2.Server.FileMonitor.Event object from it, and adds that event to events.

Parameters:ievent (pyinotify._Event) – Event to be processed

shutdown()

Handle any tasks required to shut down the monitor.

start()

The inotify notifier and manager objects in notifier and watchmgr must be created by the daemonized process, so they are created in start(). Before those objects are created, monitors are added to add_q, and are created once the pyinotify.ThreadedNotifier and pyinotify.WatchManager objects are created.

watches_by_path = None

inotify doesn’t like monitoring a path twice, so we keep a dict of pyinotify.Watch objects, keyed by monitor path, to avoid trying to create duplicate monitors. (Duplicates can happen if an object accidentally requests duplicate monitors, or if two files in a single directory are both individually monitored, since inotify can’t set monitors on the files but only on the parent directories.)

watchmgr = None

The pyinotify.WatchManager object. This is created in start() after the server is done daemonizing.