Server-side Caching System

New in version 1.4.0.

Bcfg2 caches two kinds of data:

  • The contents of all files that it reads in, including (often) an optimized representation. E.g., XML files are cached both in their raw (text) format, and also as lxml.etree._Element objects.
  • Arbitrary data, in the server-side caching system documented on this page.

The caching system keeps a single unified cache with all cache data in it. Each individual datum stored in the cache is associated with any number of “tags” – simple terms that uniquely identify the datum. This lets you very easily expire related data from multiple caches at once; for isntance, for expiring all data related to a host:

Bcfg2.Server.Cache.expire("foo.example.com")

This would expire all data related to foo.example.com, regardless of which plugin cached it, and so on.

This permits a high level of interoperation between different plugins and the cache, which is necessary due to the wide distribution of data in Bcfg2 and the many different data sources that can be incorported. More technical details about writing code that uses the caches is below.

Currently known caches are:

Tags Key(s) Values Use
Metadata Hostname ClientMetadata The Metadata cache
Probes, probegroups Hostname list of group names Groups set by Probes
Probes, probedata Hostname dict of <probe name>: ProbeData Other data set by Probes
Packages, collections Packages Collection cache key Collection Kept by Packages in order to expire repository metadata cached on disk
Packages, clients Hostname Packages Collection cache key Used by the Packages plugin to return Collection objects for clients. This is cross-referenced with the Packages, collections cache
Packages, pkg_groups Packages Collection cache key, hash of the selected package groups set of package names Cached results from looking up <Package group="..."/> entries
Packages, pkg_sets Packages Collection cache key, hash of the initial package selection set of package names Cached results from resolving complete package sets for clients

These are enumerated so that they can be expired as needed by other plugins or other code points.

Bcfg2.Server.Cache is an implementation of a simple memory-backed cache. Right now this doesn’t provide many features, but more (time-based expiration, etc.) can be added as necessary.

The normal workflow is to get a Cache object, which is simply a dict interface to the unified cache that automatically uses a certain tag set. For instance:

groupcache = Bcfg2.Server.Cache.Cache("Probes", "probegroups")
groupcache['foo.example.com'] = ['group1', 'group2']

This would create a Cache object that automatically tags its entries with frozenset(["Probes", "probegroups"]), and store the list ['group1', 'group1'] with the additional tag foo.example.com. So the unified backend cache would then contain a single entry:

{frozenset(["Probes", "probegroups", "foo.example.com"]):
 ['group1', 'group2']}

In addition to the dict interface, Cache objects (returned from Bcfg2.Server.Cache.Cache()) have one additional method, expire(), which is mostly identical to Bcfg2.Server.Cache.expire(), except that it is specific to the tag set of the cache object. E.g., to expire all foo.example.com records for a given cache, you could do:

groupcache = Bcfg2.Server.Cache.Cache("Probes", "probegroups")
groupcache.expire("foo.example.com")

This is mostly functionally identical to:

Bcfg2.Server.Cache.expire("Probes", "probegroups", "foo.example.com")

It’s not completely identical, though; the first example will expire, at most, exactly one item from the cache. The second example will expire all items that are tagged with a superset of the given tags. To illustrate the difference, consider the following two examples:

groupcache = Bcfg2.Server.Cache.Cache("Probes")
groupcache.expire("probegroups")

Bcfg2.Server.Cache.expire("Probes", "probegroups")

The former will not expire any data, because there is no single datum tagged with "Probes", "probegroups". The latter will expire all items tagged with "Probes", "probegroups" – i.e., the entire cache. In this case, the latter call is equivalent to:

groupcache = Bcfg2.Server.Cache.Cache("Probes", "probegroups")
groupcache.expire()
Bcfg2.Server.Cache.Cache(*tags)[source]

A dict interface to the cache data tagged with the given tags.

Bcfg2.Server.Cache.add_expire_hook(func)[source]

Add a hook that will be called when an item is expired from the cache. The callable passed in must take three options: the first will be the tag set that was expired; the second will be the state of the exact flag (True or False); and the third will be the number of items that were expired from the cache.

Bcfg2.Server.Cache.expire(*tags, **kwargs)[source]

Expire all items, a set of items, or one specific item from the cache. If exact is set to True, then if the given tag set doesn’t match exactly one item in the cache, nothing will be expired.