Bcfg2 Web Reporting System¶

Summary and Features¶

The new reporting system was implemented to address a number of deficiencies in the previous system. By storing statistics data in a relational database, we are now able to view and analyze more information about the state of the configuration, including information about previous configuration. Specific features in the new system include:

The ability to look at a Calendar Summary with past statistics information.
More recent data concerning hosts.
Additional information display in reports. Primarily, reasons for configuration item verification failure are now accessible.
Instead of static pages, pages are generated on the fly, allowing users to drill down to find out about a specific host, rather than only having one huge page with too much information.
Ability to store reporting data separately from other server data.

Installation¶

Quickstart¶

Web Reporting Quickstart

Prerequisites¶

sqlite3
pysqlite2 (if using python 2.4)
Django >= 1.3
mod-wsgi

Warning

There is a known issue when using an sqlite database on an ext4 filesystem. You will want to remount the filesystem without barriers (-o barrier=0) in order to speed up the operations of the database. For more information, please see http://phoronix-test-suite.com/pipermail/trondheim-pts_phoronix-test-suite.com/2009-March/000095.html.

Install¶

Be sure to include the specified fields included in the example bcfg2.conf file. These can be specified in either /etc/bcfg2.conf, if it is readable by the webserver user, or /etc/bcfg2-web.conf. Any database supported by Django can be used. As of version 1.3, South is used to control schema changes. If your database is not supported by South, any updates will need to be applied manually. Sqlite is configured by default. Please see the database section to configure alternative databases.

Warning

If you are using an sqlite database, the directory containing the database file will need to be writable by the web server. The reason for this is that sqlite will create another file for its journal when it tries to update the database file.

Note

Distributed environments can share a single remote database for reporting.
After configuring your database be sure to run bcfg2-admin reports init to create the schema.
To enable statistics collection in the bcfg2-server, add Reporting to the plugins line in your bcfg2.conf and restart the bcfg2-server. A report collecting daemon should be run to import the collected statistics into the backend. Please see the section Report Collector for more information.

Detailed installation instructions can be found here.

Apache configuration for web-based reports¶

Note

Reports no longer needs to be installed at the root URL for a given host. Therefore, reports no longer require their own virtual host.

In order to make this work, you will need to specify your web prefix by adding a web_prefix setting in the [statistics] section of your bcfg2.conf.

Warning

When running with SELINUX enabled, you can have potential problems with the WSGISocketPrefix. One solution that works without too much trouble is modifying your prefix so that it is located in a standard location:

WSGISocketPrefix /var/run/httpd/wsgi

An example site config is included below:

<IfModule mod_wsgi.c>
  #
  # Read an alternate configuration file
  #
  # SetEnv BCFG2_CONFIG_FILE /etc/bcfg2_testing.conf

  #
  # If the root is changed update the static content alias as well
  #
  WSGIScriptAlias /bcfg2 "/usr/share/bcfg2/reports.wsgi"

  WSGISocketPrefix run
  WSGIDaemonProcess Bcfg2.Server.Reports processes=1 threads=10
  WSGIProcessGroup Bcfg2.Server.Reports

  #
  # Manually set this to override the static content
  #
  #SetEnv bcfg2.media_url /bcfg2/site_media/

  #
  # This should have the same prefix as WSGIScriptAlias
  #
  Alias "/bcfg2/site_media/" "/usr/share/bcfg2/site_media/"
  <Directory "/usr/share/bcfg2/site_media/">
    Options None
    AllowOverride None
    order deny,allow
    deny from all
    allow from 127.0.0.1
  </Directory>
</IfModule>

This configuration is suitable for use with the default installation from an RPM or deb package.

At this point you should be able to point your web browser to http://localhost/bcfg2 and see the new reports.

Upgrading¶

Convert database config

Run tools/upgrade/1.3/migrate_configs.py

Beginning with 1.3 the database configuration moved from [statistics] to [database] in bcfg2.conf and bcfg2-web.conf. The old settings will be accepted but a deprecation warning will be displayed.
Replace the DBStats plugin with the Reporting plugin.
Migrate historic data.

Run tools/upgrade/1.3/migrate_dbstats.py

The reporting schema is now managed using South instead of a set of custom scripts. This creates the new schema and imports all of the historic data to the new format.

Note

After the database is upgraded all of the old tables are left intact. To remove them any table starting with reports_ can be dropped.
(Optional) Run the Report Collector

Add “transport = LocalFilesystem” under “[reporting]” in bcfg2.conf. Restart the bcfg2-server and start the bcfg2-report-collector.

Configuring¶

Most of the configuration is handled through the /etc/bcfg2.conf or alternatively /etc/bcfg2-web.conf.

An example using the defaults is listed below:

[database]
engine = sqlite3
name = /var/lib/bcfg2/etc/bcfg2.sqlite
user =
password =
host =
port =

[reporting]
transport = DirectStore
web_prefix =
file_limit = 1m

Configuration Sections¶

database¶

If you choose to use a different database, you’ll need to edit /etc/bcfg2.conf. These fields should be updated in the [database] section:

engine
- ex: engine = mysql
- ex: engine = postgresql_psycopg2
name
user
password
host
port (optional)

To store reporting data separately from the main server data, use the following options:

reporting_engine
- ex: reporting_engine = mysql
- ex: reporting_engine = postgresql_psycopg2
reporting_name
reporting_user
reporting_password
reporting_host
reporting_port (optional)

Warning

If mysql is used as a backend, it is recommended to use InnoDB for the storage engine.

Refer to Global Database Settings for a full listing of available options.

statistics¶

config: The config file to be read for additional reporting data. This is used to restrict what can be read by the web server.
time_zone: The django TIME_ZONE settings parameter.
web_debug: Set Django’s DEBUG and TEMPLATE_DEBUG settings. This is known to cause memory leaks. Use with caution!

reporting¶

transport: See Transports.
web_prefix: Prefix to be added to Django’s MEDIA_URL
file_limit: The maximum size of a diff or binary data to store in the database.
max_children: Maximum number of children for the reporting collector. Use 0 to disable the limit and spawn a thread as soon as a working file is available.

Statistics Transports¶

A transport is required to pass the data collected from the bcfg2-server to the bcfg2-report-collector. At the time of this writing two transports are available:

LocalFilesystem: Statistics are written to the local file system and collected on the local machine.
RedisTransport: Statistics are sent through a list in redis.
DirectStore: DBStats style threaded imports in the main server process.

Future transports will allow multiple servers to pass data to a single or multiple bcfg2-report-collector processes. New installations default to and should use the LocalFilesystem transport. Upgrades will use DirectStore by default in the 1.3 release.

Note

If DirectStore is used, the bcfg2-report-collector process will refuse to run since this method is not compatible with an external process.

RedisTransport¶

This transport uses a single redis instance for communication between bcfg2-server and bcfg2-report-collector. Multiple servers can write to a single redis instance and multiple report collectors may be run as well.

An example configuration with the default values:

[reporting]
transport = RedisTransport
redis_host = 127.0.0.1
redis_port = 6379
redis_db = 0

bcfg2-admin commands operate slightly differently in this mode. Instead of querying the database directly, rpc commands are issued to the report collectors. This only affects the minestruct and pull commands.

Warning

At the time of this writing the version of python-redis in EPEL is too old to use with this transport. Current versions of the python-redis package require python >= 2.5.

Usage¶

Report Collector daemon¶

Note

This section does not apply when the DirectStore transport is used.

The bcfg2-report-collector gathers statistics from the bcfg2-server process and records them in the backend database.

Options are similar to the bcfg2-server daemon:

-D <pidfile>                 Daemonize process, storing pid
-o <path>                    Set path of file log
-h                           Print this usage message
-E <encoding>                Encoding of cfg files
-W <conffile>                Web interface configuration file
-Q <repository path>         Server repository path
-C <conffile>                Specify configuration file
--version                    Print the version and exit
-d                           Enable debugging output
-v                           Enable verbose output

Note

The bcfg2-report-collector is not set to start by default

bcfg2-admin reports (command line script)¶

The bcfg2-admin tool provides management and maintenance capabilities for the reporting database. A few useful Django commands are provided as well.

init: Initialize a new database
update: Apply any updates to the reporting database. Unlike the syncdb command, this will modify existing tables.
purge: Removes unwanted clients and data.
- -c –client [client name] - Remove interactions from a single client.
- –expired - Remove all data for expired clients. –days is used to exclude clients expired within n days.
- –days [n] - Remove interactions older then n days. If not used with any other modifiers, all data older then n days is removed.
scrub: Scrub the database for any orphaned objects.

Django commands¶

dbshell: Connects to the backend database.
shell: Starts an interactive python shell with the Django environment setup.
sqlall: Print the sql statements used to create the database.
validate: Validate the database against the current models.

bcfg2-reports (command line script)¶

bcfg2-reports allows you to retrieve data from the database about clients, and the states of their current interactions. It also allows you to change the expired/unexpired states.

The utility runs as a standalone application. It does, however, use the models from /src/lib/Server/Reports/reports/models.py.

A number of different options can be used to change what bcfg2-reports displays:

Usage: python bcfg2-reports [option] ...

Options and arguments (and corresponding environment variables):
-a                        : shows all hosts, including expired hosts
-b NAME                   : single-host mode - shows bad entries from the
                            current interaction of NAME
-c                        : shows only clean hosts
-d                        : shows only dirty hosts
-e NAME                   : single-host mode - shows extra entries from the
                            current interaction of NAME
-h                        : shows help and usage info about bcfg2-reports
-m NAME                   : single-host mode - shows modified entries from the
                            current interaction of NAME
-s NAME                   : single-host mode - shows bad, modified, and extra
                            entries from the current interaction of NAME
-t NAME                   : single-host mode - shows total number of managed and
                            good entries from the current interaction of NAME
-x NAME                   : toggles expired/unexpired state of NAME
--badentry=KIND,NAME      : shows only hosts whose current interaction has bad
                            entries in of KIND kind and NAME name; if a single
                            argument ARG1 is given, then KIND,NAME pairs will be
                            read from a file of name ARG1
--modifiedentry=KIND,NAME : shows only hosts whose current interaction has
                            modified entries in of KIND kind and NAME name; if a
                            single argument ARG1 is given, then KIND,NAME pairs
                            will be read from a file of name ARG1
--extraentry=KIND,NAME    : shows only hosts whose current interaction has extra
                            entries in of KIND kind and NAME name; if a single
                            argument ARG1 is given, then KIND,NAME pairs will be
                            read from a file of name ARG1
--fields=ARG1,ARG2,...    : only displays the fields ARG1,ARG2,...
                            (name,time,state,total,good,bad)
--sort=ARG1,ARG2,...      : sorts output on ARG1,ARG2,...
                            (name,time,state,total,good,bad)
--stale                   : shows hosts which haven't run in the last 24 hours

Screenshots¶

Grid Overview¶

Detailed Overview¶

Calendar Summary¶

Client Detail¶

Common Problems¶

Bcfg2 Web Reporting System¶

Summary and Features¶

Installation¶

Quickstart¶

Prerequisites¶

Install¶

Apache configuration for web-based reports¶

Upgrading¶

Configuring¶

Configuration Sections¶

database¶

statistics¶

reporting¶

Statistics Transports¶

RedisTransport¶

Usage¶

Report Collector daemon¶

bcfg2-admin reports (command line script)¶

Django commands¶

bcfg2-reports (command line script)¶

Screenshots¶

Grid Overview¶

Detailed Overview¶

Calendar Summary¶

Client Detail¶

Common Problems¶

Item Listing¶

Item Detail¶

Table Of Contents

Previous topic

Next topic

This Page