Troubleshooting

From time to time, Bcfg2 produces results that the user finds surprising. This can happen either due to bugs or user error. This page describes several techniques to gain visibility into the bcfg2 client and server and understand what is going on.

Figure out if error is client or server side

  • Cache a copy of the client configuration using bcfg2 -qnc /tmp/config.xml
  • Look in the file and search for the entry of interest
  • If it looks correct, then there is a client issue
  • If not, it is time to inspect things on the server

This file contains all aspects of client configuration. It is structured as a series of bundles and base entries.

Note

Most often the entry is not correct and the issue lies in the specification.

Review server log messages

The bcfg2-server process logs to syslog facility LOG_DAEMON. The server produces a series of messages upon a variety of events and errors.

The server also supports two XML-RPC methods that can be used to turn up the debug level in a live server:

  • toggle_debug: Turn debug on or off, depending on the current setting.
  • set_debug: Turn debug explicitly on or off.

These can be called with bcfg2-admin xcmd, e.g.:

bcfg2-admin xcmd toggle_debug
bcfg2-admin xcmd set_debug true

Each plugin also supports these two methods, which can be used to set the debug level individually on a given plugin, e.g.:

bcfg2-admin xcmd Packages.set_debug true
bcfg2-admin xcmd Probes.toggle_debug

Finally, the File Activity Monitor has its own analogue to these two methods, for setting the debug level of the FAM:

bcfg2-admin xcmd Inotify.toggle_debug
bcfg2-admin xcmd Inotify.set_debug false

Check if all repository XML files conform to schemas

Bcfg2 comes with XML schemas describing all of the XML formats used in the server repository. A validation command bcfg2-lint is included with the source distribution and all packages.

If the bcfg2 server is not reflecting recent changes, try restarting the bcfg2-server process

If this fixes the problem, it is either a bug in the underlying file monitoring system (inotify or gamin) or a bug in Bcfg2’s file monitoring code. In either case, file a ticket in the tracking system. In the ticket, include:

  • filesystem monitoring system (inotify or gamin)
  • kernel version (if on linux)
  • if any messages of the form “Handled N events in M seconds” appeared between the modification event and the client configuration generation request appeared in the server log
  • which plugin handled the file in the repostiory (Cfg, Rules, Packages, SSHbase, Metadata, etc.)
  • if a touch of the file after the modification causes the problem to go away

bcfg2-info

Bcfg2 server operations can be simulated using the bcfg2-info command. The command is interactive, and has commands to allow several useful operations

  • clients - Current client metadata (profile and group) settings
  • groups - Current group metadata values
  • mappings - Configuration entries provided by plugins
  • buildfile [–altsrc=<altsrc>] <filename> <hostname> - Build a config file for a client
  • buildbundle <bundle> <hostname> - Render a templated bundle for a client
  • showentries <client> <type> - Build the abstract configuration (list of entries) for a client
  • build <hostname> <output-file> - Build the complete configuration for a client

Type help in bcfg2-info for more information.

Error Messages

The tables below describe error messages produced by Bcfg2 and steps that can be taken to remedy them.

Client Errors

Error Meaning Repair
Incomplete information for entry <EntryTag>:<EntryName> cannot verify The described entry is not fully specified by the server, so no verification can be performed. [c1]
Incomplete information for entry <EntryTag>:<EntryName> cannot install The described entry is not fully specified by the server, so no verification can be performed. [c1]
The following entries are not handled by any tool: <EntryTag>:<EntryName> The client cannot figure out how to handle this entry. [c2]
No ca is specified. Cannot authenticate the server with SSL. The client is unable to verify the server. [c3]
GID normalization failed for FILENAME. Does group GROUP exist? The client is unable to convert the group GROUP to a usable GID. [c4]
UID normalization failed for FILENAME. Does owner OWNER exist? The client is unable to convert the owner OWNER to a usable UID. [c5]
SSL CA error The CA certificate specified in bcfg2.conf is incorrect [c6]
[c1](1, 2) This entry is not being bound. Ensure that a version of this entry applies to this client.
[c2]It is possible that the type attribute for this generator entry is undefined. You may need to add the appropriate type attribute in the literal entry specification.
[c3]Copy the Bcfg2 server’s CA certificate to the client and specify it using the ca option in the [communication] section of bcfg2.conf
[c4]If the group doesn’t exist, you need to specify the correct one in an info.xml file or set the default group appropriately.
[c5]If the owner doesn’t exist, you need to specify the correct one in an info.xml file or set the default owner appropriately.
[c6]Check that the CA specified in bcfg2.conf is appropriate for the server you are attempting to access.

Server Errors

Error Meaning Repair
Failed to bind entry: <EntryTag> <EntryName> The server was unable to find a suitable version of entry for client. [s1]
Failed to bind to socket The server was unable to bind to the tcp server socket. [s2]
Failed to load ssl key <path> The server was unable to read and process the ssl key. [s3]
Failed to read file <path> The server failed to read the specified file [s4]
Failed to parse file <path> The server failed to parse the specified XML file [s5]
Client metadata resolution error for <IP> The server cannot resolve the client hostname or the client is associated with a non-profile group. [s6]
Failed to decode <filename> Please verify you are using the proper encoding The encoding being used is unable to decode the character present in this file. [s7]
Got unknown entries [list of unknown entries] The Packages plugin has no knowledge of the listed entries [s8]
Failed to import lxml dependency. Shutting down server. The server cannot import lxml [s9]
You need to specify base64 encoding for <path> The server cannot send the file as ascii text [s10]
ERROR: Error reading file ‘/path/to/schema’: failed to load external entity “/path/to/schema” The server cannot find the schema file [s11]
Packages: No matching sources for client <clientname>; improper group memberships? None of the sources defined in the Package plugin’s sources.xml apply to the client [s12]
[s1]This entry is not being bound. Ensure that a version of this entry applies to this client.
[s2]Ensure that another instance of the daemon (or any other process) is not listening on the same port.
[s3]Ensure that the key is readable by the user running the daemon and that it is well-formed.
[s4]Ensure that this file still exists; a frequent cause is the deletion of a temp file.
[s5]Ensure that the file is properly formed XML.
[s6]Fix hostname resolution for the client or ensure that the profile group is properly setup.
[s7]Ensure the correct encoding is specified in the [components] section of bcfg2.conf.
[s8]For packages listed other than gpg-pubkey, this error means that the Packages plugin is unable to find the package in any of the sources listed in Packages/sources.xml. The issue often arises when the client is not in one of the groups necessary for the Source listed. In the case of gpg-pubkey, you can safely ignore the message as the Packages plugin has no knowledge of these packages (however, note that this package is most often specified as a BoundPackage entry).
[s9]Ensure that you have installed all the necessary Prerequisites.
[s10]You likely need to specify a base64 encoding using an info.xml file for this entry.
[s11]Verify that you have the proper prefix set in bcfg2.conf.
[s12]Ensure that the client is a member of all the appropriate groups you may have defined in your Packages configuration.

FAQs

Why won’t bcfg2-server start?

If your server doesn’t seem to be starting and you see no error messages in your server logs, try running it in the foreground to see why.

Why am I getting a traceback?

If you get a traceback, please let us know by reporting it on Trac ticket tracker, via the mailing list, or on IRC. Your best bet to get a quick response will be to jump on IRC during the daytime (CST).

What is the most common cause of “The following entries are not handled by any tool”?

Often it corresponds to entries that aren’t bound by the server (for which you’ll get error messages on the server). You should try inspecting the logs on the server to see what may be the cause.