Systemd integration¶
- Created:
2014-Mar-26
- Status:
Implemented
- Ganeti-Version:
2.12.0
This design document outlines the implementation of native systemd support in Ganeti by providing systemd unit files. It also briefly discusses the possibility of supporting socket activation.
Current state and shortcomings¶
Ganeti currently ships an example init script, compatible with Debian (and derivatives) and RedHat/Fedora (and derivatives). The initscript treats the whole Ganeti system as a single service wrt. starting and stopping (but allows starting/stopping/restarting individual daemons).
The initscript is aided by daemon-util
, which takes care of correctly
ordering the startup/shutdown of daemons using an explicit order.
Finally, process supervision is achieved by (optionally) running
ganeti-watcher
via cron every 5 minutes. ganeti-watcher
will - among
other things - try to start services that should be running but are not.
The example initscript currently shipped with Ganeti will work with systemd’s LSB compatibility wrappers out of the box, however there are a number of areas where we can benefit from providing native systemd unit files:
systemd is the de-facto choice of almost all major Linux distributions. Since it offers a stable API for service control, providing our own systemd unit files means that Ganeti will run out-of-the-box and in a predictable way in all distributions using systemd.
systemd performs constant process supervision with immediate service restarts and configurable back-off. Ganeti currently offers supervision only via ganeti-watcher, running via cron in 5-minute intervals and unconditionally starting missing daemons even if they have been manually stopped.
systemd offers socket activation support, which may be of interest for use at least with masterd, luxid and noded. Socket activation offers two main advantages: no explicit service dependencies or ordering needs to be defined as services will be activated when needed; and seamless restarts / upgrades are possible without rejecting new client connections.
systemd offers a number of security features, primarily using the Linux kernel’s namespace support, which may be of interest to better restrict daemons running as root (noded and mond).
Proposed changes¶
We propose to extend Ganeti to natively support systemd, in addition to
shipping the init-script as is. This requires the addition of systemd
unit files, as well as some changes in daemon-util and ganeti-watcher to
use systemctl
on systems where Ganeti is managed by systemd.
systemd unit files¶
Systemd uses unit files to store information about a service, device,
mount point, or other resource it controls. Each unit file contains
exactly one unit definition, consisting of a Unit
an (optional)
Install
section and an (optional) type-specific section (e.g.
Service
). Unit files are dropped in pre-determined locations in the
system, where systemd is configured to read them from. Systemd allows
complete or partial overrides of the unit files, using overlay
directories. For more information, see systemd.unit(5).
We will create one systemd service unit per daemon (masterd, noded,
mond, luxid, confd, rapi) and an additional oneshot service for
ensure-dirs (ganeti-common.service
). All services will Require
ganeti-common.service
, which will thus run exactly once per
transaction (regardless of starting one or all daemons).
All daemons will run in the foreground (already implemented by the
-f
flag), directly supervised by systemd, using
Restart=on-failure
in the respective units. Master role units will
also treat EXIT_NOTMASTER
as a successful exit and not trigger
restarts. Additionally, systemd’s conditional directives will be used to
avoid starting daemons when they will certainly fail (e.g. because of
missing configuration).
Apart from the individual daemon units, we will also provide three target units as synchronization points:
ganeti-node.target
: Regular node/master candidate functionality, includingganeti-noded.service
,ganeti-mond.service
andganeti-confd.service
.
ganeti-master.target
: Master node functionality, includingganeti-masterd.service
,ganeti-luxid.service
andganeti-rapi.service
.
ganeti.target
: A “meta-target” depending onganeti-node.target
andganti-master.target
.ganeti.target
itself will beWantedBy
multi-user.target
, so that Ganeti starts automatically on boot.
To allow starting/stopping/restarting the different roles, all units
will include a PartOf
directive referencing their direct ancestor
target. In this way systemctl restart ganeti-node.target
or systemctl
restart ganeti.target
will work as expected, i.e. restart only the node
daemons or all daemons respectively.
The full dependency tree is as follows:
ganeti.target
├─ganeti-master.target
│ ├─ganeti-luxid.service
│ │ └─ganeti-common.service
│ ├─ganeti-masterd.service
│ │ └─ganeti-common.service
│ └─ganeti-rapi.service
│ └─ganeti-common.service
└─ganeti-node.target
├─ganeti-confd.service
│ └─ganeti-common.service
├─ganeti-mond.service
│ └─ganeti-common.service
└─ganeti-noded.service
└─ganeti-common.service
Installation¶
The systemd unit files will be built from templates under
doc/examples/systemd, much like what is currently done for the
initscript. They will not be installed with make install
, but left
up to the distribution packagers to ship them at the appropriate
locations.
SysV compatibility¶
Systemd automatically creates a service for each SysV initscript on the
system, appending .service
to the initscript name, except if a
service with the given name already exists. In our case however, the
initscript’s functionality is implemented by ganeti.target
.
Systemd provides the ability to mask a given service, rendering it
unusable, but in the case of SysV services this also results in
failure to use tools like invoke-rc.d
or service
. Thus we have
to ship a ganeti.service
(calling /bin/true
) of type
oneshot
, that depends on ganeti.target
for these tools to
continue working as expected. ganeti.target
on the other hand will
be marked as PartOf = ganeti.service
for stop and restart to be
propagated to the whole service.
The ganeti.service
unit will not be marked to be enabled by systemd
(i.e. will not be started at boot), but will be available for manual
invocation and only be used for compatibility purposes.
Changes to daemon-util¶
daemon-util
is used wherever daemon control is required:
In the sample initscript, to start and stop all daemons.
In
ganeti.backend
to start the master daemons on master failover and to stop confd when leaving the cluster.In
ganeti.bootstrap
, to start the daemons on cluster initialization.In
ganeti.cli
, to control the daemon run state during certain operations (e.g. renew-crypto).
Currently, daemon-util
uses two auxiliary tools for managing daemons
start-stop-daemon
and daemon
, in this order of preference. In
order not to confuse systemd in its process supervision, daemon-util
will have to be modified to start and stop the daemons via systemctl
in preference to start-stop-daemon
and daemon
. This
will require a basic check against run-time environment integrity:
Make sure that
systemd
runs as PID 1, which is a simple check against the existence of/run/systemd/system
.Make sure
systemd
knows how to handle Ganeti natively. This can be a check against theLoadState
of theganeti.target
unit.
Unless both of these checks pass, daemon-util
will fall back to its
current behavior.
Changes to ganeti-watcher¶
Since the daemon process supervision will be systemd’s responsibility,
the watcher must detect systemd’s presence and not attempt to start any
missing services. Again, systemd can be detected by the existence of
/run/systemd/system
.
Future work¶
Socket activation¶
Systemd offers support for socket activation. A daemon supporting socket-based activation, can inherit its listening socket(s) by systemd. This in turn means that the socket can be created and bound by systemd during early boot and it can be used to provide implicit startup ordering; as soon as a client connects to the listening socket, the respective service (and all its dependencies) will be started and the client will wait until its connection is accepted.
Also, because the socket remains bound even if the service is restarting, new client connections will never be rejected, making service restarts and upgrades seamless.
Socket activation support is trivial to implement (see sd_listen_fds(3)) and relies on information passed by systemd via environment variables to the started processes.