DRBD’s core functionality is implemented by way of a Linux kernel module. Specifically, DRBD constitutes a driver for a virtual block device, so DRBD is situated right near the bottom of a system’s I/O stack. Because of this, DRBD is extremely flexible and versatile, which makes it a replication solution suitable for adding high availability to just about any application.

DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Thus, it is impossible for DRBD to miraculously add features to upper layers that these do not possess. For example, DRBD cannot auto-detect file system corruption or add active-active clustering capability to file systems like ext3 or XFS.

DRBD comes with a set of administration tools which communicate with the kernel module in order to configure and administer DRBD resources.

The high-level administration tool of the DRBD program suite. Obtains all DRBD configuration parameters from the configuration file /etc/drbd.conf and acts as a front-end for drbdsetup and drbdmeta. drbdadm has a dry-run mode, invoked with the -d option, that shows which drbdsetup and drbdmeta calls drbdadm would issue without actually calling those commands.

Configures the DRBD module loaded into the kernel. All parameters to drbdsetup must be passed on the command line. The separation between drbdadm and drbdsetup allows for maximum flexibility. Most users will rarely need to use drbdsetup directly, if at all.

Allows to create, dump, restore, and modify DRBD meta data structures. Like drbdsetup, most users will only rarely need to use drbdmeta directly.

In DRBD, resource is the collective term that refers to all aspects of a particular replicated data set. These include:

This can be any arbitrary, US-ASCII name not containing whitespace by which the resource is referred to.

Any resource is a replication group consisting of one of more volumes that share a common replication stream. DRBD ensures write fidelity across all volumes in the resource. Volumes are numbered starting with 0, and there may be up to 65,535 volumes in one resource. A volume contains the replicated data set, and a set of metadata for DRBD internal use.

At the drbdadm level, a volume within a resource can be addressed by the resource name and volume number as <resource>/<volume>.

This is a virtual block device managed by DRBD. It has a device major number of 147, and its minor numbers are numbered from 0 onwards, as is customary. Each DRBD device corresponds to a volume in a resource. The associated block device is usually named /dev/drbdX, where X is the device minor number. DRBD also allows for user-defined block device names which must, however, start with drbd_.

A connection is a communication link between two hosts that share a replicated data set. As of the time of this writing, each resource involves only two hosts and exactly one connection between these hosts, so for the most part, the terms resource and connection can be used interchangeably.

At the drbdadm level, a connection is addressed by the resource name.

In DRBD, every resource has a role, which may be Primary or Secondary.

The resource’s role can, of course, be changed, either by manual intervention or by way of some automated algorithm by a cluster management application. Changing the resource role from secondary to primary is referred to as promotion, whereas the reverse operation is termed demotion.

In single-primary mode, a resource is, at any given time, in the primary role on only one cluster member. Since it is guaranteed that only one cluster node manipulates the data at any moment, this mode can be used with any conventional file system (ext3, ext4, XFS etc.).

Deploying DRBD in single-primary mode is the canonical approach for high availability (fail-over capable) clusters.

In dual-primary mode, a resource is, at any given time, in the primary role on both cluster nodes. Since concurrent access to the data is thus possible, this mode requires the use of a shared cluster file system that utilizes a distributed lock manager. Examples include GFS and OCFS2.

Deploying DRBD in dual-primary mode is the preferred approach for load-balancing clusters which require concurrent data access from two nodes. This mode is disabled by default, and must be enabled explicitly in DRBD’s configuration file.

See Enabling dual-primary mode for information on enabling dual-primary mode for specific resources.

DRBD supports three distinct replication modes, allowing three degrees of replication synchronicity.

Asynchronous replication protocol. Local write operations on the primary node are considered completed as soon as the local disk write has finished, and the replication packet has been placed in the local TCP send buffer. In the event of forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over, however, the most recent updates performed prior to the crash could be lost. Protocol A is most often used in long distance replication scenarios. When used in combination with DRBD Proxy it makes an effective disaster recovery solution. See Long-distance replication with DRBD Proxy for more information.

Memory synchronous (semi-synchronous) replication protocol. Local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Normally, no writes are lost in case of forced fail-over. However, in the event of simultaneous power failure on both nodes and concurrent, irreversible destruction of the primary’s data store, the most recent writes completed on the primary may be lost.

Synchronous replication protocol. Local write operations on the primary node are considered completed only after both the local and the remote disk write have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. Data loss is, of course, inevitable even with this replication protocol if both nodes (or their storage subsystems) are irreversibly destroyed at the same time.

By far, the most commonly used replication protocol in DRBD setups is protocol C.

The choice of replication protocol influences two factors of your deployment: protection and latency. Throughput, by contrast, is largely independent of the replication protocol selected.

See Configuring your resource for an example resource configuration which demonstrates replication protocol configuration.

DRBD’s replication and synchronization framework socket layer supports multiple low-level transports:

This is the canonical implementation, and DRBD’s default. It may be used on any system that has IPv4 enabled.

When configured to use standard TCP sockets for replication and synchronization, DRBD can use also IPv6 as its network protocol. This is equivalent in semantics and performance to IPv4, albeit using a different addressing scheme.

SDP is an implementation of BSD-style sockets for RDMA capable transports such as InfiniBand. SDP is available as part of the OFED stack for most current distributions. SDP uses and IPv4-style addressing scheme. Employed over an InfiniBand interconnect, SDP provides a high-throughput, low-latency replication network to DRBD.

SuperSockets replace the TCP/IP portions of the stack with a single, monolithic, highly efficient and RDMA capable socket implementation. DRBD can use this socket type for very low latency replication. SuperSockets must run on specific hardware which is currently available from a single vendor, Dolphin Interconnect Solutions.

(Re-)synchronization is distinct from device replication. While replication occurs on any write event to a resource in the primary role, synchronization is decoupled from incoming writes. Rather, it affects the device as a whole.

Synchronization is necessary if the replication link has been interrupted for any reason, be it due to failure of the primary node, failure of the secondary node, or interruption of the replication link. Synchronization is efficient in the sense that DRBD does not synchronize modified blocks in the order they were originally written, but in linear order, which has the following consequences:

The service continues to run uninterrupted on the active node, while background synchronization is in progress.

If properly configured, DRBD can detect if the replication network is congested, and suspend replication in this case. In this mode, the primary node "pulls ahead" of the secondary — temporarily going out of sync, but still leaving a consistent copy on the secondary. When more bandwidth becomes available, replication automatically resumes and a background synchronization takes place.

Suspended replication is typically enabled over links with variable bandwidth, such as wide area replication over shared connections between data centers or cloud instances.

See Configuring congestion policies and suspended replication for details on congestion policies and suspended replication.

On-line device verification enables users to do a block-by-block data integrity check between nodes in a very efficient manner.

Note that efficient refers to efficient use of network bandwidth here, and to the fact that verification does not break redundancy in any way. On-line verification is still a resource-intensive operation, with a noticeable impact on CPU utilization and load average.

It works by one node (the verification source) sequentially calculating a cryptographic digest of every block stored on the lower-level storage device of a particular resource. DRBD then transmits that digest to the peer node (the verification target), where it is checked against a digest of the local copy of the affected block. If the digests do not match, the block is marked out-of-sync and may later be synchronized. Because DRBD transmits just the digests, not the full blocks, on-line verification uses network bandwidth very efficiently.

The process is termed on-line verification because it does not require that the DRBD resource being verified is unused at the time of verification. Thus, though it does carry a slight performance penalty while it is running, on-line verification does not cause service interruption or system down time — neither during the verification run nor during subsequent synchronization.

It is a common use case to have on-line verification managed by the local cron daemon, running it, for example, once a week or once a month. See Using on-line device verification for information on how to enable, invoke, and automate on-line verification.

DRBD optionally performs end-to-end message integrity checking using cryptographic message digest algorithms such as MD5, SHA-1 or CRC-32C.

These message digest algorithms are not provided by DRBD. The Linux kernel crypto API provides these; DRBD merely uses them. Thus, DRBD is capable of utilizing any message digest algorithm available in a particular system’s kernel configuration.

With this feature enabled, DRBD generates a message digest of every data block it replicates to the peer, which the peer then uses to verify the integrity of the replication packet. If the replicated block can not be verified against the digest, the peer requests retransmission. Thus, DRBD replication is protected against several error sources, all of which, if unchecked, would potentially lead to data corruption during the replication process:

See Configuring replication traffic integrity checking for information on how to enable replication traffic integrity checking.

Split brain is a situation where, due to temporary failure of all network links between cluster nodes, and possibly due to intervention by a cluster management software or human error, both nodes switched to the primary role while disconnected. This is a potentially harmful state, as it implies that modifications to the data might have been made on either node, without having been replicated to the peer. Thus, it is likely in this situation that two diverging sets of data have been created, which cannot be trivially merged.

DRBD split brain is distinct from cluster split brain, which is the loss of all connectivity between hosts managed by a distributed cluster management application such as Heartbeat. To avoid confusion, this guide uses the following convention:

DRBD allows for automatic operator notification (by email or other means) when it detects split brain. See Split brain notification for details on how to configure this feature.

While the recommended course of action in this scenario is to manually resolve the split brain and then eliminate its root cause, it may be desirable, in some cases, to automate the process. DRBD has several resolution algorithms available for doing so:

Whether or not automatic split brain recovery is acceptable depends largely on the individual application. Consider the example of DRBD hosting a database. The discard modifications from host with fewer changes approach may be fine for a web application click-through database. By contrast, it may be totally unacceptable to automatically discard any modifications made to a financial database, requiring manual recovery in any split brain event. Consider your application’s requirements carefully before enabling automatic split brain recovery.

Refer to Automatic split brain recovery policies for details on configuring DRBD’s automatic split brain recovery policies.

When local block devices such as hard drives or RAID logical disks have write caching enabled, writes to these devices are considered completed as soon as they have reached the volatile cache. Controller manufacturers typically refer to this as write-back mode, the opposite being write-through. If a power outage occurs on a controller in write-back mode, the last writes are never committed to the disk, potentially causing data loss.

To counteract this, DRBD makes use of disk flushes. A disk flush is a write operation that completes only when the associated data has been committed to stable (non-volatile) storage — that is to say, it has effectively been written to disk, rather than to the cache. DRBD uses disk flushes for write operations both to its replicated data set and to its meta data. In effect, DRBD circumvents the write cache in situations it deems necessary, as in activity log updates or enforcement of implicit write-after-write dependencies. This means additional reliability even in the face of power failure.

It is important to understand that DRBD can use disk flushes only when layered on top of backing devices that support them. Most reasonably recent kernels support disk flushes for most SCSI and SATA devices. Linux software RAID (md) supports disk flushes for RAID-1 provided that all component devices support them too. The same is true for device-mapper devices (LVM2, dm-raid, multipath).

Controllers with battery-backed write cache (BBWC) use a battery to back up their volatile storage. On such devices, when power is restored after an outage, the controller flushes all pending writes out to disk from the battery-backed cache, ensuring that all writes committed to the volatile cache are actually transferred to stable storage. When running DRBD on top of such devices, it may be acceptable to disable disk flushes, thereby improving DRBD’s write performance. See Disabling backing device flushes for details.

If a hard drive fails which is used as a backing block device for DRBD on one of the nodes, DRBD may either pass on the I/O error to the upper layer (usually the file system) or it can mask I/O errors from upper layers.

If DRBD is configured to pass on I/O errors, any such errors occurring on the lower-level device are transparently passed to upper I/O layers. Thus, it is left to upper layers to deal with such errors (this may result in a file system being remounted read-only, for example). This strategy does not ensure service continuity, and is hence not recommended for most users.

If DRBD is configured to detach on lower-level I/O error, DRBD will do so, automatically, upon occurrence of the first lower-level I/O error. The I/O error is masked from upper layers while DRBD transparently fetches the affected block from the peer node, over the network. From then onwards, DRBD is said to operate in diskless mode, and carries out all subsequent I/O operations, read and write, on the peer node. Performance in this mode will be reduced, but the service continues without interruption, and can be moved to the peer node in a deliberate fashion at a convenient time.

See Configuring I/O error handling strategies for information on configuring I/O error handling strategies for DRBD.

DRBD distinguishes between inconsistent and outdated data. Inconsistent data is data that cannot be expected to be accessible and useful in any manner. The prime example for this is data on a node that is currently the target of an on-going synchronization. Data on such a node is part obsolete, part up to date, and impossible to identify as either. Thus, for example, if the device holds a filesystem (as is commonly the case), that filesystem would be unexpected to mount or even pass an automatic filesystem check.

Outdated data, by contrast, is data on a secondary node that is consistent, but no longer in sync with the primary node. This would occur in any interruption of the replication link, whether temporary or permanent. Data on an outdated, disconnected secondary node is expected to be clean, but it reflects a state of the peer node some time past. In order to avoid services using outdated data, DRBD disallows promoting a resource that is in the outdated state.

DRBD has interfaces that allow an external application to outdate a secondary node as soon as a network interruption occurs. DRBD will then refuse to switch the node to the primary role, preventing applications from using the outdated data. A complete implementation of this functionality exists for the Pacemaker cluster management framework (where it uses a communication channel separate from the DRBD replication link). However, the interfaces are generic and may be easily used by any other cluster management application.

Whenever an outdated resource has its replication link re-established, its outdated flag is automatically cleared. A background synchronization then follows.

See the section about the DRBD outdate-peer daemon (dopd) for an example DRBD/Heartbeat/Pacemaker configuration enabling protection against inadvertent use of outdated data.

When using three-way replication, DRBD adds a third node to an existing 2-node cluster and replicates data to that node, where it can be used for backup and disaster recovery purposes. This type of configuration generally involves Long-distance replication with DRBD Proxy.

Three-way replication works by adding another, stacked DRBD resource on top of the existing resource holding your production data, as seen in this illustration:

The stacked resource is replicated using asynchronous replication (DRBD protocol A), whereas the production data would usually make use of synchronous replication (DRBD protocol C).

Three-way replication can be used permanently, where the third node is continuously updated with data from the production cluster. Alternatively, it may also be employed on demand, where the production cluster is normally disconnected from the backup site, and site-to-site synchronization is performed on a regular basis, for example by running a nightly cron job.

DRBD’s protocol A is asynchronous, but the writing application will block as soon as the socket output buffer is full (see the sndbuf-size option in the man page of drbd.conf). In that event, the writing application has to wait until some of the data written runs off through a possibly small bandwidth network link.

The average write bandwidth is limited by available bandwidth of the network link. Write bursts can only be handled gracefully if they fit into the limited socket output buffer.

You can mitigate this by DRBD Proxy’s buffering mechanism. DRBD Proxy will place changed data from the DRBD device on the primary node into its buffers. DRBD Proxy’s buffer size is freely configurable, only limited by the address room size and available physical RAM.

Optionally DRBD Proxy can be configured to compress and decompress the data it forwards. Compression and decompression of DRBD’s data packets might slightly increase latency. However, when the bandwidth of the network link is the limiting factor, the gain in shortening transmit time outweighs the compression and decompression overhead.

Compression and decompression were implemented with multi core SMP systems in mind, and can utilize multiple CPU cores.

The fact that most block I/O data compresses very well and therefore the effective bandwidth increases well justifies the use of the DRBD Proxy even with DRBD protocols B and C.

See Using DRBD Proxy for information on configuring DRBD Proxy.

Truck based replication, also known as disk shipping, is a means of preseeding a remote site with data to be replicated, by physically shipping storage media to the remote site. This is particularly suited for situations where

In such situations, without truck based replication, DRBD would require a very long initial device synchronization (on the order of days or weeks). Truck based replication allows us to ship a data seed to the remote site, and drastically reduce the initial synchronization time. See Using truck based replication for details on this use case.

A somewhat special use case for DRBD is the floating peers configuration. In floating peer setups, DRBD peers are not tied to specific named hosts (as in conventional configurations), but instead have the ability to float between several hosts. In such a configuration, DRBD identifies peers by IP address, rather than by host name.

For more information about managing floating peer configurations, see Configuring DRBD to replicate between two SAN-backed Pacemaker clusters.

LINBIT, the DRBD project’s sponsor company, provides DRBD binary packages to its commercial support customers. These packages are available at http://www.linbit.com/support/ and are considered "official" DRBD builds.

These builds are available for the following distributions:

LINBIT releases binary builds in parallel with any new DRBD source release.

Package installation on RPM-based systems (SLES, RHEL) is done by simply invoking rpm -i (for new installations) or rpm -U (for upgrades), along with the corresponding package names.

For Debian-based systems (Debian GNU/Linux, Ubuntu) systems, drbd8-utils and drbd8-module packages are installed with dpkg -i, or gdebi if available.

A number of distributions include DRBD, including pre-built binary packages. Support for these builds, if any, is being provided by the associated distribution vendor. Their release cycle may lag behind DRBD source releases.

Releases generated by git tags on github are snapshots of the git repository at the given time. You most likely do not want to use these. They might lack things such as generated man pages, the configure script, and other generated files. If you want to build from a tarball, use the ones provided by us.

All our projects contain standard build scripts (e.g., Makefile, configure). Maintaining specific information per distribution (e.g., documenting broken build macros) is too cumbersome, and historically the information provided in this section got outdated quickly. If you don’t know how to build software the standard way, please consider using packages provided by LINBIT.

After you have installed DRBD, you must set aside a roughly identically sized storage area on both cluster nodes. This will become the lower-level device for your DRBD resource. You may use any type of block device found on your system for this purpose. Typical examples include:

You may also use resource stacking, meaning you can use one DRBD device as a lower-level device for another. Some specific considerations apply to stacked resources; their configuration is covered in detail in Creating a three-node setup.

It is not necessary for this storage area to be empty before you create a DRBD resource from it. In fact it is a common use case to create a two-node cluster from a previously non-redundant single-server system using DRBD (some caveats apply — please refer to DRBD meta data if you are planning to do this).

For the purposes of this guide, we assume a very simple setup:

It is recommended, though not strictly required, that you run your DRBD replication over a dedicated connection. At the time of this writing, the most reasonable choice for this is a direct, back-to-back, Gigabit Ethernet connection. When DRBD is run over switches, use of redundant components and the bonding driver (in active-backup mode) is recommended.

It is generally not recommended to run DRBD replication via routers, for reasons of fairly obvious performance drawbacks (adversely affecting both throughput and latency).

In terms of local firewall considerations, it is important to understand that DRBD (by convention) uses TCP ports from 7788 upwards, with every resource listening on a separate port. DRBD uses two TCP connections for every resource configured. For proper DRBD functionality, it is required that these connections are allowed by your firewall configuration.

Security considerations other than firewalling may also apply if a Mandatory Access Control (MAC) scheme such as SELinux or AppArmor is enabled. You may have to adjust your local security policy so it does not keep DRBD from functioning properly.

You must, of course, also ensure that the TCP ports for DRBD are not already used by another application.

It is not possible to configure a DRBD resource to support more than one TCP connection. If you want to provide for DRBD connection load-balancing or redundancy, you can easily do so at the Ethernet level (again, using the bonding driver).

For the purposes of this guide, we assume a very simple setup:

All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf. Normally, this configuration file is just a skeleton with the following contents:

By convention, /etc/drbd.d/global_common.conf contains the global and common sections of the DRBD configuration, whereas the .res files contain one resource section each.

It is also possible to use drbd.conf as a flat configuration file without any include statements at all. Such a configuration, however, quickly becomes cluttered and hard to manage, which is why the multiple-file approach is the preferred one.

Regardless of which approach you employ, you should always make sure that drbd.conf, and any other files it includes, are exactly identical on all participating cluster nodes.

The DRBD source tarball contains an example configuration file in the scripts subdirectory. Binary installation packages will either install this example configuration directly in /etc, or in a package-specific documentation directory such as /usr/share/doc/packages/drbd.

This section describes only those few aspects of the configuration file which are absolutely necessary to understand in order to get DRBD up and running. The configuration file’s syntax and contents are documented in great detail in the man page of drbd.conf.

After you have completed initial resource configuration as outlined in the previous sections, you can bring up your resource.

Each of the following steps must be completed on both nodes.

Please note that with our example config snippets (resource r0 { …​ }), <resource> would be r0.

This step must be completed only on initial device creation. It initializes DRBD’s metadata:

This step associates the resource with its backing device (or devices, in case of a multi-volume resource), sets replication parameters, and connects the resource to its peer:

DRBD’s virtual status file in the /proc filesystem, /proc/drbd, should now contain information similar to the following:

By now, DRBD has successfully allocated both disk and network resources and is ready for operation. What it does not know yet is which of your nodes should be used as the source of the initial device synchronization.

There are two more steps required for DRBD to become fully operational:

If you are dealing with newly-initialized, empty disk, this choice is entirely arbitrary. If one of your nodes already has valuable data that you need to preserve, however, it is of crucial importance that you select that node as your synchronization source. If you do initial device synchronization in the wrong direction, you will lose that data. Exercise caution.

This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source. To perform this step, issue this command:

After issuing this command, the initial full synchronization will commence. You will be able to monitor its progress via /proc/drbd. It may take some time depending on the size of the device.

By now, your DRBD device is fully operational, even before the initial synchronization has completed (albeit with slightly reduced performance). You may now create a filesystem on the device, use it as a raw block device, mount it, and perform any other operation you would with an accessible block device.

You will now probably want to continue with Common administrative tasks, which describes common administrative tasks to perform on your resource.

In order to preseed a remote node with data which is then to be kept synchronized, and to skip the initial device synchronization, follow these steps.

This assumes that your local node has a configured, but disconnected DRBD resource in the Primary role. That is to say, device configuration is completed, identical drbd.conf copies exist on both nodes, and you have issued the commands for initial resource promotion on your local node — but the remote node is not connected yet.

Note the absence of the --clear-bitmap option in this second invocation.

After the two peers connect, they will not initiate a full device synchronization. Instead, the automatic synchronization that now commences only covers those blocks that changed since the invocation of drbdadm --clear-bitmap new-current-uuid.

Even if there were no changes whatsoever since then, there may still be a brief synchronization period due to areas covered by the Activity Log being rolled back on the new Secondary. This may be mitigated by the use of checksum-based synchronization.

You may use this same procedure regardless of whether the resource is a regular DRBD resource, or a stacked resource. For stacked resources, simply add the -S or --stacked option to drbdadm.

DRBD allows you to reconfigure resources while they are operational. To that end,

drbdadm adjust then hands off to drbdsetup to make the necessary adjustments to the configuration. As always, you are able to review the pending drbdsetup invocations by running drbdadm with the -d (dry-run) option.

Manually switching a resource’s role from secondary to primary (promotion) or vice versa (demotion) is done using the following commands:

In single-primary mode (DRBD’s default), any resource can be in the primary role on only one node at any given time while the connection state is Connected. Thus, issuing drbdadm primary <resource> on one node while <resource> is still in the primary role on the peer will result in an error.

A resource configured to allow dual-primary mode can be switched to the primary role on both nodes.

If not using Pacemaker and looking to handle fail-overs manually in a passive/active configuration the process is as follows.

On the current primary node stop any applications or services using the DRBD device, unmount the DRBD device, and demote the resource to secondary.

Now on the node we wish to make primary promote the resource and mount the device.

Upgrading DRBD is a fairly simple process. This section will cover the process of upgrading from 8.3.x to 8.4.x, however this process should work for all upgrades.

If you’re currently running DRBD 8.4 and would like to revert to 8.3 there are several steps you will have to follow. This section assumes you still have the 8.4 kernel module and 8.4 utilities installed.

Stop any services accessing the DRBD resources, unmount, and demote the devices to Secondary. Then perform the following commands.

If you’re using the LINBIT repositories you can remove the packages using apt-get remove drbd8-utils drbd8-module-`uname -r` or yum remove drbd kmod-drbd

Now that 8.4 is removed reinstall 8.3. You can do this either by changing your repositories back to the 8.3 repos, or by following the steps located in the 8.3 User’s Guide

Once 8.3 is re-installed you can start your DRBD resources either manually using drbdadm or /etc/init.d/drbd start.

Dual-primary mode allows a resource to assume the primary role simultaneously on both nodes. Doing so is possible on either a permanent or a temporary basis.

Normally, one tries to ensure that background synchronization (which makes the data on the synchronization target temporarily inconsistent) completes as quickly as possible. However, it is also necessary to keep background synchronization from hogging all bandwidth otherwise available for foreground replication, which would be detrimental to application performance. Thus, you must configure the synchronization bandwidth to match your hardware — which you may do in a permanent fashion or on-the-fly.

Likewise, and for the same reasons, it does not make sense to set a synchronization rate that is higher than the bandwidth available on the replication network.

Checksum-based synchronization is not enabled for resources by default. To enable it, add the following lines to your resource configuration in /etc/drbd.conf:

<algorithm> may be any message digest algorithm supported by the kernel crypto API in your system’s kernel configuration. Normally, you should be able to choose at least from sha1, md5, and crc32c.

If you make this change to an existing resource, as always, synchronize your drbd.conf to the peer, and run drbdadm adjust <resource> on both nodes.

In an environment where the replication bandwidth is highly variable (as would be typical in WAN replication setups), the replication link may occasionally become congested. In a default configuration, this would cause I/O on the primary node to block, which is sometimes undesirable.

Instead, you may configure DRBD to suspend the ongoing replication in this case, causing the Primary’s data set to pull ahead of the Secondary. In this mode, DRBD keeps the replication channel open — it never switches to disconnected mode — but does not actually replicate until sufficient bandwidth becomes available again.

The following example is for a DRBD Proxy configuration:

It is usually wise to set both congestion-fill and congestion-extents together with the pull-ahead option.

A good value for congestion-fill is 90%

A good value for congestion-extents is 90% of your configured al-extents for the affected resources.

DRBD’s strategy for handling lower-level I/O errors is determined by the on-io-error option, included in the resource disk configuration in /etc/drbd.conf:

You may, of course, set this in the common section too, if you want to define a global I/O error handling policy for all resources.

<strategy> may be one of the following options:

You may reconfigure a running resource’s I/O error handling strategy by following this process:

Replication traffic integrity checking is not enabled for resources by default. To enable it, add the following lines to your resource configuration in /etc/drbd.conf:

<algorithm> may be any message digest algorithm supported by the kernel crypto API in your system’s kernel configuration. Normally, you should be able to choose at least from sha1, md5, and crc32c.

If you make this change to an existing resource, as always, synchronize your drbd.conf to the peer, and run drbdadm adjust <resource> on both nodes.

Disabling DRBD’s flushes when running without BBWC, or on BBWC with a depleted battery, is likely to cause data loss and should not be attempted.

DRBD allows you to enable and disable backing device flushes separately for the replicated data set and DRBD’s own meta data. Both of these options are enabled by default. If you wish to disable either (or both), you would set this in the disk section for the DRBD configuration file, /etc/drbd.conf.

To disable disk flushes for the replicated data set, include the following line in your configuration:

To disable disk flushes on DRBD’s meta data, include the following line:

After you have modified your resource configuration (and synchronized your /etc/drbd.conf between nodes, of course), you may enable these settings by issuing this command on both nodes:

A three-node setup involves one DRBD device stacked atop another.

How to deal with hard drive failure depends on the way DRBD is configured to handle disk I/O errors (see Disk error handling strategies), and on the type of meta data configured (see DRBD meta data).

When DRBD detects that its peer node is down (either by true hardware failure or manual intervention), DRBD changes its connection state from Connected to WFConnection and waits for the peer node to re-appear. The DRBD resource is then said to operate in disconnected mode. In disconnected mode, the resource and its associated block device are fully usable, and may be promoted and demoted as necessary, but no block modifications are being replicated to the peer node. Instead, DRBD stores internal information on which blocks are being modified while disconnected.

DRBD detects split brain at the time connectivity becomes available again and the peer nodes exchange the initial DRBD protocol handshake. If DRBD detects that both nodes are (or were at some point, while disconnected) in the primary role, it immediately tears down the replication connection. The tell-tale sign of this is a message like the following appearing in the system log:

After split brain has been detected, one node will always have the resource in a StandAlone connection state. The other might either also be in the StandAlone state (if both nodes detected the split brain simultaneously), or in WFConnection (if the peer tore down the connection before the other node had a chance to detect split brain).

At this point, unless you configured DRBD to automatically recover from split brain, you must manually intervene by selecting one node whose modifications will be discarded (this node is referred to as the split brain victim). This intervention is made with the following commands:

On the other node (the split brain survivor), if its connection state is also StandAlone, you would enter:

You may omit this step if the node is already in the WFConnection state; it will then reconnect automatically.

If the resource affected by the split brain is a stacked resource, use drbdadm --stacked instead of just drbdadm.

Upon connection, your split brain victim immediately changes its connection state to SyncTarget, and has its modifications overwritten by the remaining primary node.

After re-synchronization has completed, the split brain is considered resolved and the two nodes form a fully consistent, redundant replicated storage system again.

Pacemaker is a sophisticated, feature-rich, and widely deployed cluster resource manager for the Linux platform. It comes with a rich set of documentation. In order to understand this chapter, reading the following documents is highly recommended:

This section explains how to enable a DRBD-backed service in a Pacemaker cluster.

The ocf:linbit:drbd OCF resource agent provides Master/Slave capability, allowing Pacemaker to start and monitor the DRBD resource on multiple nodes and promoting and demoting as needed. You must, however, understand that the drbd RA disconnects and detaches all DRBD resources it manages on Pacemaker shutdown, and also upon enabling standby mode for a node.

In order to enable a DRBD-backed configuration for a MySQL database in a Pacemaker CRM cluster with the drbd OCF resource agent, you must create both the necessary resources, and Pacemaker constraints to ensure your service only starts on a previously promoted DRBD resource. You may do so using the crm shell, as outlined in the following example:

After this, your configuration should be enabled. Pacemaker now selects a node on which it promotes the DRBD resource, and then starts the DRBD-backed resource group on that same node.

This section outlines the steps necessary to prevent Pacemaker from promoting a drbd Master/Slave resource when its DRBD replication link has been interrupted. This keeps Pacemaker from starting a service with outdated data and causing an unwanted "time warp" in the process.

In order to enable any resource-level fencing for DRBD, you must add the following lines to your resource configuration:

You will also have to make changes to the handlers section depending on the cluster infrastructure being used:

Stacked resources allow DRBD to be used for multi-level redundancy in multiple-node clusters, or to establish off-site disaster recovery capability. This section describes how to configure DRBD and Pacemaker in such configurations.

This is a somewhat advanced setup usually employed in split-site configurations. It involves two separate Pacemaker clusters, where each cluster has access to a separate Storage Area Network (SAN). DRBD is then used to replicate data stored on that SAN, across an IP link between sites.

Consider the following illustration to describe the concept.

Which of the individual nodes in each site currently acts as the DRBD peer is not explicitly defined — the DRBD peers are said to float; that is, DRBD binds to virtual IP addresses not tied to a specific physical machine.

Since this type of setup deals with shared storage, configuring and testing STONITH is absolutely vital for it to work properly.

This section outlines the configuration steps necessary to get Red Hat Cluster running. Preparing your cluster configuration is fairly straightforward; all a DRBD-based Red Hat Cluster requires are two participating nodes (referred to as Cluster Members in Red Hat’s documentation) and a fencing device.

This section, like Integrating DRBD with Pacemaker clusters, assumes you are about to configure a highly available MySQL database with the following configuration parameters:

LVM2 is an implementation of logical volume management in the context of the Linux device mapper framework. It has practically nothing in common, other than the name and acronym, with the original LVM implementation. The old implementation (now retroactively named "LVM1") is considered obsolete; it is not covered in this section.

When working with LVM, it is important to understand its most basic concepts:

A PV is an underlying block device exclusively managed by LVM. PVs can either be entire hard disks or individual partitions. It is common practice to create a partition table on the hard disk where one partition is dedicated to the use by the Linux LVM.

A VG is the basic administrative unit of the LVM. A VG may include one or more several PVs. Every VG has a unique name. A VG may be extended during runtime by adding additional PVs, or by enlarging an existing PV.

LVs may be created during runtime within VGs and are available to the other parts of the kernel as regular block devices. As such, they may be used to hold a file system, or for any other purpose block devices may be used for. LVs may be resized while they are online, and they may also be moved from one PV to another (as long as the PVs are part of the same VG).

Snapshots are temporary point-in-time copies of LVs. Creating snapshots is an operation that completes almost instantly, even if the original LV (the origin volume) has a size of several hundred GiByte. Usually, a snapshot requires significantly less space than the original LV.

Since an existing Logical Volume is simply a block device in Linux terms, you may of course use it as a DRBD backing device. To use LV’s in this manner, you simply create them, and then initialize them for DRBD as you normally would.

This example assumes that a Volume Group named foo already exists on both nodes of on your LVM-enabled system, and that you wish to create a DRBD resource named r0 using a Logical Volume in that Volume Group.

First, you create the Logical Volume:

Of course, you must complete this command on both nodes of your DRBD cluster. After this, you should have a block device named /dev/foo/bar on either node.

Then, you can simply enter the newly-created volumes in your resource configuration:

Now you can continue to bring your resource up, just as you would if you were using non-LVM block devices.

While DRBD is synchronizing, the SyncTarget's state is Inconsistent until the synchronization completes. If in this situation the SyncSource happens to fail (beyond repair), this puts you in an unfortunate position: the node with good data is dead, and the surviving node has bad data.

When serving DRBD off an LVM Logical Volume, you can mitigate this problem by creating an automated snapshot when synchronization starts, and automatically removing that same snapshot once synchronization has completed successfully.

In order to enable automated snapshotting during resynchronization, add the following lines to your resource configuration:

The two scripts parse the $DRBD_RESOURCE$ environment variable which DRBD automatically passes to any handler it invokes. The snapshot-resync-target-lvm.sh script then creates an LVM snapshot for every volume the resources contains immediately before synchronization kicks off. In case the script fails, the synchronization does not commence.

Once synchronization completes, the unsnapshot-resync-target-lvm.sh script removes the snapshot, which is then no longer needed. In case unsnapshotting fails, the snapshot continues to linger around.

If at any time your SyncSource does fail beyond repair and you decide to revert to your latest snapshot on the peer, you may do so by issuing the lvconvert -M command.

In order to prepare a DRBD resource for use as a Physical Volume, it is necessary to create a PV signature on the DRBD device. In order to do so, issue one of the following commands on the node where the resource is currently in the primary role:

or

Now, it is necessary to include this device in the list of devices LVM scans for PV signatures. In order to do this, you must edit the LVM configuration file, normally named /etc/lvm/lvm.conf. Find the line in the devices section that contains the filter keyword and edit it accordingly. If all your PVs are to be stored on DRBD devices, the following is an appropriate filter option:

This filter expression accepts PV signatures found on any DRBD devices, while rejecting (ignoring) all others.

If you want to use stacked resources as LVM PVs, then you will need a more explicit filter configuration. You need to make sure that LVM detects PV signatures on stacked resources, while ignoring them on the corresponding lower-level resources and backing devices. This example assumes that your lower-level DRBD resources use device minors 0 through 9, whereas your stacked resources are using device minors from 10 upwards:

This filter expression accepts PV signatures found only on the DRBD devices /dev/drbd10 through /dev/drbd19, while rejecting (ignoring) all others.

After modifying the lvm.conf file, you must run the vgscan command so LVM discards its configuration cache and re-scans devices for PV signatures.

You may of course use a different filter configuration to match your particular system configuration. What is important to remember, however, is that you need to

In addition, you should disable the LVM cache by setting:

After disabling the LVM cache, make sure you remove any stale cache entries by deleting /etc/lvm/cache/.cache.

You must repeat the above steps on the peer node.

When you have configured your new PV, you may proceed to add it to a Volume Group, or create a new Volume Group from it. The DRBD resource must, of course, be in the primary role while doing so.

When you have created your VG, you may start carving Logical Volumes out of it, using the lvcreate command (as with a non-DRBD-backed Volume Group).

Occasionally, you may want to add new DRBD-backed Physical Volumes to a Volume Group. Whenever you do so, a new volume should be added to an existing resource configuration. This preserves the replication stream and ensures write fidelity across all PVs in the VG.

Extend your resource configuration to include an additional volume, as in the following example:

Make sure your DRBD configuration is identical across nodes, then issue:

This will implicitly call drbdsetup new-minor r0 1 to enable the new volume 1 in the resource r0. Once the new volume has been added to the replication stream, you may initialize and add it to the volume group:

This will add the new PV /dev/drbd/by-res/<resource>/1 to the <name> VG, preserving write fidelity across the entire VG.

It is possible, if slightly advanced, to both use Logical Volumes as backing devices for DRBD and at the same time use a DRBD device itself as a Physical Volume. To provide an example, consider the following configuration:

In order to enable this configuration, follow these steps:

The Logical Volumes foo and bar will now be available as /dev/replicated/foo and /dev/replicated/bar on the local node.

The process of transferring volume groups between peers and making the corresponding logical volumes available can be automated. The Pacemaker LVM resource agent is designed for exactly that purpose.

In order to put an existing, DRBD-backed volume group under Pacemaker management, run the following commands in the crm shell:

After you have committed this configuration, Pacemaker will automatically make the r0 volume group available on whichever node currently has the Primary (Master) role for the DRBD resource.

The Red Hat Global File System (GFS) is Red Hat’s implementation of a concurrent-access shared storage file system. As any such filesystem, GFS allows multiple nodes to access the same storage device, in read/write fashion, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster members.

GFS was designed, from the outset, for use with conventional shared storage devices. Regardless, it is perfectly possible to use DRBD, in dual-primary mode, as a replicated storage device for GFS. Applications may benefit from reduced read/write latency due to the fact that DRBD normally reads from and writes to local storage, as opposed to the SAN devices GFS is normally configured to run from. Also, of course, DRBD adds an additional physical copy to every GFS filesystem, thus adding redundancy to the concept.

GFS file systems are usually tightly integrated with Red Hat’s own cluster management framework, the Red Hat Cluster. This chapter explains the use of DRBD in conjunction with GFS in the Red Hat Cluster context. Additionally the connection to the Pacemaker cluster manager is explained, which will take care of resource management und STONITH.

GFS, Pacemaker and Red Hat Cluster are available in Red Hat Enterprise Linux (RHEL) and distributions derived from it, such as CentOS. Packages built from the same sources are also available in Debian GNU/Linux. This chapter assumes running GFS on a Red Hat Enterprise Linux system.

Since GFS is a shared cluster file system expecting concurrent read/write storage access from all cluster nodes, any DRBD resource to be used for storing a GFS filesystem must be configured in dual-primary mode. Also, it is recommended to use some of DRBD’s features for automatic recovery from split brain. Promoting the resource on both nodes and starting the GFS filesystem will be handled by Pacemaker. To prepare your DRBD resource, include the following lines in the resource configuration:

Once you have added these options to your freshly-configured resource, you may initialize your resource as you normally would. Since the allow-two-primaries option is set to yes for this resource, you will be able to promote the resource to the primary role on both nodes.

GFS needs cman, the Red Hat cluster manager, to work. Since cman is not as flexible and easy to configure we will put pacemaker on top of it in the next steps.

Before we start making a GFS filesystem we will configure cman.

Next create a cman cluster configuration in /etc/cluster/cluster.conf:

This tells cman that the clustername is my-cluster, the cluster node names are gfs-machine1 and gfs-machine2, and that fencing will be done by pacemaker.

After you have made the configuration start cman.

In order to create a GFS filesystem on your dual-primary DRBD resource, issue this command on (only) one (!) node (which must be Primary):

The -j option in this command refers to the number of journals to keep for GFS. This must be identical to the number of nodes in the GFS cluster; since DRBD does not support more than two nodes, the value to set here is always 2.

The -t option, defines the lock table name. This follows the format <cluster>:<name>, where <cluster> must match your cluster name as defined in /etc/cluster/cluster.conf. Thus, only members of that cluster will be permitted to use the filesystem. By contrast, <name> is an arbitrary file system name unique in the cluster.

If you want to use Pacemaker as the cluster resource manager, you will have to set up your current configuration and tell Pacemaker to manage your resources.

For Pacemaker configuration make a setup as described in 8.2. Adding a DRBD-backed service to the cluster configuration.

Since it is a dual-primary setup consider the following changes to the Master-Slave set:

Notice that master-max is set to 2, which will cause the DRBD resource to be promoted on both cluster nodes.

Furthermore we want the GFS filesystem also to be started on both nodes, so we simply add a clone of the filesystem primitive:

The Oracle Cluster File System, version 2 (OCFS2) is a concurrent access shared storage file system developed by Oracle Corporation. Unlike its predecessor OCFS, which was specifically designed and only suitable for Oracle database payloads, OCFS2 is a general-purpose filesystem that implements most POSIX semantics. The most common use case for OCFS2 is arguably Oracle Real Application Cluster (RAC), but OCFS2 may also be used for load-balanced NFS clusters, for example.

Although originally designed for use with conventional shared storage devices, OCFS2 is equally well suited to be deployed on dual-Primary DRBD. Applications reading from the filesystem may benefit from reduced read latency due to the fact that DRBD reads from and writes to local storage, as opposed to the SAN devices OCFS2 otherwise normally runs on. In addition, DRBD adds redundancy to OCFS2 by adding an additional copy to every filesystem image, as opposed to just a single filesystem image that is merely shared.

Like other shared cluster file systems such as GFS, OCFS2 allows multiple nodes to access the same storage device, in read/write mode, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster nodes. The DLM itself uses a virtual file system (ocfs2_dlmfs) which is separate from the actual OCFS2 file systems present on the system.

OCFS2 may either use an intrinsic cluster communication layer to manage cluster membership and filesystem mount and unmount operation, or alternatively defer those tasks to the Pacemakercluster infrastructure.

OCFS2 is available in SUSE Linux Enterprise Server (where it is the primarily supported shared cluster file system), CentOS, Debian GNU/Linux, and Ubuntu Server Edition. Oracle also provides packages for Red Hat Enterprise Linux (RHEL). This chapter assumes running OCFS2 on a SUSE Linux Enterprise Server system.

Since OCFS2 is a shared cluster file system expecting concurrent read/write storage access from all cluster nodes, any DRBD resource to be used for storing a OCFS2 filesystem must be configured in dual-primary mode. Also, it is recommended to use some of DRBD’s features for automatic recovery from split brain. And, it is necessary for the resource to switch to the primary role immediately after startup. To do all this, include the following lines in the resource configuration:

It is not recommended to set the allow-two-primaries option to yes upon initial configuration. You should do so after the initial resource synchronization has completed.

Once you have added these options to your freshly-configured resource, you may initialize your resource as you normally would. After you set the allow-two-primaries option to yes for this resource, you will be able to promote the resourceto the primary role on both nodes.

Now, use OCFS2’s mkfs implementation to create the file system:

This will create an OCFS2 file system with two node slots on /dev/drbd0, and set the filesystem label to ocfs2_drbd0. You may specify other options on mkfs invocation; please see the mkfs.ocfs2 system manual page for details.

Xen is a virtualization framework originally developed at the University of Cambridge (UK), and later being maintained by XenSource, Inc. (now a part of Citrix). It is included in reasonably recent releases of most Linux distributions, such as Debian GNU/Linux (since version 4.0), SUSE Linux Enterprise Server (since release 10), Red Hat Enterprise Linux (since release 5), and many others.

Xen uses paravirtualization — a virtualization method involving a high degree of cooperation between the virtualization host and guest virtual machines — with selected guest operating systems for improved performance in comparison to conventional virtualization solutions (which are typically based on hardware emulation). Xen also supports full hardware emulation on CPUs that support the appropriate virtualization extensions, in Xen parlance, this is known as HVM ( "hardware-assisted virtual machine").

Xen supports live migration, which refers to the capability of transferring a running guest operating system from one physical host to another, without interruption.

When a DRBD resource is used as a replicated Virtual Block Device (VBD) for Xen, it serves to make the entire contents of a domU’s virtual disk available on two servers, which can then be configured for automatic fail-over. That way, DRBD does not only provide redundancy for Linux servers (as in non-virtualized DRBD deployment scenarios), but also for any other operating system that can be virtualized under Xen — which, in essence, includes any operating system available on 32- or 64-bit Intel compatible architectures.

For Xen Domain-0 kernels, it is recommended to load the DRBD module with the option disable_sendpage set to 1. To do so, create (or open) the file /etc/modprobe.d/drbd.conf and enter the following line:

Configuring a DRBD resource that is to be used as a Virtual Block Device for Xen is fairly straightforward — in essence, the typical configuration matches that of a DRBD resource being used for any other purpose. However, if you want to enable live migration for your guest instance, you need to enable dual-primary modefor this resource:

Enabling dual-primary mode is necessary because Xen, before initiating live migration, checks for write access on all VBDs a resource is configured to use on both the source and the destination host for the migration.

In order to use a DRBD resource as the virtual block device, you must add a line like the following to your Xen domU configuration:

This example configuration makes the DRBD resource named resource available to the domU as /dev/xvda in read/write mode ( w).

Of course, you may use multiple DRBD resources with a single domU. In that case, simply add more entries like the one provided in the example to the disk option, separated by commas.

Under these circumstances, you must use the traditional phy: device syntax and the DRBD device name that is associated with your resource, not the resource name. That, however, requires that you manage DRBD state transitions outside Xen, which is a less flexible approach than that provided by the drbd resource type.

Once you have configured your DRBD-backed domU, you may start it as you would any other domU:

In the process, the DRBD resource you configured as the VBD will be promoted to the primary role, and made accessible to Xen as expected.

This is equally straightforward:

Again, as you would expect, the DRBD resource is returned to the secondary role after the domU is successfully shut down.

This, too, is done using the usual Xen tools:

In this case, several administrative steps are automatically taken in rapid succession:

The fact that both resources must briefly run in the primary role on both hosts is the reason for having to configure the resource in dual-primary mode in the first place.

Xen supports two Virtual Block Device types natively:

This device type is used to hand "physical" block devices, available in the host environment, off to a guest domU in an essentially transparent fashion.

This device type is used to make file-based block device images available to the guest domU. It works by creating a loop block device from the original image file, and then handing that block device off to the domU in much the same fashion as the phy device type does.

If a Virtual Block Device configured in the disk option of a domU configuration uses any prefix other than phy:, file:, or no prefix at all (in which case Xen defaults to using the phy device type), Xen expects to find a helper script named block-prefix in the Xen scripts directory, commonly /etc/xen/scripts.

The DRBD distribution provides such a script for the drbd device type, named /etc/xen/scripts/block-drbd. This script handles the necessary DRBD resource state transitions as described earlier in this chapter.

In order to fully capitalize on the benefits provided by having a DRBD-backed Xen VBD’s, it is recommended to have Heartbeat manage the associated domU’s as Heartbeat resources.

You may configure a Xen domU as a Pacemaker resource, and automate resource failover. To do so, use the Xen OCF resource agent. If you are using the drbd Xen device type described in this chapter, you will not need to configure any separate drbd resource for use by the Xen cluster resource. Instead, the block-drbd helper script will do all the necessary resource transitions for you.

When measuring the impact of using DRBD on a system’s I/O throughput, the absolute throughput the system is capable of is of little relevance. What is much more interesting is the relative impact DRBD has on I/O performance. Thus it is always necessary to measure I/O throughput both with and without DRBD.

I/O throughput estimation works by writing significantly large chunks of data to a block device, and measuring the amount of time the system took to complete the write operation. This can be easily done using a fairly ubiquitous utility, dd, whose reasonably recent versions include a built-in throughput estimation.

A simple dd-based throughput benchmark, assuming you have a scratch resource named test which is currently connected and in the secondary role on both nodes, is one like the following:

This test simply writes a 512M chunk of data to your DRBD device, and then to its backing device for comparison. Both tests are repeated 5 times each to allow for some statistical averaging. The relevant result is the throughput measurements generated by dd.

Latency measurements have objectives completely different from throughput benchmarks: in I/O latency tests, one writes a very small chunk of data (ideally the smallest chunk of data that the system can deal with), and observes the time it takes to complete that write. The process is usually repeated several times to account for normal statistical fluctuations.

Just as throughput measurements, I/O latency measurements may be performed using the ubiquitous dd utility, albeit with different settings and an entirely different focus of observation.

Provided below is a simple dd-based latency micro-benchmark, assuming you have a scratch resource named test which is currently connected and in the secondary role on both nodes:

This test writes 1,000 512-byte chunks of data to your DRBD device, and then to its backing device for comparison. 512 bytes is the smallest block size a Linux system (on all architectures except s390) is expected to handle.

It is important to understand that throughput measurements generated by dd are completely irrelevant for this test; what is important is the time elapsed during the completion of said 1,000 writes. Dividing this time by 1,000 gives the average latency of a single sector write.

DRBD throughput is affected by both the bandwidth of the underlying I/O subsystem (disks, controllers, and corresponding caches), and the bandwidth of the replication network.

I/O subsystem throughput is determined, largely, by the number of disks that can be written to in parallel. A single, reasonably recent, SCSI or SAS disk will typically allow streaming writes of roughly 40MB/s to the single disk. When deployed in a striping configuration, the I/O subsystem will parallelize writes across disks, effectively multiplying a single disk’s throughput by the number of stripes in the configuration. Thus the same, 40MB/s disks will allow effective throughput of 120MB/s in a RAID-0 or RAID-1+0 configuration with three stripes, or 200MB/s with five stripes.

Network throughput is usually determined by the amount of traffic present on the network, and on the throughput of any routing/switching infrastructure present. These concerns are, however, largely irrelevant in DRBD replication links which are normally dedicated, back-to-back network connections. Thus, network throughput may be improved either by switching to a higher-throughput protocol (such as 10 Gigabit Ethernet), or by using link aggregation over several network links, as one may do using the Linux bonding network driver.

When estimating the throughput overhead associated with DRBD, it is important to consider the following natural limitations:

The minimum between the two establishes the theoretical throughput maximum available to DRBD. DRBD then reduces that throughput maximum by its additional throughput overhead, which can be expected to be less than 3 percent.

DRBD offers a number of configuration options which may have an effect on your system’s throughput. This section list some recommendations for tuning for throughput. However, since throughput is largely hardware dependent, the effects of tweaking the options described here may vary greatly from system to system. It is important to understand that these recommendations should not be interpreted as "silver bullets" which would magically remove any and all throughput bottlenecks.

DRBD latency is affected by both the latency of the underlying I/O subsystem (disks, controllers, and corresponding caches), and the latency of the replication network.

I/O subsystem latency is primarily a function of disk rotation speed. Thus, using fast-spinning disks is a valid approach for reducing I/O subsystem latency.

Likewise, the use of a battery-backed write cache (BBWC) reduces write completion times, also reducing write latency. Most reasonable storage subsystems come with some form of battery-backed cache, and allow the administrator to configure which portion of this cache is used for read and write operations. The recommended approach is to disable the disk read cache completely and use all cache memory available for the disk write cache.

Network latency is, in essence, the packet round-trip time ( ) between hosts. It is influenced by a number of factors, most of which are irrelevant on the dedicated, back-to-back network connections recommended for use as DRBD replication links. Thus, it is sufficient to accept that a certain amount of latency always exists in Gigabit Ethernet links, which typically is on the order of 100 to 200 microseconds (μs) packet RTT.

Network latency may typically be pushed below this limit only by using lower-latency network protocols, such as running DRBD over Dolphin Express using Dolphin SuperSockets.

As for throughput, when estimating the latency overhead associated with DRBD, there are some important natural limitations to consider:

The sum of the two establishes the theoretical latency minimum incurred to DRBD. DRBD then adds to that latency a slight additional latency overhead, which can be expected to be less than 1 percent.

DRBD stores various pieces of information about the data it keeps in a dedicated area. This metadata includes:

This metadata may be stored internally and externally. Which method is used is configurable on a per-resource basis.

DRBD uses generation identifiers (GIs) to identify "generations"of replicated data.

This is DRBD’s internal mechanism used for

The quick-sync bitmap is the internal data structure which DRBD uses, on a per-resource basis, to keep track of blocks being in sync (identical on both nodes) or out-of sync. It is only relevant when a resource is in disconnected mode.

In the quick-sync bitmap, one bit represents a 4-KiB chunk of on-disk data. If the bit is cleared, it means that the corresponding block is still in sync with the peer node. That implies that the block has not been written to since the time of disconnection. Conversely, if the bit is set, it means that the block has been modified and needs to be re-synchronized whenever the connection becomes available again.

As DRBD detects write I/O on a disconnected device, and hence starts setting bits in the quick-sync bitmap, it does so in RAM — thus avoiding expensive synchronous metadata I/O operations. Only when the corresponding blocks turn cold (that is, expire from the Activity Log), DRBD makes the appropriate modifications in an on-disk representation of the quick-sync bitmap. Likewise, if the resource happens to be manually shut down on the remaining node while disconnected, DRBD flushes the complete quick-sync bitmap out to persistent storage.

When the peer node recovers or the connection is re-established, DRBD combines the bitmap information from both nodes to determine the total data set that it must re-synchronize. Simultaneously, DRBD examines the generation identifiers to determine the direction of synchronization.

The node acting as the synchronization source then transmits the agreed-upon blocks to the peer node, clearing sync bits in the bitmap as the synchronization target acknowledges the modifications. If the re-synchronization is now interrupted (by another network outage, for example) and subsequently resumed it will continue where it left off — with any additional blocks modified in the meantime being added to the re-synchronization data set, of course.

DRBD has a defined interface for the mechanism that fences the peer node in case of the replication link being interrupted. The drbd-peer-outdater helper, bundled with Heartbeat, is the reference implementation for this interface. However, you may easily implement your own peer fencing helper program.

The fencing helper is invoked only in case

The program or script specified as the fence-peer handler, when it is invoked, has the DRBD_RESOURCE and DRBD_PEER environment variables available. They contain the name of the affected DRBD resource and the peer’s hostname, respectively.

Any peer fencing helper program (or script) must return one of the following exit codes:

Commercial DRBD support, consultancy, and training services are available from the project’s sponsor company, LINBIT.

The public mailing list for general usage questions regarding DRBD is drbd-user@lists.linbit.com. This is a subscribers-only mailing list, you may subscribe at https://lists.linbit.com/listinfo/drbd-user. A complete list archive is available at https://lists.linbit.com/pipermail/drbd-user.

Some of the DRBD developers can occasionally be found on the irc.freenode.net public IRC server, particularly in the following channels:

Getting in touch on IRC is a good way of discussing suggestions for improvements in DRBD, and having developer level discussions.

LINBIT maintains an official twitter account.

If you tweet about DRBD, please include the #drbd hashtag.

DRBD’s authors have written and published a number of papers on DRBD in general, or a specific aspect of DRBD. Here is a short selection:

Volumes are a new concept in DRBD 8.4. Prior to 8.4, every resource had only one block device associated with it, thus there was a one-to-one relationship between DRBD devices and resources. Since 8.4, multiple volumes (each corresponding to one block device) may share a single replication connection, which in turn corresponds to a single resource.

This section highlights changes to the configuration syntax. It affects the DRBD configuration files in /etc/drbd.d, and /etc/drbd.conf.

In DRBD 8.4, several drbd.conf default values have been updated to match improvements in the Linux kernel and available server hardware.