Record/replay
Record/replay functions are used for the deterministic replay of qemu execution. Execution recording writes a non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewind to the specific replay moment. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.
Deterministic replay has the following features:
Deterministically replays whole system execution and all contents of the memory, state of the hardware devices, clocks, and screen of the VM.
Writes execution log into the file for later replaying for multiple times on different machines.
Supports i386, x86_64, ARM, AArch64, Risc-V, MIPS, MIPS64, S390X, Alpha, PowerPC, PowerPC64, M68000, Microblaze, OpenRISC, Nios II, SPARC, and Xtensa hardware platforms.
Performs deterministic replay of all operations with keyboard and mouse input devices, serial ports, and network.
Usage of the record/replay:
First, record the execution with the following command line:
qemu-system-x86_64 \ -icount shift=auto,rr=record,rrfile=replay.bin \ -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \ -object filter-replay,id=replay,netdev=net1After recording, you can replay it by using another command line:
qemu-system-x86_64 \ -icount shift=auto,rr=replay,rrfile=replay.bin \ -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \ -object filter-replay,id=replay,netdev=net1The only difference with recording is changing the rr option from record to replay.
Block device images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file. This behavior is enabled by using blkreplay driver. It should be used for every enabled block device, as described in Block devices section.
-net none
option should be specified when network is not used, because QEMU adds network card by default. When network is needed, it should be configured explicitly with replay filter, as described in Network devices section.Interaction with audio devices and serial ports are recorded and replayed automatically when such devices are enabled.
Core idea
Record/replay system is based on saving and replaying non-deterministic events (e.g. keyboard input) and simulating deterministic ones (e.g. reading from HDD or memory of the VM). Saving only non-deterministic events makes log file smaller and simulation faster.
The following non-deterministic data from peripheral devices is saved into the log: mouse and keyboard input, network packets, audio controller input, serial port input, and hardware clocks (they are non-deterministic too, because their values are taken from the host machine). Inputs from simulated hardware, memory of VM, software interrupts, and execution of instructions are not saved into the log, because they are deterministic and can be replayed by simulating the behavior of virtual machine starting from initial state.
Instruction counting
QEMU should work in icount mode to use record/replay feature. icount was
designed to allow deterministic execution in absence of external inputs
of the virtual machine. Record/replay feature is enabled through -icount
command-line option, making possible deterministic execution of the machine,
interacting with user or network.
Block devices
Block devices record/replay module intercepts calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following:
-drive file=disk.qcow2,if=none,snapshot,id=img-direct
-drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
-device ide-hd,drive=img-blkreplay
blkreplay driver should be inserted between disk image and virtual driver controller. Therefore all disk requests may be recorded and replayed.
Snapshotting
New VM snapshots may be created in replay mode. They can be used later to recover the desired VM state. All VM states created in replay mode are associated with the moment of time in the replay scenario. After recovering the VM state replay will start from that position.
Default starting snapshot name may be specified with icount field rrsnapshot as follows:
-icount shift=auto,rr=record,rrfile=replay.bin,rrsnapshot=snapshot_name
This snapshot is created at start of recording and restored at start of replaying. It also can be loaded while replaying to roll back the execution.
snapshot
flag of the disk image must be removed to save the snapshots
in the overlay (or original image) instead of using the temporary overlay.
-drive file=disk.ovl,if=none,id=img-direct
-drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
-device ide-hd,drive=img-blkreplay
Use QEMU monitor to create additional snapshots. savevm <name>
command
created the snapshot and loadvm <name>
restores it. To prevent corruption
of the original disk image, use overlay files linked to the original images.
Therefore all new snapshots (including the starting one) will be saved in
overlays and the original image remains unchanged.
When you need to use snapshots with diskless virtual machine, it must be started with “orphan” qcow2 image. This image will be used for storing VM snapshots. Here is the example of the command line for this:
qemu-system-x86_64 \ -icount shift=auto,rr=replay,rrfile=record.bin,rrsnapshot=init \ -net none -drive file=empty.qcow2,if=none,id=rr
empty.qcow2
drive does not connected to any virtual block device and used
for VM snapshots only.
Network devices
Record and replay for network interactions is performed with the network filter. Each backend must have its own instance of the replay filter as follows:
-netdev user,id=net1 -device rtl8139,netdev=net1
-object filter-replay,id=replay,netdev=net1
Replay network filter is used to record and replay network packets. While recording the virtual machine this filter puts all packets coming from the outer world into the log. In replay mode packets from the log are injected into the network device. All interactions with network backend in replay mode are disabled.
Audio devices
Audio data is recorded and replay automatically. The command line for recording and replaying must contain identical specifications of audio hardware, e.g.:
-soundhw ac97
Serial ports
Serial ports input is recorded and replay automatically. The command lines
for recording and replaying must contain identical number of ports in record
and replay modes, but their backends may differ.
E.g., -serial stdio
in record mode, and -serial null
in replay mode.
Reverse debugging
Reverse debugging allows “executing” the program in reverse direction. GDB remote protocol supports “reverse step” and “reverse continue” commands. The first one steps single instruction backwards in time, and the second one finds the last breakpoint in the past.
Recorded executions may be used to enable reverse debugging. QEMU can’t execute the code in backwards direction, but can load a snapshot and replay forward to find the desired position or breakpoint.
The following GDB commands are supported:
reverse-stepi
(orrsi
) - step one instruction backwards
reverse-continue
(orrc
) - find last breakpoint in the past
Reverse step loads the nearest snapshot and replays the execution until the required instruction is met.
Reverse continue may include several passes of examining the execution between the snapshots. Each of the passes include the following steps:
loading the snapshot
replaying to examine the breakpoints
if breakpoint or watchpoint was met
loading the snapshot again
replaying to the required breakpoint
else
proceeding to the p.1 with the earlier snapshot
Therefore usage of the reverse debugging requires at least one snapshot
created. This can be done by omitting snapshot
option
for the block drives and adding rrsnapshot
for both record and replay
command lines.
See the Snapshotting section to learn more about running record/replay
and creating the snapshot in these modes.
When rrsnapshot
is not used, then snapshot named start_debugging
created in temporary overlay. This allows using reverse debugging, but with
temporary snapshots (existing within the session).