FAI Guide (Fully Automatic Installation)

The man pages always include up-to-date information and a lot of details of all FAI commands. So, don’t forget to read them carefully. Now read this manual, then enjoy the fully automatic installation and your saved time.

Have you ever performed identical installations of an operating system several times? Would you like to be able to install a Linux cluster with dozens of nodes single handedly?

Repeating the same task again and again is boring — and will surely lead to errors. Also a whole lot of time could be saved if the installations were done automatically. An installation process with manual interaction does not scale. But clusters have the habit of growing over the years. Think long-term rather than planning just a few months into the future.

In 1999, I had to perform an installation of a Linux cluster with one server and 16 clients. Since I had much experience doing automatic installations of Solaris operating systems on SUN SPARC hardware, the idea to build an automatic installation for Debian was born. Solaris has an automatic installation feature called JumpStart
[Solaris 8 Advanced Installation Guide at https://docs.oracle.com/cd/E19455-01/806-0957/806-0957.pdf ]
. In conjunction with the auto-install scripts from Casper Dik
[http://www.science.uva.nl/pub/solaris/auto-install]
, I could save a lot of time not only for every new SUN computer, but also for re-installation of existing workstations. For example, I had to build a temporary LAN with four SUN workstations for a conference, which lasted only a few days. I took these workstations out of our normal research network and set up a new installation for the conference. When it was over, I simply integrated the workstations back into the research network, rebooted just once, and after half an hour, everything was up and running as before. The configuration of all workstations was exactly the same as before the conference, because everything was performed by the same installation process. I also used the automatic installation for reinstalling a workstation after a damaged hard disk had been replaced. It took two weeks until I received the new hard disk but only a few minutes after the new disk was installed, the workstation was running as before. And this is why I choose to adapt this technique to a PC cluster running Linux.

The install client which will be installed using FAI, is booted via network card or from CD or USB stick. It gets an IP address and boots a Linux kernel which mounts its root file system via NFS (the nfsroot) from the install server. After the kernel is started, the FAI startup script performs the automatic installation which doesn’t need any interaction. First, the hard disks will be partitioned, file systems are created and then software packages are installed. After that, the new installed operating system is configured to your local needs using some scripts. Finally, the new operating system will be booted from the local disk.

The details of how to install the computer (the configuration) are stored in the configuration space on the install server. Configuration files are shared among groups of computers if they are similar using the class concept. So you need not create a configuration for every new host. Hence, FAI is a scalable method to install a big cluster with a great number of nodes even if their configuration is not identical.

FAI can also be used as a rescue system or for hardware inventory. You can boot your computer, but it will not perform an installation. Instead it will run a fully functional Debian GNU/Linux without using the local hard disks. Then you can do a remote login and backup or restore a disk partition, check a file system, inspect the hardware or do any other task.

The installation time is determined by the amount of software and the speed of the hard disk. Here are some sample times. All install clients had a 1Gbit network card installed.

Without further ado, this section will provide a quick and easy demonstration of a fully automatic installation using the FAI CD and a virtual machine.

Just download the CD ISO image from https://fai-project.org/fai-cd and boot your VM using this CD. You will see a grub menu where you can select from different installation types.

This installation will run without an install server. The CD installation is the same as when run in a network environment using the FAI install server and can also be used from USB stick on a real computer.

You can also create yourself a custom fully automated installation image on the webpage https://fai-project.org/FAIme without the need of installing FAI on your computer.

We will create a private network and start two virtual machines. One will become your own FAI server, the other will be an install client.

If you intend to use VMware or VirtualBox, ensure that your client uses a bridged network connection. A detailed description is in the FAI wiki
[https://wiki.fai-project.org/index.php/VirtualBox_for_your_first_FAI_installation]
. Also, it is not possible to use bridged network interfaces over wireless, as most WiFi network cards do not support this feature.

When using Qemu/KVM and the fai-kvm wrapper you can create the network using the command fai-mk-network. First install some packages

The next command sets up a private network with a software bridge with several tap devices that belong to the user <username>.

After that, you can use fai-kvm (-h will give you some help) for starting virtual machines using KVM that are connected to this private network. Be careful. By default, fai-kvm will create the disk images for the virtual machines in /tmp, which is a RAM disk on most systems. It’s no problem to create an empty 20G disk image in /tmp (even if this partition is of 4GB size), but while the VM is writing data to its disk, this will start to consume space in /tmp.

Start the first virtual host, which will become the FAI server
[This installation will consume about 2GB of space in /tmp.]
:

In the grub menu select faiserver, using internal DHCP and a fixed IP. This will install a host called faiserver with IP 192.168.33.250 which contains all software needed for a FAI server. It will also set up a local package cache (using apt-cacher-ng). Once the installation is finished, reboot the machine. During the first boot of the new system, it will automatically set up the nfsroot. This may take some minutes.

After that you can start additional hosts using network boot. For every new host, you have to use a different value for -u, which will be used for generating different MAC addresses and using different disk image file names.

Those install clients will show you a menu, where you can select which type of installation you like to perform. If the install client does not find the server, it is usually because fai-monitor is no longer running on it. This can happen, if you reboot the faiserver after the installation. To remedy this, simply run fai-monitor on the faiserver and re-attempt the client boot.

Another client could be started with:

You can start as many machines in the network as tap devices are available. All these machines can connect to the outside internet but are only reachable from your host machine.

First, some terms used in this manual are described.

For a more in-depth description of tasks , see [tasks].

Note that you are not limited to the FAI tasks. You can also define additional programs or scripts which will be run on particular occasions. They are called hooks.

Classes are used in nearly all tasks of the installation. Classes determine which configuration files to choose from a list of available alternatives. To determine which config files to use, FAI searches the list of defined classes and uses all configuration files that match a class name
[It’s also possible to use only the configuration file with the highest priority since the order of classes define a priority from low to high within the list of classes. ]
. The following loop implements this function in pseudo shell code:

The very nice feature of this is that you can add a new configuration alternative and it will automatically be used by FAI without changing the code, if the configuration file uses a class name.

This is because the loop automatically detects new configuration files that should be used. The idea of using classes in general and using certain files matching a class name for a configuration is adopted from the installation scripts by Casper Dik for Solaris. This technique proved to be very useful and easy.

You can group multiple hosts that share the same configuration files by using the same class. You can also split the whole configuration data for all clients into several classes and use them like lego bricks and build the entire configuration for a single client by assembling the bricks together.

If a client belongs to class A, we say the class A is defined for this client. A class has no value, it is just defined or undefined.

Classes determine how the installation is performed. For example, an install client can be configured to get the XFCE desktop by just adding the class XFCE to it. Naturally, also more granular configurations are possible. For instance, classes can describe how the hard disk should be partitioned, they can define which software packages will be installed, or which customization steps are performed.

Often, a client configuration is created by only changing or appending the classes to which this client belongs, making the installation of a new client very easy. Thus no additional information needs to be added to the configuration space if the existing classes suffice for your needs.

As you can see, classes are a central pillar of customizing your configuration space and with that your client installation. On how to define your own classes, refer to [defining classes].

This will also install the packages for DHCP, TFTP and NFS server daemons.

Install the simple examples into the configuration space
[These files need not belong to the root account.]
.

These examples contain configuration for some sample hosts. Depending on the host name used, your computer will be configured as follows:

All hosts will have an account called demo with password fai. The root account also has the password fai.

If the FAI flag menu is added, instead of using the host name for determing the type of installation, a menu is presented, and the user can choose a profile for the installation.

For booting the install client via PXE, the install server needs a DHCP and a TFTP daemon running. The package fai-quickstart has already installed the software packages for those daemons. Additionally the package of the NFS server for exporting the nfsroot and the config space was installed.

The last step before booting your client for the first time is to specify what configuration the client should boot when doing PXE boot. We use the command fai-chboot(8) to create a pxelinux configuration for each install client. This includes information about the kernel, the initrd, the config space and some boot parameters. You should read the manual page, which gives you some good examples. Here’s the command for starting the installation for the host demohost.

At this point, you should have a working faiserver setup and your clients should boot into FAI and be able to install one of the examples.

In the following section, you can read about planning your installation, tailoring your configuration space to your particular needs and extending FAI using hooks.

The faiserver and its setup is by no means static. It is possible to customize and extend your server. For this, please refer to the [Customizing your install server setup] section in [advanced].

The configuration space is the collection of information about how exactly to install a client. The central configuration space for all install clients is located on the install server in /srv/fai/config and its subdirectories. This will be mounted by the install clients to /var/lib/fai/config. The main installation command fai(8) uses all these subdirectories in the order listed except for hooks.

The folder must support directory listing. FAI will not probe for potentially matching files.

See chapter [otherdists] for how to install different distributions.

There are different possibilities to define classes:

The last option is a very nice feature, since these scripts will define classes in a very flexible way. For example, several classes may be defined only if certain hardware is identified or a class is defined depending on the network subnet information.

All names of classes, except the host name, are written in uppercase. They must not contain a hyphen, a hash, a semicolon or a dot, but may contain underscores and digits.

The task defclass calls the command fai-class(1) to define classes. All scripts matching ^[0-9][0-9]* (they start with two digits) in the subdirectory $FAI/class are executed for defining classes. Everything that is printed to STDOUT is automatically defined as a class. For more information on defining class, read the manual pages for fai-class(1). The script 50-host-classes (see below a stripped version) is used to define classes depending on the host name.

Host names should rarely be used for the configuration files in the configuration space. Instead, a class should be defined and then added for a given host. This is because most of the time the configuration data is not specific for one host, but can be shared among several hosts.

The order of the classes is important because it defines the priority of the classes from low to high.

The task defvar defines the variables for the install client. Variables are defined by scripts in class/*.var. All global variables can be set in DEFAULT.var. For groups of hosts use a class file. For a single host use the file $HOSTNAME .var. Also here, it’s useful to study all the examples.

The following variables are used in the examples and may also be useful for your installation:

The tool setup-storage(8) reads a file in $FAI/disk_config for the disk configuration. This file describes how all the local disks will be partitioned, which file system types should be created (like ext3/4, xfs, btrfs), and where they are mounted to. You can also create software RAID and LVM setups using this config file. It’s also possible to preserve the disk layout or to preserve the data on certain partitions.

During the installation process all local file systems are mounted relative to /target. For example if you specify the mount point /home in a disk configuration file this will be the directory /target/home during the installation process and will become /home for the new installed system.

A base file is only needed when installing a distribution which is different from the one in the nfsroot.

You can use the format described in debconf-set-selections(1).

FAI supports http, https and NFS for accessing the package mirror. Set the variable $FAI_DEBMIRROR for using NFS.

Before installing packages, FAI will add the content of all files named package_config/class.gpg to the list of apt keys. If your local repository is signed by your keyid AB12CD34 you can easily add this key, so FAI will use it during installation. Use this command for creating the CLASS.gpg file:

The script install_packages(8) installs the selected software packages. It reads all configuration files in $FAI/package_config whose file name matches a defined class. The syntax is very simple.

Comments are starting with a hash (#) and are ending at the end of the line. Every package command begins with the word PACKAGES followed by a command name, which maps to a different package tool like apt-get, aptitude or dnf for e.g. The command defines which command will be used to install the packages named after this command. The list of all available commands can be listed using install_packages -H. Supported package tools are: apt, apt-get, aptitude, smart, yast, dnf, rpm, zypper

Multiple lines with lists of space separated names of packages follow the PACKAGES lines. All dependencies are resolved. Packages with suffix - (eg. lilo-) will be removed instead of installed. The order of the packages doesn’t matter. If you like to install packages from another release than the default, you can append the release name to the package name like in openoffice.org/etch-backports. You can also specify a certain version like apt=0.3.1. More information on these features are described in aptitude(8).

You can specify additional parameters for the package manager adding key=value after PACKAGES <command>. Currently we support release=<name> which will add -t <name> when installing packages.

Example:

This will install the nvidia-smi package from the testing release, including the dependencies. Don’t forget to add an entry into sources.list. You may also want to adjust the apt pinning (see apt_references(5)).

A line which contains the PRELOADRM commands, downloads a file using wget(1) into a directory before installing the packages. Using the file: URL, this file is copied from $FAI_ROOT to the download directory. For example the package realplayer needs an archive to install the software, so this archive is downloaded to the directory /root. After installing the packages this file will be removed. If the file shouldn’t be removed, use the command PRELOAD instead.

It’s possible to append a list of class names after the command for apt-get. So this PACKAGES command will only be executed when at least one of the corresponding classes is defined (logical OR). So you can combine many small files into the file DEFAULT. WARNING! Use this feature only in the file DEFAULT to keep everything simple. See this file for some examples.

If you want to remove a package name from a certain class was part of this class before, you should not remove the package name from the class file, but instead append a dash (-) to it. This will make sure that the package is removed during a softupdate on hosts which were installed using the old class definition which included this package name.

If you specify a package that does not exist this package will be removed automatically from the installation list only if the command install is used.

The concept of classes priority allows a higher priority class (one that comes later in the sequence of classes) to override the selection of packages of a lower priority class. For this to work correctly, the higher priority class must use the same PACKAGES command (e.g. PACKAGES install-norec instead of just PACKAGES install) as the one used by the class it is trying to override. This is useful to suppress installation of a package, for example, to avoid installing the linuxlogo package installed by class FAIBASE:

The command fai-do-scripts(1) is called to execute all scripts in this directory. If a directory with a class name exists, all scripts matching ^[0-9][0-9]* are executed in alphabetical order. So it’s possible to use scripts of different languages (shell, cfengine, Perl, Python, Ruby, expect,..) for one class.

Thoses scripts write their output to sripts.log. The file status.log contains the names of all scripts executed and their exit status.

Hooks let you specify functions or programs which are run at certain steps of the installation process. Before a task is called, FAI searches for existing hooks for this task and executes them. As you might expect, classes are also used when calling hooks. Hooks are executed for every defined class. You only have to create the hook with the name for the desired class and it will be used. If several hooks for a task exists, they are called in the order defined by the classes. If debug is included in $FAI_FLAG the option -d is passed to all hooks, so you can debug your own hooks. If some default tasks should be skipped, use the subroutine skiptask and a list of default tasks as parameters. In the examples provided, the hooks of the class CENTOS skips some Debian specific tasks.

The directory $FAI/hooks/ contains all hooks. A hook is an executable file following the naming scheme taskname.CLASSNAME[.sh] (e.g. repository.CENTOS or 'savelog.LAST.sh). The task name specifies which task to precede executing this hook, if the specified class is defined for the installing client. See section [tasks] for a complete list of default tasks that can be used.

A hook of the form hookprefix.classname can’t define variables for the installation script, because it’s a subprocess. But you can use any binary executable or any script you wrote. Hooks that have the suffix .sh (e.g. partition.DEFAULT.sh) must be Bourne shell scripts and are sourced. So it’s possible to redefine variables for the installation scripts.

In the first part of FAI, all hooks with prefix confdir are called. Those hooks can not be located in the config space, since it’s not yet available. Therefore these hooks are the only hooks located in $nfsroot/$FAI/hooks on the install server. All other hooks are found in $FAI_CONFIGDIR/hooks on the install server.

All hooks that are called before classes are defined can only use the following classes: DEFAULT $HOSTNAME LAST. If a hook for class DEFAULT should only be called if no hook for class $HOSTNAME is available, insert these lines to the default hook:

Some examples for what hooks could be used:

The variable $FAI_FLAGS contains a space separated list of flags. Flags are normally defined in the pxelinux.cfg file which should be created by fai-chboot(1). The following flags are known:

This is only the root password during the installation process, not for the new installed system. You can also log in without a password when using $SSH_IDENTITY.

After the kernel has booted, it mounts the root file system via NFS from the install server and starts the script /usr/sbin/fai
[Since the root file system on the clients is mounted via NFS, fai is located in /srv/fai/nfsroot/usr/sbin on the install server.]
. This script controls the sequence of the installation. No other scripts in /etc/init.d/ are used.

The configuration space is made available via the configured method (an NFS mount by default) from the install server to the path defined in $FAI
[$FAI is an internal variable used by the FAI scripts. By default the path is /var/lib/fai/config.]

When booting the install client from network card with PXE you will see some messages like this:

At this point the install client has successfully received the network config via DHCP and the kernel and initrd via TFTP. It now boots the Linux kernel and the initrd. If everything went right, the initrd mounts the nfsroot
[/srv/fai/nfsroot from the install server via NFS]
and the FAI scripts are started. The first thing you see is the red FAI copyright message.

You can also see the list of FAI classes, that are defined for this host. This list is very important for the rest of the installation.

The first task is called confdir, which is responsible for getting access to the config space. Here, we use an NFS mount from the install server as you can see on the console (and later in the logs).

Before the installation is started ($FAI_ACTION=install) the computer beeps three times. So, be careful when you hear three beeps but you do not want to perform an installation and let FAI erase all yout data on the local disk!

For rebooting the computer during or at the end of the installation you should use the command faireboot in favour of the normal reboot command. Use faireboot also if logged in from remote. If the installation hasn’t finished, use faireboot -s, so the log files are also copied to the install server.

If the installation has finished successfully, the computer should boot a small Debian system. You can login as user demo or root with password fai.

After the install client has booted only the script /usr/sbin/fai is executed. It will do some minimal initialization. The variable $FAI_CONFIG_SRC
[It is defined on the kernel command line]
is used to get access to the FAI configuration space which is then available in the directory $FAI
[/var/lib/fai/config]
. FAI will not proceed without the config space.

The command fai-class(1) executes scripts in $FAI/class for defining classes. If the scripts write a string to stdout, this will be defined as a class. Read all the details in the man page of fai-class(1).

After defining the classes, every file matching .var with a prefix which matches a defined class is sourced to define variables. It must contain vaild shell code.

For the disk partitioning exactly one disk configuration file from $FAI/disk_config is selected using classes.

The format of the disk configuration is similar to a fstab file.

The partitioning tool setup-storage(8) performs all commands necessary for creating the disk partition layout, software RAID, LVM and for creating the file systems. Disks and partitions may easily be referenced by disk1.1, disk2.4 etc. Read the manual page of setup-storage(8) for a detailed description and some examples of the format.

Files in $FAI/debconf are used for the usual debconf(7) preseeding if the file names match a class name.

The command install_packages(8) reads the config files from $FAI/package_config in a class based manner and installs software packages on the new file system.

It installs the packages using apt-get(8), aptitude(1), yum or other package tools without any manual interaction needed. Package dependecies are also resolved by the package tools.

The format of the configuration files is described in [packageconfig].

Often the default configurations of the software packages will not meet your site-specific needs. You can call arbitrary scripts which adjust the system configuration. Therefore the command fai-do-scripts(1) executes scripts in $FAI/scripts in a class based manner. It is possible to have several scripts of different types (shell, cfengine, …) to be executed for one class.

The default set of scripts in $FAI/scripts include examples for installing Debian and Rocky Linux machines. They set the root password, add a user account (set by '$username, default to demo), set the timezone, configure the network for DHCP or using a fixed IP address, setup grub and more. They should do a reasonable job for your installation. You can edit them or add new scripts to match your local needs.

More information about these scripts are described in [cscripts].

When all tasks are finished, the log files are written to /var/log/fai/$HOSTNAME/install/
[/var/log/fai/localhost/install/ is a link to this directory.]
on the new system and to the account on the install server if $LOGUSER is defined (you have to enable this in /srv/fai/config/class/FAIBASE.var). It is also possible to specify another host as log saving destination through the variable $LOGSERVER. If $LOGSERVER is not defined, FAI uses the variable $SERVER which is only defined during an initial installation (by get-boot-info).

Additionally, two symlinks will be created to indicated the last directory written to. The symlink last points to the log directory of the last FAI action performed. The symlinks last-install and last-sysinfo point to the directory of the last corresponding action. By default log files will be copied to the log server using scp. You can use the variable $FAI_LOGPROTO in file fai.conf(5) to choose another method for saving logs to the remote server. Here’s an example of the symlink structure:

Examples of the log files can be found at https://fai-project.org/logs.

Before rebooting, the install client calls fai-chboot -d <hostname> on the install server, to disable its own PXELINUX configuration. Otherwise, it would restart the installation during the next boot. Normally this should boot the new installed system from its second boot device, the local hard disk.

At the end, the system is automatically rebooted if "reboot" was added to $FAI_FLAGS.

If the install client boots you can check if all information from the DHCP daemon are received correctly. The received information is written to /tmp/fai/boot.log. An example of the result of a DHCP request can be found in the sample log files.

You can monitor the installation of all install clients with the command fai-monitor(8). All clients check if this daemon is running on the install server (or the machine defined by the variable $monserver). Each time a task starts or ends, a message is sent. The FAI monitor daemon prints this messages to standard output. There’s also a graphical frontend available, called fai-monitor-gui(1).

You have to collect all Ethernet (MAC) addresses of the install clients and assign a host name and IP address to each client. To collect the MAC addresses, boot your install clients. You can already do this before any DHCP daemon is running in your subnet. They will fail to boot (because of the missing DHCP or missing TFTP), but you can still collect the MAC addresses.

While the install clients are booting, they send broadcast packets to the LAN. You can log the MAC addresses of these hosts by running the following command simultaneously on the server:

After the hosts have been sent some broadcast packets abort tcpdump by typing ctrl-c. You get a list of all unique MAC addresses with these commands:

After that, you only have to assign these MAC addresses to host names and IP addresses (/etc/ethers and /etc/hosts or corresponding NIS maps). With this information you can configure your DHCP daemon (see the section [bootdhcp]).
[I recommend to write the MAC addresses (last three bytes will suffice if you have network cards from the same vendor) and the host name in the front of each chassis.]

Here we describe the details of PXE booting, which are only needed if you have problems when booting your install clients.

Almost all modern bootable network cards support the PXE boot environment. PXE is the Preboot Execution Environment. This requires the PXELINUX bootloader and a special version of the TFTP daemon, which is available in the Debian packages pxelinux and tftpd-hpa. PXE booting also needs a DHCP server, so that the network card can configure its IP parameters. This is the sequence of a PXE boot:

Example of a pxelinux.cfg file:

See /usr/share/doc/syslinux/pxelinux.doc for more detailed information about PXELINUX. FAI uses the lpxelinux.0 binary which also supports loading the kernel and initrd via FTP or HTTP. The command fai-chboot(8) supports this with the option -U.

The configuration for the FAI package (not the configuration data for the install clients) is defined in fai.conf(5). Definitions that are only used for creating the nfsroot are located in nfsroot.conf(5). Check these important variables in nfsroot.conf before calling fai-setup or fai-make-nfsroot.

The content of /etc/fai/apt/sources.list is used by the install server and also by the clients. If your install server has multiple network cards and different host names for each card (as for a Beowulf server), use the install server name which is known by the install clients.

If you have problems running fai-setup, they usually stem from fai-make-nfsroot(8) which is called by former command. Adding -v gives you a more verbose output which helps you pinpoint the error. The output is written to /var/log/fai/fai-make-nfsroot.log.
[For debugging purpose it may help to enter the chroot environment manually using this command. faiserver# chroot /srv/fai/nfsroot bash]

The setup also creates the account fai (defined by $LOGUSER) if not already available. So you can add a user before calling fai-setup(8) using the command adduser(8) and use this as your local account for saving log files. The log files of all install clients are saved to the home directory of this account. You should change the primary group of this account, so this account has write permissions to /srv/tftp/fai in order to call fai-chboot for creating the PXE configuration for the hosts.

When you make changes to fai.conf, nfsroot.conf the nfsroot has to be rebuilt by calling fai-make-nfsroot(8). If you only like to install a new kernel package to the nfsroot add the flags -k or -K to fai-make-nfsroot. This will not recreate your nfsroot, but only updates your kernel and kernel modules inside the nfsroot or add additional packages into the nfsroot.

You can easily create an installation CD (or USB stick) of your network installation setup. This will perform the same installation and configuration from CD without the need of the install server. Therefore you need to create a partial mirror of all Debian packages needed for your FAI classes (using fai-mirror(1)). Then the command fai-cd(8) will put this mirror, the nfsroot and the config space onto a bootable CD. That’s it!

To easily create the installation CD, you can use the following command (for more tuning details see fai-cd(8)):

This installation CD contains all data needed for the installation. The command fai-cd(8) puts the nfsroot, the configuration space and a subset of the Debian mirror onto a CD-ROM. A partial package mirror is created using the command fai-mirror(1) which contains all packages that are used by the classes used in your configuration space. A sample ISO image is available at https://fai-project.org/fai-cd.

Using the command dd(1) you can also create a bootable USB stick by just writing the content of the ISO file to your USB stick (here the stick is /dev/sdf).

This is no live CD of the install server.

The command fai-diskimage(8) creates disk images, which can be used with a virtual machine like KVM, VMware, VirtualBox or a cloud service like OpenStack, GCE, EC2 and others. The installation process performs the normal FAI tasks on a raw disk image. After the installation you can boot the disk image and have a running system. The disk image can also be converted to qcow2 format. You do not need to setup the FAI nfsroot when only using fai-diskimage. But you need a basefile in your configuration space. Setting the variable $FAI_BASEFILEURL will automatically download an appropriate base file into your config space.

Here’s an example how to create a raw disk image for a host called cloud3, with a small set of software packages:

This command will create a disk image called ubuntu.qcow2 for a Ubuntu 16.04 desktop with hostname set to foobar.

You can give disk images a try without installing FAI, if you visit https://fai-project.org/FAIme/cloud

Starting FAI 5.4 it’s now possible to build a disk image for different architectures than the host is running. For example you can build an image for 64-bit ARM architecture (aarch64) on a host running on amd64 architecture. Here are the steps do to:

Then you can run qemu as a normal user:

This works similar for other architectures. Keep in mind, that qemu network setup will have poor performance if not using the virtio driver as above or tap devices.

You can find the base files for many architectures at https://fai-project.org/download/basefiles/, or use mk-basefile to create your own.

If you set the variable $FAI_ACTION to sysinfo (for e.g. by using fai-chboot -S), the client will not install a new system, but will collect a lot of system information. If you set $FAI_ACTION to inventory you will only get a few hardware information. Both actions can be used for FAI as a rescue system.

Type ctrl-c to get a shell or use Alt-F2 or Alt-F3 and you will get another console terminal, if you have added createvt to $FAI_FLAGS.

You now have a running Linux system on the install client without using the local hard disk. Use this as a rescue system if your local disk is damaged or the computer can’t boot properly from hard disk. You will get a shell and you can execute various commands (dmesg, lsmod, df, lspci, …). Look at the log file in /tmp/fai. There you can find much information about the boot process.

FAI mounts all file systems it finds on the local disks read only. It also tells you on which partition a file /etc/fstab exists. When only one file system table is found, the partitions are mounted according to this information. Here’s an example:

This method can be used as a rescue environment! If you need a file system with read-write access use the rwmount command:

To boot into FAI and begin the installation sequence without using the NFS protocol, you boot the client machine using PXE as usual and then retrieve an image containing the nfsroot via http.

To create an image, use fai-cd’s -S argument and -M to avoid setting up a partial mirror within the image.

Move this image to a directory from which it can be requested via http (usually a directory served by the webserver)

To now request the squashfs image, add the following to your kernel command line, e.g. in your pxelinux configuration file for the client with fai-chboot.

Replace faiserver with the domain name or IP of the machine your squash image is served from.

You can install all sorts of Linux distributions from a single Debian nfsroot. Therefore you have to create a base.tar.xz of the distribution you like to install and place it into the basefiles directory. Then name it UBUNTU2204.tar.xz for example. An install client which belongs to the class UBUNTU2204 then extracts this base file into its empty file system. Additionally you have to adjust the sources.list or similar configuration files which are needed for specifying the location of the package repository.

The tool rinse(8) is used for creating base files for distribution like Rocky Linux, CentOS, openSUSE or Fedora. Some basefiles can be downloaded from https://fai-project.org/download/basefiles/.

The script mk-basefile in /usr/share/doc/fai-doc/examples/simple/basefiles/ helps creating this base files.

If you have to create some chroot environments, or a virtualization environment where you neither can nor want to run a normal Debian Installer in to get to a working system (for example, Xen guest domains), there is the FAI action dirinstall. By calling

and using either the option -c <classes> or -N you get a FAI installation, without the partitioning action, right into the target directory. The host name for the target installation can be specified using -u <host-name>

This, for example, can be used to combine FAI with the tool xen-tools, which helps you to build Xen guest domains. xen-tools are very nice for generating configuration files and block devices for new guests based on simple commands and/or configuration files, but they can only assign one role per installation for customization. FAI-users need and want more, as they are used to have the class system. They get them even in xen-tools installations, by using the following code as a xen-tools role script:

Then, you should set the variable install=0 in the config of xen-tools for that host.

FAI can also do updates of already running systems, without a re-installation from scratch. This is called softupdate. A FAI softupdate skips the tasks which are not suitable for updating a running system, like partitioning the hard disks and creating file systems. Instead, it only executes the tasks for updating and installing software packages and calling the customization scripts.

To run a softupdate call:

By default, a softupdate uses the list of classes defined during the initial installation. Make sure to set the variable $LOGSERVER (done in a class/*.var file) if FAI should save the log files to a remote machine.

It’s up to you, how to start a softupdate on a bigger number of hosts. You may do the softupdate on a regular basis via cron or you can use tools like clusterssh(1) to start a softupdate via a push on a list of hosts.

Keep in mind, that the customization scripts are run every time you do a softupdate. That means, they have to be idempotent i.e. the result of their operation should always produce the same result, even when they run more than once.

For example appending a line to a file must not done via this code:

Instead use the command ainsl(1) in a shell script or use cfengine’s function AppendIfNoSuchLine.

All commands in the customization script must be capable of modifying the target file system wether it’s available in /target during the initial installation or wether it’s the normal file system relative to / during softupdate.

Here are some variable that help writing these scripts:

To install a computer with a 32-bit OS, you need an i386 nfsroot. Creating this 32-bit nfsroot on an install server running amd64 is quite simple. Install and set up the FAI packages. Then copy your FAI config files to a new subdirectory.

Edit the variable $FAI_DEBOOTSTRAP_OPTS in /etc/fai-i386/nfsroot.conf and add the option --arch i386. Also choose a different directory for your new nfsroot. Here are the two lines after editing.

Now call fai-make-nfsroot which creates the 32-bit nfsroot in /srv/fai/nfsroot-i386

Creating a partial mirror using fai-mirror(1) that is needed for a bootable CD or USB stick is also possible on a different architecture. You have to specify the architecture when calling fai-mirror.

That’s all!

Every task and hook can call the function task_error to send an error value to the installation. This error will appear in the log file and can be shown in the fai-monitor-gui(1). The error value is also checked against the variable $STOP_ON_ERROR which is set to 700 by default. If the error value is greater than $STOP_ON_ERROR the installation will stop immediately. In a hook, just add a call like this:

It is comfortable to call task_error with $? as second parameter, for e.g. if you want to set an error value of 123 use this

Then the task error is only set if the preceding command failed with some error. The same syntax can be used by the scripts in the class subdirectory ending in .sh.

A special case are scripts inside the class/ subdirectory that does not end in .sh. In those scripts you have to call task_error with 1 as third parameter. As an example you can use those two variants for setting the value to 123

or

The latter command always sets the task error to 123 regardless of the value of $?. If the error value is greater than $STOP_ON_ERROR, the installation will abort immediately.

The error values are grouped into four categories

The fai-monitor-gui has 4 different icons for these error categories.

Most tasks of the installation are defined as subroutines which are defined in /usr/lib/fai/subroutines (e.g. task_instsoft). Some are external shell scripts located in /usr/lib/fai/. They are called via a superior subroutine called task. This subroutine calls hooks if available and then calls the task (defined as task_<name>). A task and its hooks can be skipped on demand by using the command skiptask().

Now follows the description of all tasks, listed in the order they are executed.

After the customization scripts are executed, FAI will execute some tests if available. Using these test, you can check for errors of the installation. Test scripts are called via fai-do-scripts(1) and should append its messages to $LOGDIR/test.log. A Perl module including some useful subroutines can be found in Faitest.pm. A test can also define a new class for executing another tests during next boot via the variable $ADDCLASSES.

In FAI 5.0 we released a feature that allows clients to search for the faiserver in their respective subnetwork. This lifts the necessity of having to collect every client’s MAC address and configuring the DHCP daemon.

This is done by booting from a small FAI autodiscover bootmedium (CD, USB, etc.), which can be created via the command:

The image is roughly 25MB in size and scans the subnet for a FAI server. By default it shows a menu with all profiles available in the configuration space in the same manner as the menu flag does. From this menu, you can select the installation type you wish to perform.

For the clients to find the faiserver, the faiserver must run fai-monitor.

Changing the boot sequence is normally done in the BIOS setup. But you can’t change the BIOS from a running Linux system.

So, the boot sequence of the BIOS will remain unchanged and your computer should always boot first from its network card and the second boot device should be the local disk. Then you can change the boot device of the client by creating different PXELINUX configurations. This will define if an installation should be performed, or if the client should to boot from local disk. This is done using fai-chboot(8).

The nfsroot is mounted via NFS v3 by default. This is done by adding the option nfsvers=3 to the kernel boot options of the kernel booted by the install clients. fai-chboot(8) will do this for you.

If you want to use NFS v4 for the nfsroot, you have to remove this option the the kernel boot options. Additionally, overlayfs - the kernel module that is used for making the read-only mounted nfsroot writeable - can only work with NFS v4 if acl(5) support is disabled on the filessystem on the NFS server side (as of 2019). If /srv/fai/nfsroot on your FAI server is located on the root partition mounted to /, you can disable acl support for the whole file system on this partition using

Then mounting the nfsroot from a FAI server using NFS v4 will work.

The utility mkdebmirror
[You can find the script in /usr/share/doc/fai-doc/examples/utils/]
can be used for creating your own local Debian mirror. This script uses debmirror(1). A partial Debian mirror for i386 and amd64 architecture for Debian 11.0 (aka bullseye) without the source packages needs about 100GB of disk space. An amd64 only mirror needs about 80GB. Accessing the mirror via HTTP will be the default way in most cases. To see more output from the script call mkdebmirror -v. A root account is not necessary to create and maintain the Debian mirror.

To use HTTP access to the local Debian mirror, install a web server and create a symlink to the local directory where your mirror is located:

Create a file sources.list(5) in /etc/fai/apt which gives access to your Debian mirror. Also add the IP-address of the HTTP server to the variable $NFSROOT_ETC_HOSTS in nfsroot.conf if the install clients have no DNS resolving.

If flag_reboot is set, by adding "reboot" to $FAI_FLAGS, your client machine will reboot after the task faiend has finished. This is true for network as well as bootmedium installations.

FAI is creating several log files. During installation they are stored in /tmp/fai on the install client itself. At the end of the installation they will be copied to the install server (see [isavelog]). After the install client rebooted into his newly installed system, you can find the FAI logs in /var/log/fai. Log files are also created when doing the softupdate or dirinstall action.

On the faiserver, you can find the (remote) log files under the ~fai directory.

Sample log files from successfully installed computers are available on https://fai-project.org/logs. These a some log files which are created by FAI.

If the installation process finishes, the hook savelog.LAST.sh searches all log files for common errors and writes them to the file error.log. So, you should first look into this file for errors. Also the file status.log give you the exit code of the last command executed in a script. To be sure, you should look for more details in all log files.

fai-make-nfsroot now uses the lpxelinux.0 binary which already supports transfer of the kernel and initrd via http (additional to tftp). You only have to enable HTTP access to the tftp directory:

Add -U URL to the fai-chboot call. For example:

The following error message indicates that your install client doesn’t get an answer from a DHCP server. Check your cables or start the dhcpd(8) daemon with the debug flag enabled.

If you do not see the following message, the install kernel could not detect your network card, for example because of a missing driver:

Check the initrd in the nfsroot (lsinird) if the kernel driver of your network card is included there and check if you like to add the package firmware-linux-nonfree in /etc/fai/NFSROOT and rebuild the initrd by calling fai-make-nfsroot -k. You may also add a driver to /srv/fai/nfsroot/etc/dracut.conf in the line add_drivers+=.

This is the error message you will see, when your network card is working, but the install server does not export the nfsroot directory to the install clients, This is often caused by missing NFS permissions on the server side.

Now, you are inside the emergency shell of the initrd which was created by dracut(8). You will get a shell prompt, and can look at the log files. For more information about debugging the early boot process using dracut see dracut.cmdline(7)

Use the following command on the install server to see which directories are exported from the install server (named faiserver):