Latest posts of series Systemd Truelite course

These are the notes of a training course on systemd I gave as part of my work with Truelite.

.socket units

Socket units tell systemd to listen on a given IPC, network socket, or file system FIFO, and use another unit to service requests to it.

For example, this creates a network service that listens on port 55555:

# /etc/systemd/system/ddate.socket
[Unit]
Description=ddate service on port 55555

[Socket]
ListenStream=55555
Accept=true

[Install]
WantedBy=sockets.target

# /etc/systemd/system/ddate@.service
[Unit]
Description=Run ddate as a network service

[Service]
Type=simple
ExecStart=/bin/sh -ec 'while true; do /usr/bin/ddate; sleep 1m; done'
StandardOutput=socket
StandardError=journal

Note that the .service file is called ddate@ instead of ddate: units whose name ends in '@' are template units which can be activated multiple times, by adding any string after the '@' in the unit name.

If I run nc localhost 55555 a couple of times, and then check the list of running units, I see ddate@… instantiated twice, adding the local and remote socket endpoints to the unit name:

$ systemctl list-units 'ddate@*'
  UNIT                                             LOAD   ACTIVE SUB     DESCRIPTION
  ddate@15-127.0.0.1:55555-127.0.0.1:36936.service loaded active running Run ddate as a network service (127.0.0.1:36936)
  ddate@16-127.0.0.1:55555-127.0.0.1:37002.service loaded active running Run ddate as a network service (127.0.0.1:37002)

This allows me to monitor each running service individually.

systemd also automatically creates a slice unit called system-ddate.slice grouping all services together:

$ systemctl status system-ddate.slice
● system-ddate.slice
   Loaded: loaded
   Active: active since Thu 2017-09-21 14:25:02 CEST; 9min ago
    Tasks: 4
   CGroup: /system.slice/system-ddate.slice
           ├─ddate@15-127.0.0.1:55555-127.0.0.1:36936.service
           │ ├─18214 /bin/sh -ec while true; do /usr/bin/ddate; sleep 1m; done
           │ └─18661 sleep 1m
           └─ddate@16-127.0.0.1:55555-127.0.0.1:37002.service
             ├─18228 /bin/sh -ec while true; do /usr/bin/ddate; sleep 1m; done
             └─18670 sleep 1m

This allows to also work with all running services for this template unit as a whole, sending a signal to all their processes and setting up resource control features for the service as a whole.

See:

• man systemd.socket
• systemd for developers: socket activation
• template units

These are the notes of a training course on systemd I gave as part of my work with Truelite.

.path units

This kind of unit can be used to monitor a file or directory for changes using inotify, and activate other units when an event happens.

For example, this activates a unit that manages a spool directory, which activates another unit whenever a .pdf file is added to /tmp/spool/:

[Unit]
Description=Monitor /tmp/spool/ for new .pdf files

[Path]
Unit=beeponce.service
PathExistsGlob=/tmp/spool/*.pdf
MakeDirectory=true

This instead activates another unit whenever /tmp/ready is changed, for example by someone running touch /tmp/ready:

[Unit]
Description=Monitor /tmp/ready

[Path]
Unit=beeponce.service
PathChanged=/tmp/ready

And beeponce.service:

[Unit]
Description=Beeps once

[Service]
Type=oneshot
ExecStart=/usr/bin/aplay /tmp/beep.wav

See man systemd.path

These are the notes of a training course on systemd I gave as part of my work with Truelite.

.device units

Several devices are automatically represented inside systemd by .device units, which can be used to activate services when a given device exists in the file system.

See systemctl --all --full -t device to see a list of all decives for which systemd has a unit in your system.

For example, this .service unit plays a sound as long as a specific USB key is plugged in my system:

[Unit]
Description=Beeps while a USB key is plugged
DefaultDependencies=false
StopWhenUnneeded=true

[Install]
WantedBy=dev-disk-by\x2dlabel-ERLUG.device

[Service]
Type=simple
ExecStart=/bin/sh -ec 'while true; do /usr/bin/aplay -q /tmp/beep.wav; sleep 2; done'

If you need to work with a device not seen by default by systemd, you can add a udev rule that makes it available, by adding the systemd tag to the device with TAG+="systemd".

It is also possible to give the device an extra alias using ENV{SYSTEMD_ALIAS}="/dev/my-alias-name".

To figure out all you can use for matching a device:

1. Run udevadm monitor --environment and plug the device
2. Look at the DEVNAME= values and pick one that addresses your device the way you prefer
3. udevadm info --attribute-walk --name=*the value of devname* will give you all you can use for matching in the udev rule.

See:

• man systemd.device
• Using systemd and udev rules

These are the notes of a training course on systemd I gave as part of my work with Truelite.

.timer units

Configure activation of other units (usually a .service unit) at some given time.

The functionality is similar to cron, with more features and a finer time granularity. For example, in Debian Stretch apt has a timer for running apt update which runs at a random time to distribute load on servers:

# /lib/systemd/system/apt-daily.timer
[Unit]
Description=Daily apt download activities
After=network-online.target
Wants=network-online.target

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target

The corresponding apt-daily.service file then only runs when the system is on mains power, to avoid unexpected batter drains for systems like laptops:

# /lib/systemd/system/apt-daily.service
[Unit]
Description=Daily apt download activities
Documentation=man:apt(8)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/lib/apt/apt.systemd.daily update

Note that if you want to schedule tasks with an accuracy under a minute (for example to play a beep every 10 seconds when running on battery), you need to also configure AccuracySec= for the timer to a delay shorter than the default 1 minute.

This is how to make your computer beep when on battery:

# /etc/systemd/system/beep-on-battery.timer
[Unit]
Description=Beeps every 10 seconds

[Install]
WantedBy=timers.target

[Timer]
AccuracySec=1s
OnUnitActiveSec=10s

# /etc/systemd/system/beep-on-battery.service
[Unit]
Description=Beeps when on battery
ConditionACPower=false

[Service]
Type=oneshot
ExecStart=/usr/bin/aplay /tmp/beep.wav

See:

• man systemd.timer
• man systemd.time
• a programmable alarm clock using systemd
• systemctl --all -t timer for examples

These are the notes of a training course on systemd I gave as part of my work with Truelite.

.mount and .swap units

Describe mount points similarly as what /etc/fstab, but with more functionality and integration with the dependency system.

It is possible to define, for example, a filesystem that should be mounted only when the network is available and a given service has successfully started, and a service that should be started only after a given filesystem has been successfully mounted.

At boot, systemd uses systemd-fstab-generator to generate mount and swap units from /etc/fstab, so that the usual fstab configuration file can still be used to configure the file system layout.

See man systemd.mount, and man systemd.swap.

See systemctl --all -t mount and systemctl --all -t swap for examples.

These are the notes of a training course on systemd I gave as part of my work with Truelite.

.service units

Describe how to start and stop system services, like daemons.

Services are described in a [Service] section. The Type= configuration describes how the service wants to be brought up:

• Type=simple: a program is started and runs forever, providing the service. systemd takes care of daemonizing it properly in the background, creating a pidfile, stopping it and so on. The service is considered active as soon as it has started.
• Type=forking: a traditional daemons that forks itself, creates a pidfile and so on. The server is considered active as soon as the parent process ends.
• Type=oneshot: a program is run once, and the service is considered started after the program ends. This can be used, for example, to implement a service to do one-off configuration, like checking a file system.
• Type=dbus: like simple but for D-Bus services: the service is considered active as soon as it appears on the D-Bus bus.
• Type=notify: like simple, but the service tells sytemd when it has finished initialization and is ready. Notification can happen via the sd_notify C function, or the systemd-notify command.
• Type=idle: like simple, but it is run after all other services has been started on a transaction. You can use this, for example, to start a shell on a terminal after the boot, so that the prompt doesn't get flooded with boot messages, or to play a happy trumped sound after the system has finished booting.

There are a lot more configuration options to fine-tune how the program should be managed, to limit its resource access or capabilities to harden the system security, to run setup/cleanup scripts before or after it started, and after it gets stopped, to control what signals to send to ask for reload or quit, and quite a lot more.

See: man systemd.service, man systemd.exec, man systemd.resource-control, and man systemd.kill.

See systemctl --all -t service for examples.

These are the notes of a training course on systemd I gave as part of my work with Truelite.

Writing .unit files

For reference, the global index with all .unit file directives is at man systemd.directives.

All unit files have a [Unit] section with documentation and dependencies. See man systemd.unit for documentation.

It is worth having a look at existing units to see what they are like. Use systemctl --all -t unittype for a list, and systemctl cat unitname to see its content wherever it is installed.

For example: systemctl cat graphical.target. Note that systemctl cat adds a line of comment at the top so one can see where the unit file is installed.

Most unit files also have an [Install] section (also documented in man systemd.unit) that controls what happens when enabling or disabling the unit.

See also:

• Systemd unit dependencies and order
• Understanding systemd units and unit files

.target units

.target units only contain [Unit] and [Install] sections, and can be used to give a name to a given set of dependencies.

For example, one could create a remote-maintenance.target unit, that when brought up activates, via dependencies, a set of services, mounts, network sockets, and so on.

See man systemd.target

See systemctl --all -t target for examples.

special units

man systemd.special has a list of units names that have a standard use associated to them.

For example, ctrl-alt-del.target is a unit that is started whenever Control+Alt+Del is pressed on the console. By default it is symlinked to reboot.target, and you can provide your own version in /etc/systemd/system/ to perform another action when Control+Alt+Del is pressed.

User units

systemd can also be used to manage services on a user session, starting them at login and stopping them at logout.

Add --user to the normal systemd commands to have them work with the current user's session instead of the general system.

See systemd/User in the Arch Wiki for a good description of what it can do.

These are the notes of a training course on systemd I gave as part of my work with Truelite.

Exploring the state of a system

• systemctl status [unitname [unitname..]] show status of one or more units, or of the whole system. Glob patterns also work: systemctl status "systemd-fsck@*"
• systemctl list-units or just systemctl show a table with all units, their status and their description
• systemctl list-sockets lists listening sockets managed by systemd and what they activate
• systemctl list-timers lists timer-activated units, with information about when they last ran and when they will run again
• systemctl is-active [pattern] checks if one or more units are in active state
• systemctl is-enabled [pattern] checks if one or more units are enabled
• systemctl is-failed [pattern] checks if one or more units are in failed state
• systemctl list-dependencies [unitname] lists the dependencies of a unit, or a system-wide dependency tree
• systemctl is-system-running check if the system is running correctly, or if some unit is in a failed state
• systemd-cgtop like top but processes are aggregated by unit
• systemd-analyze produces reports on boot time, per-unit boot time charts, dependency graphs, and more

Start and stop services

Similar to the System V service command, systemctl provides commands to start/stop/restart/reload units or services:

• start: starts a unit if it is not already started
• stop: stops a unit
• restart: starts or restarts a unit
• reload: tell a unit to reload its configuration (if it supports it)
• try-restart: restarts a unit only if it is already active, otherwise do nothing, to prevent accidentally starting a service
• reload-or-restart: tell a unit to reload its configuration if supported, otherwise restart it
• try-reload-or-restart: tell a unit to reload its configuration if supported, otherwise restart it. If the unit is not already active, do nothing to prevent accidentally starting a service.

Changing global system state

systemctl has halt, poweroff, reboot, suspend, hibernate, and hybrid-sleep commands to tell systemd to reboot, power off, suspend and so on. kexec and switch-root also work.

The rescue and emergency commands switch the system to rescue and emergency mode (see man systemd.special. systemctl default switches to the default mode, which also happens when exiting the rescue or emergency shell.

Run services at boot

systemd does not implement runlevels, and services start at boot based on their dependencies.

To start a service at boot, you add to its .service file a WantedBy= dependency on a well-known .target unit.

At boot, systemd brings up the whole chain of dependency started from a default unit, and that will eventually activate also your service.

See systemctl get-default for what unit is currently the default in your system. You can change it via the systemd.unit= kernel command line, so you can configure multiple entries in the boot loader that boot the system running different services. For example systemd.unit=rescue.target for a rescue mode, systemd.unit=multi-user.target for a non-graphical mode, or add your own .target file to implement new system modes.

See systemctl list-units -t target --all for a list of all currently available targets in your system.

• systemctl enable unitname enables the unit to start at boot, by creating symlinks to it in the .wants directory of the units listed in its WantedBy= configuration
• systemctl disable unitname removes the symlinks created by enable
• systemctl reenable unitname removes and readds the symlinks for when you changed WantedBy=

Notes: * systemctl start activates a unit right now, but does not automatically enable it at boot * systemctl enable enables a unit at boot, but does not automatically start it right now * a disabled unit can still be activated if another unit depends on it

To disable a unit so that it will never get started even if another unit depends on it, use systemctl mask unitname. Use systemctl unmask unitname to undo the masking.

Reloading / restarting systemd

systemctl daemon-reload tells systemd to reload its configuration.

systemctl daemon-reexec tells systemd to restart iself.

These are the notes of a training course on systemd I gave as part of my work with Truelite.

There is quite a lot of material, so I split them into a series of posts, running once a day for the next 9 days.

Units

Everything managed by systemd is called a unit (see man systemd.unit), and each unit is described by a configuration in ini-style format.

For example, this unit continuously plays an alarm sound when the system is in emergency or rescue mode:

[Unit]
Description=Beeps when in emergency or rescue mode
DefaultDependencies=false
StopWhenUnneeded=true

[Install]
WantedBy=emergency.target rescue.target

[Service]
Type=simple
ExecStart=/bin/sh -ec 'while true; do /usr/bin/aplay -q /tmp/beep.wav; sleep 2; done'

Units can be described by configuration files, which have different extensions based on what kind of thing they describe:

• .service: daemons
• .socket: communication sockets
• .device: hardware devices
• .mount: mount points
• .automount: automounting
• .swap: swap files or partitions
• .target: only dependencies, like Debian metapackages
• .path: inotify monitoring of paths
• .timer: cron-like activation
• .slice: group processes for common resource management
• .scope: group processes for common resource management

System unit files can be installed in:

• /lib/systemd/system/: for units provided by packaged software
• /run/systemd/system/: runtime-generated units
• /etc/systemd/system/: for units provided by systemd administrators

Unit files in /etc/ override unit files in /lib/. Note that while Debian uses /lib/, other distributions may use /usr/lib/ instead.

If there is a directory with the same name as the unit file plus a .d suffix, any file *.conf it contains is parsed after the unit, and can be used to add or override configuration options.

For example:

• /lib/systemd/system/beep.service.d/foo.conf can be used to tweak the contents of /lib/systemd/system/beep.service, so it is possible for a package to distribute a tweak to the configuration of another package.
• /etc/systemd/system/beep.service.d/foo.conf can be used to tweak the contents of /lib/systemd/system/beep.service, so it is possible a system administrator to extend a packaged unit without needing to replace it entirely.

Similarly, a unitname.wants/ or unitname.requires/ directory can be used to extend Wants= and Requires= dependencies on other units, by placing symlinks to other units in them.

See also:

• Introduction to systemd.
• systemd on the Debian Wiki
• systemd on the Arch Wiki