Assembling the custom runner

pdo debian eng sw systemd

This post is part of a series about trying to setup a gitlab runner based on systemd-nspawn. I published the polished result as nspawn-runner on GitHub.

The plan

Back to custom runners, here's my plan:

config can be a noop
prepare starts the nspawn machine
run runs scripts with machinectl shell
cleanup runs machinectl stop

The scripts

Here are the scripts based on Federico's work:

base.sh with definitions sourced by all scripts:

MACHINE="run-$CUSTOM_ENV_CI_JOB_ID"
ROOTFS="/var/lib/gitlab-runner-custom-chroots/buster"
OVERLAY="/var/lib/gitlab-runner-custom-chroots/$MACHINE"

config.sh doing nothing:

#!/bin/sh
exit 0

prepare.sh starting the machine:

#!/bin/bash

source $(dirname "$0")/base.sh
set -eo pipefail

# trap errors as a CI system failure
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

logger "gitlab CI: preparing $MACHINE"

mkdir -p $OVERLAY

systemd-run \
  -p 'KillMode=mixed' \
  -p 'Type=notify' \
  -p 'RestartForceExitStatus=133' \
  -p 'SuccessExitStatus=133' \
  -p 'Slice=machine.slice' \
  -p 'Delegate=yes' \
  -p 'TasksMax=16384' \
  -p 'WatchdogSec=3min' \
  systemd-nspawn --quiet -D $ROOTFS \
    --overlay="$ROOTFS:$OVERLAY:/"
    --machine="$MACHINE" --boot --notify-ready=yes

run.sh running the provided scripts in the machine:

#!/bin/bash
logger "gitlab CI: running $@"
source $(dirname "$0")/base.sh

set -eo pipefail
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

systemd-run --quiet --pipe --wait --machine="$MACHINE" /bin/bash < "$1"

cleanup.sh stopping the machine and removing the writable overlay directory:

#!/bin/bash
logger "gitlab CI: cleanup $@"
source $(dirname "$0")/base.sh

machinectl stop "$MACHINE"
rm -rf $OVERLAY

Trying out the plan

I tried a manual invocation of gitlab-runner, and it worked perfectly:

# mkdir /var/lib/gitlab-runner-custom-chroots/build/
# mkdir /var/lib/gitlab-runner-custom-chroots/cache/
# gitlab-runner exec custom \
    --builds-dir /var/lib/gitlab-runner-custom-chroots/build/ \
    --cache-dir /var/lib/gitlab-runner-custom-chroots/cache/ \
    --custom-config-exec /var/lib/gitlab-runner-custom-chroots/config.sh \
    --custom-prepare-exec /var/lib/gitlab-runner-custom-chroots/prepare.sh \
    --custom-run-exec /var/lib/gitlab-runner-custom-chroots/run.sh \
    --custom-cleanup-exec /var/lib/gitlab-runner-custom-chroots/cleanup.sh \
    tests
Runtime platform                                    arch=amd64 os=linux pid=18662 revision=775dd39d version=13.8.0
Running with gitlab-runner 13.8.0 (775dd39d)
Preparing the "custom" executor
Using Custom executor...
Running as unit: run-r1be98e274224456184cbdefc0690bc71.service
executor not supported                              job=1 project=0 referee=metrics
Preparing environment

Getting source from Git repository

Executing "step_script" stage of the job script
WARNING: Starting with version 14.0 the 'build_script' stage will be replaced with 'step_script': https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26426

Job succeeded

Deploy

The remaining step is to deploy all this in /etc/gitlab-runner/config.toml:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "nspawn runner"
  url = "http://gitlab.siweb.local/"
  token = "…"
  executor = "custom"
  builds_dir = "/var/lib/gitlab-runner-custom-chroots/build/"
  cache_dir = "/var/lib/gitlab-runner-custom-chroots/cache/"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.custom]
    config_exec = "/var/lib/gitlab-runner-custom-chroots/config.sh"
    config_exec_timeout = 200
    prepare_exec = "/var/lib/gitlab-runner-custom-chroots/prepare.sh"
    prepare_exec_timeout = 200
    run_exec = "/var/lib/gitlab-runner-custom-chroots/run.sh"
    cleanup_exec = "/var/lib/gitlab-runner-custom-chroots/cleanup.sh"
    cleanup_exec_timeout = 200
    graceful_kill_timeout = 200
    force_kill_timeout = 200

Next steps

My next step will be polishing all this in a way that makes deploying and maintaining a runner configuration easy.