Making a frozen raspbian repository

pdo debian sw eng

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

A month later, we tried building an himblick image and it stopped playing video with vlc, only showing a black screen.

Although we're working with Rasbian Buster, and Debian Buster is stable, it looks like Raspbian Buster is not stable at all.

Time to learn how to freeze a partial raspbian mirror.

Looking at at the output of dpkg -l in a himblick image of a month ago, we see that vlc changed in Raspbian from version 3.0.8-0+deb10u1+rpt1 to version 3.0.8-0+deb10u1+rpt7. The former works perfectly, the latter only shows black when running with --fullscreen on the Raspberry Pi 4.

Here is the relevant changelog, for reference:

vlc (3.0.8-0+deb10u1+rpt7) buster; urgency=medium

  * Apply MMAL patch 16

vlc (3.0.8-0+deb10u1+rpt6) buster; urgency=medium

  * Apply MMAL patch 15

vlc (3.0.8-0+deb10u1+rpt5) buster; urgency=medium

  * Disable vdpau, libva, aom
  * Enable dav1d

vlc (3.0.8-0+deb10u1+rpt4) buster; urgency=medium

  * Apply MMAL patch 14

vlc (3.0.8-0+deb10u1+rpt3) buster; urgency=medium

  * Apply MMAL patch 13

vlc (3.0.8-0+deb10u1+rpt2) buster; urgency=medium

  * Apply MMAL patch 12

vlc (3.0.8-0+deb10u1+rpt1) buster; urgency=medium

  * Apply MMAL patch 10

We can use aptly to setup a Debian mirror that has the parts of raspbian that we need, plus the working vlc.

aptly by default really wants to clutter the home directory with ~/.aptly and ~/.aptly.conf. So first step, we create a .aptly.conf in the himblick repo, pointing to a rootDir next to it:

{
  "rootDir": ".aptly",
  "downloadConcurrency": 4,
  "downloadSpeedLimit": 0,
  "architectures": ["armhf"],
  "dependencyFollowSuggests": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowAllVariants": false,
  "dependencyFollowSource": false,
  "dependencyVerboseResolve": false,
  "gpgDisableSign": false,
  "gpgDisableVerify": false,
  "gpgProvider": "gpg",
  "downloadSourcePackages": false,
  "skipLegacyPool": true,
  "ppaDistributorID": "ubuntu",
  "ppaCodename": "",
  "skipContentsPublishing": false,
  "FileSystemPublishEndpoints": {},
  "S3PublishEndpoints": {},
  "SwiftPublishEndpoints": {}
}

Note that the Raspberry Pi 4 would be aarch64, but raspbian runs armhf binaries on it, to avoid maintaining packages for two different architectures.

Next to .aptly.conf, we put the /etc/apt/trusted.gpg from a raspbian image.

Then we setup a raspbian mirror of the bits that we need. APT sources of raspbian pull from two different repositories, so let's mirror them both:

aptly --config=.aptly.conf --keyring=`pwd`/trusted.gpg mirror create raspbian http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free rpi
aptly --config=.aptly.conf --keyring=`pwd`/trusted.gpg mirror create debian http://archive.raspberrypi.org/debian/ buster main

# Packages we need for himblick
NEEDED="Priority (required)|ansible|dracut|…"

# Packages that are would normally come with raspbian. This can be generated
# doing a dist-upgrade of a built himblick using Raspbian's repositories,
# following by dpkg -l to build the package list
IMAGE_PACKAGES="adduser|alsa-utils|apt|apt-listchanges|…"

FILTER="$NEEDED|$IMAGE_PACKAGES"

aptly --config=.aptly.conf mirror edit --dep-verbose-resolve --filter="$FILTER" --filter-with-deps raspbian
aptly --config=.aptly.conf --keyring=`pwd`/trusted.gpg mirror update raspbian

aptly --config=.aptly.conf mirror edit --dep-verbose-resolve --filter="$FILTER" --filter-with-deps debian
aptly --config=.aptly.conf --keyring=`pwd`/trusted.gpg mirror update debian

Building the FILTER expression took some iteration, trying provisioning and seeing where it stopped, as aptly's dependency resolver is more approximate than apt's.

Next, we create a distro with the bits of vlc that we need:

aptly --config=.aptly.conf repo create himblick
aptly --config=.aptly.conf repo add himblick fixed-vlc/*

Then a merged snapshot of the three:

NAME=$(date +%Y%m%d)
aptly --config=.aptly.conf snapshot create raspbian-$NAME from mirror raspbian
aptly --config=.aptly.conf snapshot create debian-$NAME from mirror debian
aptly --config=.aptly.conf snapshot create himblick-$NAME from repo himblick
aptly --config=.aptly.conf snapshot merge $NAME debian-$NAME raspbian-$NAME himblick-$NAME
aptly --config=.aptly.conf publish snapshot $NAME
echo "Published snapshot $NAME"

Finally, aptly --config=.aptly.conf serve brings up a web server with the mirror, that we can now use to build himblick.

We have adapted the provisioning script to use the local mirror, and to restore the original ones at the end, so build himblick images can still access the whole of Raspbian if needed.

Until we'll be able to run Debian stable on the Raspberry Pi 4, at least we can use snapshots to compensate for Raspbian's volatility.

Update: make sure you also disable apt update/upgrade timers. See Himblick one day later for details.