Creating a Raspberry PI SD from tar files

pdo debian sw eng

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Provisioning a SD card starting from the official raspbian-lite is getting quite slow, since there are a lot of packages to install.

It would be significantly faster if we could take a SD card, partition it from scratch, then untar the boot and rootfs partition contents into them.

Here's how.

Partitioning a SD card from scratch

We can do almost everything with pyparted.

See this LinuxVoice article for a detailed introduction to pyparted, and the C parted documentation for some low-level reference.

Here is the pyparted recipe for the SD card, plus a media directory at the end:

def partition_reset(self, dev: Dict[str, Any]):
    """
    Repartition the SD card from scratch
    """
    try:
        import parted
    except ModuleNotFoundError:
        raise Fail("please install python3-parted")

    device = parted.getDevice(dev["path"])

    device.clobber()
    disk = parted.freshDisk(device, "msdos")

    # Add 256M fat boot
    optimal = device.optimumAlignment
    constraint = parted.Constraint(
        startAlign=optimal,
        endAlign=optimal,
        startRange=parted.Geometry(
            device=device,
            start=parted.sizeToSectors(4, "MiB", device.sectorSize),
            end=parted.sizeToSectors(16, "MiB", device.sectorSize)),
        endRange=parted.Geometry(
            device=device,
            start=parted.sizeToSectors(256, "MiB", device.sectorSize),
            end=parted.sizeToSectors(512, "MiB", device.sectorSize)),
        minSize=parted.sizeToSectors(256, "MiB", device.sectorSize),
        maxSize=parted.sizeToSectors(260, "MiB", device.sectorSize))
    geometry = parted.Geometry(
        device,
        start=0,
        length=parted.sizeToSectors(256, "MiB", device.sectorSize),
    )
    geometry = constraint.solveNearest(geometry)
    boot = parted.Partition(
            disk=disk, type=parted.PARTITION_NORMAL, fs=parted.FileSystem(type='fat32', geometry=geometry),
            geometry=geometry)
    boot.setFlag(parted.PARTITION_LBA)
    disk.addPartition(partition=boot, constraint=constraint)

    # Add 4G ext4 rootfs
    constraint = parted.Constraint(
        startAlign=optimal,
        endAlign=optimal,
        startRange=parted.Geometry(
            device=device,
            start=geometry.end,
            end=geometry.end + parted.sizeToSectors(16, "MiB", device.sectorSize)),
        endRange=parted.Geometry(
            device=device,
            start=geometry.end + parted.sizeToSectors(4, "GiB", device.sectorSize),
            end=geometry.end + parted.sizeToSectors(4.2, "GiB", device.sectorSize)),
        minSize=parted.sizeToSectors(4, "GiB", device.sectorSize),
        maxSize=parted.sizeToSectors(4.2, "GiB", device.sectorSize))
    geometry = parted.Geometry(
        device,
        start=geometry.start,
        length=parted.sizeToSectors(4, "GiB", device.sectorSize),
    )
    geometry = constraint.solveNearest(geometry)
    rootfs = parted.Partition(
            disk=disk, type=parted.PARTITION_NORMAL, fs=parted.FileSystem(type='ext4', geometry=geometry),
            geometry=geometry)
    disk.addPartition(partition=rootfs, constraint=constraint)

    # Add media partition on the rest of the disk
    constraint = parted.Constraint(
        startAlign=optimal,
        endAlign=optimal,
        startRange=parted.Geometry(
            device=device,
            start=geometry.end,
            end=geometry.end + parted.sizeToSectors(16, "MiB", device.sectorSize)),
        endRange=parted.Geometry(
            device=device,
            start=geometry.end + parted.sizeToSectors(16, "MiB", device.sectorSize),
            end=disk.maxPartitionLength),
        minSize=parted.sizeToSectors(4, "GiB", device.sectorSize),
        maxSize=disk.maxPartitionLength)
    geometry = constraint.solveMax()
    # Create media partition
    media = parted.Partition(
            disk=disk, type=parted.PARTITION_NORMAL,
            geometry=geometry)
    disk.addPartition(partition=media, constraint=constraint)

    disk.commit()

Setting MBR disk identifier

So far so good, but /boot/cmdline.txt has root=PARTUUID=6c586e13-02, and we need to change the MBR disk identifier to match:

# Fix disk identifier to match what is in cmdline.txt
with open(dev["path"], "r+b") as fd:
    buf = bytearray(512)
    fd.readinto(buf)
    buf[0x1B8] = 0x13
    buf[0x1B9] = 0x6e
    buf[0x1BA] = 0x58
    buf[0x1BB] = 0x6c
    fd.seek(0)
    fd.write(buf)

Formatting the partitions

Formatting is reasonably straightforward, and although we've tried to match the way raspbian formats partitions, it may be that not all of these options are needed:

# Format boot partition with 'boot' label
run(["mkfs.fat", "-F", "32", "-n", "boot", disk.partitions[0].path])

# Format rootfs partition with 'rootfs' label
run(["mkfs.ext4", "-F", "-L", "rootfs", "-O", "^64bit,^huge_file,^metadata_csum", disk.partitions[1].path])

# Format exfatfs partition with 'media' label
run(["mkexfatfs", "-n", "media", disk.partitions[2].path])

Now the SD card is ready for a simple untarring of the boot and rootfs partition contents.

Useful commands

These commands were useful in finding out differences between how the original Raspbian image partitions were formatted, and how we were formatting them:

sudo minfo -i /dev/sdb1 ::
sudo tune2fs -l /dev/sdb2