ext4 and 32bit arm on 64bit amd64

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

To provision Himblick systems we rely on using qemu-user-static to chroot into the ARM system.

It turns out that there's an exoteric bug that triggers when accessing an ext4 filesystem with directory hashes with a 32bit architecture which makes syscalls to a 64bit kernel.

We noticed it while investigating why shared-mime-info was creating an empty mime cache database.

We can work around this quite simply by disabling the dir_index feature of ext4 before mounting the filesystem:

tune2fs -O ^dir_index /dev/sdb2

After we're done with provisioning, we can turn on the feature again, and run e2fsck to rebuild the hashes:

tune2fs -O dir_index /dev/sdb2
e2fsck -f /dev/sdb2

This also took a while to track down, starting from okular claiming that PDF was unsupported, and evince aborting on simple PDF files with warnings about not being able to load PNG icons, which hinted at needing to run update-mime-database.

Running it in the Pi worked, while running it in the chroot produced an empty database but no error messages. It turns out that catching readdir(3) errors is not straightforward.

After getting the source of update-mime-database and debugging it, I tracked the issue down to this simple test case:

#include <sys/types.h>
#include <dirent.h>
#include <stdio.h>
#include <assert.h>
#include <errno.h>

int main(int argc, const char* argv[])
{
   DIR * dir = opendir("/usr/share/mime/packages");
   assert(dir);

   while (1)
   {
      errno = 0;
      struct dirent *e = readdir(dir);
      if (!e)
      {
         if (errno)
            perror("readdir");
         else
            fprintf(stderr, "last!\n");
         break;
      }
      fprintf(stderr, "read %s\n", e->d_name);
   }
}

# gcc -o test test.c && ./test
readdir: Value too large for defined data type

This finally was enough to hit the right spot on the internet and go from the actual cause of the issue to a useful workaround: this is the initial implementation of the workaround.

I guess the workaround would not really be needed if we could work with a proper aarch64 chroot: Raspbian quite legitimately tries to support all Raspberry Pi architectures, so they cannot do that.

I hope Debian will gain Raspberry Pi 4 support soon, so that this and other part of Himblick provisioning could become much more straightforward.