Latest posts of series Qt Creator cross-platform development in Stretch: exploration

Updating cbqt for bullseye

Back in 2017 I did work to setup a cross-building toolchain for QT Creator, that takes advantage of Debian's packaging for all the dependency ecosystem.

It ended with cbqt which is a little script that sets up a chroot to hold cross-build-dependencies, to avoid conflicting with packages in the host system, and sets up a qmake alternative to make use of them.

Today I'm dusting off that work, to ensure it works on Debian bullseye.

Resetting QT Creator

To make things reproducible, I wanted to reset QT Creator's configuration.

Besides purging and reinstalling the package, one needs to manually remove:

~/.config/QtProject
~/.cache/QtProject/
/usr/share/qtcreator/QtProject which is where configuration is stored if you used sdktool to programmatically configure Qt Creator (see for example this post and see Debian bug #1012561.

Updating cbqt

Easy start, change the distribution for the chroot:

-DIST_CODENAME = "stretch"
+DIST_CODENAME = "bullseye"

Adding LIBDIR

Something else does not work:

Test$ qmake-armhf -makefile
Info: creating stash file …/Test/.qmake.stash
Test$ make
[...]
/usr/bin/arm-linux-gnueabihf-g++ -Wl,-O1 -Wl,-rpath-link,…/armhf/lib/arm-linux-gnueabihf -Wl,-rpath-link,…/armhf/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,…/armhf/usr/lib/ -o Test main.o mainwindow.o moc_mainwindow.o   …/armhf/usr/lib/arm-linux-gnueabihf/libQt5Widgets.so …/armhf/usr/lib/arm-linux-gnueabihf/libQt5Gui.so …/armhf/usr/lib/arm-linux-gnueabihf/libQt5Core.so -lGLESv2 -lpthread
/usr/lib/gcc-cross/arm-linux-gnueabihf/10/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lGLESv2
collect2: error: ld returned 1 exit status
make: *** [Makefile:146: Test] Error 1

I figured that now I also need to set QMAKE_LIBDIR and not just QMAKE_RPATHLINKDIR:

--- a/cbqt
+++ b/cbqt
@@ -241,18 +241,21 @@ include(../common/linux.conf)
 include(../common/gcc-base-unix.conf)
 include(../common/g++-unix.conf)

+QMAKE_LIBDIR += {chroot.abspath}/lib/arm-linux-gnueabihf
+QMAKE_LIBDIR += {chroot.abspath}/usr/lib/arm-linux-gnueabihf
+QMAKE_LIBDIR += {chroot.abspath}/usr/lib/
 QMAKE_RPATHLINKDIR += {chroot.abspath}/lib/arm-linux-gnueabihf
 QMAKE_RPATHLINKDIR += {chroot.abspath}/usr/lib/arm-linux-gnueabihf
 QMAKE_RPATHLINKDIR += {chroot.abspath}/usr/lib/

Now it links again:

Test$ qmake-armhf -makefile
Test$ make
/usr/bin/arm-linux-gnueabihf-g++ -Wl,-O1 -Wl,-rpath-link,…/armhf/lib/arm-linux-gnueabihf -Wl,-rpath-link,…/armhf/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,…/armhf/usr/lib/ -o Test main.o mainwindow.o moc_mainwindow.o   -L…/armhf/lib/arm-linux-gnueabihf -L…/armhf/usr/lib/arm-linux-gnueabihf -L…/armhf/usr/lib/ …/armhf/usr/lib/arm-linux-gnueabihf/libQt5Widgets.so …/armhf/usr/lib/arm-linux-gnueabihf/libQt5Gui.so …/armhf/usr/lib/arm-linux-gnueabihf/libQt5Core.so -lGLESv2 -lpthread

Making it work in Qt Creator

Time to try it in Qt Creator, and sadly it fails:

…/armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/features/toolchain.prf:76: Variable QMAKE_CXX.COMPILER_MACROS is not defined.

`QMAKE_CXX.COMPILER_MACROS` is not defined

I traced it to this bit in armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/features/toolchain.prf (nonrelevant bits deleted):

isEmpty($${target_prefix}.COMPILER_MACROS) {
    msvc {
        # …
    } else: gcc|ghs {
        vars = $$qtVariablesFromGCC($$QMAKE_CXX)
    }
    for (v, vars) {
        # …
        $${target_prefix}.COMPILER_MACROS += $$v
    }
    cache($${target_prefix}.COMPILER_MACROS, set stash)
} else {
    # …
}

It turns out that qmake is not able to realise that the compiler is gcc, so vars does not get set, nothing is set in COMPILER_MACROS, and qmake fails.

Reproducing it on the command line

When run manually, however, qmake-armhf worked, so it would be good to know how Qt Creator is actually running qmake. Since it frustratingly does not show what commands it runs, I'll have to strace it:

strace -e trace=execve --string-limit=123456 -o qtcreator.trace -f qtcreator

And there it is:

$ grep qmake- qtcreator.trace
1015841 execve("/usr/local/bin/qmake-armhf", ["/usr/local/bin/qmake-armhf", "-query"], 0x56096e923040 /* 54 vars */) = 0
1015865 execve("/usr/local/bin/qmake-armhf", ["/usr/local/bin/qmake-armhf", "…/Test/Test.pro", "-spec", "arm-linux-gnueabihf", "CONFIG+=debug", "CONFIG+=qml_debug"], 0x7f5cb4023e20 /* 55 vars */) = 0

I run the command manually and indeed I reproduce the problem:

$ /usr/local/bin/qmake-armhf Test.pro -spec arm-linux-gnueabihf CONFIG+=debug CONFIG+=qml_debug
…/armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/features/toolchain.prf:76: Variable QMAKE_CXX.COMPILER_MACROS is not defined.

I try removing options until I find the one that breaks it and... now it's always broken! Even manually running qmake-armhf, like I did earlier, stopped working:

$ rm .qmake.stash
$ qmake-armhf -makefile
…/armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/features/toolchain.prf:76: Variable QMAKE_CXX.COMPILER_MACROS is not defined.

Debugging `toolchain.prf`

I tried purging and reinstalling qtcreator, and recreating the chroot, but qmake-armhf is staying broken. I'll let that be, and try to debug toolchain.prf.

By grepping gcc in the mkspecs directory, I managed to figure out that:

The } else: gcc|ghs { test is matching the value(s) of QMAKE_COMPILER
QMAKE_COMPILER can have multiple values, separated by space
If in armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/arm-linux-gnueabihf/qmake.conf I set QMAKE_COMPILER = gcc arm-linux-gnueabihf-gcc, then things work again.

Sadly, I failed to find reference documentation for QMAKE_COMPILER's syntax and behaviour. I also failed to find why qmake-armhf worked earlier, and I am also failing to restore the system to a situation where it works again. Maybe I dreamt that it worked? I had some manual change laying around from some previous fiddling with things?

Anyway at least now I have the fix:

--- a/cbqt
+++ b/cbqt
@@ -248,7 +248,7 @@ QMAKE_RPATHLINKDIR += {chroot.abspath}/lib/arm-linux-gnueabihf
 QMAKE_RPATHLINKDIR += {chroot.abspath}/usr/lib/arm-linux-gnueabihf
 QMAKE_RPATHLINKDIR += {chroot.abspath}/usr/lib/

-QMAKE_COMPILER          = {chroot.arch_triplet}-gcc
+QMAKE_COMPILER          = gcc {chroot.arch_triplet}-gcc

 QMAKE_CC                = /usr/bin/{chroot.arch_triplet}-gcc

Fixing a compiler mismatch warning

In setting up the kit, Qt Creator also complained that the compiler from qmake did not match the one configured in the kit. That was easy to fix, by pointing at the host system cross-compiler in qmake.conf:

 QMAKE_COMPILER          = {chroot.arch_triplet}-gcc

-QMAKE_CC                = {chroot.arch_triplet}-gcc
+QMAKE_CC                = /usr/bin/{chroot.arch_triplet}-gcc

 QMAKE_LINK_C            = $$QMAKE_CC
 QMAKE_LINK_C_SHLIB      = $$QMAKE_CC

-QMAKE_CXX               = {chroot.arch_triplet}-g++
+QMAKE_CXX               = /usr/bin/{chroot.arch_triplet}-g++

 QMAKE_LINK              = $$QMAKE_CXX
 QMAKE_LINK_SHLIB        = $$QMAKE_CXX

Updated setup instructions

Create an armhf environment:

sudo cbqt ./armhf --create --verbose

Create a qmake wrapper that builds with this environment:

sudo ./cbqt ./armhf --qmake -o /usr/local/bin/qmake-armhf

Install the build-dependencies that you need:

# Note: :arch is added automatically to package names if no arch is explicitly specified
sudo ./cbqt ./armhf --install libqt5svg5-dev libmosquittopp-dev qtwebengine5-dev

Build with qmake

Use qmake-armhf instead of qmake and it works perfectly:

qmake-armhf -makefile
make

Set up Qt Creator

Configure a new Kit in Qt Creator:

Tools/Options, then Kits, then Add
Name: armhf (or anything you like)
In the Qt Versions tab, click Add then set the path of the new Qt to /usr/local/bin/qmake-armhf. Click Apply.
Back in the Kits, select the Qt version you just created in the Qt version field
In Compilers, select the ARM versions of GCC. If they do not appear, install crossbuild-essential-armhf, then in the Compilers tab click Re-detect and then Apply to make them available for selection
Dismiss the dialog with "OK": the new kit is ready

Now you can choose the default kit to build and run locally, and the armhf kit for remote cross-development.

I tried looking at sdktool to automate this step, and it requires a nontrivial amount of work to do it reliably, so these manual instructions will have to do.

Credits

This has been done as part of my work with Truelite.

2022-06-09

pdo debian sw eng

Qt Creator cross-platform development in Stretch: chroot automation

I wrote a tool to automate the creation and maintenance of Qt cross-build environments built from packages in Debian Stretch.

It allows to:

Do cross-architecture development with Qt Creator, including remote debugging on the target architecture
Compile using native compilers and cross-compilers, to avoid running the compilers inside an emulator making the build slower
Leverage all of Debian as development environment, using existing Debian packages for cross-build-dependencies

Getting started

# Creates an armhf environment under the current directory
$ sudo cbqt ./armhf --create --verbose
2017-11-30 14:09:23,715 INFO armhf: Creating /home/enrico/lavori/truelite/system/cross/armhf
…
2017-11-30 14:14:49,887 INFO armhf: Configuring cross-build environment

# Get information about an existing chroot.
# Note: the output is machine-parsable yaml
$ cbqt ./armhf --info
name: armhf
path: ./armhf
arch: armhf
arch_triplet: arm-linux-gnueabihf
exists: true
configured: true
issues: []

# Create a qmake wrapper for this environment
$ sudo ./cbqt ./armhf --qmake -o /usr/local/bin/qmake-armhf

# Install the build-depends you need
# Note: :arch is added automatically to package names if no arch is explicitly
#       specified
$ sudo ./cbqt ./armhf --install libqt5svg5-dev libmosquittopp-dev qtwebengine5-dev

Building packages

To build a package, use the qmake wrapper generated with cbqt --qmake instead of the normal qmake:

$ qmake-armhf -makefile
$ make
arm-linux-gnueabihf-g++ … -I../armhf/usr/include/arm-linux-gnueabihf/… -I../armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/arm-linux-gnueabihf -o browser.o browser.cpp
…
/home/enrico/…/armhf/usr/bin/moc …
…
arm-linux-gnueabihf-g++ … -Wl,-rpath-link,/home/enrico/…/armhf/lib/arm-linux-gnueabihf -Wl,-rpath-link,/home/enrico/…/armhf/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,/home/enrico/…/armhf/usr/lib/ -o program browser.o … -L/home/enrico/…/armhf/usr/lib/arm-linux-gnueabihf …

Building in Qt Creator

Configure a new Kit in Qt Creator:

Tools/Options, then Build & Run, then Kits, then Add
Name: armhf (or anything you like)
Qt version: choose the one autodetected from /usr/local/bin/qmake-armhf
In Compilers, add a GCC C compiler with path arm-linux-gnueabihf-gcc, and a GCC C++ compiler with path arm-linux-gnueabihf-g++
Choose the newly created compilers in the kit
Dismiss the dialog with "OK": the new kit is ready

Now you can choose the default kit to build and run locally, and the armhf kit for remote cross-development.

Where to get it

Here!

Credits

This has been done as part of my work with Truelite.

2017-11-30

pdo debian eng qt-cross

Qt Creator cross-platform development in Stretch: chroots

My next step in creating a cross-platform Qt development environment is trying to set it up on a chroot and make it usable from Qt Creator, so that both buil-dependencies and cross-build-dependencies can be available even when they are not coinstallable.

Building a chroot

I built a chroot to host the armhf build-dependencies:

$ sudo cdebootstrap stretch arm-linux-gnueabihf
$ sudo systemd-nspawn -D arm-linux-gnueabihf
# dpkg --add-architecture armhf
# apt update
# apt install qtbase5-dev qtbase5-dev-tools qtbase5-dev:armhf

The cross-compilers need to be outside the chroot. I tried to use cross-compilers installed inside the chroot while building outside the chroot, and they fail to link at runtime, since they expect their shared libraries to be in /usr.

I put this qt.conf in the chroot:

[Paths]
Prefix=/home/enrico/…/arm-linux-gnueabihf/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=arm-linux-gnueabihf

I added the custom mkspecs to the chroot:

$ cd arm-linux-gnueabihf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/
$ sudo cp -r linux-g++ arm-linux-gnueabihf
$ sudo vi arm-linux-gnueabihf/qmake.conf

This is the content of usr/lib/arm-linux-gnueabihf/qt5/mkspecs/arm-linux-gnueabihf/qmake.conf:

#
# qmake configuration for arm-linux-gnueabihf
#

MAKEFILE_GENERATOR      = UNIX
CONFIG                 += incremental
QMAKE_INCREMENTAL_STYLE = sublib

include(../common/linux.conf)
include(../common/gcc-base-unix.conf)
include(../common/g++-unix.conf)

QMAKE_RPATHLINKDIR += /home/enrico/…/arm-linux-gnueabihf/lib/arm-linux-gnueabihf
QMAKE_RPATHLINKDIR += /home/enrico/…/arm-linux-gnueabihf/usr/lib/arm-linux-gnueabihf
QMAKE_RPATHLINKDIR += /home/enrico/…/arm-linux-gnueabihf/usr/lib/

QMAKE_COMPILER          = arm-linux-gnueabihf-gcc

QMAKE_CC                = arm-linux-gnueabihf-gcc

QMAKE_LINK_C            = $$QMAKE_CC
QMAKE_LINK_C_SHLIB      = $$QMAKE_CC

QMAKE_CXX               = arm-linux-gnueabihf-g++

QMAKE_LINK              = $$QMAKE_CXX
QMAKE_LINK_SHLIB        = $$QMAKE_CXX

load(qt_config)

Note QMAKE_RPATHLINKDIR, which was not present in the previous post: since linking needs to happen against libraries in /home/enrico/…/arm-linux-gnueabihf/…, we need to tell the linker not to go and search in /.

The extra QMAKE_RPATHLINKDIR pointing to usr/lib is a workaround for libsrtp0 installing files outside multiarch directories (#765173):

# dpkg -L libsrtp0 | grep usr/lib
/usr/lib
/usr/lib/libsrtp.so.0.0
/usr/lib/libsrtp.so.0

(libsrtp0 is a dependency of libqt5webenginecore5)

I changed the wrapper /usr/local/bin/qmake-arm-linux-gnueabihf to point to the chroot:

#!/usr/bin/python3

import sys, os

QT_CONFIG = "/home/enrico/…/arm-linux-gnueabihf/qt.conf"

argv0 = os.path.join(os.path.dirname(sys.argv[0]), "qmake")

if len(sys.argv) == 1:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:])
else:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:] + ["-qtconf", QT_CONFIG])

And it all works. Even the test application that requires QtWebEngine builds and links fine.

Credits

This has been done as part of my work with Truelite.

2017-11-29

pdo debian eng qt-cross

Qt Creator cross-platform development in Stretch: consolidation

Time to consolidate the exploration done yesterday.

qmake -qtconf

Looking at the work done by the Qt/KDE team I found that there is no need to rebuild qmake, as it can be configured by using the -qtconf option.

The work recently done on debhelper provides a starting point that I can build on.

This qtconf makes qmake use the right paths, including running rcc, moc and uic from /usr/bin/, removing the need for the dirty hack:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations

It still needs overriding the compiler settings, to make it use a compiler different than g++:

qmake -makefile -qtconf /tmp/qmake.conf QMAKE_CC=arm-linux-gnueabihf-gcc QMAKE_CXX=arm-linux-gnueabihf-g++ QMAKE_LINK=arm-linux-gnueabihf-g++

But then -m64 creeps in and breaks the cross-build (this has been fixed since qtbase 5.9.1+dfsg-12. but my target is stretch):

$ make
arm-linux-gnueabihf-g++ -c -m64 -pipe -std=c++11 -O2 -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I. -isystem /usr/include/arm-linux-gnueabihf/qt5 -isystem /usr/include/arm-linux-gnueabihf/qt5/QtSvg -isystem /usr/include/arm-linux-gnueabihf/qt5/QtWidgets -isystem /usr/include/arm-linux-gnueabihf/qt5/QtGui -isystem /usr/include/arm-linux-gnueabihf/qt5/QtCore -I. -I/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/linux-g++-64 -o main.o main.cpp
arm-linux-gnueabihf-g++: error: unrecognized command line option ‘-m64’
Makefile:401: recipe for target 'main.o' failed
make: *** [main.o] Error 1

Where does it come from? I found that I could run qmake -query:

$ qmake -query -qtconf /tmp/qmake.conf
QT_SYSROOT:
QT_INSTALL_PREFIX:/usr
QT_INSTALL_ARCHDATA:/usr/lib/arm-linux-gnueabihf/qt5
QT_INSTALL_DATA:/usr/share/qt5
QT_INSTALL_DOCS:/usr/share/qt5/doc
QT_INSTALL_HEADERS:/usr/include/arm-linux-gnueabihf/qt5
QT_INSTALL_LIBS:/usr/lib/arm-linux-gnueabihf
QT_INSTALL_LIBEXECS:/usr/lib/arm-linux-gnueabihf/qt5/libexec
QT_INSTALL_BINS:/usr/lib/qt5/bin
QT_INSTALL_TESTS:/usr/tests
QT_INSTALL_PLUGINS:/usr/lib/arm-linux-gnueabihf/qt5/plugins
QT_INSTALL_IMPORTS:/usr/lib/arm-linux-gnueabihf/qt5/imports
QT_INSTALL_QML:/usr/lib/arm-linux-gnueabihf/qt5/qml
QT_INSTALL_TRANSLATIONS:/usr/share/qt5/translations
QT_INSTALL_CONFIGURATION:/etc/xdg
QT_INSTALL_EXAMPLES:/usr/lib/arm-linux-gnueabihf/qt5/examples
QT_INSTALL_DEMOS:/usr/lib/arm-linux-gnueabihf/qt5/examples
QT_HOST_PREFIX:/usr
QT_HOST_DATA:/usr/lib/arm-linux-gnueabihf/qt5
QT_HOST_BINS:/usr/bin
QT_HOST_LIBS:/usr/lib/arm-linux-gnueabihf
QMAKE_SPEC:linux-g++-64
QMAKE_XSPEC:linux-g++-64
QMAKE_VERSION:3.0
QT_VERSION:5.7.1

I suppose that QMAKE_XSPEC should be linux-g++ instead of linux-g++-64

Poking through Qt's sources I found that I could also tweak TargetSpec, to change it to linux-g++ from its default of linux-g++-64:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=linux-g++

Now, I still need to override the compilers, but that's all I need to do to get a build:

$ qmake -makefile -qtconf /tmp/qmake.conf QMAKE_CC=arm-linux-gnueabihf-gcc QMAKE_CXX=arm-linux-gnueabihf-g++ QMAKE_LINK=arm-linux-gnueabihf-g++
$ make
…
$ file usr/bin/program
usr/bin/program: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=541e472a29a2b309af2fea40d9aa3439e6616bdd, not stripped

Can I get rid of needing to override the compilers? Let's try poking with TargetSpec:

# cd /usr/lib/arm-linux-gnueabihf/qt5/mkspecs/
# cp -r linux-g++ arm-linux-gnueabihf
# cd arm-linux-gnueabihf
# …tweak…
# cat qmake.conf
#
# qmake configuration for arm-linux-gnueabihf
#

MAKEFILE_GENERATOR      = UNIX
CONFIG                 += incremental
QMAKE_INCREMENTAL_STYLE = sublib

include(../common/linux.conf)
include(../common/gcc-base-unix.conf)
include(../common/g++-unix.conf)

QMAKE_COMPILER          = arm-linux-gnueabihf-gcc

QMAKE_CC                = arm-linux-gnueabihf-gcc

QMAKE_LINK_C            = $$QMAKE_CC
QMAKE_LINK_C_SHLIB      = $$QMAKE_CC

QMAKE_CXX               = arm-linux-gnueabihf-g++

QMAKE_LINK              = $$QMAKE_CXX
QMAKE_LINK_SHLIB        = $$QMAKE_CXX

load(qt_config)

Now, using arm-linux-gnueabihf as TargetSpec:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=arm-linux-gnueabihf

It finally works:

$ qmake -makefile -qtconf /tmp/qmake.conf
$ make
$ file usr/bin/program
usr/bin/program: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=541e472a29a2b309af2fea40d9aa3439e6616bdd, not stripped

Skipping prl file '/usr/lib/arm-linux-gnueabihf/libQt5PlatformSupport.prl', because it cannot be opened (No such file or directory).

Turn this into a kit for Qt Creator

I could not find a way to configure -qtconf in a kit in Qt Creator. Since Qt Creator detects versions of Qt by pointing at their qmake, I resorted to making a custom qmake wrapper:

#!/usr/bin/python3

# /usr/local/bin/qmake-arm-linux-gnueabihf

import sys, os

QT_CONFIG = "/home/enrico/qt-arm-linux-gnueabihf.conf"

argv0 = os.path.join(os.path.dirname(sys.argv[0]), "qmake")

if len(sys.argv) == 1:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:])
else:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:] + ["-qtconf", QT_CONFIG])

Just calling qmake -qtconf /home/enrico/qt-arm-linux-gnueabihf.conf "$@" is not enough: for some invocations of qmake, providing -qtconf gives an error:

$ qmake -qtconf /home/enrico/qt-arm-linux-gnueabihf.conf
qmake: could not find a Qt installation of 'conf'

If needed I can look at qmake's sources to see what is going on, but for now the wrapper above seems to be enough to cover all the needs of Qt Creator, and if the wrapper is in the path, Qt Creator manages to autodetect it on startup.

These are the details of a working arm-linux-gnueabihf kit for Qt Creator:

Device type: Generic Linux Device
Device: my hi-fi
Sysroot: blank
C compiler: /usr/bin/arm-linux-gnueabihf-gcc
C++ compiler: /usr/bin/arm-linux-gnueabihf-g++
Qt version: /usr/local/bin/qmake-arm-linux-gnueabihf

And with this, software cross-builds locally and can be deployed and tested remotely.

Summary of the situation so far

This is what is needed to do cross-build now:

Depend on crossbuild-essential-armhf
Install :armhf versions of Qt and all build-dependencies
Deploy a custom mkspec
Deploy a custom qt.conf
Deploy qmake wrapper
Configure a kit

No dirty hacks, everything can be shipped by a single .deb package.

Further things to investigate:

Find out if there is a way to package and install a kit for Qt Creator
Find an easy way to use a chroot as the cross-build environment, to be able to use cross-build-dependencies that cannot be coinstalled together with their native version

Credits

This has been done as part of my work with Truelite.

2017-11-29

pdo debian eng qt-cross

Qt Creator cross-platform development in Stretch: exploration

I have been asked to create a system that makes it easy to do cross-architecture Qt development with Debian Stretch.

The goal is this:

Do development with Qt Creator, including remote debugging on the target architecture
Compile using native code, to avoid running the compilers inside an emulator making the build slower
Leverage all of Debian as development environment, using existing Debian packages for build-dependencies
Set everything up just by installing a single .deb package, which ideally gives Qt Creator a new kit that just works

I did some exploration on this some time ago, and the Qt/KDE and cross-build people did some work on this, too.

Now I'm trying to build on all that. I need to target Stretch so I cannot build on the recent improvements in Qt packaging, but I can at least use the experience that went into those changes.

I have two sample Qt application to try and cross-build, one that depends on a non-Qt library (libmosquittopp-dev), and one that depends on a nasty Qt library (qtwebengine5-dev).

This post has the notes from the first day of trying out different strategies.

It begins

I imported the two projects in Qt creator, installed their amd64 dependencies and make sure they build for the current system, with the default kit, no cross-building yet.

That works, good.

Now let's see about the rest:

dpkg --add-architecture armhf
apt install crossbuild-essential-armhf

A cross-build kit for Qt Creator

I created a new armhf kit for Qt Creator:

Device type: Generic Linux Device
Device: my hi-fi
Sysroot: blank
C compiler: /usr/bin/arm-linux-gnueabihf-gcc
C++ compiler: /usr/bin/arm-linux-gnueabihf-g++
Qt version: /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake

I ran qmake from Qt Creator and go this:

10:59:49: Starting: "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" …/project.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
sh: 1: /usr/lib/arm-linux-gnueabihf/qt5/bin/rcc: not found
10:59:50: The process "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" exited normally.

rcc is provided by qtbase5-dev-tools, which cannot be coinstalled alongside other architectures' version of itself:

# apt install qtbase5-dev-tools:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  qtbase5-dev-tools
The following NEW packages will be installed:
  qtbase5-dev-tools:armhf
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 651 kB of archives.
After this operation, 985 kB disk space will be freed.
Do you want to continue? [Y/n] q

I'll try with a very dirty hack:

# cd /usr/lib/arm-linux-gnueabihf/qt5/bin
# ln -s `which rcc` .
# ln -s `which uic` .
# ln -s `which moc` .
# ls -la
total 1944
drwxr-xr-x 2 root root    4096 Nov 29 11:04 .
drwxr-xr-x 7 root root    4096 Nov 28 17:05 ..
lrwxrwxrwx 1 root root      12 Nov 29 11:04 moc -> /usr/bin/moc
-rwxr-xr-x 1 root root 1982208 Jan 11  2017 qmake
lrwxrwxrwx 1 root root      12 Nov 29 11:04 rcc -> /usr/bin/rcc
lrwxrwxrwx 1 root root      12 Nov 29 11:04 uic -> /usr/bin/uic

This is ugly:

It places files in /usr outside /usr/local that are not maintained by the package manager
It places amd64 executables in /usr/lib/arm-linux-gnueabihf which should contain armhf code

Let's see what happens in Qt Creator:

11:14:35: Starting: "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" …/project.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
11:14:35: The process "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" exited normally.
11:14:35: Starting: "/usr/bin/make" qmake_all
make: Nothing to be done for 'qmake_all'.
11:14:36: The process "/usr/bin/make" exited normally.

and build:

11:15:29: Starting: "/usr/bin/make"
g++ -c -pipe -std=c++11 -g -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_QML_DEBUG -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I../project -I. -isystem /usr/include/arm-linux-gnueabihf/qt5 -isystem /usr/include/arm-linux-gnueabihf/qt5/QtSvg -isystem /usr/include/arm-linux-gnueabihf/qt5/QtWidgets -isystem /usr/include/arm-linux-gnueabihf/qt5/QtGui -isystem /usr/include/arm-linux-gnueabihf/qt5/QtCore -I. -I/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/linux-g++ -o main.o ../project/main.cpp

Unfortunately, it is using g++ even though I configured the kit to use /usr/bin/arm-linux-gnueabihf-* instead.

Trying an explicit override in the .pro:

QMAKE_CXX = /usr/bin/arm-linux-gnueabihf-g++
QMAKE_LINK = /usr/bin/arm-linux-gnueabihf-g++

And the project builds and runs fine on the Raspberry Pi, using a kit and two simple overrides in the .pro, all build dependencies as packaged by Debian, and a toolchain that entirely runs on native code.

Entirely?

$ file /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake
/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=1f5b063926570702f5568b1e5cec6c70d214fc73, stripped
$ dpkg-architecture -q DEB_HOST_ARCH
amd64
$ /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake -v
QMake version 3.0
Using Qt version 5.7.1 in /usr/lib/arm-linux-gnueabihf

Indeed qmake is armhf code that runs happily on my amd64 laptop because I accidentally have qemu-user-static installed.

This strategy produces results, although it depends on a dirty hack. To summarise it:

Depend on crossbuild-essential-armhf, qemu-user-static, qemu-system-arm
Install :armhf versions of Qt and all build-dependencies
Symlink native moc, rcc, uic into the armhf bin directory (argh!)
Configure a kit as before
Override QMAKE_CXX and QMAKE_LINK in the .pro

Other things to investigate, besides removing the need for that dirty hack:

Find out if there is a way to package and install a kit for Qt Creator
Find out why qmake is ignoring the compiler settings from the kit and needs overrides in the .pro. Ideally the .pro should be unmodified, so that it can be used for all builds

A native qmake

Would it be possible to build a native version of qmake tweaked to point to all the right bits out of the box?

$ apt source qtbase-opensource-src-5.7.1+dfsg
$ qtbase-opensource-src-5.7.1+dfsg$
$ DEB_HOST_MULTIARCH=arm-linux-gnueabihf DEB_HOST_ARCH_BITS=32 debian/rules override_dh_auto_configure
MAKEFLAGS="-j1" ./configure \
            -confirm-license \
            -prefix "/usr" \
            -bindir "/usr/lib/arm-linux-gnueabihf/qt5/bin" \
            -libdir "/usr/lib/arm-linux-gnueabihf" \
            -docdir "/usr/share/qt5/doc" \
            -headerdir "/usr/include/arm-linux-gnueabihf/qt5" \
            -datadir "/usr/share/qt5" \
            -archdatadir "/usr/lib/arm-linux-gnueabihf/qt5" \
            -plugindir "/usr/lib/arm-linux-gnueabihf/qt5/plugins" \
            -importdir "/usr/lib/arm-linux-gnueabihf/qt5/imports" \
            -translationdir "/usr/share/qt5/translations" \
            -hostdatadir "/usr/lib/arm-linux-gnueabihf/qt5" \
            -sysconfdir "/etc/xdg" \
            -examplesdir "/usr/lib/arm-linux-gnueabihf/qt5/examples" \
            -opensource \
            -plugin-sql-mysql \
            -plugin-sql-odbc \
            -plugin-sql-psql \
            -plugin-sql-sqlite \
            -no-sql-sqlite2 \
            -plugin-sql-tds \
            -system-sqlite \
            -platform linux-g++ \
            -system-harfbuzz \
            -system-zlib \
            -system-libpng \
            -system-libjpeg \
            -system-doubleconversion \
            -openssl \
            -no-rpath \
            -verbose \
            -optimized-qmake \
            -dbus-linked \
            -no-strip \
            -no-separate-debug-info \
            -qpa xcb \
            -xcb \
            -glib \
            -icu \
            -accessibility \
            -compile-examples \
            -no-directfb \
            -gstreamer 1.0 \
            -plugin-sql-ibase -opengl desktop \
…
$ file bin/qmake
bin/qmake: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ba868730cf34c54d7ddf6df0ab4b6ce5c7d6f2a0, not stripped
$ bin/qmake -v
QMake version 3.0
Using Qt version 5.7.1 in /usr/lib/arm-linux-gnueabihf

./configure in Qt builds qmake, and there is no need to run make afterwards. Good.

DEB_HOST_ARCH_BITS=32 is a hack I added to avoid debian/rules using -platform linux-g++-64 instead of -platform linux-g++.

Let's try to use that in Qt Creator:

sudo cp bin/qmake /usr/local/bin/qmake-arm-linux-gnueabihf

Qt Creator autodetects the new qmake and offers it as one of the available versions of Qt. Nice.

The need for the symlink hack is still there:

11:46:51: Starting: "/usr/local/bin/qmake-arm-linux-gnueabihf" ../project.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=qml_debug
sh: 1: /usr/lib/arm-linux-gnueabihf/qt5/bin/rcc: not found

So is the need for the QMAKE_CXX and QMAKE_LINK overrides in the .pro.

Still, this way I could remove qemu-user-static from my system and the project still builds on my laptop and runs on my Raspberry Pi.

The qemu dependency is not needed anymore, the rest of the problems still stand. I wonder, since I'm rebuilding qmake, if there's a way to tell it to use the compilers I want, and the tools I want, removing the need for the dirty hack and the overrides in the .pro files.

qtwebengine5-dev

How about the other project that depends on qtwebengine5-dev?

# apt install qtwebengine5-dev qtwebengine5-dev:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
qtwebengine5-dev is already the newest version (5.7.1+dfsg-6.1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 qtwebengine5-dev:armhf : Depends: libqt5webengine5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
                          Depends: libqt5webenginecore5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
                          Depends: libqt5webenginewidgets5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

It turns out that something in the big chain of dependencies of qtwebengine5-dev makes the amd64 and armhf versions not coinstallable.

It seems that I have to abandon the idea of installing armhf build-dependencies in the main development system, and I need to start considering to leave the host system untouched to do the native builds, and use a chroot for the cross-compilers and the cross-build-dependencies.

In the next post, I'll see how that goes.

Credits

This has been done as part of my work with Truelite.

2017-11-28

pdo debian eng qt-cross

Latest posts of series Qt Creator cross-platform development in Stretch: exploration

Updating cbqt for bullseye

Resetting QT Creator

Updating cbqt

Adding LIBDIR

Making it work in Qt Creator

QMAKE_CXX.COMPILER_MACROS is not defined

Reproducing it on the command line

Debugging toolchain.prf

Fixing a compiler mismatch warning

Updated setup instructions

Build with qmake

Set up Qt Creator

Credits

Qt Creator cross-platform development in Stretch: chroot automation

Getting started

Building packages

Building in Qt Creator

Where to get it

Credits

Qt Creator cross-platform development in Stretch: chroots

Building a chroot

Credits

Qt Creator cross-platform development in Stretch: consolidation

qmake -qtconf

Turn this into a kit for Qt Creator

Summary of the situation so far

Credits

Qt Creator cross-platform development in Stretch: exploration

It begins

A cross-build kit for Qt Creator

A native qmake

qtwebengine5-dev

Credits

`QMAKE_CXX.COMPILER_MACROS` is not defined

Debugging `toolchain.prf`