Over the last few days I've been at the GNOME Developer Experience hackfest in Brussels, looking into xdg-app and how best to use it in Debian and Debian derivatives.
xdg-app is basically a way to run "non-core" software on Linux distributions, analogous to apps on Android and iOS. It doesn't replace distributions like Debian or packaging systems, but it adds a layer above them. It's mostly aimed towards third-party apps obtained from somewhere that isn't your distribution vendor, aiming to address a few long-standing problems in that space:
There's no single ABI that can be called "a standard Linux system" in the same way there would be for Windows or OS X or Android or whatever, apart from LSB which is rather limited. Testing that a third-party app "works on Linux", or even "works on stable Linux releases from 2015", involves a combinatorial explosion of different distributions, desktop environments and local configurations. Steam uses the Steam Runtime, a chroot environment closely resembling Ubuntu 12.04 LTS; other vendors tend to test on a vaguely recent Ubuntu LTS and leave it at that.
There's no widely-supported mechanism for installing third-party applications as an ordinary user. gog.com used to distribute Ubuntu- and Debian-compatible
.debfiles, but installing a
.debinvolves running arbitrary vendor-supplied scripts as root, which should worry anyone who wants any sort of privilege-separation. (They have now switched to executable self-extracting installers, which involve running arbitrary vendor-supplied scripts as an ordinary user... better, but not perfect.)
Relatedly, the third-party application itself runs with the user's full privileges: a malicious or security-buggy third-party application can do more or less anything, unless you either switch to a different uid to run third-party apps, or use a carefully-written, app-specific AppArmor profile or equivalent.
To address the first point, each application uses a specified "runtime", which is available as /usr inside its sandbox. This can be used to run application bundles with multiple, potentially incompatible sets of dependencies within the same desktop environment. A runtime can be updated within its branch - for instance, if an application uses the "GNOME 3.18" runtime (consisting of a basic Linux system, the GNOME 3.18 libraries, other related libraries like Mesa, and their recursive dependencies like libjpeg), it can expect to see minor-version updates from GNOME 3.18.x (including any security updates that might be necessary for the bundled libraries), but not a jump to GNOME 3.20.
To address the second issue, the plan is for application bundles to be available as a single file, containing metadata (such as the runtime to use), the app itself, and any dependencies that are not available in the runtime (which the app vendor is responsible for updating if necessary). However, the primary way to distribute and upgrade runtimes and applications is to package them as OSTree repositories, which provide a git-like content-addressed filesystem, with efficient updates using binary deltas. The resulting files are hard-linked into place.
To address the last point, application bundles run partially isolated from the wider system, using containerization techniques such as namespaces to prevent direct access to system resources. Resources from outside the sandbox can be accessed via "portal" services, which are responsible for access control; for example, the Documents portal (the only one, so far) displays an "Open" dialog outside the sandbox, then allows the application to access only the selected file.
xdg-app for Debian
One thing I've been doing at this hackfest is improving the existing Debian/Ubuntu packaging for xdg-app (and its dependencies ostree and libgsystem), aiming to get it into a state where I can upload it to Debian experimental. Because xdg-app aims to be a general freedesktop project, I'm currently intending to make it part of the "Utopia" packaging team alongside projects like D-Bus and polkit, but I'm open to suggestions if people want to co-maintain it elsewhere.
In the process of updating xdg-app, I sent various patches to Alex, mostly fixing build and test issues, which are in the new 0.4.8 release.
I'd appreciate co-maintainers and further testing for this stuff, particularly ostree: ostree is primarily a whole-OS deployment technology, which isn't a use-case that I've tested, and in particular ostree-grub2 probably doesn't work yet.
Binaries (no trust path, so only use these if you have a test VM):
deb https://people.debian.org/~smcv/xdg-app xdg-app main
The "Hello, World" of xdg-apps
Another thing I set out to do here was to make a runtime and an app out of Debian packages. Most of the test applications in and around GNOME use the "freedesktop" or "GNOME" runtimes, which consist of a Yocto base system and lots of RPMs, are rebuilt from first principles on-demand, and are extensive and capable enough that they make it somewhat non-obvious what's in an app or a runtime.
So, here's a step-by-step route through xdg-app, first using typical GNOME instructions, but then using the simplest GUI app I could find - xvt, a small xterm clone. I'm using a Debian testing (stretch) x86_64 virtual machine for all this. xdg-app currently requires systemd-logind to put users and apps in cgroups, either with systemd as pid 1 (systemd-sysv) or systemd-shim and cgmanager; I used the default systemd-sysv. In principle it could work with plain cgmanager, but nobody has contributed that support yet.
Demonstrating an existing xdg-app
Debian's kernel is currently patched to be able to allow unprivileged users to create user namespaces, but make it runtime-configurable, because there have been various security issues in that feature, making it a security risk for a typical machine (and particularly a server). Hopefully unprivileged user namespaces will soon be secure enough that we can enable them by default, but for now, we have to do one of three things to let xdg-app use them:
enable unprivileged user namespaces via sysctl:
sudo sysctl kernel.unprivileged_userns_clone=1
make xdg-app root-privileged (it will keep
CAP_SYS_ADMINand drop the rest):
sudo dpkg-statoverride --update --add root root 04755 /usr/bin/xdg-app-helper
make xdg-app slightly less privileged:
sudo setcap cap_sys_admin+ep /usr/bin/xdg-app-helper
First, we'll need a runtime. The standard xdg-app tutorial would tell you to download the "GNOME Platform" version 3.18. To do that, you'd add a remote, which is a bit like a git remote, and a bit like an apt repository:
$ wget http://sdk.gnome.org/keys/gnome-sdk.gpg $ xdg-app remote-add --user --gpg-import=gnome-sdk.gpg gnome \ http://sdk.gnome.org/repo/
(I'm ignoring considerations like trust paths and security here, for brevity; in real life, you'd want to obtain the signing key via https and/or have a trust path to it, just like you would for a secure-apt signing key.)
You can list what's available in a remote:
$ xdg-app remote-ls --user gnome ... org.freedesktop.Platform ... org.freedesktop.Platform.Locale.cy ... org.freedesktop.Sdk ... org.gnome.Platform ...
The Platform runtimes are what we want here: they are collections of runtime libraries with which you can run an application. The Sdk runtimes add development tools, header files, etc. to be able to compile apps that will be compatible with the Platform.
For now, all we want is the GNOME 3.18 platform:
$ xdg-app install --user gnome org.gnome.Platform 3.18
Next, we can install an app that uses it, from Alex Larsson's nightly builds of a subset of GNOME. The server they're on doesn't have a great deal of bandwidth, so be nice :-)
$ wget http://220.127.116.11/keys/nightly.gpg $ xdg-app remote-add --user --gpg-import=nightly.gpg nightly \ http://18.104.22.168/repo/ $ xdg-app install --user nightly org.mypaint.MypaintDevel
We now have one app, and the runtime it needs:
$ xdg-app list org.mypaint.MypaintDevel $ xdg-app run org.mypaint.MypaintDevel [you see a GUI window]
Digression: what's in a runtime?
Behind the scenes, xdg-app runtimes and apps are both OSTree trees.
This means the
ostree tool, from the package of the same name, can be
used to inspect them.
$ sudo apt install ostree $ ostree refs --repo ~/.local/share/xdg-app/repo gnome:runtime/org.gnome.Platform/x86_64/3.18 nightly:app/org.mypaint.MypaintDevel/x86_64/master
A "ref" has roughly the same meaning as in git: something like a branch or
ostree can list the directory tree that it represents:
$ ostree ls --repo ~/.local/share/xdg-app/repo \ runtime/org.gnome.Platform/x86_64/3.18 d00755 0 0 0 / -00644 0 0 493 /metadata d00755 0 0 0 /files $ ostree ls --repo ~/.local/share/xdg-app/repo \ runtime/org.gnome.Platform/x86_64/3.18 /files d00755 0 0 0 /files l00777 0 0 0 /files/local -> ../var/usrlocal l00777 0 0 0 /files/sbin -> bin d00755 0 0 0 /files/bin d00755 0 0 0 /files/cache d00755 0 0 0 /files/etc d00755 0 0 0 /files/games d00755 0 0 0 /files/include d00755 0 0 0 /files/lib d00755 0 0 0 /files/lib64 d00755 0 0 0 /files/libexec d00755 0 0 0 /files/share d00755 0 0 0 /files/src
You can see that
/files in a runtime is basically a copy of
This is not coincidental: the runtime's
/files gets mounted at
inside the xdg-app container. There is also some metadata, which is in
the ini-like syntax seen in
$ ostree cat --repo ~/.local/share/xdg-app/repo \ runtime/org.gnome.Platform/x86_64/3.18 /metadata [Runtime] name=org.gnome.Platform/x86_64/3.16 runtime=org.gnome.Platform/x86_64/3.16 sdk=org.gnome.Sdk/x86_64/3.16 [Extension org.freedesktop.Platform.GL] version=1.2 directory=lib/GL [Extension org.freedesktop.Platform.Timezones] version=1.2 directory=share/zoneinfo [Extension org.gnome.Platform.Locale] directory=share/runtime/locale subdirectories=true [Environment] GI_TYPELIB_PATH=/app/lib/girepository-1.0 GST_PLUGIN_PATH=/app/lib/gstreamer-1.0 LD_LIBRARY_PATH=/app/lib:/usr/lib/GL
Looking at an app, the situation is fairly similar:
$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master d00755 0 0 0 / -00644 0 0 258 /metadata d00755 0 0 0 /export d00755 0 0 0 /files
/files maps to what will become
/app for the application, which
was compiled with
$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /files d00755 0 0 0 /files -00644 0 0 4599 /files/manifest.json d00755 0 0 0 /files/bin d00755 0 0 0 /files/lib d00755 0 0 0 /files/share
There is also a
/export directory, which is made visible to the host system
so that the contained app can appear as a "first-class citizen" in menus:
$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /export d00755 0 0 0 /export d00755 0 0 0 /export/share user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /export/share d00755 0 0 0 /export/share d00755 0 0 0 /export/share/app-info d00755 0 0 0 /export/share/applications d00755 0 0 0 /export/share/icons user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /export/share/applications d00755 0 0 0 /export/share/applications -00644 0 0 715 /export/share/applications/org.mypaint.MypaintDevel.desktop $ ostree cat --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master \ /export/share/applications/org.mypaint.MypaintDevel.desktop [Desktop Entry] Version=1.0 Name=(Nightly) MyPaint TryExec=mypaint Exec=mypaint %f Comment=Painting program for digital artists ... Comment[zh_HK]=藝術家的电脑绘画 GenericName=Raster Graphics Editor GenericName[fr]=Éditeur d'Image Matricielle MimeType=image/openraster;image/png;image/jpeg; Type=Application Icon=org.mypaint.MypaintDevel StartupNotify=true Categories=Graphics;GTK;2DGraphics;RasterGraphics; Terminal=false
Again, there's some metadata:
$ ostree cat --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /metadata [Application] name=org.mypaint.MypaintDevel runtime=org.gnome.Platform/x86_64/3.18 sdk=org.gnome.Sdk/x86_64/3.18 command=mypaint [Context] shared=ipc; sockets=x11;pulseaudio; filesystems=host; [Extension org.mypaint.MypaintDevel.Debug] directory=lib/debug
Building a runtime, probably the wrong way
The way in which the reference/demo runtimes and containers are generated is... involved. As far as I can tell, there's a base OS built using Yocto, and the actual GNOME bits come from RPMs. However, we don't need to go that far to get a working runtime.
In preparing this runtime I'm probably completely ignoring some best-practices and tools - but it works, so it's good enough.
First we'll need a repository:
$ sudo install -d -o$(id -nu) /srv/xdg-apps $ ostree init --repo /srv/xdg-apps
I'm just keeping this local for this demonstration, but you could rsync it
to a web server's exported directory or something - a lot like a git
repository, it's just a collection of files. We want everything in
because that's what xdg-app expects, hence
$ sudo mount -t tmpfs -o mode=0755 tmpfs /mnt $ sudo debootstrap --arch=amd64 --include=libx11-6,usrmerge \ --variant=minbase stretch /mnt http://192.168.122.1:3142/debian $ sudo mkdir /mnt/runtime $ sudo mv /mnt/usr /mnt/runtime/files
This obviously has a lot of stuff in it that we don't need - most obviously init, apt and dpkg - but it's Good Enough™.
We will also need some metadata. This is sufficient:
$ sudo sh -c 'cat > /mnt/runtime/metadata' [Runtime] name=org.debian.Debootstrap/x86_64/8.20160130 runtime=org.debian.Debootstrap/x86_64/8.20160130
That's a runtime. We can commit it to ostree, and generate xdg-app metadata:
$ ostree commit --repo /srv/xdg-apps \ --branch runtime/org.debian.Debootstrap/x86_64/8.20160130 \ /mnt/runtime $ fakeroot ostree commit --repo /srv/xdg-apps \ --branch runtime/org.debian.Debootstrap/x86_64/8.20160130 $ fakeroot xdg-app build-update-repo /srv/xdg-apps
(I'm not sure why ostree and xdg-app report "Operation not permitted" when we aren't root or fakeroot - feedback welcome.)
build-update-repo would presumably also be the right place to GPG-sign your repository, if you were doing that.
We can add that as another xdg-app remote:
$ xdg-app remote-add --user --no-gpg-verify local file:///srv/xdg-apps $ xdg-app remote-ls --user local org.debian.Debootstrap
Building an app, probably the wrong way
The right way to build an app is to build a "SDK" runtime - similar to that
platform runtime, but with development files and tools - and recompile the app
and any missing libraries with
./configure --prefix=/app && make && make install. I'm not going to do that,
because simplicity is nice, and I'm reasonably sure xvt doesn't actually
/usr into the binary:
$ install -d xvt-app/files/bin $ sudo apt-get --download-only install xvt $ dpkg-deb --fsys-tarfile /var/cache/apt/archives/xvt_2.1-20.1_amd64.deb \ | tar -xvf - ./usr/bin/xvt ./usr/ ./usr/bin/ ./usr/bin/xvt ... $ mv usr xvt-app/files
Again, we'll need metadata, and it's much simpler than the more production-quality GNOME nightly builds:
$ cat > xvt-app/metadata [Application] name=org.debian.packages.xvt runtime=org.debian.Debootstrap/x86_64/8.20160130 command=xvt [Context] sockets=x11; $ fakeroot ostree commit --repo /srv/xdg-apps \ --branch app/org.debian.packages.xvt/x86_64/2.1-20.1 xvt-app $ fakeroot xdg-app build-update-repo /srv/xdg-apps Updating appstream branch No appstream data for runtime/org.debian.Debootstrap/x86_64/8.20160130 No appstream data for app/org.debian.packages.xvt/x86_64/2.1-20.1 Updating summary $ xdg-app remote-ls --user local org.debian.Debootstrap org.debian.packages.xvt
The obligatory screenshot
OK, good, now we can install it:
$ xdg-app install --user local org.debian.Debootstrap 8.20160130 $ xdg-app install --user local org.debian.packages.xvt 2.1-20.1 $ xdg-app run --branch=2.1-20.1 org.debian.packages.xvt
and you can play around with the shell in the
xvt and see what you
can and can't do in the container.
I'm sure there were better ways to do most of this, but I think there's value in having such a simplistic demo to go alongside the various GNOMEish apps.
- Betacowork Coworking Brussels and ICAB Business & Technology Incubator hosted the hackfest, with Collabora providing snacks, and the GNOME Foundation supporting the hackfest in general;
- Collabora also sponsored my travel, accommodation and time;
- my colleague Philip Withnall organised the hackfest.
Thanks to all those!