This is Simon McVittie's software development blog. Main site: pseudorandom.co.uk
I'm back from the GTK hackfest in Toronto, Canada and mostly recovered from jetlag, so it's time to write up my notes on what we discussed there.
Despite the hackfest's title, I was mainly there to talk about non-GUI parts of the stack, and technologies that fit more closely in what could be seen as the freedesktop.org platform than they do in GNOME. In particular, I'm interested in Flatpak as a way to deploy self-contained "apps" in a freedesktop-based, sandboxed runtime environment layered over the Universal Operating System and its many derivatives, with both binary and source compatibility with other GNU/Linux distributions.
I'm mainly only writing about discussions I was directly involved in: lots of what sounded like good discussion about the actual graphics toolkit went over my head completely :-) More notes, mostly from Matthias Clasen, are available on the GNOME wiki.
In no particular order:
Thinking with portals
We spent some time discussing Flatpak's portals, mostly on Tuesday. These are the components that expose a subset of desktop functionality as D-Bus services that can be used by contained applications: they are part of the security boundary between a contained app and the rest of the desktop session. Android's intents are a similar concept seen elsewhere. While the portals are primarily designed for Flatpak, there's no real reason why they couldn't be used by other app-containment solutions such as Canonical's Snap.
One major topic of discussion was their overall design and layout. Most portals will consist of a UX-independent part in Flatpak itself, together with a UX-specific implementation of any user interaction the portal needs. For example, the portal for file selection has a D-Bus service in Flatpak, which interacts with some UX-specific service that will pop up a standard UX-specific "Open" dialog — for GNOME and probably other GTK environments, that dialog is in (a branch of) GTK.
A design principle that was reiterated in this discussion is that the UX-independent part should do as much as possible, with the UX-specific part only carrying out the user interactions that need to comply with a particular UX design (in the GTK case, GNOME's design). This minimizes the amount of work that needs to be redone for other desktop or embedded environments, while still ensuring that the other environments can have their chosen UX design. In particular, it's important that, as much as possible, the security- and performance-sensitive work (such as data transport and authentication) is shared between all environments.
The aim is for portals to get the user's permission to carry out actions, while keeping it as implicit as possible, avoiding an "are you sure?" step where feasible. For example, if an application asks to open a file, the user's permission is implicitly given by them selecting the file in the file-chooser dialog and pressing OK: if they do not want this application to open a file at all, they can deny permission by cancelling. Similarly, if an application asks to stream webcam data, the UX we expect is for GNOME's Cheese app (or a similar non-GNOME app) to appear, open the webcam to provide a preview window so they can see what they are about to send, but not actually start sending the stream to the requesting app until the user has pressed a "Start" button. When defining the API "contracts" to be provided by applications in that situation, we will need to be clear about whether the provider is expected to obtain confirmation like this: in most cases I would anticipate that it is.
One security trade-off here is that we have to have a small amount of trust in the providing app. For example, continuing the example of Cheese as a webcam provider, Cheese could (and perhaps should) be a contained app itself, whether via something like Flatpak, an LSM like AppArmor or both. If Cheese is compromised somehow, then whenever it is running, it would be technically possible for it to open the webcam, stream video and send it to a hostile third-party application. We concluded that this is an acceptable trade-off: each application needs to be trusted with the privileges that it needs to do its job, and we should not put up barriers that are easy to circumvent or otherwise serve no purpose.
The main (only?) portal so far is the file chooser, in which the contained
application asks the wider system to show an "Open..." dialog, and if the user
selects a file, it is returned to the contained application through a FUSE
filesystem, the document portal. The reference implementation of the UX for
this is in GTK, and is basically a
GtkFileChooserDialog. The intention is
that other environments such as KDE will substitute their own equivalent.
Other planned portals include:
- image capture (scanner/camera)
- opening a specified URI
- this needs design feedback on how it should work for non-http(s)
- sharing content, for example on social networks (like Android's Sharing menu)
- proxying joystick/gamepad input (perhaps via Wayland or FUSE, or perhaps by modifying libraries like SDL with a new input source)
- network proxies (
GProxyResolver) and availability (
- contacts/address book, probably vCard-based
- notifications, probably based on freedesktop.org Notifications
- video streaming (perhaps using Pinot, analogous to PulseAudio but for video)
GNOME on Wayland currently has a problem with environment variables:
there are some traditional ways to set environment variables for X11
sessions or login shells using shell script fragments (
/etc/profile.d), but these do not apply to
Wayland, or to noninteractive login environments like
systemd --user. We are also keen to avoid requiring a Turing-complete shell
language during session startup, because it's difficult to reason about
and potentially rather inefficient.
Some uses of environment variables can be dismissed as unnecessary or even
unwanted, similar to the statement in Debian Policy §9.9: "A program must not
depend on environment variables to get reasonable defaults." However,
there are two common situations where environment variables can be necessary
for proper OS integration: search-paths like
$PYTHONPATH (particularly necessary for things like Flatpak), and
optionally-loaded modules like
where a package influences the configuration of another package.
There is a stopgap solution in GNOME's gdm display manager,
/usr/share/gdm/env.d, but this is gdm-specific and
insufficiently expressive to provide the functionality needed by
XDG_DATA_DIRS to its specified default value if unset,
then add a couple of extra paths".
pam_env comes closer — PAM is run at every transition from "no user logged
in" to "user can execute arbitrary code as themselves" — but it doesn't
.d fragments, which are required if we want distribution packages
to be able to extend search paths.
pam_env also turns off per-user
configuration by default, citing security concerns.
I'll write more about this when I have a concrete proposal for how to solve it.
I think the best solution is probably a PAM module similar to
.d directories, either by modifying
pam_env directly or
out-of-tree, combined with clarifying what the security concerns for
per-user configuration are and how they can be avoided.
Relocatable binary packages
On Windows and OS X, various GLib APIs automatically discover where the
application binary is located and use search paths relative to that;
for example, if
C:\myprefix\bin\app.exe is running, GLib might put
C:\myprefix\share into the result of
so that the application can ask to load
app/data.xml from the data
directories and get
C:\myprefix\share\app\data.xml. We would like
to be able to do the same on Linux, for example so that the apps in a
Flatpak or Snap package can be constructed from RPM or dpkg packages without
needing to be recompiled for a different
--prefix, and so that other
third-party software packages like the games on Steam and gog.com can
easily locate their own resources.
Relatedly, there are currently no well-defined semantics for what happens
.desktop file or a D-Bus
.service file has
The meaning of
Exec=foo is well-defined (it searches
$PATH) and the
Exec=/opt/whatever/bin/foo is obvious. When this came up in
D-Bus previously, my assertion was that the meaning should be the same as
.desktop files, whatever that is.
We agreed to propose that the meaning of a non-absolute path in a
.service file should be interpreted relative to the directory
.service file was found: for example, if
/opt/whatever/bin/foo would be the right thing to execute.
While preparing a mail to the freedesktop and D-Bus mailing lists proposing
this, I found that I had
proposed the same thing
almost 2 years ago...
this time I hope I can actually make it happen!
Flatpak and OSTree bug fixing
On the way to the hackfest, and while the discussion moved to topics that I didn't have useful input on, I spent some time fixing up the Debian packaging for Flatpak and its dependencies. In particular, I did my first upload as a co-maintainer of bubblewrap, uploaded ostree to unstable (with the known limitation that the grub, dracut and systemd integration is missing for now since I haven't been able to test it yet), got most of the way through packaging Flatpak 0.6.5 (which I'll upload soon), cherry-picked the right patches to make ostree compile on Debian 8 in an effort to make backports trivial, and spent some time disentangling a flatpak test failure which was breaking the Debian package's installed-tests. I'm still looking into ostree test failures on little-endian MIPS, which I was able to reproduce on a Debian porterbox just before the end of the hackfest.
OSTree + Debian
I also had some useful conversations with developers from Endless, who recently opened up a version of their OSTree build scripts for public access. Hopefully that information brings me a bit closer to being able to publish a walkthrough for how to deploy a simple Debian derivative using OSTree (help with that is very welcome of course!).
GTK life-cycle and versioning
The life-cycle of GTK releases has already been mentioned here and elsewhere, and there are some interesting responses in the comments on my earlier blog post.
It's important to note that what we discussed at the hackfest is only a proposal: a hackfest discussion between a subset of the GTK maintainers and a small number of other GTK users (I am in the latter category) doesn't, and shouldn't, set policy for all of GTK or for all of GNOME. I believe the intention is that the GTK maintainers will discuss the proposals further at GUADEC, and make a decision after that.
As I said before, I hope that being more realistic about API and ABI guarantees can avoid GTK going too far towards either of the possible extremes: either becoming unable to advance because it's too constrained by compatibility, or breaking applications because it isn't constrained enough. The current situation, where it is meant to be compatible within the GTK 3 branch but in practice applications still sometimes break, doesn't seem ideal for anyone, and I hope we can do better in future.
Thanks to everyone involved, particularly:
- Matthias Clasen, who organised the hackfest and took a lot of notes
- Allison Lortie, who provided on-site cat-herding and led us to some excellent restaurants
- Red Hat Inc., who provided the venue (a conference room in their Toronto office), snacks, a lot of coffee, and several participants
- my employers Collabora Ltd., who sponsored my travel and accomodation
Allison Lortie has provoked a lot of comment with her blog post on a new proposal for how GTK is versioned. Here's some more context from the discussion at the GTK hackfest that prompted that proposal: there's actually quite a close analogy in how new Debian versions are developed.
The problem we're trying to address here is the two sides of a trade-off:
- Without new development, a library (or an OS) can't go anywhere new
- New development sometimes breaks existing applications
Historically, GTK has aimed to keep compatible within a major version, where major versions are rather far apart (GTK 1 in 1998, GTK 2 in 2002, GTK 3 in 2011, GTK 4 somewhere in the future). Meanwhile, fixing bugs, improving performance and introducing new features sometimes results in major changes behind the scenes. In an ideal world, these behind-the-scenes changes would never break applications; however, the world isn't ideal. (The Debian analogy here is that as much as we aspire to having the upgrade from one stable release to the next not break anything at all, I don't think we've ever achieved that in practice - we still ask users to read the release notes, even though ideally that wouldn't be necessary.)
In particular, the perceived cost of doing a proper ABI break (a fully parallel-installable GTK 4) means there's a strong temptation to make changes that don't actually remove or change C symbols, but are clearly an ABI break, in the sense that an application that previously worked and was considered correct no longer works. A prominent recent example is the theming changes in GTK 3.20: the ABI in terms of functions available didn't change, but what happens when you call those functions changed in an incompatible way. This makes GTK hard to rely on for applications outside the GNOME release cycle, which is a problem that needs to be fixed (without stopping development from continuing).
The goal of the plan we discussed today is to decouple the latest branch of development, which moves fast and sometimes breaks API, from the API-stable branches, which only get bug fixes. This model should look quite familiar to Debian contributors, because it's a lot like the way we release Debian and Ubuntu.
In Debian, at any given time we have a development branch (testing/unstable) - currently "stretch", the future Debian 9. We also have some stable branches, of which the most recent are Debian 8 "jessie" and Debian 7 "wheezy". Different users of Debian have different trade-offs that lead them to choose one or the other of these. Users who value stability and want to avoid unexpected changes, even at a cost in terms of features and fixes for non-critical bugs, choose to use a stable release, preferably the most recent; they only need to change what they run on top of Debian for OS API changes (for instance webapps, local scripts, or the way they interact with the GUI) approximately every 2 years, or perhaps less often than that with the Debian-LTS project supporting non-current stable releases. Meanwhile, users who value the latest versions and are willing to work with a "moving target" as a result choose to use testing/unstable.
The GTK analogy here is really quite close. In the new versioning model, library users who value stability over new things would prefer to use a stable-branch, ideally the latest; library users who want the latest features, the latest bug-fixes and the latest new bugs would use the branch that's the current focus of development. In practice we expect that the latter would be mostly GNOME projects. There's been some discussion at the hackfest about how often we'd have a new stable-branch: the fastest rate that's been considered is a stable-branch every 2 years, similar to Ubuntu LTS and Debian, but there's no consensus yet on whether they will be that frequent in practice.
How many stable versions of GTK would end up shipped in Debian depends on how rapidly projects move from "old-stable" to "new-stable" upstream, how much those projects' Debian maintainers are willing to patch them to move between branches, and how many versions the release team will tolerate. Once we reach a steady state, I'd hope that we might have 1 or 2 stable-branched versions active at a time, packaged as separate parallel-installable source packages (a lot like how we handle Qt). GTK 2 might well stay around as an additional active version just from historical inertia. The stable versions are intended to be fully parallel-installable, just like the situation with GTK 1.2, GTK 2 and GTK 3 or with the major versions of Qt.
For the "current development" version, I'd anticipate that we'd probably only ship one source package, and do ABI transitions for one version active at a time, a lot like how we deal with libgnome-desktop and the evolution-data-server family of libraries. Those versions would have parallel-installable runtime libraries but non-parallel-installable development files, again similar to libgnome-desktop.
At the risk of stretching the Debian/Ubuntu analogy too far, the intermediate "current development" GTK releases that would accompany a GNOME release are like Ubuntu's non-LTS suites: they're more up to date than the fully stable releases (Ubuntu LTS, which has a release schedule similar to Debian stable), but less stable and not supported for as long.
Hopefully this plan can meet both of its goals: minimize breakage for applications, while not holding back the development of new APIs.
Quite a lot has happened in xdg-app since last time I blogged about it. Most noticeably, it isn't called xdg-app any more, having been renamed to Flatpak. It is now available in Debian experimental under that name, and the xdg-app package that was briefly there has been removed. I'm currently in the process of updating Flatpak to the latest version 0.6.4.
The privileged part has also spun off into a separate project, Bubblewrap, which recently had its first release (0.1.0). This is intended as a common component with which unprivileged users can start a container in a way that won't let them escalate privileges, like a more flexible version of linux-user-chroot.
Bubblewrap has also been made available in Debian, maintained by Laszlo Boszormenyi (also maintainer of linux-user-chroot). Yesterday I sent a patch to update Laszlo's packaging for 0.1.0. I'm hoping to become a co-maintainer to upload that myself, since I suspect Flatpak and Bubblewrap might need to track each other quite closely. For the moment, Flatpak still uses its own internal copy of Bubblewrap, but I consider that to be a bug and I'd like to be able to fix it soon.
At some point I also want to experiment with using Bubblewrap to sandbox some of the game engines that are packaged in Debian: networked games are a large attack surface, and typically consist of the sort of speed-optimized C or C++ code that is an ideal home for security vulnerabilities. I've already made some progress on jailing game engines with AppArmor, but making sensitive files completely invisible to the game engine seems even better than preventing them from being opened.
Next weekend I'm going to be heading to Toronto for the GTK Hackfest, primarily to to talk to GNOME and Flatpak developers about their plans for sandboxing, portals and Flatpak. Hopefully we can make some good progress there: the more I know about the state of software security, the less happy I am with random applications all being equally privileged. Newer display technologies like Wayland and Mir represent an opportunity to plug one of the largest holes in typical application containerization, which is a major step in bringing sandboxes like Flatpak and Snap from proof-of-concept to a practical improvement in security.
Other next steps for Flatpak in Debian:
- To get into the next stable release (Debian 9), Flatpak needs to move
testing. I've taken the first step towards that by uploading
libgsystemto unstable. Before Flatpak can follow, OSTree also needs to move.
- Now that it's in Debian, please report bugs in the usual Debian way or send patches to fix bugs: Flatpak, OSTree, libgsystem.
- In particular, there are some OSTree bugs tagged
help. I'd appreciate contributions to the OSTree packaging from people who are interested in using it to deploy
dpkg-based operating systems - I'm primarily looking at it from the Flatpak perspective, so the boot/OS side of it isn't so well tested. Red Hat have
rpm-ostree, and I believe Endless do something analogous to build OS images with
dpkg, but I haven't had a chance to look into that in detail yet.
- Co-maintainers for Flatpak, OSTree, libgsystem would also be very welcome.
Over the last few days I've been at the GNOME Developer Experience hackfest in Brussels, looking into xdg-app and how best to use it in Debian and Debian derivatives.
xdg-app is basically a way to run "non-core" software on Linux distributions, analogous to apps on Android and iOS. It doesn't replace distributions like Debian or packaging systems, but it adds a layer above them. It's mostly aimed towards third-party apps obtained from somewhere that isn't your distribution vendor, aiming to address a few long-standing problems in that space:
There's no single ABI that can be called "a standard Linux system" in the same way there would be for Windows or OS X or Android or whatever, apart from LSB which is rather limited. Testing that a third-party app "works on Linux", or even "works on stable Linux releases from 2015", involves a combinatorial explosion of different distributions, desktop environments and local configurations. Steam uses the Steam Runtime, a chroot environment closely resembling Ubuntu 12.04 LTS; other vendors tend to test on a vaguely recent Ubuntu LTS and leave it at that.
There's no widely-supported mechanism for installing third-party applications as an ordinary user. gog.com used to distribute Ubuntu- and Debian-compatible
.debfiles, but installing a
.debinvolves running arbitrary vendor-supplied scripts as root, which should worry anyone who wants any sort of privilege-separation. (They have now switched to executable self-extracting installers, which involve running arbitrary vendor-supplied scripts as an ordinary user... better, but not perfect.)
Relatedly, the third-party application itself runs with the user's full privileges: a malicious or security-buggy third-party application can do more or less anything, unless you either switch to a different uid to run third-party apps, or use a carefully-written, app-specific AppArmor profile or equivalent.
To address the first point, each application uses a specified "runtime", which is available as /usr inside its sandbox. This can be used to run application bundles with multiple, potentially incompatible sets of dependencies within the same desktop environment. A runtime can be updated within its branch - for instance, if an application uses the "GNOME 3.18" runtime (consisting of a basic Linux system, the GNOME 3.18 libraries, other related libraries like Mesa, and their recursive dependencies like libjpeg), it can expect to see minor-version updates from GNOME 3.18.x (including any security updates that might be necessary for the bundled libraries), but not a jump to GNOME 3.20.
To address the second issue, the plan is for application bundles to be available as a single file, containing metadata (such as the runtime to use), the app itself, and any dependencies that are not available in the runtime (which the app vendor is responsible for updating if necessary). However, the primary way to distribute and upgrade runtimes and applications is to package them as OSTree repositories, which provide a git-like content-addressed filesystem, with efficient updates using binary deltas. The resulting files are hard-linked into place.
To address the last point, application bundles run partially isolated from the wider system, using containerization techniques such as namespaces to prevent direct access to system resources. Resources from outside the sandbox can be accessed via "portal" services, which are responsible for access control; for example, the Documents portal (the only one, so far) displays an "Open" dialog outside the sandbox, then allows the application to access only the selected file.
xdg-app for Debian
One thing I've been doing at this hackfest is improving the existing Debian/Ubuntu packaging for xdg-app (and its dependencies ostree and libgsystem), aiming to get it into a state where I can upload it to Debian experimental. Because xdg-app aims to be a general freedesktop project, I'm currently intending to make it part of the "Utopia" packaging team alongside projects like D-Bus and polkit, but I'm open to suggestions if people want to co-maintain it elsewhere.
In the process of updating xdg-app, I sent various patches to Alex, mostly fixing build and test issues, which are in the new 0.4.8 release.
I'd appreciate co-maintainers and further testing for this stuff, particularly ostree: ostree is primarily a whole-OS deployment technology, which isn't a use-case that I've tested, and in particular ostree-grub2 probably doesn't work yet.
Binaries (no trust path, so only use these if you have a test VM):
deb https://people.debian.org/~smcv/xdg-app xdg-app main
The "Hello, World" of xdg-apps
Another thing I set out to do here was to make a runtime and an app out of Debian packages. Most of the test applications in and around GNOME use the "freedesktop" or "GNOME" runtimes, which consist of a Yocto base system and lots of RPMs, are rebuilt from first principles on-demand, and are extensive and capable enough that they make it somewhat non-obvious what's in an app or a runtime.
So, here's a step-by-step route through xdg-app, first using typical GNOME instructions, but then using the simplest GUI app I could find - xvt, a small xterm clone. I'm using a Debian testing (stretch) x86_64 virtual machine for all this. xdg-app currently requires systemd-logind to put users and apps in cgroups, either with systemd as pid 1 (systemd-sysv) or systemd-shim and cgmanager; I used the default systemd-sysv. In principle it could work with plain cgmanager, but nobody has contributed that support yet.
Demonstrating an existing xdg-app
Debian's kernel is currently patched to be able to allow unprivileged users to create user namespaces, but make it runtime-configurable, because there have been various security issues in that feature, making it a security risk for a typical machine (and particularly a server). Hopefully unprivileged user namespaces will soon be secure enough that we can enable them by default, but for now, we have to do one of three things to let xdg-app use them:
enable unprivileged user namespaces via sysctl:
sudo sysctl kernel.unprivileged_userns_clone=1
make xdg-app root-privileged (it will keep
CAP_SYS_ADMINand drop the rest):
sudo dpkg-statoverride --update --add root root 04755 /usr/bin/xdg-app-helper
make xdg-app slightly less privileged:
sudo setcap cap_sys_admin+ep /usr/bin/xdg-app-helper
First, we'll need a runtime. The standard xdg-app tutorial would tell you to download the "GNOME Platform" version 3.18. To do that, you'd add a remote, which is a bit like a git remote, and a bit like an apt repository:
$ wget http://sdk.gnome.org/keys/gnome-sdk.gpg $ xdg-app remote-add --user --gpg-import=gnome-sdk.gpg gnome \ http://sdk.gnome.org/repo/
(I'm ignoring considerations like trust paths and security here, for brevity; in real life, you'd want to obtain the signing key via https and/or have a trust path to it, just like you would for a secure-apt signing key.)
You can list what's available in a remote:
$ xdg-app remote-ls --user gnome ... org.freedesktop.Platform ... org.freedesktop.Platform.Locale.cy ... org.freedesktop.Sdk ... org.gnome.Platform ...
The Platform runtimes are what we want here: they are collections of runtime libraries with which you can run an application. The Sdk runtimes add development tools, header files, etc. to be able to compile apps that will be compatible with the Platform.
For now, all we want is the GNOME 3.18 platform:
$ xdg-app install --user gnome org.gnome.Platform 3.18
Next, we can install an app that uses it, from Alex Larsson's nightly builds of a subset of GNOME. The server they're on doesn't have a great deal of bandwidth, so be nice :-)
$ wget http://184.108.40.206/keys/nightly.gpg $ xdg-app remote-add --user --gpg-import=nightly.gpg nightly \ http://220.127.116.11/repo/ $ xdg-app install --user nightly org.mypaint.MypaintDevel
We now have one app, and the runtime it needs:
$ xdg-app list org.mypaint.MypaintDevel $ xdg-app run org.mypaint.MypaintDevel [you see a GUI window]
Digression: what's in a runtime?
Behind the scenes, xdg-app runtimes and apps are both OSTree trees.
This means the
ostree tool, from the package of the same name, can be
used to inspect them.
$ sudo apt install ostree $ ostree refs --repo ~/.local/share/xdg-app/repo gnome:runtime/org.gnome.Platform/x86_64/3.18 nightly:app/org.mypaint.MypaintDevel/x86_64/master
A "ref" has roughly the same meaning as in git: something like a branch or
ostree can list the directory tree that it represents:
$ ostree ls --repo ~/.local/share/xdg-app/repo \ runtime/org.gnome.Platform/x86_64/3.18 d00755 0 0 0 / -00644 0 0 493 /metadata d00755 0 0 0 /files $ ostree ls --repo ~/.local/share/xdg-app/repo \ runtime/org.gnome.Platform/x86_64/3.18 /files d00755 0 0 0 /files l00777 0 0 0 /files/local -> ../var/usrlocal l00777 0 0 0 /files/sbin -> bin d00755 0 0 0 /files/bin d00755 0 0 0 /files/cache d00755 0 0 0 /files/etc d00755 0 0 0 /files/games d00755 0 0 0 /files/include d00755 0 0 0 /files/lib d00755 0 0 0 /files/lib64 d00755 0 0 0 /files/libexec d00755 0 0 0 /files/share d00755 0 0 0 /files/src
You can see that
/files in a runtime is basically a copy of
This is not coincidental: the runtime's
/files gets mounted at
inside the xdg-app container. There is also some metadata, which is in
the ini-like syntax seen in
$ ostree cat --repo ~/.local/share/xdg-app/repo \ runtime/org.gnome.Platform/x86_64/3.18 /metadata [Runtime] name=org.gnome.Platform/x86_64/3.16 runtime=org.gnome.Platform/x86_64/3.16 sdk=org.gnome.Sdk/x86_64/3.16 [Extension org.freedesktop.Platform.GL] version=1.2 directory=lib/GL [Extension org.freedesktop.Platform.Timezones] version=1.2 directory=share/zoneinfo [Extension org.gnome.Platform.Locale] directory=share/runtime/locale subdirectories=true [Environment] GI_TYPELIB_PATH=/app/lib/girepository-1.0 GST_PLUGIN_PATH=/app/lib/gstreamer-1.0 LD_LIBRARY_PATH=/app/lib:/usr/lib/GL
Looking at an app, the situation is fairly similar:
$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master d00755 0 0 0 / -00644 0 0 258 /metadata d00755 0 0 0 /export d00755 0 0 0 /files
/files maps to what will become
/app for the application, which
was compiled with
$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /files d00755 0 0 0 /files -00644 0 0 4599 /files/manifest.json d00755 0 0 0 /files/bin d00755 0 0 0 /files/lib d00755 0 0 0 /files/share
There is also a
/export directory, which is made visible to the host system
so that the contained app can appear as a "first-class citizen" in menus:
$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /export d00755 0 0 0 /export d00755 0 0 0 /export/share user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /export/share d00755 0 0 0 /export/share d00755 0 0 0 /export/share/app-info d00755 0 0 0 /export/share/applications d00755 0 0 0 /export/share/icons user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /export/share/applications d00755 0 0 0 /export/share/applications -00644 0 0 715 /export/share/applications/org.mypaint.MypaintDevel.desktop $ ostree cat --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master \ /export/share/applications/org.mypaint.MypaintDevel.desktop [Desktop Entry] Version=1.0 Name=(Nightly) MyPaint TryExec=mypaint Exec=mypaint %f Comment=Painting program for digital artists ... Comment[zh_HK]=藝術家的电脑绘画 GenericName=Raster Graphics Editor GenericName[fr]=Éditeur d'Image Matricielle MimeType=image/openraster;image/png;image/jpeg; Type=Application Icon=org.mypaint.MypaintDevel StartupNotify=true Categories=Graphics;GTK;2DGraphics;RasterGraphics; Terminal=false
Again, there's some metadata:
$ ostree cat --repo ~/.local/share/xdg-app/repo \ app/org.mypaint.MypaintDevel/x86_64/master /metadata [Application] name=org.mypaint.MypaintDevel runtime=org.gnome.Platform/x86_64/3.18 sdk=org.gnome.Sdk/x86_64/3.18 command=mypaint [Context] shared=ipc; sockets=x11;pulseaudio; filesystems=host; [Extension org.mypaint.MypaintDevel.Debug] directory=lib/debug
Building a runtime, probably the wrong way
The way in which the reference/demo runtimes and containers are generated is... involved. As far as I can tell, there's a base OS built using Yocto, and the actual GNOME bits come from RPMs. However, we don't need to go that far to get a working runtime.
In preparing this runtime I'm probably completely ignoring some best-practices and tools - but it works, so it's good enough.
First we'll need a repository:
$ sudo install -d -o$(id -nu) /srv/xdg-apps $ ostree init --repo /srv/xdg-apps
I'm just keeping this local for this demonstration, but you could rsync it
to a web server's exported directory or something - a lot like a git
repository, it's just a collection of files. We want everything in
because that's what xdg-app expects, hence
$ sudo mount -t tmpfs -o mode=0755 tmpfs /mnt $ sudo debootstrap --arch=amd64 --include=libx11-6,usrmerge \ --variant=minbase stretch /mnt http://192.168.122.1:3142/debian $ sudo mkdir /mnt/runtime $ sudo mv /mnt/usr /mnt/runtime/files
This obviously has a lot of stuff in it that we don't need - most obviously init, apt and dpkg - but it's Good Enough™.
We will also need some metadata. This is sufficient:
$ sudo sh -c 'cat > /mnt/runtime/metadata' [Runtime] name=org.debian.Debootstrap/x86_64/8.20160130 runtime=org.debian.Debootstrap/x86_64/8.20160130
That's a runtime. We can commit it to ostree, and generate xdg-app metadata:
$ ostree commit --repo /srv/xdg-apps \ --branch runtime/org.debian.Debootstrap/x86_64/8.20160130 \ /mnt/runtime $ fakeroot ostree commit --repo /srv/xdg-apps \ --branch runtime/org.debian.Debootstrap/x86_64/8.20160130 $ fakeroot xdg-app build-update-repo /srv/xdg-apps
(I'm not sure why ostree and xdg-app report "Operation not permitted" when we aren't root or fakeroot - feedback welcome.)
build-update-repo would presumably also be the right place to GPG-sign your repository, if you were doing that.
We can add that as another xdg-app remote:
$ xdg-app remote-add --user --no-gpg-verify local file:///srv/xdg-apps $ xdg-app remote-ls --user local org.debian.Debootstrap
Building an app, probably the wrong way
The right way to build an app is to build a "SDK" runtime - similar to that
platform runtime, but with development files and tools - and recompile the app
and any missing libraries with
./configure --prefix=/app && make && make install. I'm not going to do that,
because simplicity is nice, and I'm reasonably sure xvt doesn't actually
/usr into the binary:
$ install -d xvt-app/files/bin $ sudo apt-get --download-only install xvt $ dpkg-deb --fsys-tarfile /var/cache/apt/archives/xvt_2.1-20.1_amd64.deb \ | tar -xvf - ./usr/bin/xvt ./usr/ ./usr/bin/ ./usr/bin/xvt ... $ mv usr xvt-app/files
Again, we'll need metadata, and it's much simpler than the more production-quality GNOME nightly builds:
$ cat > xvt-app/metadata [Application] name=org.debian.packages.xvt runtime=org.debian.Debootstrap/x86_64/8.20160130 command=xvt [Context] sockets=x11; $ fakeroot ostree commit --repo /srv/xdg-apps \ --branch app/org.debian.packages.xvt/x86_64/2.1-20.1 xvt-app $ fakeroot xdg-app build-update-repo /srv/xdg-apps Updating appstream branch No appstream data for runtime/org.debian.Debootstrap/x86_64/8.20160130 No appstream data for app/org.debian.packages.xvt/x86_64/2.1-20.1 Updating summary $ xdg-app remote-ls --user local org.debian.Debootstrap org.debian.packages.xvt
The obligatory screenshot
OK, good, now we can install it:
$ xdg-app install --user local org.debian.Debootstrap 8.20160130 $ xdg-app install --user local org.debian.packages.xvt 2.1-20.1 $ xdg-app run --branch=2.1-20.1 org.debian.packages.xvt
and you can play around with the shell in the
xvt and see what you
can and can't do in the container.
I'm sure there were better ways to do most of this, but I think there's value in having such a simplistic demo to go alongside the various GNOMEish apps.
- Betacowork Coworking Brussels and ICAB Business & Technology Incubator hosted the hackfest, with Collabora providing snacks, and the GNOME Foundation supporting the hackfest in general;
- Collabora also sponsored my travel, accommodation and time;
- my colleague Philip Withnall organised the hackfest.
Thanks to all those!
Discworld Noir was a superb adventure game, but is also notoriously unreliable, even in Windows on real hardware; using Wine is just not going to work. After many attempts at bringing it back into working order, I've settled on an approach that seems to work: now that qemu and libvirt have made virtualization and emulation easy, I can run it in a version of Windows that was current at the time of its release. Unfortunately, Windows 98 doesn't virtualize particularly well either, so this still became a relatively extensive yak-shaving exercise.
These instructions assume that
/srv/virt is a suitable place to put
disk images, but you can use anywhere you want.
The emulated PC
After some trial and error, it seems to work if I configure qemu to emulate the following:
- Fully emulated hardware instead of virtualization (
- Intel Pentium III
- Intel i440fx-based motherboard with ACPI
- Real-time clock in local time
- No HPET
- 256 MiB RAM
- IDE primary master: IDE hard disk (I used 30 GiB, which is massively overkill for this game; qemu can use sparse files so it actually ends up less than 2 GiB on the host system)
- IDE primary slave, secondary master, secondary slave: three CD-ROM drives
- PS/2 keyboard and mouse
- Realtek AC97 sound card
- Cirrus video card with 16 MiB video RAM
A modern laptop CPU is an order of magnitude faster than what Discworld Noir needs, so full emulation isn't a problem, despite being inefficient.
There is deliberately no networking, because Discworld Noir doesn't need it, and a 17 year old operating system with no privilege separation is very much not safe to use on the modern Internet!
- Windows 98 installation CD-ROM as a
cp /dev/cdrom windows98.iso) - in theory you could also use a real optical drive, but my laptop doesn't usually have one of those. I used the OEM disc, version 4.10.1998 (that's the original Windows 98, not the Second Edition), which came with a long-dead PC, and didn't bother to apply any patches.
- A Windows 98 license key. Again, I used an OEM key from a past PC.
- A complete set of Discworld Noir (English) CD-ROMs as
.isofiles. I used the UK "Sold Out Software" budget release, on 3 CDs.
- A multi-platform Realtek AC97 audio driver.
Windows 98 installation
It seems to be easiest to do this bit by running qemu-system-i386 manually:
qemu-img create -f qcow2 /srv/virt/discworldnoir.qcow2 30G qemu-system-i386 -hda /srv/virt/discworldnoir.qcow2 \ -drive media=cdrom,format=raw,file=/srv/virt/windows98.iso \ -no-kvm -vga cirrus -m 256 -cpu pentium3 -localtime
Don't start the installation immediately. Instead, boot the installation CD to a DOS prompt with CD-ROM support. From here, run
and create a single partition filling the emulated hard disk. When
finished, hard-reboot the virtual machine (press Ctrl+C on the
qemu-system-i386 process and run it again).
FORMAT.COM utility is on the Windows CD-ROM but not in the root
directory or the default
%PATH%, so you'll have to run:
to create the FAT filesystem. You might have to reboot again at this point.
The reason for doing this the hard way is that the Windows 98 installer
doesn't detect qemu as supporting ACPI. You want ACPI support, so that
Windows will issue
IDLE instructions from its idle loop, instead of
occupying a CPU core with a busy-loop. To get that, boot to a DOS
prompt again, and use:
setup /p j /iv
/p j forces ACPI support
(Thanks to "Richard S" on the Virtualbox forums
for this tip.)
/iv is unimportant, but it disables the annoying "billboards" during
installation, which advertised once-exciting features like support for
dial-up modems and JPEG wallpaper.
I used a "Typical" installation; there didn't seem to be much point in tweaking the installed package set when everything is so small by modern standards.
Windows 98 has built-in support for the Cirrus VGA card that we're emulating, so after a few reboots, it should be able to run in a semi-modern resolution and colour depth. Discworld Noir apparently prefers a 640 × 480 × 16-bit video mode, so right-click on the desktop background, choose Properties and set that up.
This is the part that took me the longest to get working. Of the sound cards that qemu can emulate, Windows 98 only supports the SoundBlaster 16 out of the box. Unfortunately, the Soundblaster 16 emulation in qemu is incomplete, and in particular version 2.1 (as shipped in Debian 8) has a tendency to make Windows lock up during boot.
I've seen advice in various places to emulate an Eqsonic ES1370 (SoundBlaster AWE 64), but that didn't work for me: one of the drivers I tried caused Windows to lock up at a black screen during boot, and the other didn't detect the emulated hardware.
The next-oldest sound card that qemu can emulate is a Realtek AC97, which was often found integrated into motherboards in the late 1990s. This one seems to work, with the "A400" driver bundle linked above. For Windows 98 first edition, you need a driver bundle that includes the old "VXD" drivers, not just the "WDM" drivers supported by Second Edition and newer.
The easiest way to get that into qemu seems to be to turn it into a CD image:
genisoimage -o /srv/virt/discworldnoir-drivers.iso WDM_A400.exe qemu-system-i386 -hda /srv/virt/discworldnoir.qcow2 \ -drive media=cdrom,format=raw,file=/srv/virt/windows98.iso \ -drive media=cdrom,format=raw,file=/srv/virt/discworldnoir-drivers.iso \ -no-kvm -vga cirrus -m 256 -cpu pentium3 -localtime -soundhw ac97
Run the installer from E:, then reboot with the Windows 98 CD inserted, and Windows should install the driver.
Installing Discworld Noir
Boot up the virtual machine with CD 1 in the emulated drive:
qemu-system-i386 -hda /srv/virt/discworldnoir.qcow2 \ -drive media=cdrom,format=raw,file=/srv/virt/DWN_ENG_1.iso \ -no-kvm -vga cirrus -m 256 -cpu pentium3 -localtime -soundhw ac97
You might be thinking "... why not insert all three CDs into D:, E: and F:?"
but the installer expects subsequent disks to appear in the same drive
where CD 1 was initially, so that won't work. Instead, when prompted for
a new CD, switch to the qemu monitor with Ctrl+Alt+2 (note that this
is 2, not F2). At the
info block to see a list of emulated drives, then issue a
change ide0-cd1 /srv/virt/DWN_ENG_2.iso
to swap the CD. Then switch back to Windows' console with Ctrl+Alt+1 and continue installation. I used a Full installation of Discworld Noir.
Transferring the virtual machine to GNOME Boxes
Having finished the "control freak" phase of installation, I wanted a slightly more user-friendly way to run this game, so I transferred the virtual machine to be used by libvirtd, which is the backend for both GNOME Boxes and virt-manager:
virsh create discworldnoir.xml
Here is the configuration I used. It's a mixture of automatic configuration from virt-manager, and hand-edited configuration to make it match the qemu-system-i386 command-line.
Running the game
If all goes well, you should now see a
discworldnoir virtual machine
in GNOME Boxes, in which you can boot Windows 98 and play the game.
This blog is powered by ikiwiki.