Block outdated Flatpak clients

[Eonfge]

This is going to be a controversial one, so hold on tight.

Flatpak has been in development now for a few years and it’s really gaining more momentum. We have made progress, and it’s cool that I’ve been able to contribute to this endeavour. But as it is, we’re at risk of tripping over our own feet.

Background

Let’s wind the clock back a few months. I distributed an important update for Lollypop, which reworks the filesystem access. As such, Lollypop is now mostly sandboxed and this really helps with the goal of Flatpakking. In the days that followed, I received numerous reports of people who could no longer access their music on a secondary harddrive or NAS. It turned out that they were running Flatpak 1.6.x.

• Github issue

This shined a light on a far bigger issue: The majority of Flatpak versions out there is outdated and about half the people that download packages from Flathub is doing so with 1.6 or lower.

• Stats
• Repology

This creates a security conundrum for two reasons:

• Users don’t have all available security patches (CVE)
• Flatpakking stagnates because clients miss newer features

Action

I don’t think that we should let this run its cause. The statistics show a lot of trailing users that make up about 20% of our downloads, that will likely never update to a secure version. This should prompt us to take action.

There are many things that can be done to improve this situation. First, we should encourage distribution maintainers to update their Flatpak packages. Only last month was there another famous Linux distribution that shipped with 1.6.x. A shame, that could have been prevented.

But there is something else that can be done, something that can be done here at Flathub, and that will have a stronger long term effect: We could over time block outdated clients.

Blocking clients

As every Android and Back-end developer will attest… it’s not a popular move. But, it’s the only thing that we can realistically do in the long term. How we do it though, can vary:

• We can leave it up to individual package maintainers
• We can slowly phase out old clients
• We can be crass and block all old clients

Now, the first option is already the case. Maintainers can already define a minimum version, although it is almost never used. As a maintainer, you don’t one to be the only one pulling the trigger on such a sensitive issue. The third option is also not really desirable, because we don’t want to victimise individual users.

So, that’s what brings me to the Post’s title. Let’s discuss the issue about slowly blocking outdated clients so that they don’t become a millstone around our necks.

Additional discussion

• Where to start? Everything < 1.2?
• How to announce? How long ahead?
• What do we discuss with (commercial) maintainers? Opt-out options?

[183231bcb]

@Eonfge

Now, the first option is already the case. Maintainers can already define a minimum version, although it is almost never used. As a maintainer, you don’t one to be the only one pulling the trigger on such a sensitive issue.

Chromium is one of the most popular apps on Flathub and it requires version 1.8.2. Granted, the Chromium flatpak has required 1.8 from the beginning, while Lollypop hasn’t. Still, I don’t think some more apps that actually use new flatpak features requiring higher flatpak versions would cause more resentment than completely blocking old flatpak versions from installing any apps, even those that don’t rely on new features.

[sonny]

UX

Something important to take into consideration is that when Android/iOS/“PWA” applications do such a thing, they offer an option to their users to continue using the service - usually it’s as easy as “Please update the application” or “Accept new terms and conditions”.

Updating flatpak wouldn’t be an option - as you said the issue lies with distributions.

I don’t think preventing users from using flathub without offering a solution is reasonable.

How would they know why it’s not working? Is there a way for a repository to return information to be displayed to the user? Does it work with graphical clients?

If so - let’s consider a warning instead.

Security

From a security standpoint I don’t see the benefit to prevent users from using the most trustworthy flatpak repository while letting them (and perhaps encourage) using third party repositories.

Breakage

It seems that setting the minimum required flatpak version to be the perfect solution for this.

For new flatpak features available to application developers - then perhaps the flatpak documentation could mention the version in which it started being available and flathub reviewers could enforce the minimum required flatpak version if the application developer hasn’t.

EDIT: typo

[Eonfge]

@sonny

Something important to take into consideration is that when Android/iOS/“PWA” applications do such a thing, they offer an option to their users to continue using the service - usually it’s as easy as “Please update the application” or “Accept new terms and conditions”.

This is a bit of a factual inaccuracy. Most of the time, when apps break compatibility with a certain version of iOS or Android, updating the app is not the solution; only upgrading the phone is. An example of this is the Hardware Backed Keystore for Android 6. As an app moves to this secure enclave for authentication tokens, it breaks all support for Android 5 and earlier. No app update can undo that and users will have to upgrade their phone instead. If you check the Android app of many banks, you’ll find that many of their 1-star ratings are thanks to breaking such backwards compatibility since manually upgrading Android is not easy.

You do have a point @sonny, related to offering ‘alternatives’. What can a user do when their Flatpak is outdated, apart from complaining to their distribution maintainers? Perhaps it’s good to also offer a series of .deb and .rpm packages that people can update at any time, if the situation calls for it. We already have a PPA for Ubuntu 18.04 and 20.04 so it’s not like we don’t try to help users.

Also the idea of enforcing a minimum Flatpak version for new submissions sounds like a good idea. If all new apps on Flathub only work with supported Flatpak versions, then that enforces updates in a more granular and workflow retaining way. What do the maintainers think of this idea?

For security, the biggest issue right now, is that apps are not embracing better sandboxing, while this technology is available at large. It becomes something akin to the network effect: Our apps will be less secure then possible because app maintainers will keep support for insecure versions. Something you also see in the Android reference I just gave.

@183231bcb I also think that we should be very careful and conservative at first. Perhaps just block everything before 1.2.x This will block less then 1% of our users but it will send a message that in the future, we will also deprecate 1.4.x and 1.6.x

[sonny]

@Eonfge

This is a bit of a factual inaccuracy. Most of the time, when apps break compatibility with a certain version of iOS or Android, updating the app is not the solution; only upgrading the phone is. An example of this is the Hardware Backed Keystore for Android 6. As an app moves to this secure enclave for authentication tokens, it breaks all support for Android 5 and earlier. No app update can undo that and users will have to upgrade their phone instead. If you check the Android app of many banks, you’ll find that many of their 1-star ratings are thanks to breaking such backwards compatibility since manually upgrading Android is not easy.

Right - I thought you were referring to a service (backend) dropping support for older versions of their own clients which is more or less the use case here with flathub/flatpak.

The equivalent to dropping support for older versions of iOS/Android is to get developers/maintainers to require a minimum flatpak version and/or SDK.

Also the idea of enforcing a minimum Flatpak version for new submissions sounds like a good idea. If all new apps on Flathub only work with supported Flatpak versions, then that enforces updates in a more granular and workflow retaining way. What do the maintainers think of this idea?

That’s not what I meant, but it sounds reasonable to me and could be automated.

For security, the biggest issue right now, is that apps are not embracing better sandboxing, while this technology is available at large.

I can’t connect this with the fragmentation of flatpak or the suggestion to block outdated flatpak clients.

Do you have examples?

---

Anyhow - it looks like 3 different problems to me

• Applications break because they don’t set the minimum flatpak version required
• Users are running flatpak versions with security holes
• Developers are not embracing better sandboxing

and while I understand the suggested solution could help mitigate them; it sounds unreasonable to me given the following issues:

• What about security updates of the applications / runtimes ?
• What about breaking auto updates ?
• How do we even let users know what is going on or what to do?

[BrunoVernay]

If the security issues get exploited, Flatpak “image” will be badly impacted and it will reach a very large audience. Trust is hard to get back.
Eventually upsetting a few users with incompatible updates has much less impact.

[popey]

@Eonfge

You do have a point @sonny, related to offering ‘alternatives’. What can a user do when their Flatpak is outdated, apart from complaining to their distribution maintainers? Perhaps it’s good to also offer a series of .deb and .rpm packages that people can update at any time, if the situation calls for it. We already have a PPA for Ubuntu 18.04 and 20.04 so it’s not like we don’t try to help users.

Given Ubuntu 18.04 LTS and 20.04 LTS are likely culprits for the swath of 1.0.9 and 1.6.x in the graphic at the top, I’d say priority is getting those updates into the archive as SRU’s (Stable Release Updates) if you can.

PPAs are horribly undiscoverable for most users. As you’ve noticed, people will try to find or install something from flathub, then it will fail and they’ll give up. Only a fraction of hardened users will go off-piste to find the solution, which may lead a fraction to use the PPA. Indeed just poking at the PPAs we can see a comparatively very low number of users actually hitting the PPA. Last column is number of hits to that package since it was published nearly a week ago.

ubuntu@ppas:~/ppastats/2021-05-02-124439$ grep ^flatpak, *
alexlarsson_flatpak_bionic_amd64:flatpak,1.11.1-0flatpak1~bionic,4784
alexlarsson_flatpak_bionic_arm64:flatpak,1.11.1-0flatpak1~bionic,38
alexlarsson_flatpak_bionic_i386:flatpak,1.11.1-0flatpak1~bionic,200
alexlarsson_flatpak_focal_amd64:flatpak,1.11.1-0flatpak1~focal,6665
alexlarsson_flatpak_focal_arm64:flatpak,1.11.1-0flatpak1~focal,19
alexlarsson_flatpak_focal_armhf:flatpak,1.11.1-0flatpak1~focal,2
alexlarsson_flatpak_xenial_amd64:flatpak,1.11.1-0flatpak2~xenial,1547
alexlarsson_flatpak_xenial_arm64:flatpak,1.11.1-0flatpak2~xenial,3
alexlarsson_flatpak_xenial_armhf:flatpak,1.11.1-0flatpak2~xenial,2
alexlarsson_flatpak_xenial_i386:flatpak,1.11.1-0flatpak2~xenial,138

(obviously some people don’t turn their computer / VM on every day, some people don’t update, but it’s a good ball-park).

There’s millions of desktop 18.04 and 20.04 systems, so a few thousand is pretty low. Getting someone in the Ubuntu community to update it in the archive is preferable. This doesn’t require Canonical intervention, because it’s a package in the universe pocket. (I appreciate you may already know this, but other readers of this post may not).

[ahayzen]

That graph looks familiar

@Eonfge

Users don’t have all available security patches

This is not totally true, just because the version is shown as 1.6 or 1.0 in the stats does not mean that it is missing CVE fixes. I have been helping to backport security patches for Ubuntu back to 18.04 (using 1.0) and that CVE you reference is currently in review to be backported to 1.0, 1.6, 1.8 series (Ubuntu 18.04, 20.04, 20.10).

@Eonfge

The statistics show a lot of trailing users that make up about 20% of our downloads, that will likely never update to a secure version. This should prompt us to take action.

See my previous comment, old version does not mean insecure.

@Eonfge

slowly blocking outdated clients so that they don’t become a millstone around our necks

Completely blocking old clients based on their advertised flatpak version seems to completely break one of the main selling points of flatpak, which was that developers can release apps to old distributions without having to wait for them to catch up with packaging. If flatpak doesn’t solve this then maybe there is a design problem, as other competing systems are able to update themselves outside of distribution updates.

However it is also understandable for individual apps to require a specific version due to a new feature, but a complete ban seems over the top (maybe a warning is the route to go, like when you use EOL runtimes?).

It also means you are unlikely to have users from stable versions of RHEL, Debian, SUSE, Ubuntu etc using flatpak, if you cut the version too aggressively. Maybe <1.0 is fine to be cut now, but <1.2 feels maybe too aggressive right now (maybe in 6 months the picture will be different though etc).

@Eonfge

We already have a PPA for Ubuntu 18.04 and 20.04 so it’s not like we don’t try to help users

Problem is the PPA is currently shipping an unstable version of flatpak, I would want the latest stable not unstable, and with Alex reducing time on flatpak he wanted to share maintenance, but this is hard with a user PPA (it would be better as a team PPA). But even then PPAs are not discoverable and can break systems.

@popey

Given Ubuntu 18.04 LTS and 20.04 LTS are likely culprits for the swath of 1.0.9 and 1.6.x in the graphic at the top, I’d say priority is getting those updates into the archive as SRU’s (Stable Release Updates) if you can.

Yup I usually come to the same conclusion when doing updates to the Ubuntu archive and watching the stats. Note that with the current Ubuntu SRU policies Ubuntu 18.04 and 20.04 are up to date as much as I can make them within the existing policies.

@popey

PPAs are horribly undiscoverable for most users.

Yup I agree PPAs have bad discoverability and can easily break systems (eg with upgrades etc), hence why I have been contributing to updating the version in the archive

@popey

Getting someone in the Ubuntu community to update it in the archive is preferable.

Hello there.

If the flatpak version really does need to be increased beyond it’s series in Ubuntu LTS then I see the following options

• Apply for a exception to allow for bumping versions within an Ubuntu release (seems unlikely to be granted and would likely require other new/updated packages) - note as the SRU policy is becoming stricter we might need to apply for an exception just to update microreleases that aren’t solving specific bugs or security fixes.

• Add the package to Ubuntu backports, this would allow users to opt in with apt install -t focal-backports flatpak however, there are likely to be other packages that need to be backported, and I suspect not many users will know to opt in as it is a manual process.

• Provide a classic snap, this would allow for anywhere where snap is available for the latest version of flatpak to be installed, however it is not clear if everything would work correctly without testing and it is possible that it would be rejected from the store due to allow multiple untrusted substores (if we limited just to flathub it could be possible).

• Provide a snap which builds the latest deb version on the users machine, however this seems like an ugly hack

• Create a team PPA to allow for multiple maintainers, eg ppa:flatpak-team/flatpak-stable which tracks stable versions, however this has poor discoverability and can break systems.

• Flatpak update itself, like other packages such as steam and snapd do.

[Eonfge]

@ahayzen

This is not totally true, just because the version is shown as 1.6 or 1.0 in the stats does not mean that it is missing CVE fixes. I have been helping to backport security patches for Ubuntu back to 18.04 (using 1.0) and that CVE you reference is currently in review to be backported to 1.0, 1.6, 1.8 series (Ubuntu 18.04, 20.04, 20.10).

I’m glad to hear that. Sorry for making a wrong assumption concerning the CVEs.

Still, the security model of Flatpak is two fold: One the one hand Flatpak itself must be secure, but it must also be able to sandbox effectively. Olders versions of Flatpak, properly patched or not, are less capable of sandboxing. In that regards I still think that outdated versions precent risks, even if those risks are indirect. This is also echo’ed by Sonny’s excellent breakdown:

@sonny

Anyhow - it looks like 3 2 different problems to me

• Applications break because they don’t set the minimum flatpak version required
• Users are running flatpak versions with security holes
• Developers are not embracing better sandboxing

As for ways of remedying this issue. I’m not sure what would be easier: Both CentOS/RHEL 7 and Ubuntu 18.04 are on Flatpak 1.0.x so any technical solution should include them.

In both cases, updating the package while it’s in a stable release, seems nigh-impossible. Both Canonical and Red Hat have a lot of reputation on the line and until either of their distributions migrates to Flatpak, I don’t think that either will ever allow such major updates.

@ahayzen

Provide a classic snap, this would allow for anywhere where snap is available for the latest version of flatpak to be installed, however it is not clear if everything would work correctly without testing and it is possible that it would be rejected from the store due to allow multiple untrusted substores

Next up, you’ll tell me that they have a competing product which conflicts with our own technical implementation and philosophical goals. Also, I don’t think that many RHEL 7 users will install Snap just so they can upgrade Flatpak… While humoristic, I think this kind of defeats the purpose.

@ahayzen

Flatpak update itself

That would even be possible with the upcoming release of Flatpak, when it becomes possible for applications to bring their own runtimes. That said… I can already see the memes. It also won’t help us today, as it will take another 5 years for the old-stables of Red Hat and Canonical use Flatpak 1.12

In summary

it’s hard, and I don’t know if there are smarter solutions. This also sums up why I don’t think that the solution should be purely technical in nature.

As in my original proposal, the idea of slowly bringing the heat to distributions, encouraging them to improve their compatibility, sounds like an implicit requirement: If we can’t slowly encourage others to go along with us, we can make it work.

As for UI integrations… Both GNOME Software 3.36 and Discover will not provide very useful messages if a user tries to install or update an app for whom they don’t have the right version. This unhappy flow is not currently accounted for.

[popey]

@Eonfge

Next up, you’ll tell me that they have a competing product which conflicts with our own technical implementation and philosophical goals. Also, I don’t think that many RHEL 7 users will install Snap just so they can upgrade Flatpak… While humoristic, I think this kind of defeats the purpose.

I think the point Andrew was making was that as PPAs are undiscoverable, and Snaps are very very discoverable on Ubuntu, making a snap to pull in flatpak might help the discoverability and bootstrapping problem. I agree it’s not likely that RHEL users will use a snap to get newer flatpak, and it’s certainly a bit weird to do, but there’s other tools in the snap store which do similar things.

[183231bcb]

It’s also worth noting that if you arbitrarily block old clients just because they are old (and not because you are using features that are only available in new versions of flatpak), then some people will find ways to bypass the block. For example, Apple blocked the 2009 iMacs from installing versions of MacOS beyond 10.13, but some people found a way to patch it to install newer MacOS versions.

On the other hand, the Mac App Store still allows clients running operating systems as old as 10.6 (the first version that got the Mac App Store), the iOS App Store still allows clients running iPhone OS 2, and Google Play still allows Android 2.3 clients. It’s up to individual app developers to declare the minimum supported operating system versions for their apps.

[contrast]

Don’t block anything. That is not in the spirit of FOSS. Just have a message that says this flatpak will not function correctly unless using Flatpak version X or above.

[tuxkrusader]

I agree with @contrast , this is the most sensible solution.

[cxrvh]

Isn’t this just a flaw/bug of flatpak which should be discussed there?

The solution of setting a minimum version for an update is not ideal either because many users are left without the option to update. It would be really bad if the users are stuck with an old version of a security relevant applications (e.g. a browser), especially if the last version grants access to the host filesystem.

I don’t know how relevant the issue will be in the future (if the sandbox won’t change that much anymore), but instead of blocking old clients, maybe it’s possible to provide different metadata? Then new clients could receive stricter sandboxing & old clients just keep the previous settings.

[ChrisOfBristol]

I’m no expert, but I have just published a Flatpak for a simple application.

First there was .deb and .rpm, then there was Flatpak and Snap, now there were several different Flatpak platforms. Mine would run on an old and deprecated Flatpak platform which had Gnome, but not the latest with Gnome, but the latest without Gnome. It has a dependency which was the bit that would only run on the older platform (and the developer isn’t going to update that), but the application itself needs Gnome. I couldn’t even find any documentation about what Gnome stuff was missing from the latest non-Gnome platform so I could perhaps add it myself. Result - nightmare! I have published it on the latest platform but without the dependency, so it is only half as good.

Flatpaks are great if they can run on both Redhat and Debian based distributions, but not much good if you end up in a very similar situation to the .rpm/.deb split with different developments on several Flatpak platforms. It needs resolving one way or another, ideally only one supported platform, but if there is more than one platform they should make it very clear what the difference are and why.

[Eonfge]

@ChrisOfBristol

Flatpaks are great if they can run on both Redhat and Debian based distributions, but not much good if you end up in a very similar situation to the .rpm/.deb split with different developments on several Flatpak platforms. It needs resolving one way or another, ideally only one supported platform, but if there is more than one platform they should make it very clear what the difference are and why.

I totally agree with you there. Recommending Flatpak will be a lot harder if it comes with all kinds of Asterisks: Flatpak works everywhere* and it offers many new security features* and stability improvements*… But I do fear that we are the minority at this point. Perhaps it’s best to shelve the discussion for now, at to look back in a year or so.

Thanks everybody for your feedback and participation: We might not share the same opinion, but all feedback and participation is welcome. Let’s see what the future brings, and let’s make sure in the mean time that Flathub is the Nr. 1 place for all applications.

[BEEDELLROKEJULIANLOC]

All of the old versions of Flatpak should be prohibited because maintenance and support of them is utterly unnecessary.