How is appstream.xml.gz generated?

Hello guys,

I was browsing flathub’s github repos, and I was wondering how appstream.xml.gz archive was generated?
Location:
https://hub.flathub.org/repo/appstream/x86_64/appstream.xml.gz

I’m trying to understand how flathub’s backend works.

So far I understood external-data-checker is being ran hourly (through github actions) to check for external package changes, if there is a change, a pull request is issued to the corresponding repository to update the app’s yml file.

From there I lost the track :smiley:

Kind regards,

me

Hi @StephenRippy,

Kinda three answers here. Under the hood, Flatpak Command Reference — Flatpak documentation is what’s used to re-scan an ostree repo for Flatpaks, pull out and concatenate all of the appdata / metainfo xml pieces into one appstream, and check it back into the ostree repo.

On Flathub, that’s done by https://github.com/flatpak/flat-manager/ which is a Rust service which maintains the ostree repos for Flathub, and exposes an API which - with the right tokens - you can talk to using the client (https://github.com/flatpak/flat-manager/blob/master/flat-manager-client - or other code) to create temporary repos for your build, push your Flatpaks to them, download and test, and merge them to the main repo. After a timeout so that metadata updates are batched up, flat-manager re-runs that update repo command to generate and publish new ostree-metadata/appstream refs and a new summary file.

The thing that builds the Flatpaks in response to the GitHub triggers, and so also sends in these flat-manager requests to update the repo is the Buildbot. The exact moment it happens is here: https://github.com/flathub/buildbot/blob/8ecc35f44763b1447b516c49f49f14a37804152f/master/buildbot/flathub_master.py#L1205-L1214

Hope that helps!

Cheers,
Rob

Aha, this certainly explains a lot! Oh what a complex system is in place :smiley: !

Thanks for the great answer!