Hello guys,
I was browsing flathub’s github repos, and I was wondering how appstream.xml.gz archive was generated?
Location:
https://hub.flathub.org/repo/appstream/x86_64/appstream.xml.gz
I’m trying to understand how flathub’s backend works.
So far I understood external-data-checker is being ran hourly (through github actions) to check for external package changes, if there is a change, a pull request is issued to the corresponding repository to update the app’s yml file.
From there I lost the track 
Kind regards,
me
Hi @StephenRippy,
Kinda three answers here. Under the hood, Flatpak Command Reference — Flatpak documentation is what’s used to re-scan an ostree repo for Flatpaks, pull out and concatenate all of the appdata / metainfo xml pieces into one appstream, and check it back into the ostree repo.
On Flathub, that’s done by https://github.com/flatpak/flat-manager/ which is a Rust service which maintains the ostree repos for Flathub, and exposes an API which - with the right tokens - you can talk to using the client (https://github.com/flatpak/flat-manager/blob/master/flat-manager-client - or other code) to create temporary repos for your build, push your Flatpaks to them, download and test, and merge them to the main repo. After a timeout so that metadata updates are batched up, flat-manager re-runs that update repo command to generate and publish new ostree-metadata/appstream refs and a new summary file.
The thing that builds the Flatpaks in response to the GitHub triggers, and so also sends in these flat-manager requests to update the repo is the Buildbot. The exact moment it happens is here: https://github.com/flathub/buildbot/blob/8ecc35f44763b1447b516c49f49f14a37804152f/master/buildbot/flathub_master.py#L1205-L1214
Hope that helps!
Cheers,
Rob
Aha, this certainly explains a lot! Oh what a complex system is in place !
Thanks for the great answer!
