I tested a flatpak file explorer and it can list my home folder files (and probably access theirs content)
Not sure what is the problem here? Flatpak explicitly tells which permissions app is going to need, and for applications using portals, it’s the portal that has access to files. The files shared by the portal are not a direct filesystem access.