Remap XDG directories to specific HOST locations

I wanted to manually override $XDG_CACHE_HOME, and bind it right to the user’s $HOME/.cache, but specifying - --env=XDG_CACHE_HOME=$HOME/.cache didn’t help, and neither did - --env=XDG_CACHE_HOME=home/.cache:rw. In both instances, I get the same result inside the sandbox:

echo $XDG_CACHE_HOME
/home/evilsupahfly/.var/app/io.github.evilsupahfly.amulet_flatpak/cache

I’m assuming that trying to manipulate the XDG strings is pointless because they’re reset at runtime?

The app needs to be able to load whatever third-party Minecraft texture pack is loaded by the user automatically downloaded from Minecraft’s official servers.

I could just symlink after the flatpak is installed, and while that might work for me or other experienced users, I can’t expect a user with little to no experience to understand how do that. So, I figured remapping the directory would be the best bet, only, if it’s possible, I’m obviously not doing it right and if it’s NOT, I need to implement a work-around of some kind.

What are my options? Where did I go wrong?

Well, if you only want to access the data files of an specific app, it would probably be sufficient to use the filesystem permission to grant access to that folder specifically?

The XDG directories are hard-coded & you’re not supposed to change its values.
Instead there are other ways to access the contents:

  • --filesystem=xdg-cache & access HOST_XDG_CACHE_HOME (see flatpak-run). It’s only set if XDG_CACHE_HOME is set and else you’ll have to fallback to ~/.cache.
  • --filesystem=xdg-cache/<SUBDIR> will bind mount <SUBDIR> into the sandboxed cache. This will require that you know which subdirectories could exist & hard-code them into your application.
  • If possible, just let the user select the folders from XDG_CACHE_HOME via the File Chooser portal.

And you have to consider: If someone is using one of the launchers from Flatpak then the cache files won’t be inside the home cache but in the sandboxed cache of the corresponding app id.

1 Like

The app I’m “Flatpaking” is a Minecraft Map Editor. It stores the Minecraft texture maps and corresponding block-name library (a combined collection of .PNG and .JSON files) in the cache, with support for third party texture packs which allows working on modded worlds with custom blocks.

Currently the user has to drop a texture pack in the correct folder for it to be used as the Editor doesn’t have an internal “Chooser” mechanism to allow users to do it via menus. Obviously, the problem here is that users don’t have access to the Sandboxed directories, which was the root of my desire to remap internal sandbox folders to external host folders.

However, the app doesn’t load any textures until after the user opens a Minecraft world, so if the folder is reassigned on (or even before) the first run, before any world is actually loaded, the changes should be mounted inside the sandbox permanently, and the users can use their custom texture packs as if the app was running directly off the host, rather than from the Flatpak sandbox, right?

Edit: The app is currently available as a Python project, which can be run from source right out of a VENV, but compiled versions are at present only available for Windows users. For the average new-to-Linux user, mucking around with Python’s virtual environments and PIP dependencies is probably quite an intimidating idea.

However, running something as a Flatpak - which most modern distros support right out of the box - makes for a much more user-friendly approach. Or, at least, that was my reasoning for starting this.

The following may be a cruft hack for your case, but at least may give you more ideas to try:

You could replace your Exec key in the .desktop file to Exec=launch.sh and you create that launch.sh script that first copies/links the host’s cache files inside the corresponding sandbox dir, and then executes the application.

Many flatpak Java apps use a similar approach to prepare some flags and arguments prior to launch the main java app. Eg. see here the .desktop file and its launch script.

I never thought of doing that. I’ll give it a try and see what happens.

Thanks!

Ultimately, things didn’t work the way I wanted them to. It’s possible I just didn’t fully understand the problem (being still wet behind the ears with Python), but mucking around with directory locations didn’t resolve my issue. However. Adding three lines to the Python source did the trick.

shrugs

I tried using a launch wrapper script, as you inventively suggested, but then I realized that the cache is reset and emptied on each use, so there’s nothing to copy into the sandbox when the program starts. This is because Amulet doesn’t know what version of Minecraft you need the textures for until you actually open a world, which makes sense. For whatever reason though, when running via a PIP install inside a VENV, this is fine and everything just works. However, when running as a flatpak, things break.

This is how Amulet initializes the directories it needs:

import os
import platformdirs

experimental_bedrock_resources = False

from ._version import get_versions

__version__ = get_versions()["version"]
del get_versions

data_dir = platformdirs.user_data_dir("AmuletMapEditor", "AmuletTeam")
os.environ.setdefault("DATA_DIR", data_dir)
config_dir = platformdirs.user_config_dir("AmuletMapEditor", "AmuletTeam")
if config_dir == data_dir:
    config_dir = os.path.join(data_dir, "Config")
os.environ.setdefault("CONFIG_DIR", config_dir)
os.environ.setdefault("CACHE_DIR", platformdirs.user_cache_dir("AmuletMapEditor", "AmuletTeam"))
os.environ.setdefault("LOG_DIR", platformdirs.user_log_dir("AmuletMapEditor", "AmuletTeam"))

from amulet_map_editor.api import config as CONFIG, lang
from amulet_map_editor.api.framework.app import open_level, close_level

When running from natively from PIP, these variables seem to be empty initially, but when running as a Flatpak, they’re not, as you can see from this side-by-side from my terminal logs:

Is this difference in behaviour possibly making things not work quite right? Theoretically, any Linux system that can run Flatpak should use the same folder structure as defined by the XDG standards, right?

Well, in an effort to better control the directory usage, I tried this:

id: io.github.evilsupahfly.amulet_flatpak
runtime: org.freedesktop.Platform
runtime-version: '24.08'
sdk: org.freedesktop.Sdk
command: amulet_map_editor

finish-args:
  - --device=all
  - --device=dri
  - --allow=devel
  - --allow=per-app-dev-shm
  - --share=network
  - --share=ipc
  - --socket=wayland
  - --socket=fallback-x11
  - --unset-env=XDG_STATE_HOME
  - --unset-env=XDG_CONFIG_HOME
  - --unset-env=XDG_DATA_HOME
  - --unset-env=XDG_CACHE_HOME
  - --filesystem=host
  - --filesystem=host-os
  - --filesystem=home:create
  - --filesystem=~/.cache/AmuletMapEditor:create
  - --filesystem=~/.local/state/AmuletMapEditor:create
  - --filesystem=~/.local/state/AmuletMapTeam:create
  - --filesystem=~/.local/share/AmuletMapEditor:create
  - --filesystem=~/.config/state/AmuletMapEditor:create
  - --env=LIBGL_ALWAYS_SOFTWARE="0"
  - --env=OPENGL_VERSION=3.3
  - --env=OPENGL_LIB=/usr/lib/x86_64-linux-gnu/libGL.so
  - --env=PS1=[ AMULET_FLATPAK > \w]\n>
# Uncomment the following options to increase debug output verbosity in the terminal
#  - --env=PYTHONDEBUG=3
#  - --env=PYTHONVERBOSE=3
#  - --env=PYTHONTRACEMALLOC=10
#  - --env=G_MESSAGES_DEBUG=all

modules:
  - shared-modules/glew/glew.json
  - shared-modules/glu/glu-9.json
  - updates.yaml
  - pip-gen.yaml
  - name: metainfo-xml
    buildsystem: simple
    build-commands:
      - install -Dm644 io.github.evilsupahfly.amulet_flatpak.metainfo.xml -t ${FLATPAK_DEST}/share/metainfo/
    sources:
      - type: file
        path: io.github.evilsupahfly.amulet_flatpak.metainfo.xml
  - name: metainfo-desktop
    buildsystem: simple
    build-commands:
      - install -Dm755 io.github.evilsupahfly.amulet_flatpak.desktop -t ${FLATPAK_DEST}/share/applications/
    sources:
      - type: file
        path: io.github.evilsupahfly.amulet_flatpak.desktop
  - name: metainfo-ico
    buildsystem: simple
    build-commands:
      - install -Dm644 io.github.evilsupahfly.amulet_flatpak.png -t ${FLATPAK_DEST}/share/icons/hicolor/256x256/apps/
    sources:
      - type: file
        path: io.github.evilsupahfly.amulet_flatpak.png

But it didn’t work and I’m still missing water and lava:

ERROR - Failed to load block model {'model': 'minecraft:block/lava'}
'/app/lib/python3.12/site-packages/minecraft_model_reader/api/resource_pack/java/java_vanilla_fix/assets/minecraft/textures/block/lava.png'
ERROR - Failed to load block model {'model': 'minecraft:block/water'}
'/app/lib/python3.12/site-packages/minecraft_model_reader/api/resource_pack/java/java_vanilla_fix/assets/minecraft/textures/block/water.png'

which being built-in, aren’t included in the official texture pack which gets downloaded, and that seems to be the issue.

The files are physically (digitally?) present in the sandbox, and the .JSON is readable:

Attempting to load model /app/lib/python3.12/site-packages/minecraft_model_reader/api/resource_pack/java/java_vanilla_fix/assets/minecraft/models/block/water.json
Attempting to load model /home/evilsupahfly/.var/app/io.github.evilsupahfly.amulet_flatpak/cache/AmuletMapEditor/resource_packs/java/vanilla/assets/minecraft/models/block/rooted_dirt.json

My gut is telling me that there’s some key difference between the way Python’s directories are handled inside versus outside the sandbox, but I’m unable to find it. I’ve loaded the app (which is actually composed of 7 different sub-projects which combine like Power Ranges to make ULTIMA AMULET) down with so many print statements now, tracking every loop, file access and directory assignment that my Flatpak terminal log is now over 10 MB from all the extra info.

To track the key differences, I also installed the modified versions of the Amulet base apps into my VENV with PIP. Those terminal logs are only 7 MB. In the error loop, at one point, I had worked out an override for water but it treated water like a solid block, not a transparent one, and that was no better.

So, now I’m back to square one: Amulet loads, worlds can be edited, but water and lava are replaced with the black-and-purple checked pattern for missing textures.

I have inadvertently solved my problem by creating a venv (quite literally) at /app so I could mirror the structure inside the sandbox as closely as possible, and after checking the logs, I’ve discovered a key difference. THE key difference, in fact.

When running from PIP, Amulet is installed to
/app/lib/python3.13/site-packages/

When Amulet is installed inside the Flatpak, the location is changed to
/app/usr/lib/python3.12/site-packages/

I guess not all the paths are relative after all. I feel like an idiot now because it was so obvious. It was right there in every error message, every time.

I was exploring ways to change the sandbox location for Python apps so I don’t have to keep modifying the Amulet source code each time they release an update, but when I change --prefix=${FLATPAK_DEST} to -t /lib, it gives me the error OSError: [Errno 30] Read-only file system - which I expected but wanted to test for the sake of being thorough.

So, the only option remaining then is to tinker under the hood with Amulet, right?

This path is incorrect. /app/usr make no sense.

I don’t know why it is the way that it is, but it works.

def get_java_vanilla_fix() -> JavaResourcePack:
    global _java_vanilla_fix
    if _java_vanilla_fix is None:
        # New: Change the path from "/app/lib/python3.12/site-packages/minecraft_model_reader" to "/app/usr/lib/python3.12/site-packages/minecraft_model_reader" - temporarily works around missing textures
        base_path = os.path.dirname(__file__)
        resource_pack_path = os.path.join(base_path, "java_vanilla_fix")
        if resource_pack_path.startswith("/app/lib/python3.12/site-packages/minecraft_model_reader"):
            resource_pack_path = resource_pack_path.replace("/app/lib/python3.12/site-packages/minecraft_model_reader", "/app/usr/lib/python3.12/site-packages/minecraft_model_reader")
        _java_vanilla_fix = JavaResourcePack(resource_pack_path)
    return _java_vanilla_fix

This modified call in Amulet’s download manager module redirects Minecraft’s hard-coded texture call to /app/usr/lib/python3.12/site-packages/minecraft_model_reader/java_vanilla_fix/assets and it works in Flatpak the way it does when running from PIP.

This is my first ever flatpak. I don’t understand why, but it works. I didn’t write Amulet, I just made the Flatpak for it.

¯\_(ツ)_/¯