Pytorch and yaml file in download stage

1

I am trying to build pytorch for a flatpak from source. I have the following in my yaml file. I don’t know if the build commands are right, but that doesn’t concern me right now. For some reason the googletest repo isn’t downloading.

  • name: torch
    buildsystem: simple
    build-commands:

    • git submodule update --init --jobs 0
    • pip3 install --prefix=/app .
      sources:
    • type: git
      url: https://github.com/pytorch/pytorch
      commit: 67ece03c8cd632cce9523cd96efde6f2d1cc8121

I get the message below.

    Downloading sources
    Fetching git repo https://github.com/pytorch/pytorch, ref refs/tags/v1.12.0-rc8
    remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    Fetching git repo https://github.com/pybind/pybind11.git, ref refs/tags/v2.6.2
    remote: Enumerating objects: 1, done.
    remote: Counting objects: 100% (1/1), done.
    remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
    Receiving objects: 100% (1/1), 176 bytes | 176.00 KiB/s, done.
    Fetching git repo https://github.com/NVlabs/cub.git, ref refs/tags/1.9.10
    remote: Enumerating objects: 1, done.
    remote: Counting objects: 100% (1/1), done.
    remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
    Receiving objects: 100% (1/1), 199 bytes | 199.00 KiB/s, done.
    Fetching git repo https://gitlab.com/libeigen/eigen.git, ref refs/tags/3.4.0
    remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    Fetching full git repo https://github.com/google/googletest.git
    remote: Enumerating objects: 30, done.
    remote: Counting objects: 100% (26/26), done.
    remote: Compressing objects: 100% (8/8), done.
    remote: Total 30 (delta 17), reused 18 (delta 17), pack-reused 4
    Receiving objects: 100% (30/30), 36.83 KiB | 7.37 MiB/s, done.
    Resolving deltas: 100% (17/17), completed with 9 local objects.
    From https://github.com/google/googletest
     + 08a2a8e7...0813e1ac refs/pull/3695/merge -> refs/pull/3695/merge  (forced update)
    Fetching git repo https://github.com/google/benchmark.git, ref refs/heads/master
    remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    Fetching git repo https://github.com/protocolbuffers/protobuf.git, ref refs/heads/3.13.x
    remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    Fetching git repo https://github.com/google/benchmark.git, ref refs/tags/v1.2.0
    remote: Enumerating objects: 1, done.
    remote: Counting objects: 100% (1/1), done.
    remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
    Receiving objects: 100% (1/1), 157 bytes | 157.00 KiB/s, done.
    Fetching full git repo https://github.com/google/googletest.git
    Fetching git repo https://github.com/Yangqing/ios-cmake.git, ref refs/heads/master
    remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    Fetching git repo https://github.com/Maratyszcza/NNPACK.git, ref refs/heads/master
    remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    Fetching full git repo https://github.com/facebookincubator/gloo
    Failed to download sources: module torch: Not a gitlink tree: third-party/googletest

Thanks for your time. Any help would be great. I was hoping there would be some kind of flag that I could set that would make this part of the flatpak build work. That may not be the case.

EDIT: it seems that the forum has messed up the indentation on my yaml snippet. I’m going to leave it as it is.

2

you can’t do git submodule in the build phase since there is not network access. However git submodules are by default automatically setup.

Also you probably need to use the flatpak-builder-tools to generate the manifest for the python dependencies so pip can install offline as needed.

3

as for google test try adding disable-shallow-clone: true to the source.

4

where does the ‘disable-shallow-clones: true’ go? Also, pytorch should be built from source (I think). Also, this is what I get when I run flatpak-pip-generator. This is with just happytransformer, but the output is similar when I add torch.

    ========================================================================
    Downloading sources
    ========================================================================
    Running: "flatpak --filesystem=/home/dave/workspace/electron-gpt/py/requirements.txt --devel --share=network --filesystem=/tmp --command=pip3 run org.freedesktop.Platform download --exists-action=i --dest /tmp/pip-generator-py-packagepex81f3g -r ./../py/requirements.txt"
    bwrap: execvp pip3: No such file or directory
    Failed to download
    Please fix the module manually in the generated file
    ========================================================================
    Downloading arch independent packages
    ========================================================================
    ========================================================================
    Obtaining hashes and urls
    ========================================================================
    ========================================================================
    Generating dependencies
    ========================================================================
    Generating dependencies for happytransformer
    bwrap: execvp pip3: No such file or directory
    Failed to download happytransformer

    Output saved to py-package.json

The actual output is below:

    {
        "name": "python3-happytransformer",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"happytransformer==2.4.1\" --no-build-isolation"
        ],
        "sources": []
    }

This is the requirements.txt. You can see, I’ve tried to use torch as a requirement:

    #--extra-index-url https://download.pytorch.org/whl/cpu
    #torch==1.12.1
    happytransformer==2.4.1

Thanks for getting back to me.

5

oh, and this is the line where I call flatpak pip generator. I include the name of the runtime…

flatpak-pip-generator -r ./../py/requirements.txt --output py-package  --runtime org.freedesktop.Platform
6

If I remove the runtime option from the flatpak-pip-generator call and comment-in the torch and extra-index-url lines in the requirements.txt file, I get a lot of output that ends up here:

    Successfully downloaded torch happytransformer datasets tqdm transformers protobuf sentencepiece typing-extensions dill huggingface-hub numpy packaging pyarrow pyyaml regex requests responses tokenizers aiohttp filelock multiprocess pandas xxhash aiosignal async-timeout attrs certifi charset-normalizer frozenlist idna multidict pyparsing python-dateutil pytz urllib3 yarl fsspec six
    ========================================================================
    Downloading arch independent packages
    ========================================================================
    Traceback (most recent call last):
      File "/home/dave/workspace/electron-gpt/flatpak/./tools/pip/flatpak-pip-generator", line 259, in <module>
        url = get_tar_package_url_pypi(name, version)
      File "/home/dave/workspace/electron-gpt/flatpak/./tools/pip/flatpak-pip-generator", line 72, in get_tar_package_url_pypi
        with urllib.request.urlopen(url) as response:
      File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
        return opener.open(url, data, timeout)
      File "/usr/lib/python3.10/urllib/request.py", line 525, in open
        response = meth(req, response)
      File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
        response = self.parent.error(
      File "/usr/lib/python3.10/urllib/request.py", line 563, in error
        return self._call_chain(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
        result = func(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
        raise HTTPError(req.full_url, code, msg, hdrs, fp)
    urllib.error.HTTPError: HTTP Error 404: Not Found

I don’t really know what this means, but I thought it would be good to show the output.