Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try bumping the version of Ubuntu. #3103

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jamescrake-merani
Copy link
Contributor

Just doing some testing as running PyInstaller on my machine creates a distribution which works on my machine but the distribution generated by Github actions does not work on my machine.

Just doing some testing as running PyInstaller on my machine creates a
distribution which works on my machine but the distribution generated by
Github actions does not work on my machine.
@llimeht
Copy link
Contributor

llimeht commented Sep 5, 2024

Interesting observation - presumably this is another case where desktop daemons or settings are not isolated from the libraries in the pyinstaller bundle :(

Did it work?

Without going down this path, does deleting additional environment variables help? My previous suggestion worked by unsetting a selection of variables until it worked ok - you could also try that (look at the output of env and progressively add things to the list of vars to unset).

Moving beyond this as an experiment, increasing the linux baseline for the installer needs some care - you're fighting difficulties with compatibility from old build-host to newer desktop machine; unfortunately the other order (old desktop and newer build host) will never work because libc6 and some essential friends are not in the pyinstaller bundle at all. Perhaps 22.04 is old enough to be a fair baseline at this stage anyway.

Not sure if you have looked at the CI failure yet. The test failure in CI hopefully goes away when updating the Ubuntu version in the installer test job to match - the mateix.os is embedded into the filename of the artefact.

(I wonder if there are better options than pyinstaller these days; I'm rapidly thinking I'd rather spend effort on that than continuing to hit my head against it)

@jamescrake-merani
Copy link
Contributor Author

Did it work?

It did actually in that it produced a distribution which worked on my machine. However, I ran it under an Ubuntu 20 container since I noticed a warning message in one of the comments, and it did indeed break there. I believe this was because it was expecting a newer glibc version than what was present on the system.

Moving beyond this as an experiment, increasing the linux baseline for the installer needs some care

Yeah I suspected this would be something we would have to discuss extensively if this change were to be merged.

Not sure if you have looked at the CI failure yet. The test failure in CI hopefully goes away when updating the Ubuntu version in the installer test job to match - the mateix.os is embedded into the filename of the artefact.

Yes I did see the failure. I only created this PR so I could get Github to generate a build as I was trying to figure out why builds done on my machine were working but not the ones done on Github (which, as I suspected, had something to do with the Ubuntu versions).

(I wonder if there are better options than pyinstaller these days; I'm rapidly thinking I'd rather spend effort on that than continuing to hit my head against it)

I've been working on a Flatpak container for Sasview, and I've just managed to get one working today. However, it does rely on the PyInstaller distribution. I did originally try to build the container from scratch but the Flatpak builder does not have network access so running the normal pip install -r build_tools/requirements.txt failed because it couldn't find the sources for any of the packages. There is a script provided by Flatpak which puts these dependencies as modules in the Flatpak manifest file so they are downloaded before the builder runs but there are several issues with this script. Firstly, it does not consider cases where one module is dependent on another module. So for example, if module A depends on module B but module A is listed first, the builder will try to install it first but the installation will fail because module B is not present at that point in time. Secondly, one of Sasview's dependencies depends on another module which is built in Rust. Normally, Python can just install the binaries straight away but the script is fetching the Rust source code which cannot be built because it has Cargo dependencies it wants to download from the network... it was at that point that I started tearing my hair out.

As an alternative, I considered building a Flatpak container with the PyInstaller distribution instead. This might be a bit problematic because it depends on those being built in separate process. However, I have managed to get it to work. I haven't been able to test this extensively yet but theoretically it should run on other (and older) distributions because all of the libraries needed are either in the PyInstaller distribution, or in the container itself. This means that, as long as we can get a PyInstaller build that will work in the Flatpak container, then it should work wherever.

Thank you so much for your help in this work :)

@jamescrake-merani
Copy link
Contributor Author

I should also add that I spoke to another member of my team today who suggested looking into Singularity containers as well, and shared with me some training material which I am going to review. That could be a potential alternative to Flatpak (or if possible we could do both).

@llimeht
Copy link
Contributor

llimeht commented Sep 5, 2024

singularity seems popular for calculation engines while desktop users seem more used to flatpak for applications (like Chrome, Teams, ...).

Once unpacked, the python package is not in general relocatable (which is why you can't copy a venv around) and that gets in the way of some approaches; I assume that flatpak does some funky bind mounts to make for reproducible locations. Incidentally, wheels can be just popped into PYTHONPATH without installing them which is an intriguing prospect if their scripts and entry points aren't needed.

It's also possible to pre-download files for pip and use its cache - could that work with the flatpak builder? I looked at https://docs.flatpak.org/en/latest/python.html some time ago and I got the impression that this supposed to be possible.

@jamescrake-merani
Copy link
Contributor Author

It's also possible to pre-download files for pip and use its cache - could that work with the flatpak builder? I looked at https://docs.flatpak.org/en/latest/python.html some time ago and I got the impression that this supposed to be possible.

I recall looking at this page previously as it references the script I was talking about (flatpak-pip-generator). As mentioned previously, there are some issues with it unfortunately.

I'm going to be leaving work soon but will continue looking into this tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants