Improve Linux Helix containers usability #1374
Conversation
|
build is green and I've done some local validation. ready for review… |
| procps-ng \ | ||
| python3 \ | ||
| python3-devel \ | ||
| python3.12 \ |
There was a problem hiding this comment.
Versioning pinning Python is a new practice. It would be good to include why this is needed and what other Dockerfile maintainers should do going forward, in the PR description.
There was a problem hiding this comment.
Yeah, I'm not sure why this is being done. The version associated with python3 is 3.9 which is the same version that exists in some other Helix images. So why does this need to be higher?
There was a problem hiding this comment.
Versioning pinning Python is a new practice
not really. it was used already in src/opensuse/15.6/helix/amd64/Dockerfile
the background is fairly simple: OS vendors sometimes have Python dependencies of their own or leave the Python packages alone after a release. this means our software can easily fall behind Python's deprecations and we have to support a broader range of Python versions than is manageable
the versioned packages move forward and are sometimes backported to prior releases, reducing our costs. see below for more about Centos Stream 9 in particular (bumping Python elsewhere in this PR e.g. on Debian 11 was required)
here and in src/centos-stream/9/helix/amd64/Dockerfile, this update avoided some messier workarounds for problems installing helix-scripts dependencies when using Python 3.9. I don't remember the exact details but believe this was the most straightforward fix
that said
- 3.9 will be out of support in October IIRC
- I'll try the
python -m pip install --upgrade pipworkaround used in src/debian/11/helix/arm64v8/Dockerfile (where 3.9 is the newest available Python)
There was a problem hiding this comment.
It would be good to document in this repo (not in this issue):
- Min required Python version
- Encouraged patterns for that
There was a problem hiding this comment.
I agree the words need to land somewhere other than this PR. where would be appropriate❓
for future reference and given pushback here, I think the rules are something like:
- try other workarounds b/4 moving to non-default OS packages for Python when problems arise
- if either ^^ is not possible or the default OS packages for Python install a deprecated version, move to the latest available Python 3 packages that are for released (not Beta or RC) versions
- today, minimum Python version should be 3.9
patterns when "pinning" is needed include
- pin version number for all
python3*packages in affected Dockerfile unless unavailable for an OS e.g.python312-pipexists for SUSE 15.6 but onlypython3-pipexists for Ubuntu 20.04 (thoughpython3.9-devis what we're using now) - don't include two Python versions in the OS packages list i.e., change the OS package list and don't add to it
- if OS itself depends on deprecated / busted-for-us Python, this will be additive where it wasn't b/4 and that's fine
- link
/usr/bin/pythonto specific/usr/bin/python3.{number}executable matching "pinned" Python version
There was a problem hiding this comment.
I resolved original comments in current iteration
There was a problem hiding this comment.
I think start with an issue on what the model should be for #1374 (comment). After we lock that down, we can leave it as an issue and link to it or move it to a repo doc.
| procps-ng \ | ||
| python3 \ | ||
| python3-devel \ | ||
| python3.12 \ |
There was a problem hiding this comment.
Yeah, I'm not sure why this is being done. The version associated with python3 is 3.9 which is the same version that exists in some other Helix images. So why does this need to be higher?
- give `helixbot` user UID 1000 in Ubuntu 24.04 and 24.10 images - delete the `ubuntu` user to free up that UID - always provide and activate a `venv` for `helixbot` user - always install `helix-scripts` module and dependencies in `venv` - some images installed this module in the system Python environment - where possible, do this installation as `helixbot` user - upgrade some Python OS package versions - avoid some older versions e.g. `python3.8` - link new version to /usr/bin/python - upgrade `pip` Python package in some images - work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003252 - bug caused problems installing `psutil` package - use `${VIRTUAL_ENV}/bin/pip` if run for `venv` before `$PATH` is changed - avoid updating system environment in some cases, defence in depth in others - install `sudo` package in Mariner 2.0 and Azure Linux 3.0 images - update /etc/sudoers.d/helixbot instead of /etc/sudoers everywhere - nits: - remove `helixbot2` users; they no longer help anything - remove `virtualenv` module and comments it - remove `helix-scripts` wheels after installing them - make some syntax a bit more consistent
4 participants
helixbotuser UID 1000 in Ubuntu 24.04 and 24.10 imagesubuntuuser to free up that UIDvenvforhelixbotuserhelix-scriptsmodule and dependencies invenvhelixbotuserpython3.8pipPython package in some imagespsutilpackage${VIRTUAL_ENV}/bin/pipif run forvenvbefore$PATHis changedsudopackage in Mariner 2.0 and Azure Linux 3.0 imageshelixbot2users; they no longer help anythingvirtualenvmodule and comments ithelix-scriptswheels after installing them