Use pip-compile for package resolution

[jemrobinson]

✅ Checklist

• You have given your pull request a meaningful title (e.g. Enable foobar integration rather than 515 foobar).
• You are targeting the appropriate branch. If you're not certain which one this is, it should be develop.
• Your branch is up-to-date with the target branch (it probably was when you started, but it may have changed since then).
• You have marked this pull request as a draft and added '[WIP]' to the title if needed (if you're not yet ready to merge).
• You have formatted your code using appropriate automated tools (for example ./tests/AutoFormat_Powershell.ps1 -TargetPath <path to file or directory> for Powershell).

⤴️ Summary

• Replace poetry with pip-compile for Python package resolution
• This should avoid the issues in SRD build issue: Python installation fails #1493 which required several packages to be dropped
• NB. I think the PyTorch version on PyPI is compatible with machines that don't have CUDA so this shouldn't be a problem. Can anyone verify?

🌂 Related issues

Closes #1493

🔬 Tests

Tested locally, but could do with independent testing.

[jemrobinson]

@edwardchalstrey1: Can you try a build from this branch and see whether everything works for you? By replacing poetry with pip-compile we should avoid the Python2Supports errors. I haven't removed torch here because I think it should work.

[JimMadge]

LGTM.

I think the long term solution is to remove these altogether. It is a pain to support and we have a better way of installing Python packages for most Tiers.

It might be more idiomatic if the unpinned requirements are in a file called requirements.in and the output of pip-compile is called requirements.txt. However, that is not a hill to die on.

[jemrobinson]

It might be more idiomatic if the unpinned requirements are in a file called requirements.in and the output of pip-compile is called requirements.txt. However, that is not a hill to die on.

Agreed - I just wanted to make the most minimal change in this PR.

[edwardchalstrey1]

Deploying a new candidate VM from this branch now. Will let you know the results

[edwardchalstrey1]

Success:

[2023-08-01 09:22:42: SUCCESS] Build started:
[2023-08-01 09:24:12: SUCCESS] Running cloud-init modules: 1m29s
[2023-08-01 09:53:57: SUCCESS] Installing/updating Ubuntu packages: 29m45s
[2023-08-01 09:54:34: SUCCESS] Preparing system for installation: 36s
[2023-08-01 09:56:59: SUCCESS] Installing additional deb/source packages: 2m25s
[2023-08-01 09:59:34: SUCCESS] Updating ClamAV database: 2m35s
[2023-08-01 10:22:46: SUCCESS] Installing Spark/Hadoop: 23m12s
[2023-08-01 10:35:19: SUCCESS] Installing Julia and packages: 12m33s
[2023-08-01 10:51:27: SUCCESS] Installing Python (3.9.17) and packages: 16m8s
[2023-08-01 11:06:04: SUCCESS] Installing Python (3.10.12) and packages: 14m37s
[2023-08-01 11:19:17: SUCCESS] Installing Python (3.11.4) and packages: 13m13s
[2023-08-01 11:19:29: SUCCESS] Configuring pyenv: 12s
[2023-08-01 11:49:50: SUCCESS] Installing R packages: 30m21s
[2023-08-01 11:50:08: SUCCESS] Cleaning up the build environment: 18s
[2023-08-01 11:50:15: SUCCESS] Checking environment configuration: 7s
[2023-08-01 11:50:16: SUCCESS] Finished build: 2h27m34s
[2023-08-01 11:50:15: INFO   ] Memory available: 17 GB
[2023-08-01 11:50:15: INFO   ] ..... mean usage:  11.12% =>  1.9 GB
[2023-08-01 11:50:15: INFO   ] ...... min usage:   5.35% =>  0.9 GB
[2023-08-01 11:50:15: INFO   ] ...... max usage:  54.65% =>  9.2 GB
[2023-08-01 11:50:15: INFO   ] CPU available: 8 cores
[2023-08-01 11:50:15: INFO   ] ..... mean usage:  30.74% =>  2.5 cores
[2023-08-01 11:50:15: INFO   ] ...... min usage:   0.10% =>  0.0 cores
[2023-08-01 11:50:15: INFO   ] ...... max usage: 100.00% =>  8.0 cores