Fix problems installing on Raspberry Pi OS 12 (bookworm) #1408

delan · 2024-01-07T10:38:25Z

This patch addresses the problems in #1407 by changing easyinstall.sh as follows:

add rsyslog to APT_PACKAGES_COMMON
~~add --break-system-packages in cachePipPackages()~~
remove the -h|--help|h|help case in runChoice(), which doesn’t work
add a --help option, explaining that both short and long options need to use =
make --help more discoverable by suggesting it at the top of showMenu()
make it clear in showMenu() that by default, we compile from scratch every time

rdmark · 2024-01-08T23:31:53Z

@delan Thank you for contributing! Sorry that you had to struggle figuring out how to use the various options. The original intention was for the script to be plug’n’play, with the command line parameters treated as hidden debug options.

The fact that this script has become so bloated that it requires a dedicated help function is an issue in itself… but that’s something for another day. I agree that adding a help function now is the right course of action.

Now about this PR: Did you see the failed CI job? I want to resolve that failure before merging. Looking at the standalone logs, something breaks down when building Python dependencies. Maybe --break-system-packages is causing side effects in the GitHub runner environment? Another approach may be required. Would you be able to look into this?

rdmark · 2024-01-08T23:37:46Z

Please note that I changed the base branch to “develop” — this is where are new code goes. The “main” branch gets rebased with develop after a stable release.

rdmark · 2024-01-09T01:11:23Z

After pondering this for a moment, I posit one solution: don’t call cachePipPackages() when installing piscsi (option 1). Then, either keep it as an opt-in feature, or remove it altogether.

It is strictly speaking an optional routine, meant to have a local cache for pip when the Python clients initialize their venv for the first time, which is happening in their respective start.sh scripts. This is in order to work around the absence of a stable network environment, a very common state for a fresh RPi system. But it hasn’t proven to consistently solve the problem.

Unfortunately pip has some kind of bug or limitation that results in a broken venv under these conditions, which it cannot recover from… It’s by far the most commonly user reported incident.

There is actually a pending PR that seeks to replace it with a different solution, but it has been stalling for a long time due to complications with architecture differences for native C glue code for certain Python libraries…

As you can probably tell the Python environment situation is something that has been a very long running problem in this project!

delan · 2024-01-09T08:52:41Z

No worries, thanks for the feedback! I’ve removed cachePipPackages from option 1, but CI fails the same way. I tried to repro locally and here’s what I found, does this help?

$ export DOCKER_BACKEND_IMAGE="piscsi/backend-standalone:`git ls-files -s python .github/workflows/web.yml | git hash-object --stdin`"
$ docker pull --quiet ${DOCKER_BACKEND_IMAGE}
$ export DOCKER_BACKEND_NEEDS_PUSH=1
$ cd docker
$ docker compose -f docker-compose.ci.yml up -d
$ docker compose -f docker-compose.ci.yml run pytest -v
[...]
dependency failed to start: container docker-web-1 exited (1)

$ docker compose logs
backend-1  | SCSI Target Emulator PiSCSI (Backend Service)
backend-1  | Version 23.11 --DEVELOPMENT BUILD--  (Jan  9 2024 08:45:15)
backend-1  | Powered by XM6 TypeG Technology / Copyright (C) 2016-2020 GIMONS
web-1      | Creating python venv for web server
web-1      | Activating venv
backend-1  | Copyright (C) 2020-2023 Contributors to the PiSCSI project
web-1      | Installing requirements.txt
backend-1  | Connection type: FULLSPEC
web-1      | Collecting wheel
backend-1  | [2024-01-09 08:45:52.745] [info] Set log level to 'trace'
web-1      |   Downloading wheel-0.42.0-py3-none-any.whl (65 kB)
backend-1  | [2024-01-09 08:45:52.745] [warning] Detected Unknown SBC
web-1      | Installing collected packages: wheel
backend-1  | [2024-01-09 08:45:52.745] [info] Reserved ID(s) set to 7
web-1      | Successfully installed wheel-0.42.0
backend-1  | [2024-01-09 08:45:52.745] [info] No devices currently attached.
web-1      | Collecting Babel==2.13.1
backend-1  | No devices currently attached.
web-1      |   Downloading Babel-2.13.1-py3-none-any.whl (10.1 MB)
backend-1  | Note: No PiSCSI hardware support, only client interface calls are supported
web-1      | Collecting bjoern==3.2.2
web-1      |   Downloading bjoern-3.2.2.tar.gz (45 kB)
web-1      | Collecting blinker==1.6.3
web-1      |   Downloading blinker-1.6.3-py3-none-any.whl (13 kB)
web-1      | Collecting charset-normalizer==2.1.1
web-1      |   Downloading charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
web-1      | Collecting click==8.1.7
web-1      |   Downloading click-8.1.7-py3-none-any.whl (97 kB)
web-1      | Collecting Flask==3.0.0
web-1      |   Downloading flask-3.0.0-py3-none-any.whl (99 kB)
web-1      | Collecting flask-babel==4.0.0
web-1      |   Downloading flask_babel-4.0.0-py3-none-any.whl (9.6 kB)
web-1      | Collecting itsdangerous==2.1.2
web-1      |   Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
web-1      | Collecting Jinja2==3.1.2
web-1      |   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
web-1      | Collecting MarkupSafe==2.1.3
web-1      |   Downloading MarkupSafe-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
web-1      | Collecting protobuf==3.20.2
web-1      |   Downloading protobuf-3.20.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
web-1      | Collecting pytz==2023.3.post1
web-1      |   Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
web-1      | Collecting requests==2.31.0
web-1      |   Downloading requests-2.31.0-py3-none-any.whl (62 kB)
web-1      | Collecting simplepam==0.1.5
web-1      |   Downloading simplepam-0.1.5.tar.gz (2.4 kB)
web-1      | Collecting ua-parser==0.16.1
web-1      |   Downloading ua_parser-0.16.1-py2.py3-none-any.whl (38 kB)
web-1      | Collecting vcgencmd==0.1.1
web-1      |   Downloading vcgencmd-0.1.1-py3-none-any.whl (7.2 kB)
web-1      | Collecting Werkzeug==3.0.1
web-1      |   Downloading werkzeug-3.0.1-py3-none-any.whl (226 kB)
web-1      | Collecting importlib-metadata>=3.6.0
web-1      |   Downloading importlib_metadata-7.0.1-py3-none-any.whl (23 kB)
web-1      | Collecting urllib3<3,>=1.21.1
web-1      |   Downloading urllib3-2.1.0-py3-none-any.whl (104 kB)
web-1      | Collecting idna<4,>=2.5
web-1      |   Downloading idna-3.6-py3-none-any.whl (61 kB)
web-1      | Collecting certifi>=2017.4.17
web-1      |   Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
web-1      | Collecting zipp>=0.5
web-1      |   Downloading zipp-3.17.0-py3-none-any.whl (7.4 kB)
web-1      | Building wheels for collected packages: bjoern, simplepam
web-1      |   Building wheel for bjoern (setup.py): started
web-1      |   Building wheel for bjoern (setup.py): finished with status 'error'
web-1      |   ERROR: Command errored out with exit status 1:
web-1      |    command: /home/pi/piscsi/python/web/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-geh2j6n9
web-1      |        cwd: /tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/
web-1      |   Complete output (15 lines):
web-1      |   running bdist_wheel
web-1      |   running build
web-1      |   running build_py
web-1      |   Generating grammar tables from /usr/lib/python3.9/lib2to3/Grammar.txt
web-1      |   Generating grammar tables from /usr/lib/python3.9/lib2to3/PatternGrammar.txt
web-1      |   creating build
web-1      |   creating build/lib.linux-x86_64-3.9
web-1      |   copying bjoern.py -> build/lib.linux-x86_64-3.9
web-1      |   running build_ext
web-1      |   building '_bjoern' extension
web-1      |   creating build/temp.linux-x86_64-3.9
web-1      |   creating build/temp.linux-x86_64-3.9/bjoern
web-1      |   creating build/temp.linux-x86_64-3.9/http-parser
web-1      |   x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DSIGNAL_CHECK_INTERVAL=0.1 -DWANT_SIGNAL_HANDLING=yes -DWANT_SIGINT_HANDLING=yes -Ihttp-parser -Istatsd-c-client -I/usr/include/libev -I/opt/local/include -I/opt/homebrew/include -I/usr/local/include -I/home/pi/piscsi/python/web/venv/include -I/usr/include/python3.9 -c bjoern/_bjoernmodule.c -o build/temp.linux-x86_64-3.9/bjoern/_bjoernmodule.o -std=c99 -fno-strict-aliasing -fcommon -fPIC -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -g
web-1      |   error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
web-1      |   ----------------------------------------
web-1      |   ERROR: Failed building wheel for bjoern
web-1      |   Running setup.py clean for bjoern
web-1      |   Building wheel for simplepam (setup.py): started
web-1      |   Building wheel for simplepam (setup.py): finished with status 'done'
web-1      |   Created wheel for simplepam: filename=simplepam-0.1.5-py3-none-any.whl size=3207 sha256=b5e7c22645eb481b30ecc8dc6e2ddb581cda3a2596f8fb115e17e739f02aea5f
web-1      |   Stored in directory: /root/.cache/pip/wheels/b4/d9/6a/d9ceab53fba269efd05f4763016f07f062a822db93c4b7e567
web-1      | Successfully built simplepam
web-1      | Failed to build bjoern
web-1      | Installing collected packages: zipp, MarkupSafe, Werkzeug, Jinja2, itsdangerous, importlib-metadata, click, blinker, urllib3, pytz, idna, Flask, charset-normalizer, certifi, Babel, vcgencmd, ua-parser, simplepam, requests, protobuf, flask-babel, bjoern
web-1      |     Running setup.py install for bjoern: started
web-1      |     Running setup.py install for bjoern: finished with status 'error'
web-1      |     ERROR: Command errored out with exit status 1:
web-1      |      command: /home/pi/piscsi/python/web/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4pao20np/install-record.txt --single-version-externally-managed --compile --install-headers /home/pi/piscsi/python/web/venv/include/site/python3.9/bjoern
web-1      |          cwd: /tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/
web-1      |     Complete output (13 lines):
web-1      |     running install
web-1      |     running build
web-1      |     running build_py
web-1      |     creating build
web-1      |     creating build/lib.linux-x86_64-3.9
web-1      |     copying bjoern.py -> build/lib.linux-x86_64-3.9
web-1      |     running build_ext
web-1      |     building '_bjoern' extension
web-1      |     creating build/temp.linux-x86_64-3.9
web-1      |     creating build/temp.linux-x86_64-3.9/bjoern
web-1      |     creating build/temp.linux-x86_64-3.9/http-parser
web-1      |     x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DSIGNAL_CHECK_INTERVAL=0.1 -DWANT_SIGNAL_HANDLING=yes -DWANT_SIGINT_HANDLING=yes -Ihttp-parser -Istatsd-c-client -I/usr/include/libev -I/opt/local/include -I/opt/homebrew/include -I/usr/local/include -I/home/pi/piscsi/python/web/venv/include -I/usr/include/python3.9 -c bjoern/_bjoernmodule.c -o build/temp.linux-x86_64-3.9/bjoern/_bjoernmodule.o -std=c99 -fno-strict-aliasing -fcommon -fPIC -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -g
web-1      |     error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
web-1      |     ----------------------------------------
web-1      | ERROR: Command errored out with exit status 1: /home/pi/piscsi/python/web/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0h06z7jo/bjoern_c4f18447b1814038ba04cce68e658717/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4pao20np/install-record.txt --single-version-externally-managed --compile --install-headers /home/pi/piscsi/python/web/venv/include/site/python3.9/bjoern Check the logs for full command output.
web-1      | compiling catalog src/translations/fr/LC_MESSAGES/messages.po to src/translations/fr/LC_MESSAGES/messages.mo
web-1      | compiling catalog src/translations/es/LC_MESSAGES/messages.po to src/translations/es/LC_MESSAGES/messages.mo
web-1      | compiling catalog src/translations/zh/LC_MESSAGES/messages.po to src/translations/zh/LC_MESSAGES/messages.mo
web-1      | compiling catalog src/translations/sv/LC_MESSAGES/messages.po to src/translations/sv/LC_MESSAGES/messages.mo
web-1      | compiling catalog src/translations/de/LC_MESSAGES/messages.po to src/translations/de/LC_MESSAGES/messages.mo
web-1      | Starting PiSCSI Web UI ...
web-1      | Traceback (most recent call last):
web-1      |   File "/home/pi/piscsi/python/web/src/web.py", line 12, in <module>
web-1      |     import bjoern
web-1      | ModuleNotFoundError: No module named 'bjoern'

Both builds seem to fail due to the x86_64-linux-gnu-gcc error: 789-web.log, 790-web.log.

rdmark · 2024-01-09T09:02:20Z

@delan the critical line is this

error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory

try adding gcc to the list of packages to install in the Dockerfile and see if that makes a difference

easyinstall.sh

rdmark

Looks good, thanks for contributing!

For the record, the pip caching function is still called from two other routines: stand alone web UI and the hidden option “99” which is used by our pigen script. I think it’s appropriate to keep those intact for now.

* Fix problems installing on Raspberry Pi OS 12 (bookworm) * don’t cachePipPackages in option 1 * add gcc to required packages for web container

delan requested review from akuker and rdmark as code owners January 7, 2024 10:38

rdmark changed the base branch from main to develop January 8, 2024 23:32

Fix problems installing on Raspberry Pi OS 12 (bookworm)

f23f990

delan force-pushed the fix-raspios-12-problems branch from f0272c7 to f23f990 Compare January 9, 2024 08:33

don’t cachePipPackages in option 1

bcc1029

rdmark reviewed Jan 9, 2024

View reviewed changes

easyinstall.sh Outdated Show resolved Hide resolved

delan added 2 commits January 11, 2024 12:20

revert --break-system-packages change

d9608a9

add gcc to required packages for web container

2654400

rdmark approved these changes Jan 11, 2024

View reviewed changes

rdmark merged commit 2f483fd into PiSCSI:develop Jan 11, 2024
3 checks passed

delan deleted the fix-raspios-12-problems branch January 11, 2024 06:23

rdmark pushed a commit that referenced this pull request May 1, 2024

Fix problems installing on Raspberry Pi OS 12 (bookworm) (#1408)

b7ecb8b

* Fix problems installing on Raspberry Pi OS 12 (bookworm) * don’t cachePipPackages in option 1 * add gcc to required packages for web container

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix problems installing on Raspberry Pi OS 12 (bookworm) #1408

Fix problems installing on Raspberry Pi OS 12 (bookworm) #1408

delan commented Jan 7, 2024 •

edited

Loading

rdmark commented Jan 8, 2024

rdmark commented Jan 8, 2024

rdmark commented Jan 9, 2024

delan commented Jan 9, 2024 •

edited

Loading

rdmark commented Jan 9, 2024

rdmark left a comment

Fix problems installing on Raspberry Pi OS 12 (bookworm) #1408

Fix problems installing on Raspberry Pi OS 12 (bookworm) #1408

Conversation

delan commented Jan 7, 2024 • edited Loading

rdmark commented Jan 8, 2024

rdmark commented Jan 8, 2024

rdmark commented Jan 9, 2024

delan commented Jan 9, 2024 • edited Loading

rdmark commented Jan 9, 2024

rdmark left a comment

Choose a reason for hiding this comment

delan commented Jan 7, 2024 •

edited

Loading

delan commented Jan 9, 2024 •

edited

Loading