Replace /bin/sh with a wrapper to /bin/bash

[martin-g]

This is needed to be able to set umask=022 for the Linux aarch64 images that use umask=027 by default.

/bin/sh does not load /etc/profile nor any other rcfile.

See bioconda/bioconda-recipes#46177 for full details and galaxyproject/galaxy#17631 for a summary.

[martin-g]

The Debian image test fails here -

bioconda-containers/images/base-glibc-debian-bash/Dockerfile.test

Lines 9 to 13 in ecfb6b5

	printf '%s\n' \
	'printf "hello "' \
	> /usr/local/env-activate.sh \
	&& \
	[ "$( sh -lc 'printf world' )" = 'hello world' ] \

It prints

root@43de1e34c911:/# sh -lc 'printf world'
hello hello world

i.e. there is a double activation ...

[martin-g]

This one does not use BASH_ENV=/etc/profile because the current tests in Dockerfile.test use sh -lc ... and -l/--login causes double activation.

IMO all Dockerfiles should use BASH_ENV=/etc/profile and the tests should be adapted to not use -l.

[martin-g]

@daler The PR is ready for review!

[daler]

I'm not sure of all of the implications of this -- @mbargull and @bgruening can you weigh in on this?

I can't think of cases where changing the umask would be problematic.

But is it OK to overwrite /bin/sh with a wrapper that uses bash?

[martin-g]

I wonder why I face this issue with umask only with the bioconductor-** recipes … Why/how others mulled tests pass ?!

…

On Tue, 12 Mar 2024 at 15:54, Ryan Dale ***@***.***> wrote: I'm not sure of all of the implications of this -- @mbargull <https://github.com/mbargull> and @bgruening <https://github.com/bgruening> can you weigh in on this? I can't think of cases where changing the umask would be problematic. But is it OK to overwrite /bin/sh with a wrapper that uses bash? — Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYUQS6VJTIU4XKIMYI4E3YX4CIXAVCNFSM6AAAAABEP53HVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRG4YDOMJYG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mbargull]

But is it OK to overwrite /bin/sh with a wrapper that uses bash?

No, not OK at all.
For one, in our containers /bin/sh is already a symlink to /bin/bash.
It is also expected that you get a POSIX-compliant shell with /bin/sh; Bash will just do that, see https://manpages.debian.org/bookworm/bash/bash.1.en.html#INVOCATION (i.e., invoking bash breaks such expectations).
Even though it has been possible to chain shebangs under Linux for a long time, I'd still avoid this, esp. for /bin/sh; see https://en.wikipedia.org/w/index.php?title=Shebang_(Unix)&oldid=1212120830#Syntax .

sh -l/bash -l (and dash -l which we don't use) all read /etc/profile as far as I'm aware.

---

Sorry, I'm aware this is not helpful to resolve you problems.
I haven't read the referenced issues/PRs yet so can't advice on what's wrong where.
Just wanted you know that the changes proposed here likely break other things and as such can't be applied.
I'll try to have a look in a few of days (would otherwise need a complete but succinct summary to give proper advice).

[martin-g]

Thanks for the comments, @mbargull !

For one, in our containers /bin/sh is already a symlink to /bin/bash.

❯ docker run -it --rm quay.io/bioconda/base-glibc-busybox-bash:latest bash -c "ls -la /bin/sh"
lrwxrwxrwx    1 root     root             9 Oct 17 11:23 /bin/sh -> /bin/bash

❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:latest bash -c "ls -la /bin/sh"
lrwxrwxrwx 1 root root 4 Jan  5  2023 /bin/sh -> dash

❯ docker run -it --rm quay.io/bioconda/create-env:latest bash -c "ls -la /bin/sh"
lrwxrwxrwx    1 root     root             9 Oct 17 11:23 /bin/sh -> /bin/bash

base-glibc-debian-bash points to /bin/dash !

sh -l/bash -l (and dash -l which we don't use) all read /etc/profile as far as I'm aware.

This is true! But the problem is that mulled-build does not pass -l:

1. https://github.com/galaxyproject/galaxy/blob/0a3e222e625959f6d829576742031dfe9fa2d878/lib/galaxy/tool_util/deps/mulled/invfile.lua#L90
2. https://github.com/galaxyproject/galaxy/blob/0a3e222e625959f6d829576742031dfe9fa2d878/lib/galaxy/tool_util/deps/mulled/invfile.lua#L106
3. https://github.com/galaxyproject/galaxy/blob/0a3e222e625959f6d829576742031dfe9fa2d878/lib/galaxy/tool_util/deps/mulled/invfile.lua#L121

And because of this /etc/profile is not taken into account for the mulled tests and the consequence is that umask on the Linux ARM64 Docker images is 027 (inherited from Debian/Ubuntu base images). And due to 027 folders created by create-env Docker container are not readable by the biocontainers image (i.e. base-glibc-**-bash).

One more (minor?!) issue: /bin/sh (without -l) does not read any rcfile (like /etc/profile). With -l it would read it but create-env Docker image specifies a custom ENV env var (

bioconda-containers/images/create-env/Dockerfile

Line 41 in ecfb6b5

ENV ENV=/etc/profile.d/conda.sh

) that tells it to read /etc/profile.d/conda.sh instead, so any global settings in /etc/profile are missed again.

I will close this issue because as you said the proposed changes may cause big differences for others!
But we can still discuss here other possible solutions!

[mbargull]

Hi @martin-g, sorry for not getting back to you earlier.

base-glibc-debian-bash points to /bin/dash !

Right, I completely forgot that I kept the larger one mostly "vanilla Debian" in that regard.
Thanks for the correction!

One more (minor?!) issue: /bin/sh (without -l) does not read any rcfile (like /etc/profile). With -l it would read it but create-env Docker image specifies a custom ENV env var (

bioconda-containers/images/create-env/Dockerfile

Line 41 in ecfb6b5

ENV ENV=/etc/profile.d/conda.sh

) that tells it to read /etc/profile.d/conda.sh instead, so any global settings in /etc/profile are missed again.

Setting ENV is for interactive use and does not prevent sourcing login shell specific files (e.g, /etc/profile).
The differences between non-login-non-interactive/non-login-interactive/login-non-interactive/login-interactive shell invocations and between POSIX-y shell/Bash-specifics/other-shell-specifics can be confusing/daunting.
There are many different things that can come into play under different circumstances (/etc/profile, ~/.profile, ~/.bash_profile, ~/.bash_login, /etc/bash.bashrc, ~/.bashrc, ENV, BASH_ENV, and more env vars/arguments -- and they all have different meanings/implications).
We should take care about understanding implications of build setup changes around this since it can (negatively) affect the products of such setups (e.g., our packages/containers).
The aforementioned https://manpages.debian.org/bookworm/bash/bash.1.en.html#INVOCATION gives a reasonably good overview about this, IMO.
(I'm somewhat familiar with the differences, but also have to refresh my memory on this regularly with all this differences/niche uses.)

Generally, we don't want to run interactive/login scripts for these cases.
Meaning, the shell invocations you mentioned should remain sh -c without any -i/-l.
(And sh -c is supposed to behave the same as sh some-script.sh in regards to not sourcing any custom scripts; if they were to allow running anything from /etc/profile or elsewhere, then a simple echo exit 1 > /etc/profile would pretty much break your system.)

---

FWIW, I cannot reproduce the behavior you described locally.
For me, all container images I tested for amd64/arm64 with or without running through their entry points, yield umask of 0022 for any sh/bash combination of -c/-i/-l (tested with mainly with podman but also checked with the latest docker in case something fishy would happen there).
(This is locally on an x86-64 machine with the ARM64 bits running via QEMU, which should not matter, but noting for completeness' sake.)
So, to me it is not clear at all where the observed 0027 stems from.

---

The discussion around this seem to be scatter around multiple issues/PRs; I don't have an overview about this -- do we have some central issue to discuss this?

[martin-g]

Hi Marcel, Did you test on Linux ARM64 machine/VM or using —platform=linux-arm64 and QEMU ? Please check bioconda/bioconda-recipes#46177 <bioconda/bioconda-recipes#46177 (comment)> It has all the history of my investigation. My latest solution uses galaxy-until-tool from their dev branch and it solves the problem by using “-l” I’m on vacation until Wednesday so I cannot test anything but I can answer questions here!

…

On Sat, 30 Mar 2024 at 11:23, Marcel Bargull ***@***.***> wrote: Hi @martin-g <https://github.com/martin-g>, sorry for not getting back to you earlier. base-glibc-debian-bash points to /bin/dash ! Right, I completely forgot that I kept the larger one mostly "vanilla Debian" in that regard. Thanks for the correction! One more (minor?!) issue: /bin/sh (without -l) does not read any rcfile (like /etc/profile). With -l it would read it but create-env Docker image specifies a custom ENV env var ( https://github.com/bioconda/bioconda-containers/blob/ecfb6b5e9bbd5944890e92e6587fabc00153c96d/images/create-env/Dockerfile#L41) that tells it to read /etc/profile.d/conda.sh instead, so any global settings in /etc/profile are missed again. Setting ENV is for interactive use and does not prevent sourcing login shell specific files (e.g, /etc/profile). The differences between non-login-non-interactive/non-login-interactive/login-non-interactive/login-interactive shell invocations and between POSIX-y shell/Bash-specifics/other-shell-specifics can be confusing/daunting. There are many different things that can come into play under different circumstances (/etc/profile, ~/.profile, ~/.bash_profile, ~/.bash_login, /etc/bash.bashrc, ~/.bashrc, ENV, BASH_ENV, and more env vars/arguments -- and they all have different meanings/implications). We should take care about understanding implications of build setup changes around this since it can (negatively) affect the products of such setups (e.g., our packages/containers). The aforementioned https://manpages.debian.org/bookworm/bash/bash.1.en.html#INVOCATION gives a reasonably good overview about this, IMO. (I'm somewhat familiar with the differences, but also have to refresh my memory on this regularly with all this differences/niche uses.) Generally, we don't want to run interactive/login scripts for these cases. Meaning, the shell invocations you mentioned should remain sh -c without any -i/-l. (And sh -c is supposed to behave the same as sh some-script.sh in regards to not sourcing any custom scripts; if they were to allow running anything from /etc/profile or elsewhere, then a simple echo exit 1 > /etc/profile would pretty much break your system.) ------------------------------ FWIW, I *cannot* reproduce the behavior you described locally. For me, all container images I tested for amd64/arm64 with or without running through their entry points, yield umask of 0022 for any sh/bash combination of -c/-i/-l (tested with mainly with podman but also checked with the latest docker in case something fishy would happen there). (This is locally on an x86-64 machine with the ARM64 bits running via QEMU, which should not matter, but noting for completeness' sake.) So, to me it is not clear at all where the observed 0027 stems from. ------------------------------ The discussion around this seem to be scatter around multiple issues/PRs; I don't have an overview about this -- do we have some central issue to discuss this? — Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYUQQUQV4DORZUX2GVKRTY2ZZAZAVCNFSM6AAAAABEP53HVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXHE4DQMZXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mbargull]

Hi Martin,

Did you test on Linux ARM64 machine/VM or using —platform=linux-arm64 and QEMU ?

QEMU (as noted above); i.e., only user space via binfmt_misc.

My latest solution uses galaxy-until-tool from their dev branch and it solves the problem by using “-l”

Hm... I don't think that's a good idea, TBH. That's rather a workaround for whatever underlying issue and might mask (no pun intended..) other problems.

I've let this run on CI in bioconda/bioconda-recipes#46900 and we get the same output for amd64/arm64 images there, i.e., umask of 0022 for root and 0002 for non-root login shells (also non-root non-login interactive Bash which I didn't expect; not that it matters, but it surprised me).

So, still no sign of the 0027 you observed. (Really no idea there; do you run some old Docker that does things differently?)

I still haven't looked into what/how happens for the CI build failures (meaning, not looked at the logs you pointed out at all).
One thing I could imagine, would be package building happening with root user (i.e., overriding the entrypoint which I think we did for some reason at some point -- I'll have to check if that's still the case) and then reusing those package files as the non-root ("conda") user when creating the environment for Biocontainers (i.e., umask 0022 from root preventing writes from conda user).
^-- But that should've also happen for x86-64 builds since the setup/umasks seem to be the same on CI...

I’m on vacation until Wednesday so I cannot test anything but I can answer questions here!

I'll try to see if I can understand/reproduce things on CI.
Enjoy your vacation!

[martin-g]

On Sat, 30 Mar 2024 at 13:12, Marcel Bargull ***@***.***> wrote: Hi Martin, Did you test on Linux ARM64 machine/VM or using —platform=linux-arm64 and QEMU ? QEMU (as noted above); i.e., only user space via binfmt_misc. My latest solution uses galaxy-until-tool from their dev branch and it solves the problem by using “-l” Hm... I don't think that's a good idea, TBH. That's rather a workaround for whatever underlying issue and might mask (no pun intended..) other problems.

Probably you are right! The mask issue is easily reproducible on the ARM64 VM I use but it might be issue with Docker, as you said. Using “-l” solved the issue locally and at CircleCI so I didn’t pursue other solutions. I've let this run on CI in bioconda/bioconda-recipes#46900

<bioconda/bioconda-recipes#46900> and we get the same output for amd64/arm64 images there, i.e., umask of 0022 for root and 0002 for non-root login shells (also non-root non-login interactive Bash which I didn't expect; not that it matters, but it surprised me). So, still no sign of the 0027 you observed. (Really no idea there; do you run some old Docker that does things differently?) I still haven't looked into what/how happens for the CI build failures (meaning, not looked at the logs you pointed out at all). One thing I could imagine, would be package building happening with root user (i.e., overriding the entrypoint which I think we did for some reason at some point -- I'll have to check if that's still the case) and then reusing those package files as the non-root ("conda") user when creating the environment for Biocontainers (i.e., umask 0022 from root preventing writes from conda user). ^-- But that should've also happen for x86-64 builds since the setup/umasks seem to be the same on CI...

Actually it have seen it for x86_64 too. If an error happens (like this one) then it falls back to “remote” execution of involucro, i.e. it tries to use a Docker image “Involucro/tool” that’s amd64 only and this hides the earlier problem. I’m on vacation until Wednesday so I cannot test anything but I can answer

questions here! I'll try to see if I can understand/reproduce things on CI. Enjoy your vacation!

Thank you for your help! —

…

Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYUQSXNBWPGVMLWRMVUQTY22FZ3AVCNFSM6AAAAABEP53HVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGAYTKOJSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mbargull]

Actually it have seen it for x86_64 too. If an error happens (like this one) then it falls back to “remote” execution of involucro, i.e. it tries to use a Docker image “Involucro/tool” that’s amd64 only and this hides the earlier problem.

Oh, wow, that's ugly! Thanks for making me aware of that!

[martin-g]

On Sat, 30 Mar 2024 at 14:06, Marcel Bargull ***@***.***> wrote: Actually it have seen it for x86_64 too. If an error happens (like this one) then it falls back to “remote” execution of involucro, i.e. it tries to use a Docker image “Involucro/tool” that’s amd64 only and this hides the earlier problem. Oh, wow, that's ugly! Thanks for making me aware of that!

The logic is in mulled-build tool (galaxy-util-tool). On ARM64 it fails with format exec error. —

…

Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYUQXUKSHASY472PCYNNLY22MDXAVCNFSM6AAAAABEP53HVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGAZDSNBZGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[martin-g]

@mbargull Here is an fresh example of the problem on x86_64 - bioconda/bioconda-recipes#47373
https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=56753&view=logs&j=e14e69ff-a0ae-55c4-b71d-229b239cfb2f&t=4dddc55b-10bc-50fc-ac68-7ff899b7a030 contains the following logs:

...
(ERR) [Apr 22 07:08:15] DEBU Creating container [step-4451144196]�[0m
2024-04-22T07:08:28.3523577Z 07:08:15 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:15] DEBU Image [quay.io/bioconda/base-glibc-busybox-bash:3.0] not present, pulling it�[0m
2024-04-22T07:08:28.3524095Z 07:08:15 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:15] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:3.0]�[0m
2024-04-22T07:08:28.3524725Z 07:08:18 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:18] DEBU Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied�[0m
2024-04-22T07:08:28.3525485Z 07:08:18 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:18] WARN Local execution errorred with error [open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied], retrying with remote execution�[0m
2024-04-22T07:08:28.3526074Z 07:08:18 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:18] DEBU Creating container [step-bfdb1470bb]�[0m
2024-04-22T07:08:28.3526541Z 07:08:18 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:18] DEBU Image [involucro/tool:latest] not present, pulling it�[0m
2024-04-22T07:08:28.3526997Z 07:08:18 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:18] DEBU Pull Image [involucro/tool:latest]�[0m
2024-04-22T07:08:28.3527479Z 07:08:19 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:19] DEBU Created container [a65c654bb544 step-bfdb1470bb], starting it�[0m
2024-04-22T07:08:28.3528019Z 07:08:19 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:19] DEBU Container [a65c654bb544 step-bfdb1470bb] started, waiting for completion�[0m
2024-04-22T07:08:28.3528541Z 07:08:19 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:19] SERR [Apr 22 07:08:19] DEBU Creating container [step-63cec14dbe]�[0m
2024-04-22T07:08:28.3529027Z 07:08:26 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 07:08:26] SERR [Apr 22 07:08:26] DEBU Packing succeeded�[0m
...

Similar logs but for aarch64 fail with:

...
 DEBU Creating container [step-d423ee17d9]�[0m
06:57:06 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:06] DEBU Image [quay.io/bioconda/base-glibc-busybox-bash:3.0] not present, pulling it�[0m
06:57:06 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:06] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:3.0]�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] DEBU Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] WARN Local execution errorred with error [open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied], retrying with remote execution�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] DEBU Creating container [step-440d147bef]�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] DEBU Image [involucro/tool:latest] not present, pulling it�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] DEBU Pull Image [involucro/tool:latest]�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] DEBU Created container [b0358490bcdf step-440d147bef], starting it�[0m
06:57:07 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:07] DEBU Container [b0358490bcdf step-440d147bef] started, waiting for completion�[0m
06:57:08 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:08] SERR exec /involucro: exec format error�[0m
06:57:08 �[32mBIOCONDA INFO�[0m (ERR) [Apr 22 06:57:08] ERRO Task processing failed: Unexpected exit code [1] of container [b0358490bcdf step-440d147bef], container preserved�[0m
.06:57:09 �[32mBIOCONDA INFO�[0m (OUT) .�[0m
...

[martin-g]

@mbargull Here is the simplest reproducer I could do:

conda create -n test123
conda activate test123
conda install font-ttf-ubuntu involucro
pip3 install galaxy-tool-util
mkdir /tmp/mulled && cd /tmp/mulled
mulled-build build 'font-ttf-ubuntu=0.83' --involucro-path $(which involucro) --verbose
ls -laR

It fails for me on both Linux aarch64 and x86_64!

x86_64 logs:

/tmp/mulled via 🅒 test123 
❯ mulled-build build 'font-ttf-ubuntu=0.83' --involucro-path $(which involucro) --verbose                                                                                                                  (test123) 
Executing: /home/martin/devel/miniconda3/envs/test123/bin/involucro -v=3 -f /home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,bioconda -set TARGETS=font-ttf-ubuntu=0.83 -set REPO=quay.io/biocontainers/font-ttf-ubuntu:0.83 -set BINDS=build/dist:/usr/local/ -set DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:latest -set CONDA_BIN=conda build
.[Apr 22 16:11:52] DEBU Run file [/home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
[Apr 22 16:11:52] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[rm -rf /data/dist]]
[Apr 22 16:11:52] DEBU Creating container [step-c1bad912eb]
[Apr 22 16:11:52] DEBU Created container [db1ea49a5a5c step-c1bad912eb], starting it
[Apr 22 16:11:53] DEBU Container [db1ea49a5a5c step-c1bad912eb] started, waiting for completion
[Apr 22 16:11:53] DEBU Container [db1ea49a5a5c step-c1bad912eb] completed with exit code [0] as expected
[Apr 22 16:11:53] DEBU Container [db1ea49a5a5c step-c1bad912eb] removed
[Apr 22 16:11:53] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[/bin/sh -c conda install  -c 'conda-forge' -c 'bioconda'  'font-ttf-ubuntu=0.83' --strict-channel-priority -p /usr/local --copy --yes --quiet]]
[Apr 22 16:11:53] DEBU Creating container [step-aadb81a5c3]
[Apr 22 16:11:53] DEBU Created container [16ec356d04e9 step-aadb81a5c3], starting it
[Apr 22 16:11:53] DEBU Container [16ec356d04e9 step-aadb81a5c3] started, waiting for completion
[Apr 22 16:11:54] SOUT Channels:
[Apr 22 16:11:54] SOUT - conda-forge
[Apr 22 16:11:54] SOUT - bioconda
[Apr 22 16:11:54] SOUT Platform: linux-64
[Apr 22 16:12:34] SOUT Collecting package metadata (repodata.json): ...working... done
[Apr 22 16:12:35] SOUT Solving environment: ...working... done
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT ## Package Plan ##
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT environment location: /usr/local
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT added / updated specs:
[Apr 22 16:12:35] SOUT - font-ttf-ubuntu=0.83
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT The following packages will be downloaded:
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT package                    |            build
[Apr 22 16:12:35] SOUT ---------------------------|-----------------
[Apr 22 16:12:35] SOUT font-ttf-ubuntu-0.83       |       h77eed37_1         1.5 MB  conda-forge
[Apr 22 16:12:35] SOUT ------------------------------------------------------------
[Apr 22 16:12:35] SOUT Total:         1.5 MB
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT The following NEW packages will be INSTALLED:
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT font-ttf-ubuntu    conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_1
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT 
[Apr 22 16:12:35] SOUT Preparing transaction: ...working... done
[Apr 22 16:12:35] SOUT Verifying transaction: ...working... done
[Apr 22 16:12:35] SOUT Executing transaction: ...working... done
[Apr 22 16:12:35] DEBU Container [16ec356d04e9 step-aadb81a5c3] completed with exit code [0] as expected
[Apr 22 16:12:35] DEBU Container [16ec356d04e9 step-aadb81a5c3] removed
[Apr 22 16:12:35] STEP Wrap [build/dist] as [quay.io/biocontainers/font-ttf-ubuntu:0.83]
[Apr 22 16:12:35] DEBU Creating container [step-3a4cb58c9d]
[Apr 22 16:12:36] DEBU Packing succeeded

Traceback (most recent call last):
  File "/home/martin/.local/bin/mulled-build", line 8, in <module>
    sys.exit(main())
  File "/home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 612, in main
    sys.exit(mull_targets(targets, **args_to_mull_targets_kwds(args)))
  File "/home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 352, in mull_targets
    ret = involucro_context.exec_command(involucro_args)
  File "/home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 422, in exec_command
    shutil.rmtree("./build")
  File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'font-ttf-ubuntu-0.83-h77eed37_1.json'

aarch64 logs:

mgrigorov in 🌐 euler-arm-22 in /tmp/mulled via 🅒 test123 took 6s 
❯ mulled-build build 'font-ttf-ubuntu=0.83' --involucro-path (which involucro) --verbose                                                                                                                   (test123) 
Executing: /home/mgrigorov/miniconda3/envs/test123/bin/involucro -v=3 -f /home/mgrigorov/miniconda3/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,bioconda -set TARGETS=font-ttf-ubuntu=0.83 -set REPO=quay.io/biocontainers/font-ttf-ubuntu:0.83 -set BINDS=build/dist:/usr/local/ -set DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:latest -set CONDA_BIN=conda build
.[Apr 22 12:48:10] DEBU Run file [/home/mgrigorov/miniconda3/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
[Apr 22 12:48:10] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[rm -rf /data/dist]]
[Apr 22 12:48:10] DEBU Creating container [step-c8f7cfdd56]
[Apr 22 12:48:10] DEBU Created container [02ce903dd31c step-c8f7cfdd56], starting it
[Apr 22 12:48:11] DEBU Container [02ce903dd31c step-c8f7cfdd56] started, waiting for completion
[Apr 22 12:48:11] DEBU Container [02ce903dd31c step-c8f7cfdd56] completed with exit code [0] as expected
[Apr 22 12:48:11] DEBU Container [02ce903dd31c step-c8f7cfdd56] removed
[Apr 22 12:48:11] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[/bin/sh -c conda install  -c 'conda-forge' -c 'bioconda'  'font-ttf-ubuntu=0.83' --strict-channel-priority -p /usr/local --copy --yes --quiet]]
[Apr 22 12:48:11] DEBU Creating container [step-cec9f68dd6]
[Apr 22 12:48:11] DEBU Created container [bb7fdf825d43 step-cec9f68dd6], starting it
[Apr 22 12:48:11] DEBU Container [bb7fdf825d43 step-cec9f68dd6] started, waiting for completion
[Apr 22 12:48:17] SOUT Channels:
[Apr 22 12:48:17] SOUT - conda-forge
[Apr 22 12:48:17] SOUT - bioconda
[Apr 22 12:48:17] SOUT Platform: linux-aarch64
[Apr 22 12:49:05] SOUT Collecting package metadata (repodata.json): ...working... done
[Apr 22 12:49:05] SOUT Solving environment: ...working... done
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT ## Package Plan ##
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT environment location: /usr/local
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT added / updated specs:
[Apr 22 12:49:06] SOUT - font-ttf-ubuntu=0.83
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT The following packages will be downloaded:
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT package                    |            build
[Apr 22 12:49:06] SOUT ---------------------------|-----------------
[Apr 22 12:49:06] SOUT font-ttf-ubuntu-0.83       |       h77eed37_1         1.5 MB  conda-forge
[Apr 22 12:49:06] SOUT ------------------------------------------------------------
[Apr 22 12:49:06] SOUT Total:         1.5 MB
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT The following NEW packages will be INSTALLED:
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT font-ttf-ubuntu    conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_1
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT 
[Apr 22 12:49:06] SOUT Preparing transaction: ...working... done
[Apr 22 12:49:06] SOUT Verifying transaction: ...working... done
[Apr 22 12:49:06] SOUT Executing transaction: ...working... done
[Apr 22 12:49:07] DEBU Container [bb7fdf825d43 step-cec9f68dd6] completed with exit code [0] as expected
[Apr 22 12:49:07] DEBU Container [bb7fdf825d43 step-cec9f68dd6] removed
[Apr 22 12:49:07] STEP Wrap [build/dist] as [quay.io/biocontainers/font-ttf-ubuntu:0.83]
[Apr 22 12:49:07] DEBU Creating container [step-836ef90575]
[Apr 22 12:49:07] DEBU Packing succeeded

Traceback (most recent call last):
  File "/home/mgrigorov/miniconda3/bin/mulled-build", line 8, in <module>
    sys.exit(main())
  File "/home/mgrigorov/miniconda3/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 612, in main
    sys.exit(mull_targets(targets, **args_to_mull_targets_kwds(args)))
  File "/home/mgrigorov/miniconda3/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 352, in mull_targets
    ret = involucro_context.exec_command(involucro_args)
  File "/home/mgrigorov/miniconda3/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 422, in exec_command
    shutil.rmtree("./build")
  File "/home/mgrigorov/miniconda3/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/mgrigorov/miniconda3/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/mgrigorov/miniconda3/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/mgrigorov/miniconda3/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/home/mgrigorov/miniconda3/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'UbuntuMono-RI.ttf'

[martin-g]

$ ls -alR build/                                                                                                                                                                                           (test123) 
build/:
total 0
drwxr-xr-x 3 mgrigorov mgrigorov 60 Apr 22 12:48 .
drwxr-xr-x 3 mgrigorov mgrigorov 60 Apr 22 12:48 ..
drwxr-xr-x 4 root      root      80 Apr 22 12:49 dist

build/dist:
total 0
drwxr-xr-x 4 root      root       80 Apr 22 12:49 .
drwxr-xr-x 3 mgrigorov mgrigorov  60 Apr 22 12:48 ..
drwxr-xr-x 2 root      root       80 Apr 22 12:49 conda-meta
drwxr-xr-x 2 root      root      320 Apr 22 12:49 fonts

build/dist/conda-meta:
total 12
drwxr-xr-x 2 root root   80 Apr 22 12:49 .
drwxr-xr-x 4 root root   80 Apr 22 12:49 ..
-rw-r--r-- 1 root root 5740 Apr 22 12:49 font-ttf-ubuntu-0.83-h77eed37_1.json
-rw-r--r-- 1 root root  292 Apr 22 12:49 history

build/dist/fonts:
total 4308
drwxr-xr-x 2 root root    320 Apr 22 12:49 .
drwxr-xr-x 4 root root     80 Apr 22 12:49 ..
-rw-r--r-- 1 root root 356980 Nov 25 20:30 Ubuntu-BI.ttf
-rw-r--r-- 1 root root 333612 Nov 25 20:30 Ubuntu-B.ttf
-rw-r--r-- 1 root root 350444 Nov 25 20:30 Ubuntu-C.ttf
-rw-r--r-- 1 root root 409608 Nov 25 20:30 Ubuntu-LI.ttf
-rw-r--r-- 1 root root 415552 Nov 25 20:30 Ubuntu-L.ttf
-rw-r--r-- 1 root root 366992 Nov 25 20:30 Ubuntu-MI.ttf
-rw-r--r-- 1 root root 216208 Nov 25 20:30 UbuntuMono-BI.ttf
-rw-r--r-- 1 root root 191400 Nov 25 20:30 UbuntuMono-B.ttf
-rw-r--r-- 1 root root 210216 Nov 25 20:30 UbuntuMono-RI.ttf
-rw-r--r-- 1 root root 205748 Nov 25 20:30 UbuntuMono-R.ttf
-rw-r--r-- 1 root root 341324 Nov 25 20:30 Ubuntu-M.ttf
-rw-r--r-- 1 root root 386440 Nov 25 20:30 Ubuntu-RI.ttf
-rw-r--r-- 1 root root 353824 Nov 25 20:30 Ubuntu-R.ttf
-rw-r--r-- 1 root root 240020 Nov 25 20:30 Ubuntu-Th.ttf

[martin-g]

One thing I could imagine, would be package building happening with root user (i.e., overriding the entrypoint which I think we did for some reason at some point -- I'll have to check if that's still the case) and then reusing those package files as the non-root ("conda") user when creating the environment for Biocontainers (i.e., umask 0022 from root preventing writes from conda user).
^-- But that should've also happen for x86-64 builds since the setup/umasks seem to be the same on CI...

I think you were right about the cause.

[aliciaaevans]

Hi @martin-g, I'm trying to get up to speed on this issue in case I can help. Thanks for all the work you've put into it so far. This seems really tricky to track down.

I did notice that the example you made above actually seems to be a different error. It's in mulled-build where it's cleaning up any leftover build directory while the CircleCI error is in the packing step. In your example, packing succeeds. Also on my linux (amd64) local test, if I do build-and-test as the mulled-build command, which is what is happening in the CI, the error doesn't occur because involucro cleans up that dir itself. If I modify the invfile.lua to ls that directory, there is no LICENCE.txt (your ls -alR build/ shows that as well) and so no LICENCE.txt error. But it is there if I do the bioconductor-biocgenerics build via bioconda-utils.

Have you seen the error in CircleCI for any other files except this particular LICENCE.txt? I noticed that if I install font-ttf-ubuntu from conda-forge and look in my pkgs, the file shows -rw-------. Looks like that might be coming from the source that way. unzip -Z 0cef8205-ubuntu-font-family-0.83.zip shows all the files permissions are the same, though. I'm not sure why the LICENCE.txt would have the issue and not the fonts themselves, though.

I'll let you know if I figure anything out.

[martin-g]

Hi @aliciaaevans !
Thanks for helping here!

The error in Bioconda recipes build is always the same - this licence from font-ttf-ubuntu.
If font-ttf-ubuntu is not a dependency then the mulled build is OK.
I also wasn't able to find where this file comes from! It is not in the sources tarball of this package!

[aliciaaevans]

I made a comment on the original PR. It's probably better to continue the discussion there. bioconda/bioconda-recipes#46177 (comment)