Failed builds with dind 20.10.9 and/or sysbox 0.4.1

[nudgegoonies]

Since the update to dind 20.10.9 and/or sysbox 0.4.1 we have the problem, that installation of the python gem cyclonedx-bom fails when building a stretch or buster image within gitlab-ci via dind. Building a bullseye image works. I am sorry, but i cannot provide our whole project with Dockerfile, because it contains lots of internal tools, files etc. Providing a minimal example is useless because it works installing the gem. I checked all 4 of dind version and sysbox version combinations:

• Works: Docker 20.10.9 on Host / Docker 20.10.7 as dind / sysbox-ce 0.4.0
• Fails: Docker 20.10.9 on Host / Docker 20.10.9 as dind / sysbox-ce 0.4.0
• Fails: Docker 20.10.9 on Host / Docker 20.10.7 as dind / sysbox-ce 0.4.1
• Fails: Docker 20.10.9 on Host / Docker 20.10.9 as dind / sysbox-ce 0.4.1
  Sorry, that i cannot provide an example. At least i can provide sysbox-fs log entries that never appeared before:

level=warning msg="Unexpected error during NotifReceive() execution (no such file or directory) on fd 10 pid 23246"

By the way, i see many of these in the logs but they appeard before and after the updates and they seem to have no negative effect:

time="2021-10-14 12:59:39" level=warning msg="Unexpected error during NotifRespond() execution (no such file or directory) on fd 7547 pid 23546"
time="2021-10-14 12:59:06" level=info msg="TOCTOU check failed on fd 7547 pid 26894 cntr 085edf9ae473: req.Id is no longer valid (no such file or directory)"

Please see the difference between NotifReceive (new since the update that breaks our docker gitlab-ci pipelines) and NotifRespond that also happened before.

[ctalledo]

Hi @nudgegoonies , thanks for filing this issue.

Since the update to dind 20.10.9 and/or sysbox 0.4.1 we have the problem, that installation of the python gem cyclonedx-bom fails when building a stretch or buster image within gitlab-ci via dind

What is the failure message you are seeing?

Is that error message the same on all failures?

Is the problem 100% reproducible?

The warning messages in the log may not be related to the failure, those sometimes occur when Sysbox containers are stopped suddenly.

[nudgegoonies]

Please excuse my long delay. The circumstances are reproducible but i was not able to create a minimal example. I am going to test everything again. Now on Debian 11, Docker 20.10.12 and sysbox 0.4.1.

[nudgegoonies]

The combination of Debian 11, Docker 20.10.12 and sysbox 0.4.1 does not fix the strange Python problems.

• Works: Docker 20.10.12 on Host / Docker 20.10.7 as dind / sysbox-ce 0.4.0
• Fails: Docker 20.10.12 on Host / Docker 20.10.12 as dind / sysbox-ce 0.4.0
• Fails: Docker 20.10.12 on Host / Docker 20.10.7 as dind / sysbox-ce 0.4.1
• Fails: Docker 20.10.12 on Host / Docker 20.10.12 as dind / sysbox-ce 0.4.1

[ctalledo]

Hi @nudgegoonies, thanks for the update. I think this may have been fixed recently via this PR. If possible, could you try to build Sysbox from source and retry. If you need some help on this please let me know.

[ctalledo]

Closing as issue is stale now; please re-open if you hit this problem again, but try with the upcoming v0.5.0 release before reopening.

[nudgegoonies]

This issue sounds very similar to this one: #485

[nudgegoonies]

Fixed with sysbox 0.5.0 on docker 20.10.14 and with docker-dind 20.10.7 and docker-dind 20.10.14