Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sciunit fails with invalid literal for int() with base 10 #31

Closed
charmoniumQ opened this issue Dec 18, 2023 · 6 comments · Fixed by #35
Closed

Sciunit fails with invalid literal for int() with base 10 #31

charmoniumQ opened this issue Dec 18, 2023 · 6 comments · Fixed by #35

Comments

@charmoniumQ
Copy link
Contributor

Using podman run -it --rm ubuntu:22.04 /bin/bash,

root@86d0ed87d5be:/# apt update && apt install -y python3 python3-pip
...
root@86d0ed87d5be:/# export PATH=/root/.local/bin:$PATH
root@86d0ed87d5be:/# pip install --user sciunit2
...
root@86d0ed87d5be:/# sciunit create test3
Opened empty sciunit at /root/sciunit/test3
root@86d0ed87d5be:/# sciunit exec /bin/true
... lots of hex numbers ... - Committing Traceback (most recent call last):
  File "/root/.local/bin/sciunit", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.10/site-packages/sciunit2/cli.py", line 64, in main
    _main(sys.argv[1:])
  File "/root/.local/lib/python3.10/site-packages/sciunit2/cli.py", line 101, in _main
    r = cmd.run(args[1:])
  File "/root/.local/lib/python3.10/site-packages/sciunit2/command/exec_/__init__.py", line 36, in run
    return self.do_commit('cde-package', rev, emgr, repo)
  File "/root/.local/lib/python3.10/site-packages/sciunit2/command/mixin.py", line 18, in do_commit
    sz = repo.checkin(rev, pkgdir, sp)
  File "/root/.local/lib/python3.10/site-packages/sciunit2/version_control.py", line 43, in checkin
    return int(err)
ValueError: invalid literal for int() with base 10: b'tar: cde-package/cde-root/usr/bin: file changed as we read it\ntar: cde-package/cde-root/usr/lib/x86_64-linux-gnu: file changed as we read it\n3256320\n'
@TanuMalik
Copy link
Member

Did you also try the same sequence on Ubuntu 20.04? For us, it works.

Ubuntu 22.04 has been giving us some issues for a while. We are working on it during the break. So bear with us.

A little more explanation on what you are trying to achieve will allow us to help you better.

@charmoniumQ
Copy link
Contributor Author

A little more explanation on what you are trying to achieve will allow us to help you better.

I am just trying to use sciunit in a basic example: create test3 and execute /bin/true in test3 (/bin/true should just exit 0). I'm only using Podman, so I can test on a blank-slate environment, and hopefully those errors will be reproducible.

I tried in 20.04 and 18.04, same error. It can be seen executing just one line of code code (~3 mins):

podman run -it --rm ubuntu:20.04 /bin/bash -c 'apt update && apt install -y python3 python3-pip && pip3 install --user sciunit2 && /root/.local/bin/sciunit create test3 && /root/.local/bin/sciunit exec /bin/true'

Did you also try the same sequence on Ubuntu 20.04? For us, it works.

So the exact sequence of commands at the beginning works in Ubuntu 20.04? Perhaps errors are not as reproducible in Podman as I thought.

@TanuMalik
Copy link
Member

TanuMalik commented Jan 19, 2024

Is your Podman configured to execute ptrace?
We can look into Podman configuration and post an update.

What is the H/W on which Podman is working on? ARMs won't work, only Intel CPUs

Also, does the same sequence work for you on an Ubuntu 20.04 machine--may be a nano instance of AWS or a Docker container?

I didn't quite get what you mean by "exact sequence of commands at the beginning works in Ubuntu 20.04?
The error certainly seems to be a conflict with the base image of Podman and sciunit library conflict.

@moazreyad
Copy link
Contributor

Actually, to be more accurate, I would say the host operating system must be Linux, Windows, or Intel-based macOS regardless of the actual hardware CPU.
If someone manages to install (a supported version of) Linux on macOS M1/M2 (Maybe with something like VirtualBox or Parallels, or natively using some hacks), sciunit could probably work.

@charmoniumQ
Copy link
Contributor Author

charmoniumQ commented Jan 19, 2024

Is your Podman configured to execute ptrace?

I believe so, because the following strace command works:

podman run -it --rm ubuntu:22.04 /bin/bash -c 'apt update && apt install -y strace && strace ls'

What is the H/W on which Podman is working on? ARMs won't work, only Intel CPUs

My host (outside the Podman container) is NixOS Linux, Intel x86_64.

$ uname -a
Linux laptop 6.1.64 #1-NixOS SMP PREEMPT_DYNAMIC Tue Nov 28 17:07:23 UTC 2023 x86_64 GNU/Linux

$ lscpu | head
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      39 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             8
On-line CPU(s) list:                0-7
Vendor ID:                          GenuineIntel
Model name:                         11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
CPU family:                         6
Model:                              140

Also, does the same sequence work for you on an Ubuntu 20.04 machine--may be a nano instance of AWS or a Docker container?

Very interesting, this podman run ... sciunit... command succeeds in AWS but not on my local machine.

The error certainly seems to be a conflict with the base image of Podman and sciunit library conflict.

That command calls for a "vanilla" base image of Ubuntu.


Anyway, now that I know this error does not occur on AWS, I'm going to try more debugging on my side.

@charmoniumQ
Copy link
Contributor Author

I think the underlying reason is actually to do with files being changed while the tar is running. The output of tar is:

tar: cde-package/cde-root/usr/bin: file changed as we read it
tar: cde-package/cde-root/usr/lib/x86_64-linux-gnu: file changed as we read it
3256320

Perhaps the virtual file system provided by Podman is slower than the native file system used in other places.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants