test_benchmarks.py Fatal Python error: Aborted

[yuzibo]

Hi,

There is one ftbfs issue on Debian:

...
I: pybuild base:311: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_plotpy/build; python3.12 -m pytest 
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.1.2, pluggy-1.5.0
rootdir: /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_plotpy/build
configfile: pyproject.toml
collected 253 items

plotpy/tests/benchmarks/test_benchmarks.py Fatal Python error: Aborted

Current thread 0x00007fc4fce9d040 (most recent call first):
  File "/usr/lib/python3/dist-packages/guidata/__init__.py", line 31 in qapplication
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_plotpy/build/plotpy/tests/benchmarks/test_benchmarks.py", line 133 in test_benchmarks
  File "/usr/lib/python3/dist-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1779 in runtest
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 172 in pytest_runtest_call
  File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
...

The full log is here and I tested it on 2.3.2 failed also.

[PierreRaybaut]

Can't reproduce on all my test platforms (CentOS, Ubuntu, ...).

See here for example.

What would you recommend as the easiest way of reproducing this?

[yuzibo]

Thanks for your reply.

First I am not the maintainer of the package but just as a member of Debian Python Team to try to fix the ftbfs issue.

Because the original maintainer of the package did not create a repo for it, so I just create its' debian repo to debug the issue easily for us.:)

Before your deep debug on Debian build environment, I suspect the build dependences of plotpy that has issue.

For 2.2.0, the debian/control has the build dependences is below:

Build-Depends: debhelper-compat (= 13),
               dh-python,
               cython3,
               python3-all,
               python3-guidata,
               python3-numpy,
               python3-scipy,
               python3-qwt,
               python3-pytest,
               python3-all-dev,

Using the same B-D, the 2.3.2 will be failed with the same reason.

As I update the debian/control, now it will be hang without anything output.

I: pybuild base:311: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_plotpy/build; python3.12 -m pytest
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0
rootdir: /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_plotpy/build
configfile: pyproject.toml
plugins: xvfb-3.0.0
collected 287 items

E: ABORT: Received INT signal (requesting cleanup and shutdown)
...

[PierreRaybaut]

Thanks for the additional details.
Investigating this will require some time as we can't do it with any of out current platforms (we will have to create a Debian VM dedicated to this and try to reproduce the issue).

I'm adding this task to the to-do list with the highest priority.

[PierreRaybaut]

Hi @yuzibo ,

Could you please help me reproduce this issue?

I have created a VM based on the latest Debian Unstable ISO.

Here is what I've done.
And it's working without any error....

# At this point, this is a fresh install of Debian Unstable
nano /etc/apt/sources.list  # to add `deb http://deb.debian.org/debian/ unstable main non-free-firmware`
sudo apt-get update
git clone https://github.com/PlotPyStack/PlotPy.git
cd PlotPy/
sudo apt-get install dh-python cython3 python3-all python3-guidata python3-numpy python3-scipy python3-qwt python3-pytest python3-all-dev
python3 setup.py build_ext --inplace
pytest

[yuzibo]

Hi @PierreRaybaut, thanks for your work here, appreciate it!

To reproduce this issue, we have to setup Debian packaging env, let me try simply to setup it.

On Debian unstable:

sudo apt install sbuild mmdebstrap

sudo sbuild-createchroot --debootstrap=mmdebstrap --arch=amd64 \
        --include=ca-certificates  \
        --make-sbuild-tarball=/srv/sid-amd64-sbuild.tgz \
        sid /tmp/chroots/sid-amd64-sbuild/ \
         http://deb.debian.org/debian/

Once got the log, the schroot was created successfully.

...
Next, copy the example sbuildrc file to the home directory of each user and
set the variables for your system:

cp /usr/share/doc/sbuild/examples/example.sbuildrc /home/vimer/.sbuildrc

Now try a build:

cd /path/to/source
sbuild-update -ud <distribution>
(or "sbuild-apt <distribution> apt-get -f install"
first if the chroot is broken)

sbuild -d <distribution> <package>_<version>

If target for 2.2.0, see #1071775, just

# first put deb-src URL into your `/etc/apt/sources.list` on VM host to get Debian source code
# deb-src http://deb.debian.org/debian/  sid main
apt source
apt source plotpy
 $: ls
plotpy-2.2.0  plotpy_2.2.0-3.debian.tar.xz  plotpy_2.2.0-3.dsc  plotpy_2.2.0.orig.tar.gz
cd plotpy-2.2.0
sudo sbuild -c sid-amd64-sbuild 

Then you should see the result was the same with #1071775.

If target for 2.3.2, please git clone my repo salsa first:

git clone https://salsa.debian.org/vimerbf-guest/plotpy
(add your pubkey into gitlab)

cd plotpy
git checkout upstream && git checkout master

 gbp buildpackage --git-tarball-dir=../ --git-upstream-tree=upstream --git-upstream-branch=upstream  --git-builder=sbuild  -c sid-amd64-sbuild --no-clean-source --source  --git-debian-branch=debian/main --git-export-dir=../build-area/ --git-ignore-new  --verbose

you will have to install some gbp something here.

If debug the package env during packaging, you can:

 sudo schroot -c sid-amd64-sbuild
apt update
apt install devscripts 
 mk-build-deps --install (to install depences from debian/control file)
dpkg-buildpackage --sanitize-env -us -uc -b -rfakeroot (to build the package again)

Sorry, maybe there are too many Debian stuff for you. Thank you very much for this again. Please let me know if any issue and next two days I should have some time on this.

[eamanu]

Hi,

Seems to be that iterator is deprecated. Similar issue here iree-org/iree@88b1d4d

[PierreRaybaut]

Hi,

Seems to be that iterator is deprecated. Similar issue here iree-org/iree@88b1d4d

Thanks for the feedback.
(But I'll try and reproduce the issue before going further.)

[PierreRaybaut]

Hi @PierreRaybaut, thanks for your work here, appreciate it!

To reproduce this issue, we have to setup Debian packaging env, let me try simply to setup it.

On Debian unstable:

sudo apt install sbuild mmdebstrap

sudo sbuild-createchroot --debootstrap=mmdebstrap --arch=amd64 \
        --include=ca-certificates  \
        --make-sbuild-tarball=/srv/sid-amd64-sbuild.tgz \
        sid /tmp/chroots/sid-amd64-sbuild/ \
         http://deb.debian.org/debian/

Once got the log, the schroot was created successfully.

...
Next, copy the example sbuildrc file to the home directory of each user and
set the variables for your system:

cp /usr/share/doc/sbuild/examples/example.sbuildrc /home/vimer/.sbuildrc

Now try a build:

cd /path/to/source
sbuild-update -ud <distribution>
(or "sbuild-apt <distribution> apt-get -f install"
first if the chroot is broken)

sbuild -d <distribution> <package>_<version>

If target for 2.2.0, see #1071775, just

# first put deb-src URL into your `/etc/apt/sources.list` on VM host to get Debian source code
# deb-src http://deb.debian.org/debian/  sid main
apt source
apt source plotpy
 $: ls
plotpy-2.2.0  plotpy_2.2.0-3.debian.tar.xz  plotpy_2.2.0-3.dsc  plotpy_2.2.0.orig.tar.gz
cd plotpy-2.2.0
sudo sbuild -c sid-amd64-sbuild 

Then you should see the result was the same with #1071775.

If target for 2.3.2, please git clone my repo salsa first:

git clone https://salsa.debian.org/vimerbf-guest/plotpy
(add your pubkey into gitlab)

cd plotpy
git checkout upstream && git checkout master

 gbp buildpackage --git-tarball-dir=../ --git-upstream-tree=upstream --git-upstream-branch=upstream  --git-builder=sbuild  -c sid-amd64-sbuild --no-clean-source --source  --git-debian-branch=debian/main --git-export-dir=../build-area/ --git-ignore-new  --verbose

you will have to install some gbp something here.

If debug the package env during packaging, you can:

 sudo schroot -c sid-amd64-sbuild
apt update
apt install devscripts 
 mk-build-deps --install (to install depences from debian/control file)
dpkg-buildpackage --sanitize-env -us -uc -b -rfakeroot (to build the package again)

Sorry, maybe there are too many Debian stuff for you. Thank you very much for this again. Please let me know if any issue and next two days I should have some time on this.

Hey @yuzibo ,

So I've tried to follow the steps of the first part of your instructions.
But I'm stuck here:

pierre@DebianTesting:~/plotpy-2.2.0$ sudo sbuild -c sid-amd64-sbuild --no-clean
dpkg-source: info: using options from plotpy-2.2.0/debian/source/options: --extend-diff-ignore=^[^/]+.(egg-info|dist-info)/
dpkg-source: info: using options from plotpy-2.2.0/debian/source/options: --extend-diff-ignore=^[^/]+.(egg-info|dist-info)/
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building plotpy using existing ./plotpy_2.2.0.orig.tar.gz
dpkg-source: info: building plotpy in plotpy_2.2.0-3.debian.tar.xz
dpkg-source: info: building plotpy in plotpy_2.2.0-3.dsc
dpkg-source: info: using options from plotpy-2.2.0/debian/source/options: --extend-diff-ignore=^[^/]+.(egg-info|dist-info)/
sbuild (Debian sbuild) 0.85.10 (30 May 2024) on DebianTesting.myguest.virtualbox.org

+==============================================================================+
| plotpy 2.2.0-3 (amd64)                       Tue, 18 Jun 2024 08:39:13 +0000 |
+==============================================================================+

Package: plotpy
Version: 2.2.0-3
Source Version: 2.2.0-3
Distribution: unstable
Machine Architecture: amd64
Host Architecture: amd64
Build Architecture: amd64
Build Type: binary

I: NOTICE: Log filtering will replace 'var/run/schroot/mount/sid-amd64-sbuild-23cfc17d-90c9-4035-a1fe-3ee1e560c658' with '<<CHROOT>>'
I: NOTICE: Log filtering will replace 'build/plotpy-qHDBUm/resolver-0Sj5LE' with '<<RESOLVERDIR>>'

+------------------------------------------------------------------------------+
| Update chroot                                                                |
+------------------------------------------------------------------------------+

Get:1 http://deb.debian.org/debian sid InRelease [198 kB]
/usr/bin/apt-key: 95: cannot create /dev/null: Permission denied
/usr/bin/apt-key: 95: cannot create /dev/null: Permission denied
/usr/bin/apt-key: 95: cannot create /dev/null: Permission denied
E: gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
Sub-process apt-key returned an error code (29)Err:1 http://deb.debian.org/debian sid InRelease
  gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
Reading package lists...
W: GPG error: http://deb.debian.org/debian sid InRelease: gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
E: The repository 'http://deb.debian.org/debian sid InRelease' is not signed.
E: apt-get update failed

+------------------------------------------------------------------------------+
| Summary                                                                      |
+------------------------------------------------------------------------------+

Build Architecture: amd64
Build Type: binary
Build-Space: 0
Build-Time: 0
Distribution: unstable
Fail-Stage: apt-get-update
Host Architecture: amd64
Install-Time: 0
Job: /home/pierre/plotpy_2.2.0-3.dsc
Machine Architecture: amd64
Package: plotpy
Package-Time: 0
Source-Version: 2.2.0-3
Space: 0
Status: failed
Version: 2.2.0-3
--------------------------------------------------------------------------------
Finished at 2024-06-18T08:39:13Z
Build needed 00:00:00, 0k disk space
E: apt-get update failed

Any idea?
(I've tried a lot of things, without success)
(My VM is quite clean: fresh Debian 12 install, sources.list with only "sid main" (deb and deb-src))

[yuzibo]

hi,

Could you try it with the instructions?

From man null:

   These devices are typically created by:

          mknod -m 666 /dev/null c 1 3
          mknod -m 666 /dev/zero c 1 5
          chown root:root /dev/null /dev/zero
These commands would have to be executed in the chroot environment.

To enter into chroot env, we have to

 sudo sbuild-shell  sid-amd64-sbuild

In fact, I do not remember I have faced the issue on my physical host. but maybe we can try it.

[PierreRaybaut]

Just did it:

pierre@DebianTesting:~/plotpy-2.2.0$ sudo sbuild-shell  sid-amd64-sbuild
[sudo] password for pierre: 
I: /bin/sh
# mknod -m 666 /dev/null c 1 3
mknod: /dev/null: File exists
# mknod -m 666 /dev/zero c 1 5
# chown root:root /dev/null /dev/zero
# exit

But, unfortunately, same result as before.

[PierreRaybaut]

There may be an issue with the permissions of /dev/null, but I don't know how to fix this permanently:

pierre@DebianTesting:~/plotpy-2.2.0$ sudo sbuild-shell  sid-amd64-sbuild
I: /bin/sh
# ls -l /dev/null     
-rw-r--r-- 1 root root 0 Jun 18 07:52 /dev/null

[yuzibo]

I think we can change permissions of /dev/null just like we do on a normal system then exit.

Under chroot with sbuild-shell, we can save the change I understand.

vimer@dev:~/dev$ sudo sbuild-shell  sid-amd64-sbuild
I: /bin/sh
# ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Apr  2 01:10 /dev/null
#

[PierreRaybaut]

Sorry, my bad, that's because I needed to remove and recreate it to change the first "-" into "c" in permissions:

pierre@DebianTesting:~/plotpy-2.2.0$ sudo sbuild-shell  sid-amd64-sbuild
I: /bin/sh
# ls -l /dev/null     
-rw-r--r-- 1 root root 0 Jun 18 07:52 /dev/null
# rm -f /dev/null
# mknod /dev/null c 1 3
# chmod 666 /dev/null
# ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Jun 18 12:31 /dev/null
# exit

So now, the building process is running smoothly... (in progress)

[PierreRaybaut]

OK, so now I have exactly the same output as you first posted.

Now I'm wondering how I could debug this.
Is there a way to execute any given Python script from the repository inside that environment, in order to reproduce the failure and debug it?