Conda-forge test failures redux #2562
Labels
Milestone
Comments
|
Attn: @DennisHeimbigner |
5 tasks
|
Unfortunately, many of our early tests in nc_test, nc_test4, and ncdump
especially
use files supposedly created in tests presumed to have run previously.
Doing parallel make tests has completely screwed this up.
The only practical solution I can figure are:
1. give up on parallel testing or
2. build a special global test script for these directories that
guarantees serial execution.
3. ?
=Dennis
In 11/21/2022 4:06 PM, Ryan May wrote:
…
Found this when building 4.9.1rc2
<conda-forge/libnetcdf-feedstock#154>. The
relevant build log lines are:
|2022-11-21T22:42:54.5509815Z Start 58: ncdump_tst_nccopy4
2022-11-21T22:42:54.5515223Z 2022-11-21T22:42:54.5521378Z 58: Test
command: /usr/bin/bash "-c" "export srcdir=$SRC_DIR/ncdump;export
TOPSRCDIR=$SRC_DIR;$SRC_DIR/build-shared/ncdump/tst_nccopy4.sh
tst_nccopy4.sh " 2022-11-21T22:42:54.5527574Z 58: Working Directory:
$SRC_DIR/build-shared/ncdump 2022-11-21T22:42:54.5533277Z 58: Test
timeout computed to be: 1500 2022-11-21T22:42:54.5801585Z 58:
2022-11-21T22:42:54.5807946Z 58: *** Testing groups.
2022-11-21T22:42:54.5814936Z 58: *** creating nested group file
tst_group_data.nc...ok. 2022-11-21T22:42:54.5821116Z 58: *** Tests
successful! 2022-11-21T22:42:54.5961033Z 58:
2022-11-21T22:42:54.5961571Z 58: *** Testing enums.
2022-11-21T22:42:54.5962067Z 58: *** creating enum test file
tst_enum_data.nc...ok. 2022-11-21T22:42:54.5962545Z 58: *** Tests
successful! 2022-11-21T22:42:54.6100110Z 58:
2022-11-21T22:42:54.6100746Z 58: *** Testing compound types.
2022-11-21T22:42:54.6101274Z 58: *** creating compound test file
tst_comp.nc...ok. 2022-11-21T22:42:54.6101750Z 58: *** Tests
successful! 2022-11-21T22:42:54.6238944Z 58:
2022-11-21T22:42:54.6239360Z 58: *** Testing compound types some more.
2022-11-21T22:42:54.6239893Z 58: *** creating another compound test
file tst_comp2.nc...ok. 2022-11-21T22:42:54.6240378Z 58: *** Tests
successful! 2022-11-21T22:42:54.6251984Z 58:
2022-11-21T22:42:54.6259605Z 58: *** Testing netCDF-4 features of
nccopy on ncdump/*.nc files 2022-11-21T22:42:54.6268757Z 58: *** Test
nccopy tst_comp.nc copy_of_tst_comp.nc ...
2022-11-21T22:42:54.6669010Z 58: *** compare with copy_of_tst_comp.cdl
2022-11-21T22:42:54.6696766Z 58: *** Test nccopy tst_comp2.nc
copy_of_tst_comp2.nc ... 2022-11-21T22:42:54.7220679Z 58: *** compare
with copy_of_tst_comp2.cdl 2022-11-21T22:42:54.7251077Z 58: *** Test
nccopy tst_enum_data.nc copy_of_tst_enum_data.nc ...
2022-11-21T22:42:54.7643154Z 58: *** compare with
copy_of_tst_enum_data.cdl 2022-11-21T22:42:54.7671076Z 58: *** Test
nccopy tst_fillbug.nc copy_of_tst_fillbug.nc ...
2022-11-21T22:42:54.8070545Z test 42 2022-11-21T22:42:54.8070955Z
Start 42: ncdump_tst_netcdf4 2022-11-21T22:42:54.8071197Z
2022-11-21T22:42:54.8072694Z 42: Test command: /usr/bin/bash "-c"
"export srcdir=$SRC_DIR/ncdump;export
TOPSRCDIR=$SRC_DIR;$SRC_DIR/build-shared/ncdump/tst_netcdf4.sh
tst_netcdf4.sh " 2022-11-21T22:42:54.8073642Z 42: Working Directory:
$SRC_DIR/build-shared/ncdump 2022-11-21T22:42:54.8074331Z 42: Test
timeout computed to be: 1500 2022-11-21T22:42:54.8089713Z 58: ***
compare with copy_of_tst_fillbug.cdl 2022-11-21T22:42:54.8120318Z 58:
*** Test nccopy tst_group_data.nc copy_of_tst_group_data.nc ...
2022-11-21T22:42:54.8203182Z 42: 2022-11-21T22:42:54.8204441Z 42: ***
Testing ncgen and ncdump for netCDF-4 format.
2022-11-21T22:42:54.8573796Z 58: *** compare with
copy_of_tst_group_data.cdl 2022-11-21T22:42:54.8616944Z 58: *** Test
nccopy tst_nans.nc copy_of_tst_nans.nc ...
2022-11-21T22:42:54.8953182Z 58: *** compare with copy_of_tst_nans.cdl
2022-11-21T22:42:54.8975355Z 42: *** Creating test output
tst_netcdf4_c0.nc. 2022-11-21T22:42:54.8990426Z 58: *** Test nccopy
tst_opaque_data.nc copy_of_tst_opaque_data.nc ...
2022-11-21T22:42:54.9342465Z 42: *** Testing that program tst_h_rdc0
can read tst_netcdf4_c0.nc. 2022-11-21T22:42:54.9396887Z 58: ***
compare with copy_of_tst_opaque_data.cdl 2022-11-21T22:42:54.9426702Z
58: *** Test nccopy tst_solar_1.nc copy_of_tst_solar_1.nc ...
2022-11-21T22:42:54.9487930Z 42: 2022-11-21T22:42:54.9488529Z 42: ***
Checking HDF5 file c0.nc. 2022-11-21T22:42:54.9489005Z 42: ***
Checking HDF5 objcts...ok. 2022-11-21T22:42:54.9493342Z 42: *** Tests
successful! 2022-11-21T22:42:54.9493902Z 42: *** Running
tst_create_files.c to create test files. 2022-11-21T22:42:54.9585756Z
42: 2022-11-21T22:42:54.9586277Z 42: *** Creating test files for
ncdump. 2022-11-21T22:42:54.9586932Z 42: *** creating nested group
file tst_solar_1.nc...Sorry! Unexpected result,
$SRC_DIR/ncdump/tst_create_files.c, line: 71
2022-11-21T22:42:54.9607207Z 49/229 Test #42: ncdump_tst_netcdf4
....................***Failed 0.15 sec 2022-11-21T22:42:54.9607700Z
2022-11-21T22:42:54.9608794Z *** Testing ncgen and ncdump for netCDF-4
format. 2022-11-21T22:42:54.9609342Z *** Creating test output
tst_netcdf4_c0.nc. 2022-11-21T22:42:54.9609907Z *** Testing that
program tst_h_rdc0 can read tst_netcdf4_c0.nc.
2022-11-21T22:42:54.9610229Z 2022-11-21T22:42:54.9610601Z *** Checking
HDF5 file c0.nc. 2022-11-21T22:42:54.9611028Z *** Checking HDF5
objcts...ok. 2022-11-21T22:42:54.9611448Z *** Tests successful!
2022-11-21T22:42:54.9611900Z *** Running tst_create_files.c to create
test files. 2022-11-21T22:42:54.9612197Z 2022-11-21T22:42:54.9612567Z
*** Creating test files for ncdump. 2022-11-21T22:42:54.9614238Z ***
creating nested group file tst_solar_1.nc...Sorry! Unexpected result,
$SRC_DIR/ncdump/tst_create_files.c, line: 71
2022-11-21T22:42:54.9719951Z 58: $SRC_DIR/build-shared/ncdump/ncdump:
tst_solar_1.nc: NetCDF: Unknown file format
2022-11-21T22:42:54.9748238Z 50/229 Test #58: ncdump_tst_nccopy4
....................***Failed 0.42 sec |
Full log
<https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/616373/logs/64>
is available if needed; here I snipped out unrelated tests.
Again, the fact that the tests failing are running concurrently is
suspicious, so poke around and find that both |ncdump/tst_netcdf4.sh|
and |ncdump/tst_nccopy4.sh| reference |tst_solar_1.nc| (and cdl) as
well as |tst_solar_2.nc| (and cdl).
I also see mention of |tst_solar| across the codebase:
|ncdump/tst_create_files.c 16:#define FILE_NAME_1 "tst_solar_1.nc"
17:#define FILE_NAME_2 "tst_solar_2.nc" 18:#define FILE_NAME_CMP
"tst_solar_cmp.nc" ncdump/tst_nccopy4.sh 23: tst_group_data tst_nans
tst_opaque_data tst_solar_1 tst_solar_2 24: tst_solar_cmp
tst_special_atts' ncdump/tst_ncgen_shared.sh 90:ref_tst_solar_1 \
91:ref_tst_solar_2 \ 140:BIGTESTS4="ref_tst_solar_1"
ncdump/tst_netcdf4.sh 47:${NCDUMP} tst_solar_1.nc | sed 's/e+0/e+/g' >
tst_solar_1.cdl ; ERR 48:diff -b tst_solar_1.cdl
$srcdir/ref_tst_solar_1.cdl ; ERR 49:${NCDUMP} tst_solar_2.nc | sed
's/e+0/e+/g' > tst_solar_2.cdl ; ERR 50:diff -b tst_solar_2.cdl
$srcdir/ref_tst_solar_2.cdl ; ERR ncgen/ctests.sh 84:ref_tst_solar_1 \
85:ref_tst_solar_2 \ 104:BIGTESTS4="ref_tst_solar_1" ncgen/ftests.sh
85:ref_tst_solar_1 \ 86:ref_tst_solar_2 \
105:BIGTESTS4="ref_tst_solar_1" ncgen/jtests.sh 84:ref_tst_solar_1 \
85:ref_tst_solar_2 \ 108:BIGTESTS4="ref_tst_solar_1"
nczarr_test/run_ncgen4.sh |
Is there a better way to approach this testing than playing "whack a
mole" by manually renaming files? This is proving pretty brittle.
—
Reply to this email directly, view it on GitHub
<#2562>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG47W5VV5RD4E5KHXO7NT3WJP6BFANCNFSM6AAAAAASHE2OM4>.
You are receiving this because you are subscribed to this
thread.Message ID: ***@***.***>
|
|
If you used a random temp directory for the intermediate test output that would let the scripts avoid stomping on each other. |
|
Adding another case here, there's also a conflict between those two same test scripts and |
|
And then another with |
|
Ah, I see the reason we don't see such issues here isn't that we're not relying on it, it's that they're run with |
|
I think we can kill the interference cases. The real problem is |
|
As long as we know which order the tests need to be run in, the mechanisms to ensure "test A" finishes before "test B" is run exist in both autotools and cmake. We already make copious use of them. So adding more shouldn't be a problem as we establish what depends on what. |
|
I'd also argue that relying on side effects of tests instead of having that as an explicit separate dependency (e.g. a fixture/setup) is a problem with the tests that should at some point be addressed rather than continually worked around. Just because they're tests doesn't mean they're any less deserving of programmers' love. 😆 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Found this when building 4.9.1rc2. The relevant build log lines are:
Full log is available if needed; here I snipped out unrelated tests.
Again, the fact that the tests failing are running concurrently is suspicious, so poke around and find that both
ncdump/tst_netcdf4.shandncdump/tst_nccopy4.shreferencetst_solar_1.nc(and cdl) as well astst_solar_2.nc(and cdl).I also see mention of
tst_solaracross the codebase:Is there a better way to approach this testing than playing "whack a mole" by manually renaming files? This is proving pretty brittle.
The text was updated successfully, but these errors were encountered: