current main branch fails make -j distcheck due to missing dap4_test subdirectories in the autotools build files

[edwardhartnett]

When I build like:

ed@koko:~/netcdf-c$ export LD_LIBRARY_PATH=/usr/local/hdf5-1.12.2/lib
ed@koko:~/netcdf-c$ export CPPFLAGS="-I/usr/local/hdf5-1.12.2/include -I/usr/local/szip-2.1.1/include"
ed@koko:~/netcdf-c$ export LDFLAGS="-L/usr/local/hdf5-1.12.2/lib -L/usr/local/szip-2.1.1/lib"
ed@koko:~/netcdf-c$ export DISTCHECK_CONFIGURE_FLAGS=--disable-dap-remote-tests
ed@koko:~/netcdf-c$ ./configure --disable-dap-remote-tests

Then make -j check works. (Yea!)

But make -j distcheck does not work:

make[5]: Entering directory '/home/ed/netcdf-c/netcdf-c-4.9.1-development/_build/sub/dap4_test'
FAIL: test_meta.sh
FAIL: test_parse.sh
FAIL: test_raw.sh
FAIL: test_fillmismatch.sh
FAIL: test_data.sh

Note that I set DISTCHECK_CONFIGURE_FLAGS to --disable-dap-remote-tests, so should these tests even be running?

[edwardhartnett]

I also fails make distcheck, so this is not a parallel build/test issue.

When I look in the log for the failed tests I see:

../../../dap4_test/test_parse.sh: 27: cd: can't cd to /home/ed/netcdf-c/netcdf-c-4.9.1-development/_build/sub/dap4_test/baselinethredds
FAIL test_parse.sh (exit status: 2)

The problem here is the dap4_test/baseline (and perhaps other directories under dap4_test) does not have a Makefile.am.

If you just want to distribute the files in this directory for testing, we need a Makefile.am which lists all the files as EXTRA_DIST.

I will submit a PR and see if that fixes my distcheck...

[edwardhartnett]

...BTW it seems like this problem would also be present in the tarball release. That is, the tarball release, if constructed with make dist, will only contain the files listed in the Makefile.am files.

@DennisHeimbigner unfortunately we cannot list subdirectories in EXTRA_DIST, only files. ;-)

[edwardhartnett]

I have a fix to this but I have to turn off nczarr_test/run_scalar.sh for it to work. See #2545

[DennisHeimbigner]

I am doing a distcheck right now, and it looks like I can
put directories in EXTRA_DIST.
There was, however, an error in d4test_common.sh that specified
an incorrect path to one of those directories.

[WardF]

This path issue is fixed in the 4.9.1 wellspring branch and will be back propagated to main in due time. I'll take a look at doing that early, but no changes to fix it are needed now. I'll be back in town next Tuesday and will bring this fix to Main if nothing else.

[DWesl]

Putting globs or directories in an Autotools file list is discouraged, but putting files in a subdirectory is perfectly allowed, and sometimes encouraged.

[DennisHeimbigner]

I am surprised; Autotools seems to handle directories with no problems.
What is the argument against them?

[DWesl]

For wildcards/globs, there's this section of the documentation explaining the reasons; requesting tests in PRs and running tests in CI seem to cover most of their concerns. The one exception is lack of portability to non-GNU make, which may not be a concern here.

I wasn't aware that Autotools accepted directories in EXTRA_DIST, but that gets described just before this bit of the documentation. The Autotools developers also recommend against using directories in EXTRA_DIST due to the likelihood of picking up extraneous files, unless you add something to dist-hook to take the most likely extraneous files back out. If the release archive is created on CI rather than an active developer's computer, there are fewer places for extra files to creep into the directory.

In short: it looks like I failed to consider the Autotools advice in the context of CI.

[edwardhartnett]

I believe this has been resolved, and I will close this issue.