make check fails for parallel classic build of 4.4.1.1 (but make all check works)

[edhartnett]

Trying to build 4.4.1.1 (what is with that version number BTW? What's wrong with 4.1.2?)

My configure:

CC=mpicc ./configure --enable-parallel-tests --prefix=/usr/local/netcdf-classic-4.4.1.1 --disable-netcdf4

When I do a make check the ncgen tests fail. I went into the ncgen directory and ran run_tests.sh:

[ed@localhost ncgen]$ ./run_tests.sh
*** Testing ncgen.
*** creating classic file c0.nc from c0.cdl...
*** generating c0.nc ***
./run_tests.sh: line 22: ../ncdump/ncdump: No such file or directory
*** comparing c0.nc against /home/ed/Downloads/netcdf-4.4.1.1/ncgen/c0.cdl ***
1,388d0
< netcdf c0 {
< dimensions:
< Dr = UNLIMITED ; // (2 currently)
< D1 = 1 ;
< D2 = 2 ;
< D3 = 3 ;
< dim-name-dashes = 4 ;
< dim.name.dots = 5 ;
< dim+name+plusses = 6 ;
< dim@name@ats = 7;
< variables:
< char c ;
< c:att-name-dashes = 4 ;
< c:att.name.dots = 5 ;
< c:att+name+plusses = 6 ;
< c:att@name@ats = 7 ;
< byte b ;
< b:c = "" ;
< short s ;
< s:b = 0b, 127b, -128b, -1b ;
< s:s = -32768s, 0s, 32767s ;
< int i ;
< i:i = -2147483647, 0, 2147483647 ;
< i:f = -1.e+36f, 0.f, 1.e+36f ;
< i:d = -1.e+308, 0., 1.e+308 ;
< float f ;
< f:c = "x" ;
< double d ;
< d:c = "abcd\tZ$&" ;
< char cr(Dr) ;
< byte br(Dr) ;
< short sr(Dr) ;
< int ir(Dr) ;
< float fr(Dr) ;
< double dr(Dr) ;
< char c1(D1) ;
< byte b1(D1) ;
< short s1(D1) ;
< int i1(D1) ;
< float f1(D1) ;
< double d1(D1) ;
< char c2(D2) ;
< byte b2(D2) ;
< short s2(D2) ;
< int i2(D2) ;
< float f2(D2) ;
< double d2(D2) ;
< char c3(D3) ;
< byte b3(D3) ;
< short s3(D3) ;
< int i3(D3) ;
< float f3(D3) ;
< double d3(D3) ;
< char cr1(Dr, D1) ;
< byte br2(Dr, D2) ;
< short sr3(Dr, D3) ;
< float f11(D1, D1) ;
< double d12(D1, D2) ;
< char c13(D1, D3) ;
< short s21(D2, D1) ;
< int i22(D2, D2) ;
< float f23(D2, D3) ;
< char c31(D3, D1) ;
< byte b32(D3, D2) ;
< short s33(D3, D3) ;
< short sr11(Dr, D1, D1) ;
< int ir12(Dr, D1, D2) ;
< float fr13(Dr, D1, D3) ;
< char cr21(Dr, D2, D1) ;
< byte br22(Dr, D2, D2) ;
< short sr23(Dr, D2, D3) ;
< float fr31(Dr, D3, D1) ;
< double dr32(Dr, D3, D2) ;
< char cr33(Dr, D3, D3) ;
< char c111(D1, D1, D1) ;
< byte b112(D1, D1, D2) ;
< short s113(D1, D1, D3) ;
< float f121(D1, D2, D1) ;
< double d122(D1, D2, D2) ;
< char c123(D1, D2, D3) ;
< short s131(D1, D3, D1) ;
< int i132(D1, D3, D2) ;
< float f133(D1, D3, D3) ;
< float f211(D2, D1, D1) ;
< double d212(D2, D1, D2) ;
< char c213(D2, D1, D3) ;
< short s221(D2, D2, D1) ;
< int i222(D2, D2, D2) ;
< float f223(D2, D2, D3) ;
< char c231(D2, D3, D1) ;
< byte b232(D2, D3, D2) ;
< short s233(D2, D3, D3) ;
< short s311(D3, D1, D1) ;
< int i312(D3, D1, D2) ;
< float f313(D3, D1, D3) ;
< double var-name-dashes ;
< double var.name.dots ;
< double var+name+plusses ;
< double var@name@ats ;
<
< // global attributes:
< :Gc = "" ;
< :Gb = -128b, 127b ;
< :Gs = -32768s, 0s, 32767s ;
< :Gi = -2147483647, 0, 2147483647 ;
< :Gf = -1.e+36f, 0.f, 1.e+36f ;
< :Gd = -1.e+308, 0., 1.e+308 ;
< :Gatt-name-dashes = -1 ;
< :Gatt.name.dots = -2 ;
< :Gatt+name+plusses = -3 ;
< :Gatt@name@ats = -4 ;
< data:
<
< c = "2" ;
<
< b = -2 ;
<
< s = -5 ;
<
< i = -20 ;
<
< f = -9 ;
<
< d = -10 ;
<
< cr = "ab" ;
<
< br = -128, 127 ;
<
< sr = -32768, 32767 ;
<
< ir = -2147483646, 2147483647 ;
<
< fr = -1e+36, 1e+36 ;
<
< dr = -1e+308, 1e+308 ;
<
< c1 = "" ;
<
< b1 = -128 ;
<
< s1 = -32768 ;
<
< i1 = -2147483646 ;
<
< f1 = -1e+36 ;
<
< d1 = -1e+308 ;
<
< c2 = "ab" ;
<
< b2 = -128, 127 ;
<
< s2 = -32768, 32767 ;
<
< i2 = -2147483646, 2147483647 ;
<
< f2 = -1e+36, 1e+36 ;
<
< d2 = -1e+308, 1e+308 ;
<
< c3 = "\001\177." ;
<
< b3 = -128, 127, -1 ;
<
< s3 = -32768, 0, 32767 ;
<
< i3 = -2147483646, 0, 2147483647 ;
<
< f3 = -1e+36, 0, 1e+36 ;
<
< d3 = -1e+308, 0, 1e+308 ;
<
< cr1 =
< "x",
< "y" ;
<
< br2 =
< -24, -26,
< -20, -22 ;
<
< sr3 =
< -375, -380, -385,
< -350, -355, -360 ;
<
< f11 =
< -2187 ;
<
< d12 =
< -3000, -3010 ;
<
< c13 =
< "\tb\177" ;
<
< s21 =
< -375,
< -350 ;
<
< i22 =
< -24000, -24020,
< -23600, -23620 ;
<
< f23 =
< -2187, -2196, -2205,
< -2106, -2115, -2124 ;
<
< c31 =
< "+",
< "-",
< " " ;
<
< b32 =
< -24, -26,
< -20, -22,
< -16, -18 ;
<
< s33 =
< -375, -380, -385,
< -350, -355, -360,
< -325, -330, -335 ;
<
< sr11 =
< 2500,
< 2375 ;
<
< ir12 =
< 640000, 639980,
< 632000, 631980 ;
<
< fr13 =
< 26244, 26235, 26226,
< 25515, 25506, 25497 ;
<
< cr21 =
< "@",
< "D",
< "H",
< "L" ;
<
< br22 =
< 64, 62,
< 68, 66,
< 56, 54,
< 60, 58 ;
<
< sr23 =
< 2500, 2495, 2490,
< 2525, 2520, 2515,
< 2375, 2370, 2365,
< 2400, 2395, 2390 ;
<
< fr31 =
< 26244,
< 26325,
< 26406,
< 25515,
< 25596,
< 25677 ;
<
< dr32 =
< 40000, 39990,
< 40100, 40090,
< 40200, 40190,
< 39000, 38990,
< 39100, 39090,
< 39200, 39190 ;
<
< cr33 =
< "1",
< "two",
< "3",
< "4",
< "5",
< "six" ;
<
< c111 =
< "@" ;
<
< b112 =
< 64, 62 ;
<
< s113 =
< 2500, 2495, 2490 ;
<
< f121 =
< 26244,
< 26325 ;
<
< d122 =
< 40000, 39990,
< 40100, 40090 ;
<
< c123 =
< "one",
< "2" ;
<
< s131 =
< 2500,
< 2525,
< 2550 ;
<
< i132 =
< 640000, 639980,
< 640400, 640380,
< 640800, 640780 ;
<
< f133 =
< 26244, 26235, 26226,
< 26325, 26316, 26307,
< 26406, 26397, 26388 ;
<
< f211 =
< 26244,
< 25515 ;
<
< d212 =
< 40000, 39990,
< 39000, 38990 ;
<
< c213 =
< "",
< "" ;
<
< s221 =
< 2500,
< 2525,
< 2375,
< 2400 ;
<
< i222 =
< 640000, 639980,
< 640400, 640380,
< 632000, 631980,
< 632400, 632380 ;
<
< f223 =
< 26244, 26235, 26226,
< 26325, 26316, 26307,
< 25515, 25506, 25497,
< 25596, 25587, 25578 ;
<
< c231 =
< "@",
< "D",
< "H",
< "H",
< "L",
< "P" ;
<
< b232 =
< 64, 62,
< 68, 66,
< 72, 70,
< 56, 54,
< 60, 58,
< 64, 62 ;
<
< s233 =
< 2500, 2495, 2490,
< 2525, 2520, 2515,
< 2550, 2545, 2540,
< 2375, 2370, 2365,
< 2400, 2395, 2390,
< 2425, 2420, 2415 ;
<
< s311 =
< 2500,
< 2375,
< 2250 ;
<
< i312 =
< 640000, 639980,
< 632000, 631980,
< 624000, 623980 ;
<
< f313 =
< 26244, 26235, 26226,
< 25515, 25506, 25497,
< 24786, 24777, 24768 ;
<
< var-name-dashes = -1 ;
<
< var.name.dots = -2 ;
<
< var+name+plusses = _ ;
<
< var@name@ats = _;
< }
[ed@localhost ncgen]$

I've attached the config.log.
config.log.txt

[WardF]

Hi Ed,

I went 4.4.1.1 instead of 4.4.2 because it was an bug-fix/maintenance release we had to get out in short order; 4.4.2 is still coming down the pike pretty quickly, hopefully, now that AGU is over :). Let me see if I can recreate the other issue and then figure out what the heck is going on. I presume it's correct when it says no such file ../ncdump/ncdump?

[edhartnett]

OK, this is pretty simple. make check did not work, but make all check did. I should have tried that, but I am used to make check working. ;-)

[edhartnett]

I just checked out the netCDF C library from github and it also has this issue.

The funny thing is, I distinctly remember having this same bug when I re-did the netCDF autoconf, and added automake and libtool. But unfortunately, I don't remember how it was solved. You somehow have to indicate that ncdump depends on the ncgen executable.

Actually it is not enough to do it by directory order, IIRC, because then make -j check won't work.

You should be using make -j check in your CI servers. It is the most demanding form of the build, and therefore good for testing. And on modern multi-core systems, make -j check is a LOT faster than make check. So it would be great if that worked.

I know it worked at one time, but I don't know when it stopped working. ;-)

I think the answer is that you want to move ncgen/run_tests.sh into ncdump and run it there, since it depends on ncdump to exist. THe ncdump testing already depends on ncgen. THat is the directory where tests go which need both ncgen and ncdump to exist.

[DennisHeimbigner]

I formally propose we create a new directory: utilitiestests
and move all the ncgen3, ncgen, and ncdump tests into that directory.
Then the dependencies will all be confined to that directory.

[WardF]

Yeah the issue can be solved as Dennis suggests, although I'm going to hold off a bit just because there will be some overhead in modifying some of the test scripts, which always breaks things and somehow I manage to miss some regardless of how well I check :).

The problem I encountered early on in my time here with make -j check is that some individual tests rely on each other, and I was unclear how to ensure the order at an individual test level. I can look into that again I suppose :)

[edhartnett]

I agree with Denis, that sounds very sensible.

I believe that should be the only dependency in the tests.

[WardF]

It should be, perhaps, but currently it's not; some tests assume that other tests have already completed running, and expect to find "output from previous test" to use as input. I'll have to dig through for the specifics, but I encountered this in multiple places when wiring tests in to cmake; e.g., the order in which tests were run mattered.

[edhartnett]

Well I just remembered that the examples have to be run in order. One way to handle that is to make a bash script that runs the examples in order, and then have the makefile run that. I seem to recall there is also a way to turn off parallel builds in a subdirectory.

[WardF]

I'd prefer to avoid shell scripts where possible, to accommodate Windows builds. I'll also look to see what the process would be for a cmake-based build; I'm certain we can disable parallel builds there as well, I just need to figure out how.

…

On Mon, Dec 19, 2016 at 2:35 PM, Imperial Fleet Commander Tarth < ***@***.***> wrote: Well I just remembered that the examples have to be run in order. One way to handle that is to make a bash script that runs the examples in order, and then have the makefile run that. I seem to recall there is also a way to turn off parallel builds in a subdirectory. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#339 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEH-UtFKa1Ml0ezw_YKTzR6GqbUAvBY_ks5rJviUgaJpZM4LQx7E> .

[WardF]

This appears to be fixed, closing out in prep for the 4.5.0 release candidate.

[WardF]

Actually this needs correcting still. Oops, shouldn't be a big fix.

[WardF]

Moved the tests around so that make check will work even if make or make all had not been run first. Funny what you learn about alternative work flows with bugs like this; I had never considered running make check without first running make :)