The Reunion [POC]

[jhunkeler]

based on #322

EDIT: This is necessary because gcc is no longer shipped with latest version of conda.

Replaces WAF with CMAKE.

• Source files are explicitly tracked per project
• HSTCAL is now 100% circular dependency-free (fixed ctegen2)
• Patched up a few inter-library linkage issues (i.e. calstis0 requires symbols from cs2 but those symbols were not defined/exported correctly)
• Dropped nested directories for each instrument
• Refactored instrument directories to use a lib and src design.
  • PKGNAME/lib contains all of the source code for the instrument
  • PKGNAME/src contains the main program(s)
• Rolled a few libraries into hstcalib because it saves time and decreases the burden of tracking common symbols between different sub-projects (i.e. imphtab and ncarfft)

How do I play with this?

1. Install CMake (with or without conda)
2. Install GCC and GFortran (with or without conda)
3. Install cfitsio (with or without conda)
4. Install pkgconfig (with or without conda)
5. Add remote target to existing clone

git remote add jhunkeler https://github.com/jhunkeler/hstcal.git
git fetch jhunkeler

6. Checkout the branch

git checkout thereunion

7. Build

# If using conda:
# conda create -n hstcaldev -c conda-forge cmake compilers cfitsio pkgconfig
# conda activate hstcaldev
mkdir _build
cd _build
cmake ..
make

To set an installation prefix, do:

cmake -DCMAKE_INSTALL_PREFIX=/some/path ..
make
make install

This supports DESTDIR as well for testing of installations:

cmake ..
make
make install DESTDIR=/some/path

For example...

$ make install DESTDIR=/tmp/hstcal
$ find /tmp/hstcal
/tmp/hstcal
/tmp/hstcal/usr
/tmp/hstcal/usr/local
/tmp/hstcal/usr/local/bin
/tmp/hstcal/usr/local/bin/cs12.e
/tmp/hstcal/usr/local/bin/cs11.e
/tmp/hstcal/usr/local/bin/cs8.e
/tmp/hstcal/usr/local/bin/cs7.e
/tmp/hstcal/usr/local/bin/cs6.e
/tmp/hstcal/usr/local/bin/cs4.e
/tmp/hstcal/usr/local/bin/cs2.e
/tmp/hstcal/usr/local/bin/cs1.e
/tmp/hstcal/usr/local/bin/cs0.e
/tmp/hstcal/usr/local/bin/calwf3.e
/tmp/hstcal/usr/local/bin/wf3sum.e
/tmp/hstcal/usr/local/bin/wf3rej.e
/tmp/hstcal/usr/local/bin/wf3ir.e
/tmp/hstcal/usr/local/bin/wf3cte.e
/tmp/hstcal/usr/local/bin/wf3ccd.e
/tmp/hstcal/usr/local/bin/wf32d.e
/tmp/hstcal/usr/local/bin/calacs.e
/tmp/hstcal/usr/local/bin/acssum.e
/tmp/hstcal/usr/local/bin/acsrej.e
/tmp/hstcal/usr/local/bin/acscte.e
/tmp/hstcal/usr/local/bin/acsccd.e
/tmp/hstcal/usr/local/bin/acs2d.e
/tmp/hstcal/usr/local/lib
/tmp/hstcal/usr/local/lib/libhststis.so
/tmp/hstcal/usr/local/lib/libhstwf3.so
/tmp/hstcal/usr/local/lib/libhstacs.so
/tmp/hstcal/usr/local/lib/libhsttables.so
/tmp/hstcal/usr/local/lib/libhstcalib.so
/tmp/hstcal/usr/local/lib/libhstio.so
/tmp/hstcal/usr/local/lib/libhstcvos.so
/tmp/hstcal/usr/local/lib/libhstctegen2.a

$ /tmp/hstcal/usr/local/bin/calacs.e --version
10.3.5 (08-Feb-2022)

$ /tmp/hstcal/usr/local/bin/calacs.e --gitinfo
git tag: v1.0.0-370-g1a4826f
git branch: thereunion
HEAD @: 1a4826f0e21cdb99c528c6463b13a1f81d962c6b

$ readelf -d /tmp/hstcal/usr/local/bin/calacs.e

Dynamic section at offset 0x2ca0 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libhstacs.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgomp.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libhstcalib.so]
 0x0000000000000001 (NEEDED)             Shared library: [libhsttables.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [/home/jhunk/miniconda3/envs/hstcaldev/lib:$ORIGIN/../lib]
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x182c
 0x0000000000000019 (INIT_ARRAY)         0x3c90
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3c98
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x2e8
 0x000000006ffffef5 (GNU_HASH)           0x498
 0x0000000000000005 (STRTAB)             0x840
 0x0000000000000006 (SYMTAB)             0x4e0
 0x000000000000000a (STRSZ)              745 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x3ef0
 0x0000000000000002 (PLTRELSZ)           24 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0xed0
 0x0000000000000007 (RELA)               0xbb8
 0x0000000000000008 (RELASZ)             792 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x0000000000000018 (BIND_NOW)           
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x000000006ffffffe (VERNEED)            0xb78
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0xb2a
 0x000000006ffffff9 (RELACOUNT)          3
 0x0000000000000000 (NULL)               0x0

$ ldd /tmp/hstcal/usr/local/bin/calacs.e
        linux-vdso.so.1 (0x00007ffeb9ff1000)
        libhstacs.so => /tmp/hstcal/usr/local/bin/../lib/libhstacs.so (0x00007f62dddfc000)
        libgomp.so.1 => /home/jhunk/miniconda3/envs/hstcaldev/lib/libgomp.so.1 (0x00007f62dddc3000)
        libhstcalib.so => /tmp/hstcal/usr/local/bin/../lib/libhstcalib.so (0x00007f62dddae000)
        libhsttables.so => /tmp/hstcal/usr/local/bin/../lib/libhsttables.so (0x00007f62ddd9e000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f62ddb84000)
        libhstio.so => /tmp/hstcal/usr/local/bin/../lib/../lib/libhstio.so (0x00007f62ddb68000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f62ddb63000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f62ddb5e000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f62ddb59000)
        libcfitsio.so.10 => /home/jhunk/miniconda3/envs/hstcaldev/lib/libcfitsio.so.10 (0x00007f62dd800000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f62dd718000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f62dde59000)
        libhstcvos.so => /tmp/hstcal/usr/local/bin/../lib/../lib/../lib/libhstcvos.so (0x00007f62ddb52000)
        libcurl.so.4 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./libcurl.so.4 (0x00007f62dd66c000)
        libz.so.1 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./libz.so.1 (0x00007f62ddb38000)
        libbz2.so.1.0 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./libbz2.so.1.0 (0x00007f62dd658000)
        libnghttp2.so.14 => /home/jhunk/miniconda3/envs/hstcaldev/lib/././libnghttp2.so.14 (0x00007f62dd62b000)
        libssh2.so.1 => /home/jhunk/miniconda3/envs/hstcaldev/lib/././libssh2.so.1 (0x00007f62dd5e7000)
        libssl.so.3 => /home/jhunk/miniconda3/envs/hstcaldev/lib/././libssl.so.3 (0x00007f62dd545000)
        libcrypto.so.3 => /home/jhunk/miniconda3/envs/hstcaldev/lib/././libcrypto.so.3 (0x00007f62dd000000)
        libgssapi_krb5.so.2 => /home/jhunk/miniconda3/envs/hstcaldev/lib/././libgssapi_krb5.so.2 (0x00007f62dcfae000)
        libzstd.so.1 => /home/jhunk/miniconda3/envs/hstcaldev/lib/././libzstd.so.1 (0x00007f62dceeb000)
        libkrb5.so.3 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./././libkrb5.so.3 (0x00007f62dce14000)
        libk5crypto.so.3 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./././libk5crypto.so.3 (0x00007f62dd52c000)
        libcom_err.so.3 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./././libcom_err.so.3 (0x00007f62ddb2e000)
        libkrb5support.so.0 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./././libkrb5support.so.0 (0x00007f62dd51d000)
        libkeyutils.so.1 => /home/jhunk/miniconda3/envs/hstcaldev/lib/./././libkeyutils.so.1 (0x00007f62dd516000)
        libresolv.so.2 => /usr/lib/libresolv.so.2 (0x00007f62dce02000)

[mdlpstsci]

@jhunkeler I see all these changes and I am nervous and excited all at the same time 😉. Can you please write more of a description regarding this work so the "HSTCAL" folks have a better sense of the essence of the changes? Also, at some point in the coming weeks, it would be beneficial for all of us to get together and discuss this update. Best, M

[jhunkeler]

Hey @mdlpstsci, the bulk of the changes were mapped out and described in the previous PR (#322) in 2018. My memory is failing me but I think I created this one because I screwed up a rebase and didn't feel like backtracking to fix it. 😓 I'll carry over the description from 322 and expand on whatever I've done differently since then.

I feel like I'm several years too late to mention this, but I'll throw it out there anyway. cmake provides a test runner/suite called ctest and it makes writing unit and regression tests a breeze. It is also possible to run arbitrary programs with arguments and trap their return value, so if you want to execute the *.e binaries with a matrix of args its only a few lines of code away. I've used it in a few personal projects and haven't been disappointed yet.

Example unit tests:
https://github.com/jhunkeler/spmc/tree/master/tests
https://github.com/jhunkeler/reloc/tree/master/test
https://github.com/jhunkeler/cleanpath/tree/master/tests (you can see a couple regtests defined in the CMakeLists.txt that calls cleanpath with arguments)

[pllim]

Even Friends and Full House already have their reunions. When can we do this one? 😸

[mdlpstsci]

After all our recent exchanges and testing done by me, Robert, and DSB, I think we are as ready as we can be. You have responded to our issues (last one I know about is the segmentation fault). We should proceed with the merge and see what happens sooner rather than later.

[pllim]

It's about time! (5 years)

[mdlpstsci]

Now, now. We do not want to be hasty!

[jhunkeler]

Here goes nothing.

[pllim]