Reader for matched-table catalog and config files for composite add-o…

[evevkovacs]

@yymao @fjaviersanchez Here is a reader for the new table, created by Javi to find objects in the dc2 object catalog that have matches in cosmoDC2. This reader just reads in the fits file and generates some useful quantities. I should be able to run some descqa tests on this table/mini-catalog. I have requested both of you as reviwers. The next steps are 1)to make it an add-on to the dc2-object catalog, since people might want to check other quantities and 2) to make cosmoDC2 an add-on so other tests can be run. I have also pushed my first attempt at the config file for using the new reader as part of a composite catalog. It doesn't crash, but it is not giving me the result that I expected. The object catalog and the matched table both have 7548496 rows, but when I read in the composite catalog (which is matched on objectId) I get 78649132 rows, so I have not understood something about how composite catalogs work. Could one of you take possibly take a look and see if you can spot the problem?

[fjaviersanchez]

It looks good to me. Thanks for putting this together Eve!

[yymao]

Thanks @evevkovacs @fjaviersanchez --- about the questions @evevkovacs raised:

1. to make it an add-on to the dc2-object catalog

This can work by either matching the object_id column like you did, or, if the object catalog and the matched table have the same row order, use the MATCHING_ORDER method. (It seems that the matched table is a single fits file, so we can't use theMATCHING_FORMAT method, since it requires the add-on catalog to have exactly the same order.)

In the composite config, it appears that you are matching to dc2_object_run2.1i_dr1 not dc2_object_run2.1.1i. This is probably what causes the issue?

2. to make cosmoDC2 an add-on so other tests can be run

I now realized that this is a bit beyond the original design of the composite catalog, which was to add add-on columns to a main catalog by matching to a single column of the main catalog. Here, the cosmoDC2 needs to be matched to a different column (truth_id).

While this possible, the implementation would look quite ugly unless we go change the underlying composite catalog code. But before we further expand GCR's functionality, we should really ask if we should instead explore using database instead.

[johannct]

I propose that we use this pull request to fix the NERSC-specific paths problem. @yymao @boutigny @heather999 @wmwv

[heather999]

We need to fully understand what code needs to change, see: #346
We also need to identify all of the catalogs and their current locations at NERSC. With a list, we can create an area at NERSC with a set prefix and agreed upon directory structure.

[evevkovacs]

@yymao @fjaviersanchez The code is tested and works but fails the travis check with a warning I don't fully understand. The offending line is 127. The warning goes away if I add the argument "handle" to the call signature for the native_quantity_getter, but then the code crashes. Can one of you please take a look and see if you know how to make travis happy. Thanks

[evevkovacs]

@fjaviersanchez @yymao I made one last attempt to fix the travis issue and this time it passed! So as soon as one of you approves, I can merge this code. Thanks a lot! I'd like to merge so that other people can run descqa tests if they want.

[fjaviersanchez]

It looks good to me but I don't know if we want to put the matching scripts somewhere in the DC2-production repo or something like that.

Currently on leave

[heather999]

@fjaviersanchez Please go ahead and add the matching scripts to DC2-production via a PR and then we should organize a tag of DC2-production.