Merge pull request #31 from gmbrandt/feature/hip2_dvd_source_rejection

Feature/hip2 dvd source rejection

.coveragerc

@@ -4,5 +4,6 @@ omit =
     */site-packages/nose/*
     *__init__*
     */validation/*
+    *.ipynb
 exclude_lines =
     pragma: no cover

CHANGES.md

@@ -1,3 +1,15 @@
+0.4.0 (2021-04-30)
+------------------
+- Implemented a better data fix for Hip2 java data. This fixes most of the 6400 discrepant sources
+that have the Hip2 file-write error. Note that this is not perfect, there are some minor
+  degeneracies between which epochs to reject. See the note in parse.find_epochs_to_reject()
+- Note that this write-out bug fix does not work on the dvd data (note, 
+  the dvd still has the same bug, it is just not easily correctable).
+- Updated the flagged source list (hip2_javatool_flagged.txt).
+- Parallax factors from the IAD are now loaded on data.parse()
+- Updated parse to use the new version of the java tool IAD. Old versions of the java
+tool IAD will not work.
+
 0.3.5 (2021-04-19)
 ------------------
 - Fixed improper syntax calls to panda etc. so that warnings are silence in python 3.8 and beyond.

README.rst

@@ -34,11 +34,21 @@ while in the root directory of this repo. It can also be installed directly from
 
     pip install git+https://github.com/gmbrandt/htof
 
+Usage
+-----
+
+HTOF has a rich variety of usages. We encourage the reader to consult the examples.ipynb jupyter notebook
+for a set of usage examples (e.g., fitting the standard astrometric model to data, combining astrometric missions).
+However, we also go into a few basic and specific use cases in this readme.
+
 Usage: Fits without Parallax
 ----------------------------
 The following examples show how one would both load in and fit a line to the astrometric intermediate data
 from either Hipparcos data reduction or Gaia. Gaia requires you to first download a .csv of the
-predicted scans and scan epochs from GOST (https://gaia.esac.esa.int/gost/).
+predicted scans and scan epochs from GOST (https://gaia.esac.esa.int/gost/). In particular, using the 'submit for
+events forecast' feature on the website. One should select the widest range of dates
+possible because \codename automatically restricts the predicted epochs of observations
+to the desired data release range (e.g., EDR3) and removes any astrometric gaps.
 
 Let ra_vs_epoch, dec_vs_epoch be 1d arrays of ra and dec positions.
 Assume we want to fit to data from GaiaDR2 on the star with hip id 027321. The choices of data
@@ -253,9 +263,14 @@ Flagged Sources
 There are a small number of sources in the original reduction of Hipparcos and the re-reductions (e.g. the DVD version
 of the 2007 re-reduction) that HTOF cannot well refit. These sources should be used cautiously and are listed by HIP ID in
 the files in the htof/data directory, e.g. htof/data/hip1_flagged.txt for the 1997 reduction and
-htof/data/hip2_dvd_flagged.txt for the 2007 re-reduction which came on the DVD accompanying the book. Every source in
-these lists have a difference in the catalog best fit proper motions and the HTOF refit proper motions in excess
-of 0.02 mas/yr in either RA or DEC or both.
+htof/data/hip2_dvd_flagged.txt for the 2007 re-reduction which came on the DVD accompanying the book.
+Every source in these lists have a difference in the catalog best fit proper motions and the HTOF refit proper motions in excess
+of 0.1 mas/yr in either RA or DEC or both.
+There is an additional data corruption in the 2007 DVD version of the IAD, that makes nearly 8000 sources hard to refit.
+One will notice that hip2_dvd_flagged.txt has 7550 items.
+We have corrected this corruption in the Java tool IAD, and so we recommend using this IAD (when it is available).
+By contrast, hip2_Javatool_flagged.txt has only 39 entries. These 39 sources are the only ones that htof cannot handle
+satisfactorially. This list may shrink in the future.
 
 
 Astrometric Gaps

htof/data/epoch_reject_shortlist.csv

@@ -0,0 +1,85 @@
+hip_id, residual/along_scan_error, orbit/scan_angle/time
+100135,'[153, 152, 151, 150, 149]','[5, 28, 29, 33, 75]'
+57607,'[106, 105, 104, 103, 102]','[18, 42, 43, 61, 86]'
+39751,'[131, 130, 129, 128, 127, 126, 125]','[60, 61, 69, 80, 100, 111, 113]'
+18517,'[139, 138, 137, 136, 135, 134]','[4, 11, 30, 45, 82, 93]'
+56533,'[91, 90, 89, 88, 87]','[14, 49, 54, 68, 85]'
+94489,'[126, 125, 124, 123, 122]','[6, 22, 59, 60, 106]'
+52661,'[133, 132, 131, 130, 129, 128]','[10, 27, 28, 96, 105, 106]'
+68137,'[83, 82, 81, 80, 79]','[4, 24, 28, 30, 31]'
+89669,'[132, 131, 130, 129, 128, 127]','[11, 12, 75, 76, 115, 119]'
+41201,'[57, 56, 55, 54, 53, 52, 51]','[0, 1, 29, 30, 37, 42, 48]'
+13502,'[131, 130, 129, 128, 127]','[4, 12, 16, 69, 105]'
+114114,'[82, 81, 80, 79]','[1, 17, 22, 56]'
+78721,'[117, 116, 115, 114, 113, 112]','[9, 51, 63, 83, 91, 96]'
+7145,'[167, 166, 165, 164, 163]','[6, 73, 94, 100, 155]'
+96458,'[122, 121, 120, 119, 118]','[15, 39, 63, 66, 73]'
+54543,'[120, 119, 118, 117, 116, 115, 114, 113]','[1, 11, 28, 37, 38, 46, 51, 118]'
+21046,'[124, 123, 122, 121, 120]','[4, 92, 93, 110, 117]'
+106583,'[124, 123, 122, 121, 120]','[10, 13, 61, 70, 120]'
+14542,'[224, 223, 222, 221, 220, 219]','[4, 46, 55, 92, 170, 224]'
+94162,'[149, 148, 147, 146, 145, 144]','[4, 15, 31, 36, 63, 101]'
+43541,'[162, 161, 160, 159, 158]','[32, 52, 64, 134, 154]'
+90241,'[49, 48, 47, 46, 45]','[7, 9, 10, 38, 44]'
+93785,'[88, 87, 86, 85, 84]','[3, 8, 19, 30, 53]'
+112626,'[126, 125, 124, 123, 122]','[15, 32, 88, 93, 120]'
+79932,'[118, 117, 116, 115, 114]','[24, 27, 59, 86, 97]'
+105788,'[94, 93, 92, 91, 90]','[7, 13, 38, 56, 78]'
+21479,'[99, 98, 97, 96, 95]','[6, 53, 55, 62, 85]'
+108630,'[166, 165, 164, 163, 162, 161]','[7, 12, 38, 50, 105, 126]'
+9306,'[134, 133, 132, 131, 130, 129, 128]','[2, 9, 16, 91, 95, 110, 127]'
+4008,'[175, 174, 173, 172, 171, 170]','[2, 40, 97, 129, 146, 147]'
+82516,'[96, 95, 94, 93, 92]','[6, 15, 22, 81, 96]'
+112887,'[154, 153, 152, 151, 150]','[33, 70, 122, 123, 154]'
+54101,'[126, 125, 124, 123, 122, 121]','[7, 24, 32, 68, 89, 126]'
+84213,'[95, 94, 93, 92, 91, 90]','[4, 17, 29, 30, 66, 79]'
+12193,'[92, 91, 90, 89, 88, 87, 86]','[10, 12, 13, 17, 36, 55, 85]'
+24169,'[87, 86, 85, 84, 83, 82]','[10, 13, 23, 41, 56, 57]'
+60180,'[144, 143, 142, 141, 140]','[4, 6, 16, 42, 53]'
+106642,'[134, 133, 132, 131, 130, 129]','[11, 20, 67, 76, 113, 129]'
+102949,'[142, 141, 140, 139, 138, 137, 136]','[21, 31, 47, 93, 111, 134, 138]'
+53300,'[124, 123, 122, 121, 120, 119, 118, 117]','[8, 14, 31, 33, 50, 79, 87, 103]'
+67439,'[80, 79, 78, 77, 76]','[9, 17, 26, 30, 46]'
+97629,'[115, 114, 113, 112, 111, 110]','[22, 23, 41, 81, 82, 98]'
+81835,'[128, 127, 126, 125, 124, 123]','[19, 21, 53, 75, 89, 90]'
+80259,'[157, 156, 155, 154, 153, 152]','[5, 18, 31, 89, 94, 143]'
+99653,'[130, 129, 128, 127, 126]','[3, 9, 20, 40, 106]'
+50843,'[137, 136, 135, 134, 133, 132]','[20, 28, 38, 47, 98, 133]'
+34922,'[115, 114, 113, 112, 111]','[7, 9, 73, 103, 113]'
+108728,'[150, 149, 148, 147, 146]','[7, 34, 39, 42, 93]'
+83674,'[94, 93, 92, 91, 90, 89]','[4, 11, 12, 39, 57, 84]'
+92235,'[98, 97, 96, 95, 94, 93]','[3, 4, 8, 12, 14, 59]'
+66696,'[126, 125, 124, 123, 122, 121, 120]','[0, 26, 48, 53, 79, 80, 107]'
+59844,'[174, 173, 172, 171, 170]','[8, 60, 61, 109, 119]'
+48405,'[104, 103, 102, 101, 100, 99]','[24, 39, 45, 46, 54, 69]'
+47131,'[130, 129, 128, 127, 126]','[13, 20, 24, 55, 63]'
+101023,'[133, 132, 131, 130, 129]','[19, 26, 56, 96, 112]'
+57260,'[135, 134, 133, 132, 131, 130]','[10, 26, 59, 60, 72, 115]'
+80432,'[154, 153, 152, 151, 150]','[7, 18, 42, 107, 138]'
+52599,'[100, 99, 98, 97, 96, 95]','[13, 48, 84, 90, 95, 100]'
+66825,'[146, 145, 144, 143, 142, 141]','[26, 27, 43, 50, 53, 142]'
+109070,'[128, 127, 126, 125, 124, 123]','[0, 80, 81, 90, 113, 128]'
+45949,'[131, 130, 129, 128, 127]','[54, 77, 105, 106, 110]'
+74337,'[125, 124, 123, 122, 121, 120, 119]','[8, 9, 41, 93, 108, 111, 121]'
+116758,'[81, 80, 79, 78, 77]','[3, 15, 29, 52, 81]'
+12302,'[130, 129, 128, 127, 126]','[20, 72, 73, 92, 93]'
+38965,'[184, 183, 182, 181, 180, 179]','[20, 70, 74, 96, 158, 175]'
+28770,'[146, 145, 144, 143, 142, 141]','[22, 52, 104, 127, 132, 139]'
+118188,'[173, 172, 171, 170, 169]','[89, 90, 118, 142, 156]'
+69695,'[179, 178, 177, 176, 175, 174, 173]','[4, 8, 126, 137, 143, 158, 166]'
+30785,'[160, 159, 158, 157, 156]','[11, 22, 71, 87, 107]'
+21000,'[40, 39, 38, 37, 36, 35, 34, 33, 32, 31]','[13, 14, 15, 16, 21, 22, 23, 24, 34, 36]'
+94312,'[112, 111, 110, 109, 108]','[23, 97, 110, 111, 112]'
+516,'[150, 149, 148, 147, 146]','[123, 124, 144, 145, 150]'
+84004,'[105, 104, 103, 102, 101]','[15, 20, 69, 86, 105]'
+36685,'[116, 115, 114, 113, 112]','[2, 10, 15, 22, 96]'
+22170,'[109, 108, 107, 106, 105]','[9, 13, 35, 56, 79]'
+116883,'[142, 141, 140, 139, 138]','[2, 19, 77, 128, 131]'
+17447,'[135, 134, 133, 132, 131]','[13, 27, 41, 55, 98]'
+20429,'[149, 148, 147, 146, 145, 144]','[4, 5, 80, 92, 112, 120]'
+114489,'[71, 70, 69, 68, 67]','[8, 13, 34, 50, 68]'
+41588,'[122, 121, 120, 119, 118, 117]','[24, 25, 38, 51, 89, 113]'
+117185,'[93, 92, 91, 90, 89]','[27, 33, 64, 79, 92]'
+89596,'[79, 78, 77, 76, 75, 74]','[3, 39, 40, 47, 56, 69]'
+92400,'[127, 126, 125, 124, 123, 122]','[68, 104, 112, 114, 126, 127]'
+89968,'[81, 80, 79, 78, 77]','[9, 39, 40, 52, 69]'