Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

issue/161: Fix bug where --subset flag fails with data subscriber #162

Merged
merged 2 commits into from
Mar 7, 2024

Conversation

skorper
Copy link
Contributor

@skorper skorper commented Mar 4, 2024

See #161

This fixes the above issue and allows the data subscriber to run as expected with the --subset flag enabled.


Testing

Able to run the subscriber with the --subset flag present:

➜ poetry run python3 subscriber/podaac_data_subscriber.py -c SWOT_L2_NALT_OGDR_SSHA_2.0 -d ./data --start-date 2023-12-01T00:00:00Z --end-date 2023-12-21T00:00:00Z --subset -b="120,-30,160,20"
[2024-03-04 14:23:05,497] {podaac_data_subscriber.py:169} INFO - NOTE: Making new data directory at ./data(This is the first run.)
[2024-03-04 14:23:12,353] {subsetting.py:102} INFO - Waiting for Harmony subsetting job to complete...
./data/SWOT_OPR_2PfS007_212_20231130_230255_20231201_004855_subsetted.nc4
./data/SWOT_OPR_2PfS007_225_20231201_105209_20231201_120837_subsetted.nc4
./data/SWOT_OPR_2PfS007_228_20231201_124642_20231201_140008_subsetted.nc4
./data/SWOT_OPR_2PfS007_238_20231201_213512_20231201_230405_subsetted.nc4
./data/SWOT_OPR_2PfS007_240_20231201_230341_20231202_004949_subsetted.nc4
./data/SWOT_OPR_2PfS007_254_20231202_110559_20231202_121546_subsetted.nc4
./data/SWOT_OPR_2PfS007_255_20231202_121544_20231202_134525_subsetted.nc4
./data/SWOT_OPR_2PfS007_266_20231202_213518_20231202_230500_subsetted.nc4
./data/SWOT_OPR_2PfS007_281_20231203_105407_20231203_113614_subsetted.nc4
./data/SWOT_OPR_2PfS007_268_20231202_230459_20231203_005037_subsetted.nc4
./data/SWOT_OPR_2PfS007_296_20231203_232721_20231204_005119_subsetted.nc4
./data/SWOT_OPR_2PfS007_284_20231203_124811_20231203_140005_subsetted.nc4
./data/SWOT_OPR_2PfS007_310_20231204_110817_20231204_121209_subsetted.nc4
./data/SWOT_OPR_2PfS007_322_20231204_213508_20231204_230632_subsetted.nc4
./data/SWOT_OPR_2PfS007_324_20231204_232915_20231205_005205_subsetted.nc4
./data/SWOT_OPR_2PfS007_338_20231205_110431_20231205_113752_subsetted.nc4
./data/SWOT_OPR_2PfS007_294_20231203_213514_20231203_230546_subsetted.nc4
./data/SWOT_OPR_2PfS007_350_20231205_212132_20231205_230718_subsetted.nc4
./data/SWOT_OPR_2PfS007_363_20231206_090732_20231206_110515_subsetted.nc4
./data/SWOT_OPR_2PfS007_352_20231205_233020_20231206_005255_subsetted.nc4
./data/SWOT_OPR_2PfS007_366_20231206_110513_20231206_113813_subsetted.nc4
./data/SWOT_OPR_2PfS007_378_20231206_212210_20231206_230802_subsetted.nc4
./data/SWOT_OPR_2PfS007_391_20231207_091220_20231207_110731_subsetted.nc4
./data/SWOT_OPR_2PfS007_380_20231206_233116_20231207_005107_subsetted.nc4
./data/SWOT_OPR_2PfS007_408_20231207_233204_20231208_005429_subsetted.nc4
./data/SWOT_OPR_2PfS007_406_20231207_212256_20231207_230840_subsetted.nc4
./data/SWOT_OPR_2PfS007_394_20231207_110729_20231207_120924_subsetted.nc4
./data/SWOT_OPR_2PfS007_418_20231208_080521_20231208_100330_subsetted.nc4
./data/SWOT_OPR_2PfS007_434_20231208_212339_20231208_230923_subsetted.nc4
./data/SWOT_OPR_2PfS007_462_20231209_212417_20231209_231027_subsetted.nc4
./data/SWOT_OPR_2PfS007_422_20231208_111601_20231208_122109_subsetted.nc4
./data/SWOT_OPR_2PfS007_448_20231209_092506_20231209_103657_subsetted.nc4
./data/SWOT_OPR_2PfS007_449_20231209_103654_20231209_121237_subsetted.nc4
./data/SWOT_OPR_2PfS007_476_20231210_093030_20231210_103739_subsetted.nc4
./data/SWOT_OPR_2PfS007_488_20231210_195721_20231210_212530_subsetted.nc4
./data/SWOT_OPR_2PfS007_477_20231210_103737_20231210_115354_subsetted.nc4
./data/SWOT_OPR_2PfS007_504_20231211_092644_20231211_110853_subsetted.nc4
./data/SWOT_OPR_2PfS007_490_20231210_212507_20231210_231111_subsetted.nc4
./data/SWOT_OPR_2PfS007_506_20231211_110850_20231211_122108_subsetted.nc4
./data/SWOT_OPR_2PfS007_518_20231211_212546_20231211_231153_subsetted.nc4
./data/SWOT_OPR_2PfS007_532_20231212_093245_20231212_110938_subsetted.nc4
./data/SWOT_OPR_2PfS007_516_20231211_195735_20231211_212609_subsetted.nc4
./data/SWOT_OPR_2PfS007_560_20231213_093054_20231213_103909_subsetted.nc4
./data/SWOT_OPR_2PfS007_534_20231212_110935_20231212_122244_subsetted.nc4
./data/SWOT_OPR_2PfS007_544_20231212_195741_20231212_212658_subsetted.nc4
./data/SWOT_OPR_2PfS007_546_20231212_212635_20231212_231250_subsetted.nc4
./data/SWOT_OPR_2PfS007_574_20231213_214905_20231213_231330_subsetted.nc4
./data/SWOT_OPR_2PfS007_572_20231213_195756_20231213_212754_subsetted.nc4
./data/SWOT_OPR_2PfS007_561_20231213_103908_20231213_121941_subsetted.nc4
./data/SWOT_OPR_2PfS008_004_20231214_092523_20231214_103049_subsetted.nc4
./data/SWOT_OPR_2PfS008_016_20231214_195740_20231214_212839_subsetted.nc4
./data/SWOT_OPR_2PfS008_006_20231214_112147_20231214_125443_subsetted.nc4
./data/SWOT_OPR_2PfS008_031_20231215_091715_20231215_095926_subsetted.nc4
./data/SWOT_OPR_2PfS008_043_20231215_193658_20231215_215454_subsetted.nc4
./data/SWOT_OPR_2PfS008_018_20231214_215108_20231214_231418_subsetted.nc4
./data/SWOT_OPR_2PfS008_046_20231215_215451_20231215_232939_subsetted.nc4
./data/SWOT_OPR_2PfS008_072_20231216_194648_20231216_212938_subsetted.nc4
./data/SWOT_OPR_2PfS008_057_20231216_071756_20231216_091317_subsetted.nc4
./data/SWOT_OPR_2PfS008_059_20231216_091254_20231216_102902_subsetted.nc4
./data/SWOT_OPR_2PfS008_074_20231216_212936_20231216_225821_subsetted.nc4
./data/SWOT_OPR_2PfS008_086_20231217_075012_20231217_095952_subsetted.nc4
./data/SWOT_OPR_2PfS008_100_20231217_194505_20231217_213048_subsetted.nc4
./data/SWOT_OPR_2PfS008_113_20231218_072839_20231218_092851_subsetted.nc4
./data/SWOT_OPR_2PfS008_116_20231218_092849_20231218_102734_subsetted.nc4
./data/SWOT_OPR_2PfS008_102_20231217_215412_20231217_231640_subsetted.nc4
./data/SWOT_OPR_2PfS008_128_20231218_194552_20231218_213131_subsetted.nc4
./data/SWOT_OPR_2PfS008_141_20231219_072921_20231219_092935_subsetted.nc4
./data/SWOT_OPR_2PfS008_144_20231219_092933_20231219_102849_subsetted.nc4
./data/SWOT_OPR_2PfS008_130_20231218_215458_20231218_231630_subsetted.nc4
./data/SWOT_OPR_2PfS008_156_20231219_194630_20231219_213223_subsetted.nc4
./data/SWOT_OPR_2PfS008_170_20231220_074922_20231220_085927_subsetted.nc4
./data/SWOT_OPR_2PfS008_182_20231220_181941_20231220_194745_subsetted.nc4
./data/SWOT_OPR_2PfS008_184_20231220_194722_20231220_213324_subsetted.nc4
./data/SWOT_OPR_2PfS008_171_20231220_085926_20231220_103308_subsetted.nc4
[2024-03-04 14:24:32,305] {podaac_data_subscriber.py:298} INFO - END

But cannot run the downloader with both --subset and --cycles:

➜ poetry run python3 subscriber/podaac_data_downloader.py -c SWOT_L2_NALT_GDR_SSHA_2.0 -d .data/ -e .nc --cycle 474 --subset
[2024-03-04 14:31:30,886] {podaac_data_downloader.py:138} ERROR - Error: Incompatible Parameters. You've provided both cycles and subset, which is not allowed. Please provide either cycles or subset separately, but not both.

@skorper
Copy link
Contributor Author

skorper commented Mar 7, 2024

@jasonmlkang @yenes56 @hkryeung @voxparcxls @phoeberepo Can someone review this PR? I want to get this patch out ASAP. Thanks so much!

@skorper skorper merged commit 9782d1f into develop Mar 7, 2024
24 checks passed
skorper added a commit that referenced this pull request Mar 20, 2024
* initial implementaiton of subset code

* added tests, more updates

* removed skip_errors (not implemented yet)

* Upgrade to harmony-py 0.4.9

* subsetting capability initial commit

* update spacing

* Update docs

* update poetry lock

* upgrade to poetry 1.5.1 (latest)

* fix failing checksum test

* fixed regression test issues

* bump poetry version in second gh action file

* Update poetry version to fix build issue

* bump poetry version

* bump poetry version

* bump version to 1.15.0

* bump version to 1.15.0

* add unreleased section to changelog

* issue/161: Fix bug where --subset flag fails with data subscriber (#162)

* fix bug where the subset flag fails for subscriber

* move arg validation to correct spot

* bump version

* bump version

* Issue/164: Upgrade harmony-py to avoid 413 request entity too large issue (#165)

* bump harmony-py version to 0.4.12

* /version 1.15.1-alpha.2

---------

Co-authored-by: mike-gangl <michael.e.gangl@jpl.nasa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants