add an option to upload the open data #247
Conversation
src/hyp3_autorift/process.py
Outdated
| return var.latitude, var.longitude | ||
|
|
||
|
|
||
| def point_to_prefix(dir_path: str, lat: float, lon: float) -> str: |
There was a problem hiding this comment.
Since we're hard-coding the prefix anyways, I think it'd be better to take platform as input and instead return the entire prefix from this function so that we'd end up calling it like:
>>> platform = get_platform(scene)
>>> lat, lon = get_lon_lat_from_ncfile(file)
>>> point_to_prefix(platform, lat, lon)
'velocity_image_pair/landsatOLI/v02/S80W100'
src/hyp3_autorift/process.py
Outdated
| return dirstring | ||
|
|
||
|
|
||
| def get_opendata_prefix(file: Path, scene: str): |
There was a problem hiding this comment.
you can also get the scene name from the file name
scene = file.name.split('_X_')[0]which is better since we sort the granules on input. Either that, or when this is called, you should pass it g1:
https://github.com/ASFHyP3/hyp3-autorift/blob/develop/src/hyp3_autorift/process.py#L538
Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>
|
OK @cirrusasf and @jhkennedy I believe I've addressed all the notes from the code review. Next thing we should do is add tests for |
src/hyp3_autorift/process.py
Outdated
| if args.publish: | ||
| prefix = get_opendata_prefix(product_file) | ||
| upload_file_to_s3(product_file, OPEN_DATA_BUCKET, prefix) | ||
| upload_file_to_s3(browse_file, OPEN_DATA_BUCKET, prefix) | ||
| upload_file_to_s3(thumbnail_file, OPEN_DATA_BUCKET, prefix) |
There was a problem hiding this comment.
Thinking about it, I think I'd rather do that after we upload to the HyP3 content bucket -- that way, if this part breaks, we still have a copy in HyP3 and can copy the files over without having to reprocess.
What do you thinK?
There was a problem hiding this comment.
very good idea. I will change the sequence.
| @@ -538,6 +592,13 @@ def main(): | |||
| g1, g2 = sorted(args.granules, key=get_datetime) | |||
|
|
|||
| product_file, browse_file = process(g1, g2, parameter_file=args.parameter_file, naming_scheme=args.naming_scheme) | |||
| thumbnail_file = create_thumbnail(browse_file) | |||
There was a problem hiding this comment.
moving this here means we can remove the duplicate line 606
src/hyp3_autorift/process.py
Outdated
| lat_lon_prefix_component = point_to_prefix(platform_shortname, lat, lon) | ||
|
|
||
| dir_path = f'velocity_image_pair/{PLATFORM_SHORTNAME_LONGNAME_MAPPING[platform_shortname]}/v02' | ||
| prefix = os.path.join(dir_path, lat_lon_prefix_component) |
There was a problem hiding this comment.
I'd still prefer all of this logic in point_to_prefix so it's actually returning the prefix. Otherwise we should change the function name.
Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>
…/hyp3-autorift into auto_upload_its_live_data_bucket
Set up access key use for open data upload
There was a problem hiding this comment.
@cirrusasf a couple small suggestions, but I'll be ready to approve after this!
CHANGELOG.md
Outdated
| @@ -6,6 +6,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), | |||
| and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/) | |||
| and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). | |||
|
|
|||
| ## [0.14.2] | |||
| ### Added | |||
| * `--publish` option has been added to the HyP3 entry point to publish product to the ITS_LIVE AWS Open Data bucket, `s3://its-live-data`. | |||
There was a problem hiding this comment.
| * `--publish` option has been added to the HyP3 entry point to publish product to the ITS_LIVE AWS Open Data bucket, `s3://its-live-data`. | |
| * `--publish` option has been added to the HyP3 entry point to publish product to the ITS_LIVE AWS Open Data bucket, `s3://its-live-data`. | |
| * `upload_file_to_s3_with_upload_access_keys` to perform S3 uploads with credentialed S3 clients. | |
| * use of `UPLOAD_ACCESS_KEY_ID` and `UPLOAD_ACCESS_KEY_SECRET` to upload products to write-protected bucket. |
Change to a publish-bucket parameter to enable testing
Co-authored-by: Forrest Williams <31411324+forrestfwilliams@users.noreply.github.com>
No description provided.