Skip to content

feature: convertToPNG --> sendToHarmony #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 2, 2024
Merged

feature: convertToPNG --> sendToHarmony #21

merged 4 commits into from
May 2, 2024

Conversation

voxparcxls
Copy link
Contributor

@voxparcxls voxparcxls commented Apr 25, 2024 •
edited
Loading

Github Issue: #15

Description

Change State_Machine (boolean Choice) convertToPNG --> sendToHarmony to accommodate OPERA product's use of BIGnBIT Workflow, and improve dataset config identification for OPERA with sendToHarmony.

Overview of work done

Changed State_Machine step convertToPNG --> sendToHarmony (true/false)

Overview of verification done

Ran in SIT bignbit (browseimageworkflow)

  • Test data: OPERA_L3_DSWX-S1_V1
"sendToHarmony": false,
Screenshot 2024-04-25 at 3 54 06 PM
  • Test data: MUR-JPL-L4-GLOB-v4.1
"sendToHarmony": true,
Screenshot 2024-04-25 at 3 55 48 PM

State Test:

{
    "nextState": "Get Collection Concept Id",
    "output": ...
    "status": "SUCCEEDED"
},
Screenshot 2024-04-25 at 3 58 49 PM

Overview of integration done

mock pytest tests

PR checklist:

  • Linted
  • Updated unit tests
  • Updated changelog
  • Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

@voxparcxls voxparcxls requested a review from frankinspace April 25, 2024 23:19
frankinspace
frankinspace reviewed May 1, 2024
View reviewed changes
Copy link
Member

@frankinspace frankinspace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also update the configs in https://github.com/podaac/bignbit-config/tree/develop/dataset-config

@frankinspace frankinspace merged commit 341c23b into develop May 2, 2024
@frankinspace frankinspace deleted the feature/sendToHarmony branch May 2, 2024 17:18
@voxparcxls voxparcxls mentioned this pull request May 2, 2024
Closed
frankinspace added a commit that referenced this pull request Feb 4, 2025
@frankinspace @voxparcxls
@torimcd @jake-herrmann @jamesfwood
Merge pull request #49 from podaac/release/0.2.0
43d56cb 
* /version 0.2.0a0

* feature: convertToPNG --> sendToHarmony (#21)

* update state_machine, sendToHarmony

* --changelog Issue-15

* moved to unreleased, changelog

* /version 0.2.0a1

* Update README.md (#22)

* /version 0.2.0a2

* Use -j to remove top-level directory in zip artifact (#25)

* /version 0.2.0a3

* Issue#16: Change apply opera treatment choice and lambda to be specific to HLS (#24)

* update for HLS -- lambda and state machine

* py lint error

* update file name

* /version 0.2.0a4

* /version 0.2.0a5

* Feature/issue 7 - Remove wait for GITC response (#30)

* fix type in readme

* change task token to uuid in send to gitc

* remove wait for task token & gitc response handler from state machine

* change gitc response handler to invoke save cma message

* fix typo

* fix context parameter

* convert uuid to string

* fix json typo

* remove uuid lib to use stdlib uuid

* change identifier to image set name instead of uuid

* remove uuid import

* fix json formatting of cma invoke

* add granule concept ID to identifier

* update image set name with granule conceptid

* remove whitespace

* move get umm json to utils and save gitc outgoing cnm

* remove unused import

* try setting cmr query env based on stage

* remove task token from tests

* fix granule index

* change how image set name defined

* add region to ssm client in utils

* add ssm parameters to gitc lambdas

* change region reference

* update how granule name referenced in send to gitc

* remove unused cmr var

* fix case

* fix cnm parsing

* reformat save cnm to separate step

* update tf vars for save cnm

* fix type in module definition

* fix cnm in cma

* fix collection reference in save cnm

* add debugging log statements

* fix input

* update save cnm input

* debugging

* debugging

* linting

* debugging

* debugging

* linting

* fix state machine

* update state machine

* reorg state machine

* missing comma

* move save cnm into map

* fix state machine transitions

* fix sm

* fix boolean

* debugging cnm input

* fix input

* change input

* debug input

* add cnm as config parameter

* lint

* fix cnm payload

* remove debugging statements

* add prefix to cnm path and fix gitc response

* increase handle gitc response timeout

* update changelog

* add original shortname as cnm prefix

* revert shortname and change collection ref to save cnm & cnm-r in same location

* remove save cma lambda no longer used

* remove EDL env params from sendtogitc

* change parsing of granule concept id

* /version 0.2.0a6

* Update README.md

* /version 0.2.0a7

* Support datetimes without microseconds (#37)

* Support datetimes without microseconds

* changelog

* Update generate_image_metadata.py (#39)

add docstring

* /version 0.2.0a8

* /version 0.2.0a9

* Issues/23: Enable Harmony write output to user bucket (#46)

* add s3 destination to harmony job

* syntax

* job success

* harmony block update

* --variable

* temp test in uat

* manual to uat

* test s1-v1

* lint

* deploy --sit

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

---------

Co-authored-by: Josh <joshhaile7@gmail.com>

* /version 0.2.0a10

* fix deploy failure

* /version 0.2.0a11

* Add new message attribute response_topic_arn to all outgoing messages to GIBS (#47)

* /version 0.2.0a12

* prep for 0.2.0

* /version 0.2.0a13

* Fix build error

* /version 0.2.0a14

* /release 0.2.0

* Update graph

* /version 0.2.0rc1

* Turn prevent_destroy back on for gibs response topic

* /version 0.2.0rc2

* Improve error message when missing config

* /version 0.2.0rc3

* Improve error message when missing config

* /version 0.2.0rc4

* UAT is in east-1 region

* /version 0.2.0rc5

* fix gibs queue name

* /version 0.2.0rc6

* calculate md5sum for files from harmony

* calculate md5sum for files from harmony

* /version 0.2.0rc7

* bump chunk size to 100 MB

* /version 0.2.0rc8

* Update test message for UAT, add checksum to generated tif, add logging, fix type error

* Lint

* Lint

* /version 0.2.0rc9

* Change named variables for audit path from pobit to bignbit

* /version 0.2.0rc10

* Update docs

---------

Co-authored-by: frankinspace <frankinspace@users.noreply.github.com>
Co-authored-by: Josh Haile <35245966+voxparcxls@users.noreply.github.com>
Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com>
Co-authored-by: Jake Herrmann <jtherrmann1@gmail.com>
Co-authored-by: James Wood <James.F.Wood@jpl.nasa.gov>
Co-authored-by: jamesfwood <jamesfwood@users.noreply.github.com>
Co-authored-by: Josh <joshhaile7@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@frankinspace frankinspace frankinspace approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@voxparcxls @frankinspace