CUMULUS-3756 #3790
Merged
CUMULUS-3756 #3790
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…file-links Updated tests to exercise new file-exclusion feature
etcart
changed the base branch from
master
to
ecarton/nsidc-NDCUM-1264-excludeFileRegex
etcart
changed the base branch from
ecarton/nsidc-NDCUM-1264-excludeFileRegex
to
master
etcart
changed the base branch from
master
to
ecarton/nsidc-NDCUM-1264-excludeFileRegex
etcart
merged commit 03a54c0
into
nasa:ecarton/nsidc-NDCUM-1264-excludeFileRegex
2 checks passed
etcart
added a commit
that referenced
this pull request
Oct 11, 2024
…e-granules-cmr-metadata-… (#3791) * Added excludeFileRegex configuration to update-granules-cmr-metadata-file-links (#3790) Updated tests to exercise new file-exclusion feature * linter fixes * remove explicit null for un-found regexpattern * switch to logging when no excludable files found * changelog broken into multiple lines * linter fixes in changelog * name in changelog after lambda function name * remove TODO. non-mocked is a truer representation of function * small refactor * typo in passthrough of fileregex * nyc values with new tests * version requirement update * fixed merge weirdness * fix jsonpath in the other places it's flagged * remove unneeded explicit pin in aws-client * check like instead of deepequal on credentials return --------- Co-authored-by: Mike Dorfman <42116953+mikedorfman@users.noreply.github.com>
Jkovarik
pushed a commit
that referenced
this pull request
Oct 22, 2024
…e-granules-cmr-metadata-… (#3791) * Added excludeFileRegex configuration to update-granules-cmr-metadata-file-links (#3790) Updated tests to exercise new file-exclusion feature * linter fixes * remove explicit null for un-found regexpattern * switch to logging when no excludable files found * changelog broken into multiple lines * linter fixes in changelog * name in changelog after lambda function name * remove TODO. non-mocked is a truer representation of function * small refactor * typo in passthrough of fileregex * nyc values with new tests * version requirement update * fixed merge weirdness * fix jsonpath in the other places it's flagged * remove unneeded explicit pin in aws-client * check like instead of deepequal on credentials return --------- Co-authored-by: Mike Dorfman <42116953+mikedorfman@users.noreply.github.com>
Jkovarik
added a commit
that referenced
this pull request
Oct 24, 2024
* CUMULUS-3891/3773: Add sftpFastDownload config to SyncGranule (#3818) * CUMULUS-3891: Add fastGet download option to sftp data file download * add sftpFastDownload config * fix fastDownload boolean vs string * add unit tests * fix aws-client services unit test * test SFTP_DEBUG * remove only * add changelog entry * remove unused code * remove jsonpath from common * update latest-version and add jsonpath-plus to audit-ci * serial * update readme remove serial * add sftp test * CUMULUS-3756 NDCUM-1264 Added excludeFileRegex configuration to update-granules-cmr-metadata-… (#3791) * Added excludeFileRegex configuration to update-granules-cmr-metadata-file-links (#3790) Updated tests to exercise new file-exclusion feature * linter fixes * remove explicit null for un-found regexpattern * switch to logging when no excludable files found * changelog broken into multiple lines * linter fixes in changelog * name in changelog after lambda function name * remove TODO. non-mocked is a truer representation of function * small refactor * typo in passthrough of fileregex * nyc values with new tests * version requirement update * fixed merge weirdness * fix jsonpath in the other places it's flagged * remove unneeded explicit pin in aws-client * check like instead of deepequal on credentials return --------- Co-authored-by: Mike Dorfman <42116953+mikedorfman@users.noreply.github.com> * update dependencies to latest cma, cma-python, cumulus-process (#3810) * update dependencies to latest cma, cma-python, cumulus-process * changelog * fixed shas in locks * whitelist jsonpath for buiggy audit behavior * remove incorrect changelog entry --------- Co-authored-by: etcart <amberhosen@gmail.com> * CUMULUS-3919:Added terraform variables disableSSL and rejectUnauthorized (#3832) * CUMULUS-3919:Added terraform variables disableSsl and rejectUnauthorized * disableSsl->disableSSL * Fix bad merge * Fix audit-ci * Address bad merge on #3818 * Update audit-ci * [CUMULUS-3902]: Update AWS-SDK errors to use the V3 error classes (#3807) i* Update AWS errors to use the V3 error classes * Fix lint * Import aws sdk directly to avoid circular dep * Update CL * Remove module in favor of aws imports directly * Revert change to ThrottlingException error type * Add comments * Fix lint * Remove unnecessary dependency * add debug logging for CI * update type and debug comment * temporarily revert to name checks * Remove logging and type check on conditional exception. Instance of does not work in this case. I believe we're calling the service "dynamodbDocClient" using the non-V3 syntax. * Fix lint * Update tests to throw correct aws-sdk error * Update tests with new aws-sdk error types * Import error type correctly * Correctly import sfn error * Instantiate errors like I know what I'm doing * Basic syntax 🤦 * update tests * Remove unnecessary comment * fixup for clarity * Update test for clarity * Update test fixture and logging for consistency * Address PR feedback/fix bad backport merge * Correct aws-client dev dependency/version --------- Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: etcart <37375117+etcart@users.noreply.github.com> Co-authored-by: Mike Dorfman <42116953+mikedorfman@users.noreply.github.com> Co-authored-by: cumulus-bot <141277837+cumulus-bot@users.noreply.github.com> Co-authored-by: etcart <amberhosen@gmail.com> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
merged to #3791 for further testing
Added excludeFileRegex configuration to update-granules-cmr-metadata-file-links and updated unit tests to exercise new file-exclusion feature.
Addresses CUMULUS-3756: Add the ability to exclude URLs matching a specific pattern in the CMR metadata
Changes
PR Checklist