v4.0.0

This release includes changes since the last Software Integration Test (SIT-3) and roughly serves as a 'half way' mark to the next SIT (SIT-4, scheduled for April 2024).

Features

• API query results are now sorted
• A single database session is used and passed around the code instead of new instances being created
• Various updates to database schema
• Added S3 mounts to containers for I-ALiRT
• Implemented SQS processing
• Now using shared Lambda layer for all lambdas
• Added region to the S3 presigned URLs
• Added AWS stack for SDC website deployments
• Various updates to instrument dependency table
• Added a database synchronizer to file database
• Now using separate database tables for different file types

What's Changed

• MNT: Sort query return results when returning from the API by @greglucas in #303
• MNT: Remove unused dependencies by @greglucas in #304
• MNT: Pass around database session rather than creating new ones by @greglucas in #307
• DB unique constraint by @greglucas in #308
• Db refactor by @greglucas in #309
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #310
• Db session fix by @tech3371 in #313
• Add S3 Mount to Containers for I-ALiRT by @laspsandoval in #305
• Sqs processing by @maxinelasp in #315
• Share Lambda layer for all lambda(s) by @tech3371 in #323
• I-ALiRT: Added DynamoDB Streams for Packet Ingest by @laspsandoval in #314
• Stack to construct by @greglucas in #326
• MNT: Use direct inheritance for simple resource like LambdaLayer by @greglucas in #331
• FIX: Add region to the s3 presigned urls by @greglucas in #330
• ENH: Add a stack for website deployments by @greglucas in #333
• MNT: Refactor batch processing by @greglucas in #332
• Harrison 1692 - SWYN/TLAY - added L3 SWAPI dependency configuration by @pleasant-menlo in #336
• Updated IDEX dependency to distinguish l1a and l1b by @bourque in #335
• SWAPI L2 dependency by @tech3371 in #338
• Lo dependency table update by @sdhoyt in #340
• Changes to DynamoDB to accomodate query patterns by @laspsandoval in #337
• HIT - Use Raw L0 File as Single L1A Dependency by @sdhoyt in #347
• ENH: Add a DB synchronizer for our file database by @greglucas in #342
• Fixing unique constraint for dependency table by @maxinelasp in #349
• Update EventBridge Rule to match new spice path in data bucket by @tech3371 in #345
• Adding SWE L2 dependency by @tech3371 in #354
• MNT: Change the dependency database table into a local lookup by @greglucas in #352
• MNT/ENH: Separate database tables for different file types by @greglucas in #344
• BUG - Adding fixes for psycopg by @tech3371 in #365
• Add Final DynamoDB Table by @laspsandoval in #353
• Add dependency entry for swapi l3b combined by @pleasant-menlo in #363
• update with versioning command by @laspsandoval in #369
• I-ALiRT: Switch out ECR for Nexus by @laspsandoval in #370
• I-ALiRT: update to readthedocs by @laspsandoval in #371
• V4.0.0 Release by @bourque in #378

New Contributors

• @pleasant-menlo made their first contribution in #336

Full Changelog: v3.0.0...v4.0.0

v3.0.0

This release includes updates to be used for the IMAP SDC Software Integration Test 3 (SIT-3). More information can be found in this galaxy page.

Features

• Added tests for backup S3 buckets
• Added RDS database schemas and tables
• Now using shared resources in a common environment/account
• Added support for ancillary data access from EFS
• Added cloudwatch monitoring
• Added API endpoint for the universal spin table
• Various updates to /upload, /download, and /query endpoints
• Added documentation on how to use personal AWS accounts for deployment
• Removed use of OpenSearch and StepFunctions
• Replaced processing step stack with Batch job stack
• Added batch job event handler to indexer lambda
• Added pre-processing dependency table
• Added I-ALiRT architecture for IOIS
• Standardized logging across the package
• Now using imap-data-access for file validation
• Added SNS Notification
• Added ECS to I-ALiRT Architecture
  Enabled SPICE file uploads to the SDC
• Added dependencies for all instruments
• Removed char limit on descriptor
• Automated Docker processing deployments

What's Changed

• Remove API Wrapper Scripts by @sdhoyt in #126
• Backup bucket tests by @maxinelasp in #146
• SIT-2 Release [dev] by @bourque in #151
• First pass at processing architecture by @laspsandoval in #154
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #155
• Starting point for RDS by @laspsandoval in #180
• Share resources in a common environment/account and auto deploy resources by @greglucas in #181
• CI: Update codecov configuration by @greglucas in #189
• CI: Use poetry run when deploying resources by @greglucas in #190
• Ancillary data access from EFS by @tech3371 in #188
• Remove old processing pipeline from SIT1 by @tech3371 in #195
• ENH: Add cloudwatch monitoring by @greglucas in #197
• Added endpoint to the API for universal spin table by @tech3371 in #199
• RDS with Schema Creation Custom Resource by @sdhoyt in #196
• CI: Remove version pin on npm cdk version installation by @greglucas in #201
• Processing Trigger Mechanisms by @laspsandoval in #191
• fix to upload folder path by @tech3371 in #198
• Setup Personal AWS Deploy Documentation by @sdhoyt in #200
• Added some more info to personal AWS account docs by @bourque in #205
• Upload config remove by @tech3371 in #202
• Create spin table by @tech3371 in #204
• ENH: Add redirect response from download url by @greglucas in #206
• Remove OpenSearch by @sdhoyt in #207
• Metadata Schema by @sdhoyt in #209
• update to batch env by @laspsandoval in #212
• Metadata Ingest by @sdhoyt in #210
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #211
• Remove Step Functions by @laspsandoval in #214
• Reorganize sds manager stack by @tech3371 in #215
• Replace processing step stack with Batch job stack by @tech3371 in #216
• Indexer events by @tech3371 in #217
• Api dockerfile for testing by @laspsandoval in #219
• InstrumentLambda from Construct to Stack by @laspsandoval in #220
• Update to indexer lambda to make pytest work locally by @tech3371 in #224
• Update File Catalog and Add Status Table by @sdhoyt in #223
• Query RDS with API by @sdhoyt in #228
• Added pre-processing dependency table by @tech3371 in #231
• I-ALiRT: First pass at architecture for IOIS by @laspsandoval in #208
• Batch starter by @laspsandoval in #230
• Indexer events by @tech3371 in #229
• added batch job event handler to indexer lambda by @tech3371 in #233
• Add event rule to batch starter lambda by @tech3371 in #236
• Various updates to database schema by @bourque in #232
• Added custom PutEvent to Batch Starter Lambda by @tech3371 in #234
• Updates to Batch Job by @laspsandoval in #238
• setup to send event from indexer lambda. by @tech3371 in #237
• FIX: Change to path params instead of query params in APIs by @greglucas in #235
• Batch starter fix by @tech3371 in #239
• Rename s3 uri by @tech3371 in #240
• Return list of dicts in query API by @bourque in #241
• Added name for database extensions enum by @sdhoyt in #243
• Prevent database duplicates by @sdhoyt in #244
• Fixes from Integration tests by @tech3371 in #245
• FIX: Change download API responses by @greglucas in #248
• MNT: Standardize logging across the package by @greglucas in #247
• Indexer lambda update post demo by @tech3371 in #253
• Updated ruff by @sdhoyt in #256
• update to use imap-data-access file validator by @tech3371 in #255
• Batch starter updates by @tech3371 in #257
• updated --data_level to --data-level to follow best practices by @tech3371 in #259
• Update dependency from json to database by @tech3371 in #260
• Consistent ruff checks with imap_processing by @bourque in #258
• added few instruments to list by @tech3371 in #261
• Infrastructure changes to match filename convention updates by @tech3371 in #262
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #263
• Added SNS Notification by @laspsandoval in #266
• Add ECS to I-ALiRT Architecture by @laspsandoval in #265
• ENH: Enable SPICE file uploads to the SDC by @greglucas in https://githu...

v2.0.0

This release marks the successful completion of SIT-2. Further information about what was tested in SIT-2 can be found in this galaxy page and the SIT-2 GitHub Project Board.

Features

• Ability to deploy and test software in the SDC Staging environment.
• Software can be successfully promoted from the SDC Staging Environment to the SDC Data Production Environment.
• Delivery of data files to the SPDF for routine archival purposes.
  • Backing up both OpenSearch and the data bucket to the backup account.
• Use of the API data interfaces by both SDC personnel as well as instrument teams. APIs:
  • The /upload endpoint for uploading new data to SDC
  • The /download endpoint for downloading data from SDC
  • The /query endpoint for querying information about data and data location.
• Demonstrate the SDC's ability to trigger jobs based on data availability and launch necessary processing containers.
  • Implementing a data watcher (Database) that keeps track of processing status.
  • Establishing a processing pipeline that initiates processing tasks.

What's Changed

• Primarily adding more fine-grained permissions to the lambda by @bryan-harter in #30
• Adding a way to create unique stack names each deploy by @bryan-harter in #29
• Adding the upload API to the stack and adding the Upload API code by @bryan-harter in #33
• MNT: Style updates by @greglucas in #19
• Adding poetry to the project by @maxinelasp in #38
• TST: Add test for s3 bucket properties by @greglucas in #42
• MNT: Ignore deprecation warnings by @greglucas in #44
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #51
• 1 create a spot to store configjson in a centralized location on aws by @sdhoyt in #50
• Improve stack creation testing by @sdhoyt in #52
• Update issue templates to include spike by @maxinelasp in #58
• 24 mock opensearch for unit testing by @sdhoyt in #69
• PR template by @sdhoyt in #70
• moved pull request template one directory up by @sdhoyt in #78
• Code of Conduct by @bourque in #77
• Modularize CDK code by @laspsandoval in #101
• API Gateway by @laspsandoval in #102
• TST: Reorganizing tests and moving to pytest-style by @greglucas in #103
• Removing templates now that we have a project-wide template for issues by @maxinelasp in #106
• Documentation by @maxinelasp in #109
• Add pre-commit hooks by @bourque in #115
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #80
• Updated infrastructure tests by @laspsandoval in #118
• adding .rst to docs/cdk by @GFMoraga in #114
• MNT: Add more information in the app template file by @greglucas in #112
• Implementation of Track Data Availability by @tech3371 in #120
• Changing the Upload API to use a GET rather than a POST by @bryan-harter in #110
• Initial Implementation of Trigger jobs based on data availability by @tech3371 in #119
• Opensearch snapshot by @sdhoyt in #121
• S3 backups by @maxinelasp in #125
• Updating permissions for data bucket role - SIT2 fix by @maxinelasp in #128
• SIT-2 Release [main] by @bourque in #150

Full Changelog: v1.0.1...v2.0.0

v1.0.0

This is the first successful release of the SDS Data Management software. It was released released after SOC SDC System Integration Test #1.

What's Changed

• 2 have indexerpy store file metadata in the opensearch cluster by @sdhoyt in #11
• TST/CI: Add automated testing for pull requests by @greglucas in #18
• 7 create an api for downloading the data by @tech3371 in #20
• 4 create api for querying the data by @sdhoyt in #21

Full Changelog: https://github.com/IMAP-Science-Operations-Center/sds-data-manager/commits/1.0.0

v1.0.1

SIT1 Fixes

Only one defect was found in SOC SDC System Integration Test (SIT) 1. The defect was that we failed to block public access on the data bucket. It was fixed in this release.

What's Changed

• Post SIT1 fix by @tech3371 in #39

Full Changelog: 1.0.0...v1.0.1