Epic/attribution lambda generalization #1022
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## 🎫 Ticket https://jira.cms.gov/browse/BCDA-8405 ## 🛠 Changes <!-- What was added, updated, or removed in this PR? --> Added the main entry to the CCLF lambda. Added the logic flow for checking if the SQS event contains a CSV file path, or CCLF. Renamed CCLF lambda functions to attribution agnostic. Added parser functions for checking CSV file. Added testing for CSV parser functions. ## ℹ️ Context These changes will be the entry for the lambda changes in order to make the CCLF lambda more agonistic to an attribution lambda. ## 🧪 Validation This PR is a part of a larger group of PR's that will come later. Some of these entries don't function properly, such as the CSVImporter. Therefore this PR is more-or less a check for making sure the functionality is up to standards for the team.
## 🎫 Ticket https://jira.cms.gov/browse/BCDA-4443 https://jira.cms.gov/browse/BCDA-4444 ## 🛠 Changes - CSV file import logic has been created that works with local file handler and s3 file handler - tests added/updated - removed most logging statements so that errors are pushed to `handlecsvimport`, where there is logging already. this reduces log noise - added custom errors to improve readability and error handling ## ℹ️ Context Changes were made to support attribution files in a csv file format. Follows the flow and structure of the CCLF file importer and was adapted to fit expected csv file format. <!-- If any of the following security implications apply, this PR must not be merged without Stephen Walter's approval. Explain in this section and add @SJWalter11 as a reviewer. - Adds a new software dependency or dependencies. - Modifies or invalidates one or more of our security controls. - Stores or transmits data that was not stored or transmitted before. - Requires additional review of security implications for other reasons. --> ## 🧪 Validation Added new tests and current tests passing. --------- Co-authored-by: Parwinder Bhagat <Parwinder.Bhagat@e14s.com>
## 🎫 Ticket https://jira.cms.gov/browse/BCDA-8548 ## 🛠 Changes - github workflow for attribution integration test has been updated to include the csv file type ## ℹ️ Context Made to support csv attribution types for models that utilize csv attribution files. <!-- If any of the following security implications apply, this PR must not be merged without Stephen Walter's approval. Explain in this section and add @SJWalter11 as a reviewer. - Adds a new software dependency or dependencies. - Modifies or invalidates one or more of our security controls. - Stores or transmits data that was not stored or transmitted before. - Requires additional review of security implications for other reasons. --> ## 🧪 Validation Integration test github action has been updated to include steps to generate a csv attribution file and ingest it. Test is passing and manually validated that database entries looks OK.
bhagatparwinder
approved these changes
Dec 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🎫 Ticket
https://jira.cms.gov/browse/BCDA-8176
🛠 Changes
Convert file attribution lambda to be more generalized by allowing for CSV files to be ingested as well.
ℹ️ Context
This is a large epic including multiple feature branches. Overall the goal is to convert the file attribution lambda to be more generalized by allowing for CSV files to be ingested as well.
🧪 Validation
Each individual feature branch was PRd before merging into epic branch. Each feature branch was also passing local and git linting and testing.