Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DIG-1344: make sure no dup IDs in schemas #36

Merged
merged 29 commits into from
Oct 30, 2023
Merged

DIG-1344: make sure no dup IDs in schemas #36

merged 29 commits into from
Oct 30, 2023

Conversation

daisieh
Copy link
Member

@daisieh daisieh commented Oct 19, 2023

Validate that there are no duplicate IDs within an ingest dataset, at least within each schema.

There are tests to check for this in the test file.

NOTE: there are also extra bonus fixes here:

  • when something fails during CSVConvert, it will write out the current state of INDEXED_DATA to x_indexed.json, including all calculated fields in a virtual sheet called CALCULATED
  • added a mapping function that just concatenates all the values passed to it (seemed possibly useful)
  • added a bunch of extra checks in the validation methods to check for keys that exist but are null. (this is so that you can validate the output of katsu's authorized/donor_with_clinical_data/ endpoint and have that pass)

None of these bonus fixes should affect behavior, at least as far as I can tell.

@daisieh daisieh requested a review from yavyx October 19, 2023 22:54
@daisieh daisieh marked this pull request as draft October 26, 2023 02:28
@daisieh
Copy link
Member Author

daisieh commented Oct 26, 2023

I'm pulling this back to draft because I keep pushing changes onto the branch and Javi hasn't had time to review anyway.

@daisieh daisieh force-pushed the daisieh/redcap branch 2 times, most recently from fb21eb7 to 60f9ebf Compare October 26, 2023 03:01
@daisieh
Copy link
Member Author

daisieh commented Oct 26, 2023

Rebased this branch to remove the jsonschema stuff. That will go into a different PR.

@daisieh daisieh marked this pull request as ready for review October 26, 2023 03:03
@daisieh daisieh requested review from kcranston and removed request for yavyx October 26, 2023 18:02
@daisieh daisieh merged commit fb5d76e into main Oct 30, 2023
@daisieh daisieh deleted the daisieh/redcap branch October 30, 2023 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants