Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DIG-1079, DIG-1226: validate ETL beyond jsonschema #20

Merged
merged 35 commits into from
Jul 18, 2023
Merged

DIG-1079, DIG-1226: validate ETL beyond jsonschema #20

merged 35 commits into from
Jul 18, 2023
+912 −88

Conversation

daisieh
Copy link
Member

@daisieh daisieh commented Jul 14, 2023

  • Make defaults explicit: default mappings are implemented as mohschema.add_default_mappings and don't add back template lines not in the mapping file.
  • The mohschema class now implements its own validations beyond jsonschema. Fields required in the data model throw MoHValidationErrors; everything else gets logged as warnings.
  • Miscellaneous fixes that I noted while updating https://github.com/CanDIG/clinical_ETL_data/pull/21

Test by pulling this and then using the datasets in https://github.com/CanDIG/clinical_ETL_data/pull/21

daisieh added 27 commits June 26, 2023 22:03
@daisieh
consolidate template mapping into mohschema

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode
ea4c20d
@daisieh
bring back coverage check
7c2f17c
@daisieh
use mohschema.template
8514101
@daisieh
Update test_data_ingest.py
ede9acf
@daisieh
Update validate_coverage.py
e5c999a
@daisieh
bring back coverage check
fc701cc
@daisieh
use mohschema.template
70f21bb
@daisieh
Update test_data_ingest.py
92e8590
@daisieh
Update validate_coverage.py
02350e1
@daisieh
Update requirements.txt
aa0fa18
@daisieh
Merge branch 'daisieh/validate-coverage' of https://github.com/CanDIG…
2e02957 
…/clinical_ETL_code into daisieh/validate-coverage
@daisieh
first pass at additional validations
ae8e70f
@daisieh
we require 3.10
f64d11e
@daisieh
Update test.yml
9297c95
@daisieh
Update test.yml
2da8d3a
@daisieh
I missed allOf/anyOf/oneOf before
ac33b63
@daisieh
can factor out generate_schema_array
ea39e0b
@daisieh
deal with INDEX before matching
411f677
@daisieh
we no longer use * for required, so can simplify
1e54f33
@daisieh
Add default mappings to template
23d9416
@daisieh
Don't use default mappings implicitly
61869eb
@daisieh
Fix indexing
1357c16
@daisieh
better integer handling
9f25753
@daisieh
fix indexing
5cbb9e2
@daisieh
don't interpolate by default
cc60434
@daisieh
save messages off
3e92b2f
@daisieh
comment out coverage
dd24e2b
@daisieh daisieh requested a review from kcranston July 14, 2023 19:09
@daisieh daisieh changed the title DIG-1079: validate ETL beyond jsonschema DIG-1079, DIG-1226: validate ETL beyond jsonschema Jul 15, 2023
@daisieh daisieh requested a review from yavyx July 17, 2023 20:09
yavyx
yavyx approved these changes Jul 17, 2023
View reviewed changes
Copy link
Contributor

@yavyx yavyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and working

@kcranston
Copy link
Member

Can you also update the README to describe the new 'beyond jsonschema' validation?

@daisieh daisieh merged commit 187565a into main Jul 18, 2023
@daisieh daisieh deleted the daisieh/validate-coverage branch July 18, 2023 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yavyx yavyx

@kcranston kcranston

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@daisieh @kcranston @yavyx