fix(ingest/json-schema): convert non-string enums to strings #8479
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benjamin-awd
force-pushed
the
fix-non-string-jsonschema-enum
branch
from
6b5c5a7
to
72122fc
Compare
benjamin-awd
changed the title
github-actions
bot
added
the
ingestion
|
Thanks for the contribution! |
benjamin-awd
force-pushed
the
fix-non-string-jsonschema-enum
branch
2 times, most recently
from
e2b3623
to
95c6995
Compare
Hey @shirshanka, I've added a test -- let me know if there's anything else I'm missing.
|
Prior to this change, the schema enums were processed without being converted to their string representations. This caused an ingestion failure for schemas with non-string enum elements. To address this, the code has been refactored to ensure that all elements in the schema's enum list are now converted to their string representations using the `repr()` function. This change adds quotation marks around string elements, in order to clearly delineate strings from other special values. e.g. "Foo", "Bar", 0, null
Capitalization with the introduction of a colon creates a clean visual break - it gives the reader a good starting point since it distinguishes the actual elements of the enum from the general text e.g. one of foo,bar,baz -> One of: 'foo', 'bar', 'baz'
benjamin-awd
force-pushed
the
fix-non-string-jsonschema-enum
branch
from
95c6995
to
65889df
Compare
The current implementation using the repr function does not align with
JSON conventions. Notably, the repr function encloses strings in
single quotes ('baz') instead of using the standard double
quotes ("baz"), and it represents null values as
Python's None, rather than using JSON's explicit null keyword.
To ensure consistency and better adherence to JSON conventions,
we should switch to json.dumps() for the conversion.
json.dumps() ensures that strings are enclosed in double quotes and
null values are represented correctly as null.
benjamin-awd
force-pushed
the
fix-non-string-jsonschema-enum
branch
from
66f35fd
to
0798a81
Compare
hsheth2
approved these changes
Jul 24, 2023
hsheth2
added
the
merge-pending-ci
yoonhyejin
pushed a commit
that referenced
this pull request
Aug 24, 2023
spadhi7
added a commit
to spadhi7/datahub
that referenced
this pull request
Aug 29, 2023
* tag 'v0.10.5': (222 commits) fix(test): increase siblings.js test stability (datahub-project#8542) feat(search): Allow aggregating on facets that are not explicitly part of default filter set (datahub-project#8540) fix(ui) Make multiple small updates to new search and browse (datahub-project#8524) feat(presto-on-hive): allow v1 fieldpaths in the presto-on-hive source (datahub-project#8474) feat(cli): Adds ability to upload recipes to DataHub's UI (datahub-project#8317) feat(browseV2): add browseV2 logic to system update (datahub-project#8506) fix(ingest/json-schema): convert non-string enums to strings (datahub-project#8479) feat(ingestion/tableau): support column level lineage for custom sql (datahub-project#8466) test(ingest): test case statements with sql parser (datahub-project#8437) feat(ingest/vertica): performance improvement and bug fixes (datahub-project#8328) ci: reduce git fetch depth (datahub-project#8473) fix(ingest): remove duplication of tags (datahub-project#8532) docs: small update to homepage (datahub-project#8483) fix(ingest): pin boto3-stubs in CI (datahub-project#8527) feat(siblings): hiding non-existant siblings in FE (datahub-project#8528) fix(ingest/build): Fix sagemaker mypy and flake8 issues (datahub-project#8530) feat(metrics): add metrics for aspect write and bytes (datahub-project#8526) feat(elasticsearch): allow bulk delete (datahub-project#8424) fix(ui): use locale lowercase when filtering columns of an entity in the lineage (datahub-project#8213) fix(auth): ignore case when comparing http headers (datahub-project#8356) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to this change, the schema enums were processed without being converted to their string representations. This caused an ingestion failure for schemas with non-string enum elements.
To address this, the code has been refactored to ensure that all elements in the schema's enum list are now converted
to their string representations using the
json.dumps()function.This change adds quotation marks around string elements, which clearly delineates strings from other special values like integers or None types. (similar to the json-schema-for-humans library: 1, 2)
Checklist
Related issue: #8480