fix(ingestion/airflow-plugin): warning log for non-materialized iolets #10421
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ingestion
dushayntAW
changed the title
dushayntAW
changed the title
dushayntAW
force-pushed
the
fix/ING-601/logging-for-non-materialized-iotlets-in-airflow-ingestion
branch
from
5a18105
to
efb5e6d
Compare
dushayntAW
force-pushed
the
fix/ING-601/logging-for-non-materialized-iotlets-in-airflow-ingestion
branch
from
d7a9823
to
c99b5b7
Compare
…terialize_iolets is false
dushayntAW
force-pushed
the
fix/ING-601/logging-for-non-materialized-iotlets-in-airflow-ingestion
branch
from
c99b5b7
to
6fcfe44
Compare
anshbansal
requested changes
May 7, 2024
Comment on lines
431
to
432
| clone_inlets=False if config and not config.materialize_iolets else True, | ||
| clone_outlets=False if config and not config.materialize_iolets else True, |
There was a problem hiding this comment.
These could just be config.materialize_iolets in both cases. That is a boolean, right?
There was a problem hiding this comment.
yes, but implemented this way as there might be a case where config is not having materialize_iolets as config is of type Optional[DatahubLineageConfig]
There was a problem hiding this comment.
Please use config and not config.materialize_iolets instead
There was a problem hiding this comment.
this is generating linter error
src/datahub_airflow_plugin/client/airflow_generator.py:431: error: Argument "clone_inlets" to "from_datajob" of "DataProcessInstance" has incompatible type "Union[DatahubLineageConfig, None, bool]"; expected "bool" [arg-type]
used config is None or config.materialize_iolets, instead
anshbansal
approved these changes
May 8, 2024
hsheth2
added a commit
that referenced
this pull request
May 29, 2024
This would cause it to not be respected for data process instances. Also fixes a bug from #10421.
5 tasks
sleeperdeep
pushed a commit
to sleeperdeep/datahub
that referenced
this pull request
Jun 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist