Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐞 CDK: fix error when logging warning for types that could not be transformed #17367

Merged
merged 2 commits into from
Sep 29, 2022

Conversation

pedroslopez
Copy link
Contributor

What

The warnings recently introduced to the TypeTransformer in #16695 were causing an error to be thrown when the mismatched type was inside an array

Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 123, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 114, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 127, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 113, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 182, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 246, in _read_incremental
    yield self._as_airbyte_record(stream_name, record_data)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 316, in _as_airbyte_record
    transformer.transform(data, schema)  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/utils/transform.py", line 189, in transform
    logger.warning(self.get_error_message(e))
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/utils/transform.py", line 193, in get_error_message
    key_path = "." + ".".join(e.path)
TypeError: sequence item 2: expected str instance, int found

This is beacuse e.path in this case was ['event_properties', 'items', 0].

How

Fix the issue by mapping all path parts to string

@pedroslopez pedroslopez requested a review from a team as a code owner September 29, 2022 03:38
@github-actions github-actions bot added the CDK Connector Development Kit label Sep 29, 2022
Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not forget to update the changelog + bump cdk version 😄

@pedroslopez
Copy link
Contributor Author

pedroslopez commented Sep 29, 2022

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/3155046386
https://github.com/airbytehq/airbyte/actions/runs/3155046386

@pedroslopez pedroslopez merged commit 58f5128 into master Sep 29, 2022
@pedroslopez pedroslopez deleted the pedroslopez/fix-transformer-warn-err branch September 29, 2022 22:48
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
…nsformed (airbytehq#17367)

* fix error when logging warning for types that could not be transformed

* changelog and bump
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CDK Connector Development Kit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants