Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Pipeline fails if archive enabled with s3 and tap table name is not in format <<schema_name>>__<<table>> #235

Open
lfpll opened this issue Nov 30, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@lfpll
Copy link

lfpll commented Nov 30, 2021

When using s3 with archive like:
archive_load_files: true

With taps that generate streams with naming <<table>> instead of <<schema_name>>__<<table>> it will break on archiving into s3.
Example taps: tap-pend, tap-gitlab

This happens because in STREAM_UTILS_L97-L11, where the schema_name will be set to None if the name do not follow the pattern <schema_name>_<table>.

In UPLOAD_CLIENTS_L107 if the metadata has {"schema_name":None} that will throw an error on trying to convert to ascii.

To Reproduce

To reproduce use use in meltano.yaml

s3_bucket:          "your_bucket"
stage:                  '"your_stage'
archive_load_files: true

Use a tap like tap-gitlab or ta-pendo
Error: AttributeError: 'NoneType' object has no attribute 'encode'
From:

File "..../target_snowflake/upload_clients/s3_upload_client.py", line 111, in copy_object
target-snowflake | loader    |     Metadata=metadata, MetadataDirective="REPLACE")

Expected behavior
I want to be able to archive data into s3 when data is comming from taps that only send <<table_name>> on stream list instead of <<schema_name>>_<<table_name>>

@lfpll lfpll added the bug Something isn't working label Nov 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant