tap-google-sheets not syncing all columns in source sheet

[henriblancke]

Subject of the issue

I'm trying to integrate tap-google-sheets into pipelinewise. Not all columns in my sheet are propagated to the target. The logs show the tap is identifying and selecting to correct range of cols in the sheet and when running the tap without pipelinewise with the standard csv or json target, the resulting file contains all the data in the sheet.

Your environment

• pipelinewise==0.16.0
• tap-google-sheets
• pipelinewise-target-snowflake

Steps to reproduce

Follow the contribution guide on how to add a new tap and rebuild the docker image now that tap-google-sheets is included. Add a yaml config with client_id, client_secret, refresh_token and spreadsheet_id under db_conn. Add the sheet name as table_name you want to move into your target. Import the yaml and run the tap.

Expected behaviour

I'd expect all columns in the sheet to show up in snowflake.

Actual behaviour

The tap runs, but the csv that gets uploaded to S3 by pipelinewise doesn't contain all columns in the source sheet (3 in my case).

Further notes

Am I missing something vital for this tap to integrate with pipelinewise? Thank you for your help!

[Samira-El]

Hey, have you tried running the tap in a standalone mode to verify if the streams it generates are accurate?

[henriblancke]

@Samira-El thanks for the response! Yep I did and things run as expected and are accurate.

[Samira-El]

Ok, then it sounds like the issue is in the target, because it's the one that parses the tap messages and creates csv file in S3.
Can you try feeding the tap streams to pipelinewise-target-snowflake and see if it processes every property in the SCHEMA type stream?

[henriblancke]

@Samira-El I think the problem here might be that tap-google-sheets has the following as typedefs for schemas and pipelinewise-target-snowflake doesn't know how to handle that:

"anyOf": [
    {
        "type": "null"
    },
    {
        "type": "number"
    },
    {
        "type": "string"
    }
]

[Samira-El]

Hey, glad to see that you've made progress on this issue!

Indeed, anyOf in stream's schema is currently not supported by ppw-target-snowflake.

There is an issue similar to this in the ppw-target-snowflake repo: transferwise/pipelinewise-target-snowflake#88

What do you think should be the best way to handle these typedefs?

[Samira-El]

Closing as this is not a bug in Pipelinewise, but rather target-snowflake lacking support for anyOf.