feat: Schema Only Flag

[joshua-janicas]

Feature scope

Taps (catalog, state, tests, etc.)

Description

Feature Idea

• A configurable option or command of a tap that would perform a schema/header extract only with no data attached. This would not update any state files.
• Should be flexible with most taps (including those that are not database focused) where we just pull the "fields/column names" from our data sources and ignore the data itself.
• Example: pg_dump with --schema-only in PostGres, BACPAC vs DACPAC in MSSQL.
• --schema-only or some more appropriate generic naming

Use Case

• Populating a target with empty fields
• Creating a "EMPTY_DB" in Snowflake with only empty tables, no data

it's a weird intersection of our business intelligence software and how our upstream data is handled. we need to template reports with no data to give to our clients to publish. the rub is that each client is on a different version of our database which we import data from using tap-mssql and those versions may have schema differences. we can pin a client to a specific version of our elt pipeline but that results in potentially different tables (a column added, renamed...) (edited) 

[visch]

I wonder if a mapper would do this today with filters. Try setting the filter to False for every record https://sdk.meltano.com/en/v0.43.1/stream_maps.html#filtering-out-records-from-a-stream-using-filter-operation

Might work 🤷

[visch]

If I really needed this, I'd mess with the CLI test options and manually do it so something like

1. meltano invoke tap-name --test schema > out
2. cat out | meltano invoke target-name