New fastsync branch #11
Conversation
| import singer | ||
| from singer import metadata | ||
| from singer import utils | ||
| import singer.metrics as metrics |
| @@ -32,7 +42,19 @@ def generate_bookmark_keys(catalog_entry): | |||
| bookmark_keys = base_bookmark_keys | |||
|
|
|||
| return bookmark_keys | |||
|
|
|||
| def write_dataframe_record(row, catalog_entry, stream_version, columns, table_stream, time_extracted): | |||
There was a problem hiding this comment.
Are catalog_entry or columns used anywhere? If not can we remove them?
| ) | ||
|
|
||
| columns.sort() | ||
| select_sql = common.fast_sync_generate_select_sql(catalog_entry, columns) |
There was a problem hiding this comment.
I think we already discussed this, but again just to confirm - we want full syncs to always default to fast sync right? We don't even want the option to do a full sync without it?
There was a problem hiding this comment.
It's ok if that's what we want to do, but I think my preference is to provide the option to use either fast sync or the Singer spec for the full table refresh. That could look something like moving all of this logic into another method called fastsync_table and keeping the old logic in sync_table. From there in init.py in the method do_sync_full_table we could add a line for if fastsync: fastsync_table else: sync_table.
This way we could turn fast sync on on a table by table basis if we wanted.
No description provided.