Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pull transformer outside of record write loop to minimize logging #61

Merged
merged 3 commits into from
Mar 13, 2019
Merged

pull transformer outside of record write loop to minimize logging #61

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fcedd23
pull transformer outside of record write loop to minimize logging
Mar 13, 2019
0d1b690
trigger circle
Mar 13, 2019
ff323fd
Merge remote-tracking branch 'origin/master' into move_transformer
Mar 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions tap_salesforce/sync.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,23 +60,23 @@ def resume_syncing_bulk_query(sf, catalog_entry, job_id, state, counter):

# Iterate over the remaining batches, removing them once they are synced
for batch_id in batch_ids[:]:
for rec in bulk.get_batch_results(job_id, batch_id, catalog_entry):
counter.increment()
with Transformer(pre_hook=transform_bulk_data_hook) as transformer:
with Transformer(pre_hook=transform_bulk_data_hook) as transformer:
for rec in bulk.get_batch_results(job_id, batch_id, catalog_entry):
counter.increment()
rec = transformer.transform(rec, schema)
rec = fix_record_anytype(rec, schema)
singer.write_message(
singer.RecordMessage(
stream=(
stream_alias or stream),
record=rec,
version=stream_version,
time_extracted=start_time))

# Update bookmark if necessary
replication_key_value = replication_key and singer_utils.strptime_with_tz(rec[replication_key])
if replication_key_value and replication_key_value <= start_time and replication_key_value > current_bookmark:
current_bookmark = singer_utils.strptime_with_tz(rec[replication_key])
rec = fix_record_anytype(rec, schema)
singer.write_message(
singer.RecordMessage(
stream=(
stream_alias or stream),
record=rec,
version=stream_version,
time_extracted=start_time))

# Update bookmark if necessary
replication_key_value = replication_key and singer_utils.strptime_with_tz(rec[replication_key])
if replication_key_value and replication_key_value <= start_time and replication_key_value > current_bookmark:
current_bookmark = singer_utils.strptime_with_tz(rec[replication_key])

state = singer.write_bookmark(state,
catalog_entry['tap_stream_id'],
Expand Down