Skip to content

ingestion-sink update
Compare
Choose a tag to compare
@relud relud released this 15 Apr 20:41
20200415
b43df65
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

changes since release 20200325 for ingestion-sink/:

  • Always provide executor to CompletableFuture::*Async (#1243)
    • Fix BIG_QUERY_OUTPUT_MODE=file_loads performance issue where the only executor thread would block via Thread.sleep
  • Record gcs and bq batch stats (#1240)
  • Update maven profiles to fix IDE issues and skip missing submodules (#1242)
  • Reduce DEFAULT_BATCH_MAX_DELAY to avoid OOM errors (#1233)
    • Reduce DEFAULT_BATCH_MAX_DELAY to 10s reverted by #1243
    • Reduced gcs and bigQueryFiles defaults for MAX_OUTSTANDING_ELEMENT_COUNT to 50,000 and MAX_OUTSTANDING_REQUEST_BYTES to 100MB
  • Close GCS batches from a synchronized context (#1237)
    • Fix message loss case where BatchWrite$Batch#write could be called after BatchWrite$Batch#close
  • Limit Gcs.Write to single storage.create api call (#1238)
    • Minimize storage api cost for very small batches
Assets 2
Loading