Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async Destinations - Make debug messages less scary #32526

Merged
merged 9 commits into from
Nov 15, 2023

Conversation

evantahler
Copy link
Contributor

@evantahler evantahler commented Nov 14, 2023

Closes #32465

This PR adjusts the logging frequency and content of Destination Async framework. Prior to this PR logs like this would happen every 10 seconds:

2023-11-14 22:46:59 destination > 2023-11-14 22:46:59 INFO i.a.c.i.d.b.BufferManager(printQueueInfo):120 - START OF QUEUE INFO
2023-11-14 22:46:59 destination > This represents an estimation of the size of the elements contain in the in memory buffer.
2023-11-14 22:46:59 destination >   Global Mem Manager -- max: 5.27 GB, allocated: 70 MB (70.0 MB), % used: 0.01298219584569733
2023-11-14 22:46:59 destination >   Queue name: users, num records: 61980, num bytes: 40.72 MB, allocated bytes: 50 MB
2023-11-14 22:46:59 destination >   Queue name: products, num records: 100, num bytes: 30.71 KB, allocated bytes: 10 MB
2023-11-14 22:46:59 destination > State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2023-11-14 22:46:59 destination > END OF QUEUE INFO
2023-11-14 22:46:59 destination > 2023-11-14 22:46:59 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - WORKER INFO
2023-11-14 22:46:59 destination >   Pool queue size: 0, Active threads: 0

These logs are verbose, slow down log rendering, and look like something is wrong with the destination.

After this PR:

  • Log frequency changed from every 10s to every 60s
  • Both loggers (printWorkerInfo and printQueueInfo) only emit one line per log message
2023-11-14 22:52:53 destination > 2023-11-14 22:52:53 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - [ASYNC WORKER INFO]   Pool queue size: 0, Active threads: 1
2023-11-14 22:52:53 destination > 2023-11-14 22:52:53 INFO i.a.c.i.d.b.BufferManager(printQueueInfo):118 - [ASYNC QUEUE INFO] Global: max: 5.27 GB, allocated: 250 MB (250.0 MB), % used: 0.046364985163204746 | Queue `users`, num records: 35017, num bytes: 23.08 MB, allocated bytes: 30 MB | Queue `products`, num records: 100, num bytes: 30.71 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000

Copy link

vercel bot commented Nov 14, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Nov 15, 2023 6:47pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Nov 14, 2023
@evantahler evantahler marked this pull request as ready for review November 14, 2023 22:57
@evantahler evantahler requested a review from a team as a code owner November 14, 2023 22:57
@evantahler
Copy link
Contributor Author

FYI @benmoriceau and @tryangul

@evantahler
Copy link
Contributor Author

evantahler commented Nov 15, 2023

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6881455526
✅ Successfully published Java CDK version=0.4.11!

@evantahler evantahler enabled auto-merge (squash) November 15, 2023 18:47
@evantahler
Copy link
Contributor Author

evantahler commented Nov 15, 2023

CDK Published, useLocalCdk back to false, automerge enabled

@evantahler evantahler merged commit 0e166eb into master Nov 15, 2023
22 checks passed
@evantahler evantahler deleted the evan/printQueueInfo branch November 15, 2023 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/destination/bigquery connectors/destination/snowflake
Projects
None yet
Development

Successfully merging this pull request may close these issues.

printQueueInfo is too noisy
4 participants