[exporter] Flip on queue batcher #11637

sfc-gh-sili · 2024-11-09T05:25:59Z

Description

This PR solves #10368.

Previously we use a pushing model between the queue and the batch, resulting the batch size to be constrained by the sending_queue.num_consumers, because the batch cannot accumulate more than sending_queue.num_consumers blocked goroutines provide.

This PR changes it to a pulling model. We read from the queue until threshold is met or timeout, then allocate a worker to asynchronously send out the request.

Link to tracking issue

Fixes #10368
#8122

Testing

This PR swaps out batch_sender directly and still passes all the existing tests.

Documentation

codecov · 2024-11-11T08:37:30Z

Codecov Report

Attention: Patch coverage is 70.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 91.43%. Comparing base (32abecb) to head (2982282).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
exporter/exporterhelper/internal/queue_sender.go	68.42%	4 Missing and 2 partials ⚠️
exporter/internal/queue/default_batcher.go	25.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11637      +/-   ##
==========================================
- Coverage   91.45%   91.43%   -0.02%     
==========================================
  Files         447      447              
  Lines       23721    23743      +22     
==========================================
+ Hits        21694    21710      +16     
- Misses       1653     1657       +4     
- Partials      374      376       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bogdandrutu · 2024-11-12T18:01:06Z

exporter/internal/queue/batcher.go

-// Shutdown ensures that queue and all Batcher are stopped.
-func (qb *BaseBatcher) Shutdown(_ context.Context) error {
-	qb.stopWG.Wait()
-	return nil
-}


Why this change?

See the other comment.

bogdandrutu · 2024-11-12T18:01:26Z

exporter/internal/queue/default_batcher.go

-				qb.currentBatchMu.Lock()
-				if qb.currentBatch == nil || qb.currentBatch.req == nil {
-					qb.currentBatchMu.Unlock()
-					continue
-				}
-				batchToFlush := *qb.currentBatch
-				qb.currentBatch = nil
-				qb.currentBatchMu.Unlock()
-
-				// flushAsync() blocks until successfully started a goroutine for flushing.
-				qb.flushAsync(batchToFlush)
-				qb.resetTimer()


Not sure I understand this, can we do this in a separate PR?

Thanks! Here it is: #11666
batch_sender_test helped me detect that the original implementation is missing a flush on shutdown.

dmitryax · 2024-11-12T23:28:58Z

Given the impact of this change (every collector user with sending_queue enabled, which is the default), I suggest we introduce it with a feature gate, e.g. exporter.batchingQeueue.

sfc-gh-sili · 2024-11-15T01:57:45Z

@dmitryax Hi Dimitrii, I wonder if you know what would be a better way to make sure existing tests pass with both feature gate on and off. Manually enabling then disabling in every single exporter test could work but I wonder if there's other option

dmitryax · 2024-11-20T19:02:57Z

exporter/exporterhelper/internal/base_exporter.go

 	"go.opentelemetry.io/collector/pipeline"
 )

+var usePullingBasedExporterQueueBatcher = featuregate.GlobalRegistry().MustRegister(
+	"telemetry.UsePullingBasedExporterQueueBatcher",
+	featuregate.StageBeta,


Why starting with Beta? That sounds too aggressive. Let's start with Alpha

#### Description This PR proceeds #11637. It * Introduces a noop feature gate that will be used for queue batcher. * Updates exporter tests to run with both the feature gate on and off.  #### Link to tracking issue #10368 #8122  #### Testing  #### Documentation

.chloggen/11637-exporter-queue-batcher.yaml

exporter/exporterhelper/internal/base_exporter.go

.chloggen/11637-exporter-queue-batcher.yaml

exporter/exporterhelper/internal/queue_sender.go

exporter/exporterhelper/internal/base_exporter.go

exporter/exporterhelper/internal/batch_sender_test.go

dmitryax

One question about tests. Otherwise LGTM

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

#### Description This PR proceeds open-telemetry#11637. It * Introduces a noop feature gate that will be used for queue batcher. * Updates exporter tests to run with both the feature gate on and off.  #### Link to tracking issue open-telemetry#10368 open-telemetry#8122  #### Testing  #### Documentation

#### Description This PR solves open-telemetry#10368. Previously we use a pushing model between the queue and the batch, resulting the batch size to be constrained by the `sending_queue.num_consumers`, because the batch cannot accumulate more than `sending_queue.num_consumers` blocked goroutines provide. This PR changes it to a pulling model. We read from the queue until threshold is met or timeout, then allocate a worker to asynchronously send out the request.  #### Link to tracking issue Fixes open-telemetry#10368 open-telemetry#8122 --------- Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

sfc-gh-sili force-pushed the sili-flip-on branch 7 times, most recently from 2900101 to 55aae5c Compare November 11, 2024 08:30

sfc-gh-sili marked this pull request as ready for review November 11, 2024 08:58

sfc-gh-sili requested a review from a team as a code owner November 11, 2024 08:58

sfc-gh-sili requested a review from songy23 November 11, 2024 08:58

songy23 requested review from bogdandrutu and dmitryax and removed request for songy23 November 11, 2024 15:08

bogdandrutu reviewed Nov 12, 2024

View reviewed changes

sfc-gh-sili changed the title ~~[exporter] Flip on queue batcher~~ [PAUSED] [exporter] Flip on queue batcher Nov 12, 2024

sfc-gh-sili force-pushed the sili-flip-on branch 8 times, most recently from 555baa2 to 6bb9b7f Compare November 15, 2024 01:29

sfc-gh-sili changed the title ~~[PAUSED] [exporter] Flip on queue batcher~~ [exporter] Flip on queue batcher Nov 15, 2024

sfc-gh-sili force-pushed the sili-flip-on branch 3 times, most recently from 844cfc6 to d158ac8 Compare November 20, 2024 01:56

dmitryax reviewed Nov 20, 2024

View reviewed changes

sfc-gh-sili mentioned this pull request Nov 21, 2024

[exporter] Feature gate for queue batcher #11721

Merged

sfc-gh-sili force-pushed the sili-flip-on branch 3 times, most recently from 97bd281 to 415dcb8 Compare November 22, 2024 21:29

sfc-gh-sili requested review from dmitryax and bogdandrutu November 22, 2024 21:46

dmitryax reviewed Nov 23, 2024

View reviewed changes

sfc-gh-sili force-pushed the sili-flip-on branch from 2a3200d to 7e1c11b Compare November 25, 2024 21:37

sfc-gh-sili requested a review from dmitryax November 25, 2024 21:47

dmitryax reviewed Dec 2, 2024

View reviewed changes

exporter/exporterhelper/internal/batch_sender_test.go Show resolved Hide resolved

dmitryax approved these changes Dec 2, 2024

View reviewed changes

sfc-gh-sili and others added 6 commits December 2, 2024 12:44

Flip on queue batcher

be9037a

Update .chloggen/11637-exporter-queue-batcher.yaml

e80d2f8

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

Update exporter/exporterhelper/internal/base_exporter.go

a112d27

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

Update .chloggen/11637-exporter-queue-batcher.yaml

9f4d67f

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

Update exporter/exporterhelper/internal/base_exporter.go

2af506c

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

edit according to Dmitrii's feedback

2982282

sfc-gh-sili force-pushed the sili-flip-on branch from d297257 to 2982282 Compare December 2, 2024 20:44

dmitryax merged commit 4782ad0 into open-telemetry:main Dec 2, 2024
49 of 50 checks passed

github-actions bot added this to the next release milestone Dec 2, 2024

sfc-gh-sili deleted the sili-flip-on branch December 4, 2024 00:10

swiatekm mentioned this pull request Dec 6, 2024

[testbed] Add batcher performance tests open-telemetry/opentelemetry-collector-contrib#36206

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter] Flip on queue batcher #11637

[exporter] Flip on queue batcher #11637

sfc-gh-sili commented Nov 9, 2024 •

edited

Loading

codecov bot commented Nov 11, 2024 •

edited

Loading

bogdandrutu Nov 12, 2024

sfc-gh-sili Nov 12, 2024

bogdandrutu Nov 12, 2024

sfc-gh-sili Nov 12, 2024

dmitryax commented Nov 12, 2024

sfc-gh-sili commented Nov 15, 2024

dmitryax Nov 20, 2024

dmitryax left a comment

[exporter] Flip on queue batcher #11637

[exporter] Flip on queue batcher #11637

Conversation

sfc-gh-sili commented Nov 9, 2024 • edited Loading

Description

Link to tracking issue

Testing

Documentation

codecov bot commented Nov 11, 2024 • edited Loading

Codecov Report

bogdandrutu Nov 12, 2024

Choose a reason for hiding this comment

sfc-gh-sili Nov 12, 2024

Choose a reason for hiding this comment

bogdandrutu Nov 12, 2024

Choose a reason for hiding this comment

sfc-gh-sili Nov 12, 2024

Choose a reason for hiding this comment

dmitryax commented Nov 12, 2024

sfc-gh-sili commented Nov 15, 2024

dmitryax Nov 20, 2024

Choose a reason for hiding this comment

dmitryax left a comment

Choose a reason for hiding this comment

sfc-gh-sili commented Nov 9, 2024 •

edited

Loading

codecov bot commented Nov 11, 2024 •

edited

Loading