Add batching to Graphite inputs #2683
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This means that batch sizes of 0 mean send immediately (like a batch size of 1).
This is more useful for client code, and is a more common Go pattern. As a result the goroutine is launched within start.
otoolep
force-pushed
the
batch_integration
branch
from
4852655 to
a9ba189
Compare
otoolep
changed the title
otoolep
force-pushed
the
batch_integration
branch
from
deddc0d to
0f385d2
Compare
| for { | ||
| // Read up to the next newline. | ||
| buf, err := reader.ReadBytes('\n') | ||
| if err != nil { | ||
| close(done) |
There was a problem hiding this comment.
Do we need some type of flusher on the batcher so that when we shut down we can ensure it drains the batch before we exit?
|
This looks good. My only concern is that we don't drain a batch on shutdown, which could mean we lose data that was in flight. We could always add that later as well if we don't want to support that now. |
|
Thanks @corylanou -- I agree that is a good idea. Let me get on that immediately after merging this, OK? |
|
+1 🚢 |
otoolep
added a commit
that referenced
this pull request
May 28, 2015
Add batching to Graphite inputs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change integrates the new
PointBatcherwith the Graphite inputs. Most of the work involves passing batching configuration parameters to the Graphite inputs and having those same inputs use the batcher.There was also one change made to the batcher itself. Previously client code had to pass in channels for sending and receiving points (and point-batches). Now the batcher creates those same channels itself, and returns them. That can be seen in this commit: a9ba189
All Graphite inputs come up with a batch size of 1 by default. This means the lowest performance, but the maximum safety with regards to data loss.