WIP: batch points in kafka messages

[woodsaj]

No description provided.

[Dieterbe]

interesting. do you have instructions to start kafka to repeat this experiment? can we just use the service as defined in one of the docker stacks?

[woodsaj]

do you have instructions to start kafka to repeat this experiment

docker run -d -p 9092:9092 -p 2181:2181  --name kafka -h localhost raintank/kafka

[replay]

nice. if tsdb needs to batch the msgs before submitting to kafka, does that mean that crng will have to wait longer until it gets the acknowledgement? if we deploy that change we might need to tell the customers to increase their crng concurrency

[woodsaj]

@replay see #1002

short answer. yes, but we already do that. Kafka messages are already sent over the wire in batches. We already buffer in memory then flush to kafka, then ack the POST from crng. So this change wouldnt alter the crng->tsdb-gw interaction much. We would still be sending the same number of metrics in each "batch" sent to kafka, the only difference is that instead of sending a batch of 5000 metrics in a batch of 5000 kafka messages, we would be sending 5000 metrics in a batch of 50 kafka messages.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.