feat(kafka): Add Ingestion from Kafka in Ingesters #14192
Conversation
github-actions
bot
added
the
type/docs
Co-authored-by: Joao Marcal <joao.marcal12@gmail.com>
|
|
||
| // ExtractIngesterPartitionID returns the partition ID owner the the given ingester. | ||
| func ExtractIngesterPartitionID(ingesterID string) (int32, error) { | ||
| if strings.Contains(ingesterID, "local") { |
There was a problem hiding this comment.
This is probably fine for now, but I don't like it as a long-term solution. I don't think its that common practice to include .local in the hostname across all major OSes.
It would be better to define an arg for this to set it at runtime. We could use the Downward pod API in kubernetes to inject the pod name, for example.
pkg/loki/modules.go
Outdated
| @@ -616,6 +616,9 @@ func (t *Loki) initIngester() (_ services.Service, err error) { | |||
| t.Server.HTTP.Methods("POST", "GET", "DELETE").Path("/ingester/prepare_shutdown").Handler( | |||
| httpMiddleware.Wrap(http.HandlerFunc(t.Ingester.PrepareShutdown)), | |||
| ) | |||
| t.Server.HTTP.Methods("POST", "GET", "DELETE").Path("/ingester/prepare-partition-downscale").Handler( | |||
There was a problem hiding this comment.
nit: prepare_shutdown uses underscores, prepare-partition-downscale uses hyphens. Ideally, these would be in sync. Except hyphens seems more web-friendly and changing prepare_shutdown to hyphens would be a breaking change.
Not a big deal, but would be nice to align these now before they get some usage.
There was a problem hiding this comment.
Sounds like Mimir uses dash everywhere. But It's a bit too late since prepare_shutdown is used.
This is used by the rollout operator and can be configured so I'll use _ everywhere. see https://github.com/grafana/rollout-operator?tab=readme-ov-file#delayed-scaledown
Co-authored-by: benclive <ben.clive@grafana.com>
What this PR does / why we need it:
This adds a new flag to allow starting ingesting from Kafka. It's heavily inspired by Mimir Kafka ingestion.
This is meant to be used as a new set of replica of ingesters.
The idea is simple, we keep the same ingestion from Ingester but allow to ingest from Kafka.
Ingesters now shares partition ownership through the partition ring.
A new downscale partition endpoint is added for downscaling and keeping partition alive until the ingester query window (2h) is passed. The new endpoint is used by the new rollout operator.
Which issue(s) this PR fixes:
Fixes https://github.com/grafana/loki-private/issues/1115
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mdproduction/helm/loki/Chart.yamland updateproduction/helm/loki/CHANGELOG.mdandproduction/helm/loki/README.md. Example PRdeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR