TempoDB: Restrict Ingestion time range #1332
Conversation
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
joe-elliott
requested review from
KMiller-Grafana,
annanay25,
mdisibio,
dgzlopes,
mapno,
yvrhdn and
zalegrala
as code owners
Signed-off-by: Joe Elliott <number101010@gmail.com>
I'm concerned about a wide time window b/c the larger the window the more unnecessary data you're pulling in during search. Also, if something is just outside the window (say 30s) it will likely still be picked up during a search b/c the whole block will be scanned. |
|
|
||
| if start < startOfRange { | ||
| metricWarnings.WithLabelValues(a.meta.TenantID, reasonOutsideIngestionSlack).Inc() | ||
| start = uint32(now.Unix()) |
There was a problem hiding this comment.
Should these clip to the range instead of now?
There was a problem hiding this comment.
The reason I decided to do use time.Now() is b/c we know we are regularly receiving backdated spans from multiple weeks ago.
If we use start instead of time.Now() then every block's time range will always start ingestion_time_range_slack earlier than it actually does. Also, every replayed block's start time will always be replay time - max_block_duration - ingestion_time_range_slack.
| @@ -342,7 +342,7 @@ func (i *Ingester) replayWal() error { | |||
| return 0, 0, err | |||
| } | |||
| return start, end, nil | |||
| }, log.Logger) | |||
| }, i.cfg.MaxBlockDuration, log.Logger) | |||
There was a problem hiding this comment.
Yeah agree this is a hard spot but understand the reasons, and this is reasonable. Maybe a comment ?
Signed-off-by: Joe Elliott <number101010@gmail.com>
What this PR does:
Adds a new config option to restrict the time range around now that Tempo will honor when updating the current block start and end times. The new option defaults to
2mmeaning that if a span is received outside of this time range it will be stored and findable by trace id, but the block's time range it is stored in will not reflect this. This will make it less likely to be found using a trace search query.One gross corner is that during WAL replay we have to assume that we should adjust the block's time range based on the maximum time that a WAL file will be kept since we don't know when it was created.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]