tempodb integer divide by zero error

[kroksys]

I'm just letting you know about potential problem. I did some experiments and somehow I broke tempo storage connected with minio (without replication). I have no idea how to reproduce it, but from stack trace it seems like it needs to check for 0 value before dividing it somewhere. I'm obviously using docker-compose setup and after I removed volume and started everything over again the issue is gone.

stack-tempo-1    | panic: runtime error: integer divide by zero
stack-tempo-1    | 
stack-tempo-1    | goroutine 632 [running]:
stack-tempo-1    | github.com/grafana/tempo/tempodb.(*timeWindowBlockSelector).windowForTime(...)
stack-tempo-1    |      /drone/src/tempodb/compaction_block_selector.go:182
stack-tempo-1    | github.com/grafana/tempo/tempodb.newTimeWindowBlockSelector({0x4000284cf0, 0x1, 0xd?}, 0x0, 0x0, 0x0, 0x2, 0x4)
stack-tempo-1    |      /drone/src/tempodb/compaction_block_selector.go:58 +0x604
stack-tempo-1    | github.com/grafana/tempo/tempodb.(*readerWriter).doCompaction(0x40003006c0)
stack-tempo-1    |      /drone/src/tempodb/compactor.go:92 +0xf4
stack-tempo-1    | github.com/grafana/tempo/tempodb.(*readerWriter).compactionLoop(0x40003006c0)
stack-tempo-1    |      /drone/src/tempodb/compactor.go:74 +0x64
stack-tempo-1    | created by github.com/grafana/tempo/tempodb.(*readerWriter).EnableCompaction
stack-tempo-1    |      /drone/src/tempodb/tempodb.go:407 +0x1d0
stack-tempo-1 exited with code 2

[joe-elliott]

The divide by zero is here:

tempo/tempodb/compaction_block_selector.go

Line 182 in 62ddaac

return t.Unix() / int64(twbs.MaxCompactionRange/time.Second)

and should only happen if compaction_window is set to 0:

compactor:
    compaction:
        compaction_window: 5m

We should error more gracefully and log something meaningful in this case instead of just panicing.

[kroksys]

In my case compactor is set to null. I guess googling and copy/paste don't always work out. But after full reset it works even with this config.
If compactor: null then all objects inside that config will be 0, nil or "", etc. ?

compactor: null

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:

overrides:
  ingestion_burst_size_bytes: 10_000_000
  max_traces_per_user: 1000000

server:
  http_listen_port: 3200

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo
      endpoint: minio:9000
      access_key: minio
      secret_key: 9reatsecr3t
      insecure: true
    pool:
      queue_depth: 2000
    wal:
      path: /var/tempo/wal

[joe-elliott]

If compactor: null then all objects inside that config will be 0, nil or "", etc. ?

Honestly, I don't know if that would take Tempo defaults are be set to all 0s.

[mghildiy]

Can I take this issue?

[ie-pham]

@mghildiy go ahead! :)

[mghildiy]

The divide by zero is here:

tempo/tempodb/compaction_block_selector.go

Line 182 in 62ddaac

return t.Unix() / int64(twbs.MaxCompactionRange/time.Second)

and should only happen if compaction_window is set to 0:

compactor:
    compaction:
        compaction_window: 5m

We should error more gracefully and log something meaningful in this case instead of just panicing.

I see 2 divisions going on here. I need to take care of both, I guess.

[mghildiy]

For testing, a unit test would suffice I guess.
Also, is it possible to simulate 'distributed' setup locally to perform some sort of user testing?

[mapno]

There are a few example set-ups in this folder https://github.com/grafana/tempo/tree/main/example

[mghildiy]

I have PR here:

#2333

But its build is going on since ages.