Data set filled with tombstones

[amnonh]

Describe the issue logically
Alot of tombestones makes reads slow

Describe the condition when the issue happens
No good indication

What the user should do when this happens
Change compaction strategy.
run a major compaction
Any Other info or Links.

[slivne]

@bhalevy / @cmm - amnon can add an indication to monitoring on range tombstones / tombstones - we are intrested mainly in the amount that are hit during reads

I kmow we added some metrics (I hope they are merged) can you please point out the metrics and the releases that have those metrics.

[amnonh]

@slivne @bhalevy does the metrics in:
https://github.com/scylladb/scylla-enterprise/pull/1926

Can cover this?

[cmm]

the *_tombstone_reads metrics there seem to be what you want here, yes. they should have been back-ported to 21.1, if I understand correctly

[slivne]

Detailed:

• add a graph for cache tombstones
• add a graph for tombstones for sstables

Advisor - add a warning - use a ratio between number of tombstones to number of reads - if the ratio is > 10 indicate the user should check the data model

[amnonh]

Add enable probabilistic tracing and slow query tracing

[amnonh]

@bhalevy I didn't see the metrics in 4.5, do I need to do somthing for it to appear?

[bhalevy]

@bhalevy I didn't see the metrics in 4.5, do I need to do somthing for it to appear?

@amnonh they are available only since 4.6 (and backported to 2021.1)

[amnonh]

@bhalevy I want to create some graphs for the release notes, can you explain how can I test it?

[bhalevy]

@bhalevy I want to create some graphs for the release notes, can you explain how can I test it?

I think a good test will be:

• Setup
  • set gc_grace_seconds to a small number
  • disable auto_compaction
• Insert data
  • flush
• do a bunch of range deletions
  • at this point the graph should show them
  • flush, so to generate tombstones in the sstables
• run repair (in master, since tombstone purging now depends on successful repair)
• run major compaction
• See that the tombstones were purged and went away

@raphaelsc @asias please confirm