fix(tsm1): fix wal's totalOldDiskSize statistics #20811
Conversation
Tests added. I compared monitor data from measurement |
tsdb/engine/tsm1/wal_test.go
Outdated
| @@ -701,6 +703,124 @@ func TestWALRollSegment(t *testing.T) { | |||
| } | |||
| } | |||
|
|
|||
| func TestWAL_DistSize(t *testing.T) { | |||
There was a problem hiding this comment.
| func TestWAL_DistSize(t *testing.T) { | |
| func TestWAL_DiskSize(t *testing.T) { |
tsdb/engine/tsm1/wal_test.go
Outdated
| "testing" | ||
|
|
||
| "github.com/golang/snappy" | ||
| "github.com/influxdata/influxdb/v2/pkg/slices" | ||
| "github.com/influxdata/influxdb/v2/tsdb/engine/tsm1" | ||
| "github.com/stretchr/testify/assert" |
There was a problem hiding this comment.
| "github.com/stretchr/testify/assert" | |
| "github.com/stretchr/testify/require" |
IMO makes the testing simpler, since it bails at the first mismatched expectation
tsdb/engine/tsm1/wal_test.go
Outdated
| test := func(w *tsm1.WAL, oldZero, curZero bool) () { | ||
| // get disk size by reading file | ||
| files, err := ioutil.ReadDir(w.Path()) | ||
| if err != nil { |
There was a problem hiding this comment.
This if could be replaced by require.NoError
tsdb/engine/tsm1/wal_test.go
Outdated
| } | ||
| if !curZero && cur == 0 { | ||
| t.Fatalf("expect current disk size larger than 0, got 0") | ||
| } |
There was a problem hiding this comment.
These ifs could all be replaced by require functions
tsdb/engine/tsm1/wal_test.go
Outdated
| // test method DiskSizeBytes | ||
| if got, exp := w.DiskSizeBytes(), old+cur; got != exp { | ||
| t.Fatalf("expect total disk size: %d, got: %d", exp, got) | ||
| } |
There was a problem hiding this comment.
Same here (require.Equal I think)
tsdb/engine/tsm1/wal_test.go
Outdated
|
|
||
| if err := w.Remove(closedSegments); err != nil { | ||
| t.Fatalf("close segment error: %v", err) | ||
| } |
tsdb/engine/tsm1/wal_test.go
Outdated
| }) | ||
|
|
||
| var old, cur int64 | ||
| for i, f := range files { |
There was a problem hiding this comment.
This loop confuses me. Is the end goal to have:
cur = // size of last file
old = // sum of sizes of all but last file
?
There was a problem hiding this comment.
If so, would we assign cur directly outside of the loop, and have the range only loop over the files we want to sum?
| } | ||
|
|
||
| // test Statistics | ||
| ss := w.Statistics(nil) |
There was a problem hiding this comment.
nil means no need to add additional tags
tsdb/engine/tsm1/wal_test.go
Outdated
|
|
||
| test(w, true, false) | ||
|
|
||
| // write with rollSegment |
There was a problem hiding this comment.
Not super familiar with this code, so I'm confused by this comment when I don't see rollSegment below. Can you clarify? Something like:
| // write with rollSegment | |
| // write enough points to force the segment to roll |
tsdb/engine/tsm1/wal_test.go
Outdated
|
|
||
| test(w, true, true) | ||
|
|
||
| // write without rollSegment |
There was a problem hiding this comment.
| // write without rollSegment | |
| // write some points (not enough to force the segment to roll) |
Thank you very much for your suggestions, the code is indeed a lot concise. |
The value of the totalOldDiskSize in WAL is wrong. There are two reason caused the inaccuracy.
Reason 1 caused great inaccuracies: TotalOldDiskSize should add rather than set the current segment bytes when rolling a new segment.
Reason 2 caused minor inaccuracies: Must skip the current segment size when statistic the totalOldDiskSize, because total disk size is computed by following method: