Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(v2): metastore index #3586

Merged
merged 45 commits into from
Oct 7, 2024
Merged

feat(v2): metastore index #3586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
9709881
Move error checking elsewhere
aleks-p Sep 17, 2024
228b6af
Add metastore index (wip)
aleks-p Sep 17, 2024
0ef2169
Fix a few bugs and tests
aleks-p Sep 17, 2024
f57cc4d
Remove unneeded file
aleks-p Sep 19, 2024
b126d7b
Fix race condition
aleks-p Sep 19, 2024
38cce6f
Enable new write path for local deployments
aleks-p Sep 19, 2024
49c20e9
Return level 0 blocks, remove duplicates
aleks-p Sep 19, 2024
69028ae
Add simple logging
aleks-p Sep 19, 2024
fe950a5
Remove unused var
aleks-p Sep 19, 2024
ceb4f83
Refactor out metastore index, add partition meta
aleks-p Sep 20, 2024
689dfe7
Refactor, minor perf improvements
aleks-p Sep 23, 2024
4809966
Fix local setup for v2
aleks-p Sep 23, 2024
c532d7b
Replace block truncation with block retention
aleks-p Sep 23, 2024
a7a0f89
Refactor some more
aleks-p Sep 23, 2024
b204efe
Add tests, basic config
aleks-p Sep 23, 2024
928bf97
Add documentation
aleks-p Sep 23, 2024
9e07370
Add todo
aleks-p Sep 23, 2024
b396cde
Remove unused struct field tags
aleks-p Sep 23, 2024
876dc8f
Add todo
aleks-p Sep 23, 2024
130105a
Run make fmt
aleks-p Sep 23, 2024
bb28bfd
Run make fmt
aleks-p Sep 24, 2024
92c401d
Allow longer queries in v2 locally
aleks-p Sep 24, 2024
f36e344
Add test for changing the partition duration
aleks-p Sep 25, 2024
81b60c9
Update test
aleks-p Sep 25, 2024
f139937
Address issue with finding / deleting blocks following a partition du…
aleks-p Sep 25, 2024
684c8f5
Add test cases
aleks-p Sep 25, 2024
588df8b
Remove nested locking primitives
aleks-p Sep 27, 2024
dd298b1
Remove index/models.go
aleks-p Sep 27, 2024
1fa40cf
Improve error handling in ForEachPartition
aleks-p Sep 27, 2024
a547f1a
Fix access without a lock
aleks-p Sep 27, 2024
57383b3
Validate block ids at the API level
aleks-p Sep 27, 2024
87e3651
Improve FindBlocksInRange, add tests
aleks-p Sep 27, 2024
0681853
Switch to a LRU cache for loaded partitions
aleks-p Oct 1, 2024
8c6e23d
Load partition meta on the fly
aleks-p Oct 1, 2024
79217ef
Add logging
aleks-p Oct 1, 2024
557a146
Load the active partition in memory on startup
aleks-p Oct 2, 2024
e098c9b
Switch away from using a LRU cache for loaded partitions
aleks-p Oct 3, 2024
891aab2
Switch the index to a partition+tenant structure
aleks-p Oct 3, 2024
2c23770
Fix nil pointer dereference error on startup
aleks-p Oct 4, 2024
ef400d7
Fix compile error
aleks-p Oct 4, 2024
4270517
Update cache entry access ts on reads
aleks-p Oct 4, 2024
d5471f4
Add missing tenant check
aleks-p Oct 4, 2024
d27c875
Run make fmt
aleks-p Oct 4, 2024
bebfb42
Fix panic in query frontend
aleks-p Oct 4, 2024
020e00d
Fixed issue with unreachable anonymous blocks
aleks-p Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .mockery.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,7 @@ packages:
github.com/grafana/pyroscope/pkg/experiment/metastore/dlq:
interfaces:
LocalServer:
github.com/grafana/pyroscope/pkg/experiment/metastore/index:
interfaces:
Store:
config:
13 changes: 8 additions & 5 deletions pkg/experiment/ingester/segment_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@ import (
"bytes"
"context"
"fmt"
"github.com/grafana/dskit/flagext"
"github.com/grafana/pyroscope/pkg/experiment/metastore"
"github.com/grafana/pyroscope/pkg/test/mocks/mockdlq"
"io"
"math/rand"
"path/filepath"
Expand All @@ -16,7 +13,13 @@ import (
"testing"
"time"

"github.com/grafana/dskit/flagext"

"github.com/grafana/pyroscope/pkg/experiment/metastore"
"github.com/grafana/pyroscope/pkg/test/mocks/mockdlq"

gprofile "github.com/google/pprof/profile"

profilev1 "github.com/grafana/pyroscope/api/gen/proto/go/google/v1"
ingesterv1 "github.com/grafana/pyroscope/api/gen/proto/go/ingester/v1"
"github.com/grafana/pyroscope/api/gen/proto/go/ingester/v1/ingesterv1connect"
Expand Down Expand Up @@ -414,9 +417,9 @@ func TestDLQRecoveryMock(t *testing.T) {

func TestDLQRecovery(t *testing.T) {
const tenant = "tb"
const ts = 239
var ts = time.Now().UnixMilli()
chunk := inputChunk([]input{
{shard: 1, tenant: tenant, profile: cpuProfile(42, ts, "svc1", "kek", "foo", "bar")},
{shard: 1, tenant: tenant, profile: cpuProfile(42, int(ts), "svc1", "kek", "foo", "bar")},
})

sw := newTestSegmentWriter(t, segmentWriterConfig{
Expand Down
Loading
Loading