Skip to content

Implementing Parquet Converter #6716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Apr 29, 2025
Merged

Implementing Parquet Converter #6716

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
5f3f55c
Inital Commit - Skeleton
alanprot Apr 22, 2025
d820fd5
converting block
alanprot Apr 22, 2025
35719a4
lint + update common
alanprot Apr 24, 2025
040e699
Using MetaFetcher to discover the blocks
alanprot Apr 24, 2025
6397507
moving the marker to the common storage package
alanprot Apr 24, 2025
adca9f7
wrap bucket with the global markers bucket
alanprot Apr 24, 2025
46e3114
adding tests / hiding documentation for now
alanprot Apr 25, 2025
4fa4df6
improving some logs
alanprot Apr 25, 2025
1c8eda0
using filesystem bucket on test
alanprot Apr 25, 2025
a81b78e
update parquet common
alanprot Apr 28, 2025
f72e3d3
Using another global prefix for the parquet markers
alanprot Apr 28, 2025
9c0dcb1
Changelog
alanprot Apr 28, 2025
db9de10
Cleaning sync metas for deleted tenants
alanprot Apr 28, 2025
c04560b
Merge branch 'master' into parquet-converter
alanprot Apr 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
* [FEATURE] Ingester: Support out-of-order native histogram ingestion. It automatically enabled when `-ingester.out-of-order-time-window > 0` and `-blocks-storage.tsdb.enable-native-histograms=true`. #6626 #6663
* [FEATURE] Ruler: Add support for percentage based sharding for rulers. #6680
* [FEATURE] Ruler: Add support for group labels. #6665
* [FEATURE] Support Parquet format: Implement parquet converter service to convert a TSDB block into Parquet. #6716
* [ENHANCEMENT] Query Frontend: Change to return 400 when the tenant resolving fail. #6715
* [ENHANCEMENT] Querier: Support query parameters to metadata api (/api/v1/metadata) to allow user to limit metadata to return. #6681
* [ENHANCEMENT] Ingester: Add a `cortex_ingester_active_native_histogram_series` metric to track # of active NH series. #6695
Expand Down
92 changes: 55 additions & 37 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@ require (
github.com/gorilla/mux v1.8.1
github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
github.com/hashicorp/consul/api v1.31.0
github.com/hashicorp/consul/api v1.31.2
github.com/hashicorp/go-cleanhttp v0.5.2
github.com/hashicorp/go-sockaddr v1.0.7
github.com/hashicorp/memberlist v0.5.1
github.com/json-iterator/go v1.1.12
github.com/klauspost/compress v1.17.11
github.com/klauspost/compress v1.18.0
github.com/lib/pq v1.10.9
github.com/minio/minio-go/v7 v7.0.82
github.com/mitchellh/go-wordwrap v1.0.1
Expand All @@ -45,12 +45,12 @@ require (
github.com/prometheus/client_model v0.6.1
github.com/prometheus/common v0.62.0
// Prometheus maps version 2.x.y to tags v0.x.y.
github.com/prometheus/prometheus v0.302.1
github.com/prometheus/prometheus v0.303.0
github.com/segmentio/fasthash v1.0.3
github.com/sony/gobreaker v1.0.0
github.com/spf13/afero v1.11.0
github.com/stretchr/testify v1.10.0
github.com/thanos-io/objstore v0.0.0-20241111205755-d1dd89d41f97
github.com/thanos-io/objstore v0.0.0-20250317105316-a0136a6f898d
github.com/thanos-io/promql-engine v0.0.0-20250329215917-4055a112d1ea
github.com/thanos-io/thanos v0.38.0
github.com/uber/jaeger-client-go v2.30.0+incompatible
Expand All @@ -61,15 +61,15 @@ require (
go.opentelemetry.io/contrib/propagators/aws v1.33.0
go.opentelemetry.io/otel v1.35.0
go.opentelemetry.io/otel/bridge/opentracing v1.33.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.35.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.35.0
go.opentelemetry.io/otel/sdk v1.35.0
go.opentelemetry.io/otel/trace v1.35.0
go.uber.org/atomic v1.11.0
golang.org/x/net v0.38.0
golang.org/x/sync v0.12.0
golang.org/x/time v0.9.0
google.golang.org/grpc v1.70.0
golang.org/x/time v0.10.0
google.golang.org/grpc v1.71.0
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
)
Expand All @@ -81,28 +81,31 @@ require (
github.com/google/go-cmp v0.7.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
github.com/parquet-go/parquet-go v0.25.0
github.com/prometheus-community/parquet-common v0.0.0-20250428074311-306c8486441d
github.com/prometheus/procfs v0.15.1
github.com/sercand/kuberesolver/v5 v5.1.1
github.com/tjhop/slog-gokit v0.1.3
go.opentelemetry.io/collector/pdata v1.24.0
go.opentelemetry.io/collector/pdata v1.27.0
go.uber.org/automaxprocs v1.6.0
google.golang.org/protobuf v1.36.4
google.golang.org/protobuf v1.36.5
)

require (
cloud.google.com/go v0.115.1 // indirect
cloud.google.com/go/auth v0.14.0 // indirect
cloud.google.com/go/auth v0.15.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.7 // indirect
cloud.google.com/go/compute/metadata v0.6.0 // indirect
cloud.google.com/go/iam v1.1.13 // indirect
cloud.google.com/go/storage v1.43.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.17.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.8.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.8.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.2 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.3 // indirect
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-sdk-go-v2 v1.16.16 // indirect
github.com/aws/aws-sdk-go-v2/config v1.15.1 // indirect
Expand Down Expand Up @@ -147,6 +150,7 @@ require (
github.com/go-openapi/runtime v0.28.0 // indirect
github.com/go-openapi/spec v0.21.0 // indirect
github.com/go-openapi/validate v0.24.0 // indirect
github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/gofrs/uuid v4.4.0+incompatible // indirect
github.com/gogo/googleapis v1.4.0 // indirect
Expand All @@ -156,10 +160,10 @@ require (
github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.5 // indirect
github.com/googleapis/gax-go/v2 v2.14.1 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.25.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
Expand All @@ -174,35 +178,44 @@ require (
github.com/jpillora/backoff v1.0.0 // indirect
github.com/julienschmidt/httprouter v1.3.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
github.com/knadh/koanf/maps v0.1.1 // indirect
github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
github.com/knadh/koanf/v2 v2.1.2 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 // indirect
github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/mdlayher/socket v0.4.1 // indirect
github.com/mdlayher/vsock v1.2.1 // indirect
github.com/metalmatze/signal v0.0.0-20210307161603-1c9aa721a97a // indirect
github.com/miekg/dns v1.1.63 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
github.com/ncw/swift v1.0.53 // indirect
github.com/oklog/run v1.1.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/exp/metrics v0.116.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil v0.116.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/processor/deltatocumulativeprocessor v0.116.0 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/exp/metrics v0.121.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil v0.121.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/processor/deltatocumulativeprocessor v0.121.0 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus-community/prom-label-proxy v0.8.1-0.20240127162815-c1195f9aabc0 // indirect
github.com/prometheus/exporter-toolkit v0.13.2 // indirect
github.com/prometheus/sigv4 v0.1.1 // indirect
github.com/prometheus/exporter-toolkit v0.14.0 // indirect
github.com/prometheus/sigv4 v0.1.2 // indirect
github.com/redis/rueidis v1.0.45-alpha.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rs/cors v1.11.1 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 // indirect
Expand All @@ -221,15 +234,16 @@ require (
go.mongodb.org/mongo-driver v1.14.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/collector/component v0.118.0 // indirect
go.opentelemetry.io/collector/config/configtelemetry v0.118.0 // indirect
go.opentelemetry.io/collector/consumer v1.24.0 // indirect
go.opentelemetry.io/collector/pipeline v0.118.0 // indirect
go.opentelemetry.io/collector/processor v0.118.0 // indirect
go.opentelemetry.io/collector/semconv v0.118.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.59.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.59.0 // indirect
go.opentelemetry.io/collector/component v1.27.0 // indirect
go.opentelemetry.io/collector/confmap v1.27.0 // indirect
go.opentelemetry.io/collector/confmap/xconfmap v0.121.0 // indirect
go.opentelemetry.io/collector/consumer v1.27.0 // indirect
go.opentelemetry.io/collector/pipeline v0.121.0 // indirect
go.opentelemetry.io/collector/processor v0.121.0 // indirect
go.opentelemetry.io/collector/semconv v0.121.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.60.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
go.opentelemetry.io/contrib/propagators/autoprop v0.54.0 // indirect
go.opentelemetry.io/contrib/propagators/b3 v1.29.0 // indirect
go.opentelemetry.io/contrib/propagators/jaeger v1.29.0 // indirect
Expand All @@ -242,23 +256,23 @@ require (
go4.org/intern v0.0.0-20230525184215-6c62f75575cb // indirect
go4.org/unsafe/assume-no-moving-gc v0.0.0-20230525183740-e7c30c78aeb2 // indirect
golang.org/x/crypto v0.36.0 // indirect
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8 // indirect
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
golang.org/x/mod v0.24.0 // indirect
golang.org/x/oauth2 v0.25.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.23.0 // indirect
golang.org/x/tools v0.31.0 // indirect
gonum.org/v1/gonum v0.15.1 // indirect
google.golang.org/api v0.218.0 // indirect
google.golang.org/api v0.224.0 // indirect
google.golang.org/genproto v0.0.0-20240823204242-4ba0660f739c // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250115164207-1a7da9e5054f // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250227231956-55c901821b1e // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/telebot.v3 v3.3.8 // indirect
k8s.io/apimachinery v0.31.3 // indirect
k8s.io/client-go v0.31.3 // indirect
k8s.io/apimachinery v0.32.2 // indirect
k8s.io/client-go v0.32.2 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
k8s.io/utils v0.0.0-20241104163129-6fe5fd82f078 // indirect
)

// Using cortex fork of weaveworks/common
Expand Down Expand Up @@ -290,3 +304,7 @@ replace gopkg.in/alecthomas/kingpin.v2 => github.com/alecthomas/kingpin v1.3.8-0

// gRPC 1.66 introduced memory pooling which breaks Cortex queries. Pin 1.65.0 until we have a fix.
replace google.golang.org/grpc => google.golang.org/grpc v1.65.0

replace github.com/thanos-io/objstore => github.com/thanos-io/objstore v0.0.0-20241111205755-d1dd89d41f97

replace github.com/prometheus/prometheus => github.com/prometheus/prometheus v0.302.1
Loading
Loading