Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instrument Otel #125

Merged
merged 44 commits into from
Jul 22, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
cbdda80
initial tracing try with background.get
louiseschmidtgen Jul 8, 2024
ecda7a8
more updates backend.get
louiseschmidtgen Jul 9, 2024
6b95d6c
parent spans
louiseschmidtgen Jul 9, 2024
38e3b21
logs to jaeger
louiseschmidtgen Jul 9, 2024
8514849
svc name
louiseschmidtgen Jul 9, 2024
0be0a10
traces mostly list
louiseschmidtgen Jul 10, 2024
87e5548
instrument create
louiseschmidtgen Jul 10, 2024
4e0dd87
instrument delete and update
louiseschmidtgen Jul 10, 2024
089c4d3
dbsize
louiseschmidtgen Jul 10, 2024
3be9a84
cleanup
louiseschmidtgen Jul 10, 2024
478cbb7
cleanup 2
louiseschmidtgen Jul 10, 2024
788fcf3
Merge remote-tracking branch 'origin' into KU-858/tracing
louiseschmidtgen Jul 10, 2024
755e113
cleanup 3
louiseschmidtgen Jul 10, 2024
10ac5db
add service name to meterprovider
louiseschmidtgen Jul 10, 2024
6913487
cleanup
louiseschmidtgen Jul 10, 2024
778a82c
comments
louiseschmidtgen Jul 10, 2024
67dc71f
move otel setup
louiseschmidtgen Jul 10, 2024
988deca
cleanup
louiseschmidtgen Jul 10, 2024
213b745
remove wrong place compact logs
louiseschmidtgen Jul 12, 2024
85af5c3
add traces to compaction
louiseschmidtgen Jul 12, 2024
0534bef
on update's create if errkeyexists do a get
louiseschmidtgen Jul 12, 2024
81ef2bb
trace names
louiseschmidtgen Jul 12, 2024
9143cc9
ctx
louiseschmidtgen Jul 12, 2024
2681a0e
ctx
louiseschmidtgen Jul 12, 2024
38c31b3
move run jaeger instance to script
louiseschmidtgen Jul 12, 2024
0febbb9
comments part 1
louiseschmidtgen Jul 16, 2024
f56eb46
move counters closer to where they are used
louiseschmidtgen Jul 17, 2024
2733d4e
traces in logstructured
louiseschmidtgen Jul 17, 2024
ceb9c8f
capture more logs on compaction
louiseschmidtgen Jul 18, 2024
ada277b
cleanup
louiseschmidtgen Jul 18, 2024
cda87ce
naming
louiseschmidtgen Jul 18, 2024
8577851
pass conn address
louiseschmidtgen Jul 18, 2024
0bb3126
cleanup
louiseschmidtgen Jul 18, 2024
04ab821
rewrite otel setup, disable otel by default
louiseschmidtgen Jul 18, 2024
f081376
move otel shutdown
louiseschmidtgen Jul 19, 2024
b42ceea
typo
louiseschmidtgen Jul 19, 2024
bd4d75f
counter init and naming
louiseschmidtgen Jul 22, 2024
fcdae60
more renaming
louiseschmidtgen Jul 22, 2024
da1622a
comments
louiseschmidtgen Jul 22, 2024
90a04ec
shutdown conn
louiseschmidtgen Jul 22, 2024
2b1887c
handling errs
louiseschmidtgen Jul 22, 2024
7dbfb74
adjust ignore compacted
louiseschmidtgen Jul 22, 2024
b9c235e
spaces
louiseschmidtgen Jul 22, 2024
9023e1e
tidy
louiseschmidtgen Jul 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ Steps:

7. While developing and making changes to `k8s-dqlite`, just restart k8s-dqlite

Note: When develop k8s-dqlite against Canonical Kubernetes use the following flags:
Note: When developing k8s-dqlite against Canonical Kubernetes use the following flags:

- `--storage-dir /var/snap/k8s/common/var/lib/k8s-dqlite`
- `--listen unix:///var/snap/k8s/common/var/lib/k8s-dqlite/k8s-dqlite.sock`
11 changes: 11 additions & 0 deletions cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package cmd

import (
"context"
"errors"
"net/http"
_ "net/http/pprof"
"os"
Expand Down Expand Up @@ -60,6 +61,16 @@ var (

if rootCmdOpts.metrics {
go func() {
otelShutdown, err := setupOTelSDK(cmd.Context())
if err != nil {
logrus.WithError(err).Warning("Failed to setup OpenTelemetry SDK")
}
defer func() {
err = errors.Join(err, otelShutdown(cmd.Context()))
if err != nil {
logrus.WithError(err).Warning("Failed to shutdown OpenTelemetry SDK")
}
}()
logrus.WithField("address", rootCmdOpts.metricsAddress).Print("Enable metrics endpoint")
mux := http.NewServeMux()
mux.Handle("/metrics", promhttp.Handler())
Expand Down
114 changes: 114 additions & 0 deletions cmd/tracing.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
package cmd

import (
"context"
"errors"
"fmt"
"time"

"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/exporters/stdout/stdoutmetric"
"go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/resource"
"go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)

// setupOTelSDK bootstraps the OpenTelemetry pipeline.
// If it does not return an error, make sure to call shutdown for proper cleanup.
func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
var shutdownFuncs []func(context.Context) error

shutdown = func(ctx context.Context) error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pattern seems a bit obscure IMHO. The logic is opaque as it's just a list of functions, but maybe we should instead have it explicit? Consider that having a list of functions allows for a dynamic number of cleanup logic, but in this function the number of objects to close is statically known.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True this makes sense if you collect logs etc here too.

var err error
for _, fn := range shutdownFuncs {
err = errors.Join(err, fn(ctx))
}
shutdownFuncs = nil
return err
}

handleErr := func(inErr error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be handled in a defer instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed it alltogether.

err = errors.Join(inErr, shutdown(ctx))
}

res, err := resource.New(ctx,
marco6 marked this conversation as resolved.
Show resolved Hide resolved
resource.WithAttributes(
semconv.ServiceNameKey.String("k8s-dqlite"),
),
)

tracerProvider, err := newTraceProvider(ctx, res)
if err != nil {
handleErr(err)
return
}
shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
marco6 marked this conversation as resolved.
Show resolved Hide resolved
otel.SetTracerProvider(tracerProvider)

meterProvider, err := newMeterProvider(res)
if err != nil {
handleErr(err)
return
}
shutdownFuncs = append(shutdownFuncs, meterProvider.Shutdown)
otel.SetMeterProvider(meterProvider)

return
}

func initConn() (*grpc.ClientConn, error) {
// It connects the OpenTelemetry Collector through local gRPC connection.
// You may replace `localhost:4317` with your endpoint.
conn, err := grpc.NewClient("localhost:4317",
louiseschmidtgen marked this conversation as resolved.
Show resolved Hide resolved
louiseschmidtgen marked this conversation as resolved.
Show resolved Hide resolved
// Note the use of insecure transport here. TLS is recommended in production.
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err)
}

return conn, err
louiseschmidtgen marked this conversation as resolved.
Show resolved Hide resolved
}

func newExporter(ctx context.Context) (trace.SpanExporter, error) {
louiseschmidtgen marked this conversation as resolved.
Show resolved Hide resolved
conn, _ := initConn()

traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("failed to create trace exporter: %w", err)
}
return traceExporter, nil
}

func newTraceProvider(ctx context.Context, res *resource.Resource) (*trace.TracerProvider, error) {
traceExporter, err := newExporter(ctx)
marco6 marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, err
}

traceProvider := trace.NewTracerProvider(
trace.WithBatcher(traceExporter,
trace.WithBatchTimeout(time.Second),
),
trace.WithResource(res),
)
return traceProvider, nil
}

func newMeterProvider(res *resource.Resource) (*metric.MeterProvider, error) {
metricExporter, err := stdoutmetric.New()
if err != nil {
return nil, err
}

meterProvider := metric.NewMeterProvider(
metric.WithReader(metric.NewPeriodicReader(metricExporter,
metric.WithInterval(30*time.Second))),
metric.WithResource(res),
)
return meterProvider, nil
}
43 changes: 23 additions & 20 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -15,32 +15,39 @@ require (
go.etcd.io/etcd/client/pkg/v3 v3.5.12
go.etcd.io/etcd/client/v3 v3.5.12
go.etcd.io/etcd/server/v3 v3.5.12
golang.org/x/sync v0.6.0
golang.org/x/sys v0.18.0
google.golang.org/grpc v1.62.1
go.opentelemetry.io/otel v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.28.0
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.28.0
go.opentelemetry.io/otel/metric v1.28.0
go.opentelemetry.io/otel/sdk v1.28.0
go.opentelemetry.io/otel/sdk/metric v1.28.0
golang.org/x/sync v0.7.0
golang.org/x/sys v0.21.0
google.golang.org/grpc v1.65.0
gopkg.in/yaml.v2 v2.4.0
)

require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/renameio v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/websocket v1.5.1 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jonboulle/clockwork v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
Expand All @@ -58,23 +65,19 @@ require (
go.etcd.io/etcd/pkg/v3 v3.5.12 // indirect
go.etcd.io/etcd/raft/v3 v3.5.12 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.24.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/sdk v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.opentelemetry.io/proto/otlp v1.1.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/net v0.22.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/crypto v0.24.0 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/text v0.16.0 // indirect
golang.org/x/time v0.5.0 // indirect
google.golang.org/genproto v0.0.0-20240311173647-c811ad7063a7 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240311173647-c811ad7063a7 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240311173647-c811ad7063a7 // indirect
google.golang.org/protobuf v1.33.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
Expand Down
Loading
Loading