Migrate weaveworks/common packages used by Mimir #342

charleskorn · 2023-08-03T03:56:52Z

What this PR does:

This PR moves all github.com/weaveworks/common packages used by Mimir to this repo.

We're migrating these packages because CI for github.com/weaveworks/common is broken for master, and Weaveworks has expressed a desire to archive this repo.

The following packages have been migrated, all to new dskit packages with the same name unless noted otherwise:

github.com/weaveworks/common/errors to github.com/grafana/dskit/errors
github.com/weaveworks/common/grpc to github.com/grafana/dskit/grpcutil (merged into existing grpcutil package)
github.com/weaveworks/common/httpgrpc to github.com/grafana/dskit/httpgrpc
github.com/weaveworks/common/instrument to github.com/grafana/dskit/instrument
github.com/weaveworks/common/logging to github.com/grafana/dskit/log (merged into existing log package)
github.com/weaveworks/common/middleware to github.com/grafana/dskit/middleware
github.com/weaveworks/common/mtime to github.com/grafana/dskit/mtime
github.com/weaveworks/common/server to github.com/grafana/dskit/server
github.com/weaveworks/common/signals to github.com/grafana/dskit/signals
github.com/weaveworks/common/tracing to github.com/grafana/dskit/tracing
github.com/weaveworks/common/user to github.com/grafana/dskit/user

Tempo depends only on the same packages. Loki also requires aws and test, which I'll migrate in a separate PR.

Other changes of note are:

all generated protobuf encoders / decoders now use gogo/protobuf, rather than Google's implementation: 062ea8d
some fields have had minor name changes to satisfy the linter (eg. TCP instead of Tcp): bc7854c
replaced the use of golang.org/x/net/context with context: 2ea3a37
replaced the use of prometheus with promauto, except in instrument package, where this would require changing the interface of Collector, HistogramCollector and JobCollector: 273c830

If this migration needs to be repeated / redone in the future, these are the steps I followed:

Create packages.txt (example) with source and destination package names separated by a tab

I ran go list -test -deps ./... | sort | grep -E '^github.com/weaveworks/common/[^/]*$' | sed -e "s#^github.com/weaveworks/common/##" in a working copy of Mimir to get top-level packages to move
Run migrate.sh (source) to move the packages between repos, preserving history, rewriting file paths and adding provenance comments.
Manually correct any compilation or linting issues in dskit directory before committing

Which issue(s) this PR fixes:

(none)

Checklist

[n/a] Tests updated
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Part of cortexproject/cortex#810

* Add gRPC streaming middleware. Adds a series of gRPC stream middleware for logging, user header propagation and tracing. This is laying groundwork for using gRPC streaming between ingesters and queriers in Cortex. * Update the vendored versions. * Pin kuberesolver because the API has changed. * Always put code under github.com/weaveworks/common; Dependancies between packages won't work if the code is checked out in github.com/tomwilkie, for instance. Also, common is a very common name for a repo, so its not uncommon for the repo to be called something like weaveworks-common. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

…sked (#102) * Don't enable trace sampling by default * Don't export new unused member, simplify docstring

* Move dedupe code to dedupe.go * Add unified logging interface. * Adapt middleware/, user/, signals/ and server/ to new logging interfaces. Allow user to specify logging implementation in server, if non is specified default to logrus. As part of this we need log level as server config, and for convenience we expose the server logger. * Update vendoring * Expose the various level types. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

gRPC behaviour should be the same as HTTP: Run() exits when a signal is received, and Stop() stops the handling of requests. And as we no longer call GracefulShutdown in Run, call it in Stop - one should always try to shutdown gracefully. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Propagate tracing context over httpgrpc * Log warnings for failed trace inject/extract

Random people on the Internet can send in arbitrary paths, so collapse them all into "other" when reporting metrics.

Primary motivation is to allow a standard Go profiling request to complete, for which I only need the write timeout, but I raised the other 5-second timeouts too since it isn't that unusual to have operations or data uploads take longer.

* Return errors when http/grpc servers exit. There was a case where the HTTP server died but the process exited. We need to exit when the servers exit. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Improve http request logging When logging an http request, e.g. on 500 error, put all the headers and response on one line. Also strip cookies and csrf tokens on security grounds. * Add a test to generate an http error to check formatting

* Refactor: extract functions HTTPRequest(), WriteResponse(), WriteError() * Regenerate httpgrpc.pb.go with protoc-gen-gogo from https://github.com/gogo/protobuf Change taken from cortexproject/cortex@f807690 where no explanation was given.

* Expose the HTTP server from the server struct. This allows users to inject HTTP requests and have the logged, instrumented and traced correctly. * Fix #120: buffer the errChan, as one of the servers could return an error before Run starts listening on errChan. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Refactor - s/logusEntry/logrusEntry * Also set global logging when initializing logrus This PR sets the global logging instance when calling `logging.Setup(level)`. Previously, only the standard logrus logger was set up and if anyone used the new funcs such as `logging.Info()` it was discarded.

* Don't trace http requests when using httpgrpc. * Demonstrate how the same can be acheived with middleware.Tracer.

* Print logrus output in logfmt format This changes the way logrous outputs log messages in case there is no TTY attached. Changes from ERRO: 2018/08/23 20:28:10.075952 Sample log message extra=foo to time="2018-08-23T20:28:10Z" level=info msg="Sample log message" extra="foo" which follows the `logfmt` key-value format by also providing `time`, `level`, and `msg` in adition to the extra fields. The PR removes the custom `textFormatter` to have the default `logrus.TextFormatter` in place which takes care of detecting whether a terminal is attached or not and switches between text and logfmt output automatically. It appears the custom formatter was to prevent differences in rendering depending on color/non-color output which I don't necessarily see as an issue. * dep ensure to fix tests

* Update 'dep' to 0.5.0 * Update more golang/protobuf to gogo/protobuf * add constraints for downstream users

…126) * Pass options to tracing middleware Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use the route matcher instead Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Update to latest kuberesolver and use a load balancing policy Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

This can be used to serve an API from a prefix (e.g. /prometheus/) without having to re-write every HTTP Handler.

server: add PathPrefix to config for all routes

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Cancellations are always caused by something else, so having them show up as errors in the tracing UI is distracting.

pracucci

Great work. I checked only your new commits (there’s no point in re-reviewing all weaveworks/common code), and LGTM. Looking forward to use it in Mimir ;)

bboreham

I don't feel this has to block the merge, but wanted to note some possible structure improvements.

The user package seems close to the existing tenant package.
errors has a Go standard library replacement.

bboreham · 2023-08-03T13:40:32Z

grpcutil/cancel.go

+)
+
+// IsCanceled checks whether an error comes from an operation being canceled
+func IsCanceled(err error) bool {


This is kinda duplicative of IsGRPCContextCanceled; do we need both?

In the interests of keeping this migration as simple as possible, I'd like to keep both for now as they're not identical, but I'll come back and review this separately afterwards.

bboreham · 2023-08-03T13:47:56Z

errors/error.go

+package errors
+
+// Error see https://dave.cheney.net/2016/04/07/constant-errors.
+type Error string


I think this could be replaced by the Go standard library errors.New.

In the interests of keeping this migration as simple as possible, I'd like to keep this for now, but I'll come back and review this separately afterwards.

See grafana/dskit#342.

**What this PR does / why we need it**: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See grafana/dskit#342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: (none) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>

juliusv and others added 30 commits June 4, 2018 16:51

Exempt 503 (Service Unavailable) from error logging (#97)

e9928ad

Part of cortexproject/cortex#810

When trace reporting is enabled, don't enable trace sampling unless a…

ee261fb

…sked (#102) * Don't enable trace sampling by default * Don't export new unused member, simplify docstring

Add user and org ID to span if known (#100)

af93c00

Propagate tracing context over httpgrpc (#105)

d58f15d

* Propagate tracing context over httpgrpc * Log warnings for failed trace inject/extract

Don't use full URL as metric label on unmatched path (#108)

cf37392

Random people on the Internet can send in arbitrary paths, so collapse them all into "other" when reporting metrics.

Return errors when http/grpc servers exit. (#92)

ef408c8

* Return errors when http/grpc servers exit. There was a case where the HTTP server died but the process exited. We need to exit when the servers exit. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Expose jaeger metrics (#114)

8f75acd

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Improve logging of http errors (#115)

c1317dc

* Improve http request logging When logging an http request, e.g. on 500 error, put all the headers and response on one line. Also strip cookies and csrf tokens on security grounds. * Add a test to generate an http error to check formatting

Add HTTP tracing middleware (#119)

742cd9f

Print traceID with request (#123)

e199adb

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Don't trace http requests when using httpgrpc. (#124)

a89a3f3

* Don't trace http requests when using httpgrpc. * Demonstrate how the same can be acheived with middleware.Tracer.

Update more golang/protobuf to gogo/protobuf (#127)

47e82f5

* Update 'dep' to 0.5.0 * Update more golang/protobuf to gogo/protobuf * add constraints for downstream users

Label http request handler tracing spans with the matched route name (#…

ca1cf91

…126) * Pass options to tracing middleware Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use the route matcher instead Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Fix dropped namespace in url parsing (#128)

c4ad22a

Update grpc-opentracing to latest version at new home (#129)

61043e1

go-grpc-middleware has moved (#130)

14ddecd

Update to latest kuberesolver (#131)

f789056

* Update to latest kuberesolver and use a load balancing policy Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Log which ports the server is listening on. (#143)

b4958f7

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

server: add PathPrefix to config for all routes

eb93014

This can be used to serve an API from a prefix (e.g. /prometheus/) without having to re-write every HTTP Handler.

Merge pull request #144 from moov-io/path-prefix

f1a9352

server: add PathPrefix to config for all routes

Allow server.Config to be unmarshaled from YAML.

302c7b9

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Dont flag context.Canceled as an error in tracing

d0f7449

Cancellations are always caused by something else, so having them show up as errors in the tracing UI is distracting.

charleskorn marked this pull request as ready for review August 3, 2023 05:38

Add changelog entry.

0e1e573

pracucci self-requested a review August 3, 2023 08:14

pracucci approved these changes Aug 3, 2023

View reviewed changes

bboreham reviewed Aug 3, 2023

View reviewed changes

charleskorn mentioned this pull request Aug 3, 2023

Upgrade dskit to remove dependency on weaveworks/common grafana/mimir#5672

Merged

charleskorn merged commit 740f56b into main Aug 4, 2023

charleskorn deleted the charleskorn/migrate-weaveworks-common branch August 4, 2023 00:36

charleskorn added a commit to grafana/mimir that referenced this pull request Aug 4, 2023

Upgrade dskit to remove dependency on weaveworks/common

c031ac4

See grafana/dskit#342.

charleskorn mentioned this pull request Aug 4, 2023

Upgrade dskit to use packages migrated from weaveworks/common grafana/loki#10164

Merged

1 task

charleskorn added a commit to grafana/loki that referenced this pull request Aug 4, 2023

Upgrade dskit to use packages migrated from weaveworks/common

796ae69

See grafana/dskit#342.

charleskorn added a commit to grafana/tempo that referenced this pull request Aug 4, 2023

Upgrade dskit to remove dependency on weaveworks/common

f23effb

See grafana/dskit#342.

charleskorn mentioned this pull request Aug 4, 2023

Upgrade dskit to remove dependency on weaveworks/common grafana/tempo#2763

Merged

charleskorn added a commit to grafana/tempo that referenced this pull request Aug 4, 2023

Upgrade dskit to remove dependency on weaveworks/common

2d6b9a6

See grafana/dskit#342.

charleskorn added a commit to charleskorn/pyroscope that referenced this pull request Aug 4, 2023

Upgrade dskit to remove direct dependency on weaveworks/common

f8e527e

See grafana/dskit#342.

charleskorn mentioned this pull request Aug 4, 2023

Upgrade dskit to remove direct dependency on weaveworks/common grafana/pyroscope#2207

Merged

pracucci pushed a commit to grafana/mimir that referenced this pull request Aug 4, 2023

Upgrade dskit to remove dependency on weaveworks/common (#5672)

ff4a68a

See grafana/dskit#342.

simonswine pushed a commit to grafana/pyroscope that referenced this pull request Aug 4, 2023

Upgrade dskit to remove direct dependency on weaveworks/common (#2207)

4832d64

See grafana/dskit#342.

zalegrala pushed a commit to grafana/tempo that referenced this pull request Aug 4, 2023

Upgrade dskit to remove dependency on weaveworks/common (#2763)

b5f541a

See grafana/dskit#342.

This was referenced Aug 11, 2023

Migrate weaveworks/common packages used by Loki #356

Merged

Reduce duplication introduced in weaveworks/common migration #357

Merged

duricanikolic mentioned this pull request Aug 13, 2023

Getting rid of sirupsen/logrus and log.Interface #359

Merged

2 tasks

wildum mentioned this pull request Sep 22, 2023

Replace weaveworks/common by grafana/dskit grafana/agent#5273

Closed

tpaschalis mentioned this pull request Oct 9, 2023

Avoid using promauto for metrics registration #394

Open

julienduchesne mentioned this pull request Oct 4, 2024

Import github.com/weaveworks/common/user #44

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate weaveworks/common packages used by Mimir #342

Migrate weaveworks/common packages used by Mimir #342

charleskorn commented Aug 3, 2023 •

edited

Loading

pracucci left a comment •

edited

Loading

bboreham left a comment

bboreham Aug 3, 2023

charleskorn Aug 3, 2023

charleskorn Aug 3, 2023

bboreham Aug 3, 2023

charleskorn Aug 3, 2023

charleskorn Aug 3, 2023

Migrate weaveworks/common packages used by Mimir #342

Migrate weaveworks/common packages used by Mimir #342

Conversation

charleskorn commented Aug 3, 2023 • edited Loading

pracucci left a comment • edited Loading

Choose a reason for hiding this comment

bboreham left a comment

Choose a reason for hiding this comment

bboreham Aug 3, 2023

Choose a reason for hiding this comment

charleskorn Aug 3, 2023

Choose a reason for hiding this comment

charleskorn Aug 3, 2023

Choose a reason for hiding this comment

bboreham Aug 3, 2023

Choose a reason for hiding this comment

charleskorn Aug 3, 2023

Choose a reason for hiding this comment

charleskorn Aug 3, 2023

Choose a reason for hiding this comment

charleskorn commented Aug 3, 2023 •

edited

Loading

pracucci left a comment •

edited

Loading