Registry Counter panic: duplicate metrics collector registration attempted

[bigpigeon]

client version

github.com/prometheus/client_golang v1.4.1

go.mod

module github.com/bigpigeon/Test/go/prometheus_example_http
go 1.13

require (
	github.com/prometheus/client_golang v1.4.1
	github.com/prometheus/common v0.9.1
)

Problem

when I Registry prometheus.Counter with prometheus must be panic with duplicate metrics collector registration attempted

source code

package main

import (
	"github.com/prometheus/common/expfmt"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

func recordMetrics(opsProcessed prometheus.Counter) {
	go func() {
		for {
			opsProcessed.Inc()
			time.Sleep(2 * time.Second)
		}
	}()
}

func main() {

	opsProcessed := promauto.NewCounter(prometheus.CounterOpts{
		Namespace: "bigpigeon",
		Subsystem: "test",
		Name:      "xxx",
		Help:      "The total number of processed events",
	})

	prometheus.MustRegister(opsProcessed)

	recordMetrics(opsProcessed)

	mfs, err := prometheus.DefaultGatherer.Gather()
	if err != nil {
		panic(err)
	}
	http.HandleFunc("/metrics", func(writer http.ResponseWriter, request *http.Request) {
		for _, mf := range mfs {
			if mf.Help != nil && *mf.Help == "test histogram bucket" {
				if _, err := expfmt.MetricFamilyToText(writer, mf); err != nil {
					if err != nil {
						panic(err)
					}
				}
			}
		}
	})
	http.ListenAndServe(":9090", nil)
}

[bwplotka]

Hi 👋

This is expected, as you used promauto which automatically registers the metric. (:

You have few options (you can choose any of those three):

1. Use prometheus package for creating metrics: prometheus.NewCounter(prometheus.CounterOpts{
2. Don't register prometheus.MustRegister(opsProcessed). promauto already does that.
3. Better: Use custom registry to have explicit control on what is registered and when:

reg := prometheus.NewRegistry()
opsProcessed := promauto.With(reg).NewCounter(prometheus.CounterOpts{
		Namespace: "bigpigeon",
		Subsystem: "test",
		Name:      "xxx",
		Help:      "The total number of processed events",
	})
// No need to register with prometheus.MustRegister(opsProcessed) anymore.

Let us know if you have more questions! 🤗 Otherwise closing (:

[beorn7]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[1a1a11a]

@beorn7 GitHub is easier to find for someone not on the mailing list and benefits more people

[jeeftor]

@1a1a11a - Fully agree with this

[meetsubhojit]

@beorn7 GitHub is easier to find for someone not on the mailing list and benefits more people

and its google searchable

[clintonShopify]

Going to just add my notes in case it's helpful for anyone else. I think @bwplotka gave a solid answer but I'll add to it.

Bottom Line Upfront

Use prometheus client instead of promauto to handle fallible registry of new prometheus metrics.

1. Sometimes one might have a metric already register. There is an example on how to handle already registered metrics that one can follow.

Why?

I was using promauto but as the notes in the code suggests:

This appears very handy. So why are these constructors locked away in a
separate package?

The main problem is that registration may fail, e.g. if a metric inconsistent
with or equal to the newly to be registered one is already registered.
Therefore, the Register method in the prometheus.Registerer interface returns
an error, and the same is the case for the top-level prometheus.Register
function that registers with the global registry. The prometheus package also
provides MustRegister versions for both. They panic if the registration
fails, and they clearly call this out by using the Must… idiom. Panicking is
problematic in this case because it doesn't just happen on input provided by
the caller that is invalid on its own. Things are a bit more subtle here:
Metric creation and registration tend to be spread widely over the
codebase. It can easily happen that an incompatible metric is added to an
unrelated part of the code, and suddenly code that used to work perfectly
fine starts to panic (provided that the registration of the newly added
metric happens before the registration of the previously existing
metric). This may come as an even bigger surprise with the global registry,
where simply importing another package can trigger a panic (if the newly
imported package registers metrics in its init function). At least, in the
prometheus package, creation of metrics and other collectors is separate from
registration. You first create the metric, and then you decide explicitly if
you want to register it with a local or the global registry, and if you want
to handle the error or risk a panic. With the constructors in the promauto
package, registration is automatic, and if it fails, it will always
panic. Furthermore, the constructors will often be called in the var section
of a file, which means that panicking will happen as a side effect of merely
importing a package.

A separate package allows conservative users to entirely ignore it. And
whoever wants to use it will do so explicitly, with an opportunity to read
this warning.