Intermittent extra space characters in prometheus.Metric String() calls after upgrade from v0.3.0 to v0.6.0

[blkperl]

The argo workflow projects recently updated the following dependencies in commit argoproj/argo-workflows@33ad82f. Afterwards, we started seeing intermittent test failures where sometimes extra spaces would be present in the String() response.

--- FAIL: TestCounterMetric (0.01s)
    operator_metrics_test.go:224: 
        	Error Trace:	/home/runner/work/argo-workflows/argo-workflows/workflow/controller/operator_metrics_test.go:224
        	Error:      	"label:{name:\"name\"  value:\"flakey\"}  counter:{value:1  created_timestamp:{seconds:1709341821  nanos:61663999}}" does not contain "label:{name:\"name\" value:\"flakey\"} counter:{value:1"
        	Test:       	TestCounterMetric

The following versions were changed in the commit. I've also tested the v1.19.0 of client_golang and it fails as well.

• github.com/prometheus/client_golang v1.16.0 => v1.18.0
• github.com/prometheus/client_model v0.3.0 => v0.6.0
• github.com/prometheus/common v0.42.0 => v0.48.0

Here is the relevant test code that is failing

func TestCounterMetric(t *testing.T) {

	// .... additional setup ...
	woc = newWorkflowOperationCtx(woc.wf, controller)
	woc.operate(ctx)

        // .... additional assertions ....
	metricTotalCounterString, err := getMetricStringValue(metricTotalCounter)
	assert.NoError(t, err)
	assert.Contains(t, metricTotalCounterString, `label:{name:"name" value:"flakey"} counter:{value:1`)
}

func getMetricStringValue(metric prometheus.Metric) (string, error) {
	metricString := &dto.Metric{}
	err := metric.Write(metricString)
	if err != nil {
		return "", err
	}
	return metricString.String(), nil
}

I've opened argoproj/argo-workflows#12737 as a workaround for now.

[SuperQ]

This is sadly an intentionally breaking change by the upstream protobuf library. They intentionally inject random spaces into the string output in order to prevent people from doing string comparisons in tests.

The upstream considers this working as intended. golang/protobuf#1121

We ended up having to change all our tests to marshal string to proto and compare protos.

See prometheus/client_golang#1323

[agilgur5]

Oh, did not expect that, thanks for the explanation and links/references @SuperQ!