Expose activation working set as a metric

[ahydrax]

This should resolve #8279.

Microsoft Reviewers: Open in CodeFlow

[ahydrax]

reviewers fyi: dotnet/diagnostics#3542
I verified that the bug still present in dotnet-counters tool however the fix is merged and we just have to wait for new version.
Anyway, we can test correctness of UpDownCounter output by using prometheus-net.

[ReubenBond]

Thank you for opening this! I believe we should use an observable counter for performance reasons, particularly since we are already tracking the count in an interlocked variable.

[ahydrax]

Yeah, I was considering using ObservableUpDownCounter but then I had to deal with static/nonstatic context. As metrics live in static context we're unable to use observable counters until we made ActivationWorkingSet static somehow.

[ahydrax]

For sure I could make that ActivationWorkingSet initializes static field with observable counter but for me it will look like a bad design.

[ahydrax]

The way I see it is to make CatalogInstruments nonstatic then add it to DI and inject ActivationWorkingSet into it.
cc @ReubenBond

[ReubenBond]

Good point, that is a limitation of how metrics work in .NET today. We don't expect coherent results when multiple silos are hosted within a single process and this problem exists with most of the other counters we publish.

If/when .NET adopts a non-static model for counters, we can adjust our approach to accommodate.

An alternative is to also have a global count property

[ReubenBond]

The way I see it is to make CatalogInstruments nonstatic then add it to DI and inject ActivationWorkingSet into it.
cc @ReubenBond

We should probably do this, too.

[ahydrax]

Just a couple of observations made:

• Potentionally big performance issue: we allow to observe ConcurrentDictionary.Count property which could acquire more than one lock at once https://github.com/dotnet/orleans/pull/8291/files#diff-afca937af4caf78f3e9f6df7b74bb6f52eee5e1e4cf9a1764b1f710043b0f0a2L66
• Using approach I suggested will require a bigger refactoring than I expected: metrics are tightly coupled with the domain code in both directions. For example Catalog wants to use metric counters but on other side some of counters want to observe Catalog properties (see link above).

So I decided to fix major issue and proceed with current approach.