added a lightweight GC profiling option

[Maoni0]

added a profiling event mask in the high 32-bit, COR_PRF_HIGH_BASIC_GC, for basic GC monitoring. it can be set via ICorProfilerInfo5::SetEventMask2. all this gives you is

• GC start callback
• GC end callback
• update generational bounds

note that one different behavior between this and the existing COR_PRF_MONITOR_GC is, aside from the obvious that it doesn't give you any info beyond the above, is that the GC end callback + update generational bounds are enabled for all GCs, not just non concurrent GCs. I kept the behavior the same for COR_PRF_MONITOR_GC because I don't want to risk breaking existing profiling tools that do not anticipate these for concurrent GCs.

I marked it as [WIP] because I have NOT tested it!

@sywhang @noahfalk

@mjsabby I thought there was an issue that you opened for this but I can't find it...do you know what happened to it?

[sywhang]

Would probably want to test this but for the time being these changes look good to me.

[Maoni0]

@dotnet-bot test Windows_NT x64 Checked Innerloop Build and Test

[mjsabby]

@Maoni0 I guess I didn't? I can't find it.

One question on your change. To now determine if there is an STW GC, would we look at runtime suspension?

[Maoni0]

ok...taking off [WIP]. found one thing during testing which was an assert in profiling code was fired because I now need to check for the GC event mask in eetoprofintefaceimpl.cpp.
I tested the following -

• only having COR_PRF_MONITOR_GC on
• only having COR_PRF_HIGH_BASIC_GC on
• having both COR_PRF_MONITOR_GC and COR_PRF_HIGH_BASIC_GC on (COR_PRF_MONITOR_GC trumps COR_PRF_HIGH_BASIC_GC) - I thought about just not setting COR_PRF_HIGH_BASIC_GC in this case but I'm not sure if there's a chance that the user might want to unset just COR_PRF_MONITOR_GC
  and expect to still have the basic monitoring on. the perf diff here is extremely small anyway so I left both on.

all interested parties please feel free to CR

[Maoni0]

@mjsabby GarbageCollectStarted is always during STW. GarbageCollectFinished will not be during STW for BGC. unfortunately GarbageCollectFinished has no args to indicate if it's a BGC. so yes you'd need to turn on the COR_PRF_MONITOR_SUSPENDS mask. that's not a problem right?

[mjsabby]

Nopes, it isn’t. I just wanted to confirm and we can make a documentation note too. Thanks.

[mjsabby]

Gonna apply your change and try it out to see if GetGenerationBounds is valid between RuntimeSuspendFinished and RuntimeResumeStarted

[mattwarren]

@Maoni0 you might want to consider posting a comment about this change in https://github.com/dotnet/coreclr/issues/15136 as it's a central place that people working on .NET Profilers subscribe to.

[Maoni0]

@mattwarren seems appropriate. I'll add it; thanks!

[ygc369]

@Maoni0
Are there any documents or examples about how to use these GC events?

[noahfalk]

Adding @davmason

I talked with @Maoni0 offline but one question for you @mjsabby - can you describe the problem these events are solving for you? Whenever possible I like to have canonical examples of what customers wanted to do with various APIs.

A couple of things we'll need to note in the docs:

1. Turning on this creates the possibility that the profiler will observe GCs nest, which was never visible before. Conveniently the only pattern of start/end you might get is either:
   a) Start, Finish - this could be a non-concurrent GC (stop-the-world) or a background GC (minimal pauses)
   b) Start, Start, Finish, Finish - the outer GC is a background GC, the inner GC is a foreground GC (stop the world)

2. GetGenerationBounds currently says it can't be called inside a GC, we should update the docs to say it is legal to call it during GC finished including the case where it is a foreground GC finishing that is nested inside the time interval of a background GC.

3. Ditto for GetObjectGeneration

[noahfalk]

@ygc369 - There isn't a GC specific profiler example, however there are a few other profiler examples here: https://github.com/Microsoft/clr-samples/tree/master/ProfilingAPI
It should be relatively easy to modify one to receive the GC callbacks if you wanted to. It would involve modifying the call to SetEventMask to include the flag COR_PRF_MONITOR_GC. This new feature is similar using a different flag to enable a subset of the same events.

Updating the profiling docs is still in progress.

[mjsabby]

@noahfalk Knowing GC ranges helps with production monitoring. Often we will have some ObjectIDs, either because a set of machines has ObjectAllocated monitoring turned on, or we acquire them through other means. In our case our DI container provides that info, so knowing these sentinel objects moving from a generation to the other helps with analysis, because we can log that info in high level logs.

Also this setting allows us to have these machines have Concurrent GC turned on which wasn't possible previously.

https://docs.microsoft.com/en-us/dotnet/framework/unmanaged-api/profiling/icorprofilerinfo2-getgenerationbounds-method says it can be called in GCStart and GCEnd, but not in between. What wording would you like to change?

[noahfalk]

Often we will have some ObjectIDs either because a set of machines has ObjectAllocated monitoring turned on, or we acquire them through other means... knowing these sentinel objects moving from a generation

Do you have a mechanism to rediscover the ObjectID after a GC changes it?

What wording would you like to change?

Imagine your profiler got a series of events GCStart, GCStart, GCEnd, GCEnd. From the implementation side it is legal to call GetGenerationBounds() at the first GCEnd. However the doc makes it ambiguous if that is legal. It says don't call in between GCStart and GCEnd, and the first GCEnd is indeed between the outermost GCStart and GCEnd. I wanted to clarify in the doc that GCEnd is legal, even when nested like that.

[mjsabby]

Yes, these objects periodically write events with their own addresses. Then some of these are also in a Dictionary which refers to them, although not sure if that mechanism is being used by us.

Ahh yes, your documentation note does make it clearer. Sounds good to me.

[mjsabby]

@Maoni0 would the perf impact still be low if we also added MovedReferences2 and SurvivingReferences2 to this option?

[Maoni0]

it would certainly be much higher overhead. not as high as what you get right now with the old GC profiling mask. and you can still have BGC enabled but BGC would need an extra STW in order to give you this info and that will make BGC's STW pause way longer.

[mjsabby]

Thanks. I'm wondering if it makes sense to add another mask that gives this info OR since we haven't shipped this version of the runtime that this option only gives you this info non-BGC.

[Maoni0]

can you please explain how you use info you get with MovedReferences2 and SurvivingReferences2?

[mjsabby]

It would be used to implement Object Tracking without a custom way to rediscover ObjectIDs after a GC. Its use case is again mainly approximate analytics and will therefore be used to beef up production monitoring.

[ygc369]

@Maoni0
Can I register a callback to be called before or after GC? Just in my program, not in profile.

[Maoni0]

so do you need Surviving refs at all or you will be fine with Moved refs only? if you just want to track Moved, it means we can cut down some overhead and not do anything for BGC.

[noahfalk]

@ygc369 - You might be interested in this technique. If that isn't what you are looking for its probably better to create a new issue.

[mjsabby]

Yes, surviving references are not needed.

[ygc369]

@ygc369 - You might be interested in this technique. If that isn't what you are looking for its probably better to create a new issue.

@noahfalk Yes, that's what I want. But unfortunately, @Maoni0 didn't paste her complete code in that artical, so I still don't know how to use.

[noahfalk]

@ygc369 - I moved your issue to #24104