Profiling API

[gregoryyoung]

Is there any timeframe on a profiling API? Are there any public design docs available at this point?

Greg

[jkotas]

ICorProfiler profiling APIs are available in CoreCLR (on Windows, on Unix - not yet). They work same as in the full .NET Framework. The main difference is that the environment variables to setup the profiler have CoreClr_ prefix to avoid colliding with the full .NET Framework (ie CORECLR_ENABLE_PROFILING, CORECLR_PROFILER_PATH for CoreCLR vs. COR_ENABLE_PROFILING, COR_PROFILER_PATH for full .NET Framework).

Are you looking for anything particular?

[sawilde]

Do you know why *_PROFILER_PATH doesn't come in 32 and 64 bit versions - it limits this feature as the processes being profiled could be a combination of both but you can only register one.

[jkotas]

32-bit vs. 64-bit profiler path: It is a good suggestion. Could you please open a separate issue for it?

[MattWhilden]

I've opened dotnet/coreclr#601 to track the suggestion by @sawilde.

@gregoryyoung do you have enough information or are there other specific's you'd like?

[MattWhilden]

Alright, I'm closing this down. Let me know if that was in error.

[schourode]

@jkotas You mention that the Profiling API is already available on Windows, but not yet on Unix. Is it the plan to provide this API (or something similar) on all platforms in the future?

If I am not mistaken, this API is what the guys over at NewRelic are missing in order to instrument .NET apps on non-Windows hosts. For reference: https://docs.newrelic.com/docs/agents/net-agent/getting-started/new-relic-net#app-frameworks

[gregoryyoung]

We are basically in the same boat.

On Thu, Jun 4, 2015 at 5:03 PM, Jørn Schou-Rode notifications@github.com
wrote:

@jkotas https://github.com/jkotas You mention that the Profiling API is
already available on Windows, but not yet on Unix. Is it the plan to
provide this API (or something similar) on all platforms in the future?

If I am not mistaken, this API is what the guys over at NewRelic are
missing in order to instrument .NET apps on non-Windows hosts. For
reference:
https://docs.newrelic.com/docs/agents/net-agent/getting-started/new-relic-net#app-frameworks

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-108908467.

Studying for the Turing test

[jkotas]

This issue should not have been closed. Yes, we should enable the profiling APIs on Unix to support the ecosystem of tools build on top of them going cross-plat. We are aiming for as much parity as possible for the runtime capabilities across platforms.

cc @sergiy-k @mikem8361 @noahfalk

[amanda-mitchell]

I've begun to work on this a bit. You can track my progress (or tell me that I'm doing something terrible) at https://github.com/david-mitchell/coreclr/tree/enable-profiling

[gregoryyoung]

👍

On Wed, Jul 15, 2015 at 5:56 PM, David Mitchell notifications@github.com
wrote:

I've begun to work on this a bit. You can track my progress (or tell me
that I'm doing something terrible) at
https://github.com/david-mitchell/coreclr/tree/enable-profiling

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-121678593.

Studying for the Turing test

[sawilde]

👍

[noahfalk]

Thanks @david-mitchell ! Last I heard a new coworker here at msft will looking at this area soon (he is still in process of transferring teams). In the meantime feel free to reach out to me for any questions, feedback, PRs, etc.

[amanda-mitchell]

I've got the Profiling API building on OS X (haven't tried Linux yet), and I've created a proof-of-concept profiler at https://github.com/david-mitchell/CoreCLRProfiler

[gregoryyoung]

Just my .02 I actually prefer the mono model on this one. It is far easier to comprehend than transporting the windows model.

[amanda-mitchell]

@gregoryyoung the MS Profiling API has a number of features that are not present in mono's profiler. The New Relic profiler, for example, could not be built on top of Mono's Profiling API as it exists today. (see https://github.com/david-mitchell/NewRelicProfiler for more information on this)

[sawilde]

@david-mitchell - all I see is code, no details on the issues - perhaps you could expand more (even if you put it in the wiki).

As @gregoryyoung says the mono approach does seem much cleaner (once you get past the low documentation footprint) than the windows COM way; all that constant querying via COM interfaces (I know I've done my fair share of it) is awfully tedious.

I'd be interested to know where the gaps were when/if I decide to go ahead on the mono version of OpenCover.

[amanda-mitchell]

The main issue is that the MS version of the API supplies for method rewriting—along with provisions for allocating memory to be used for this purpose and other related functionality—which mono does not.

In any case, the COM API already exists in CoreCLR, and there are advantages to supporting it—for example, porting existing Windows profilers to OS X/Linux should be much easier if a similar API is preserved.

[kangaroo]

Basically this boils down to rejit, which is a big thing for profiler vendors. No?

[sawilde]

Basically this boils down to rejit, which is a big thing for profiler vendors. No?

Nope, looked into it but in the end never really found a need for it - can't talk for the commercial guys but I've yet to see any code in the open-source community that used it outside just exploratory testing. If however is @david-mitchell says there is no way to rewrite a method (I just assumed the profilers in the Open Source space had never needed to and so no obvious code sample) then that is a bit of a downer.

[ayende]

TypeMock does rejit, using the profiler API, IIRC.

*Hibernating Rhinos Ltd *

Oren Eini* l CEO l *Mobile: + 972-52-548-6969

Office: +972-4-622-7811 *l *Fax: +972-153-4-622-7811

On Sat, Jul 18, 2015 at 9:34 AM, Shaun Wilde notifications@github.com
wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says there
is no way to rewrite a method (I just assumed the profilers in the Open
Source space had never needed to and so no obvious code sample) then that
is a bit of a downer.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-122498114.

[mattwarren]

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post
http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde notifications@github.com wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says there
is no way to rewrite a method (I just assumed the profilers in the Open
Source space had never needed to and so no obvious code sample) then that
is a bit of a downer.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-122498114.

[mattwarren]

Sorry that should've been private eye by @gregoryyoung

On Saturday, 18 July 2015, Matt Warren matt.warren@live.co.uk wrote:

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post
http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says
there is no way to rewrite a method (I just assumed the profilers in the
Open Source space had never needed to and so no obvious code sample) then
that is a bit of a downer.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-122498114.

[gregoryyoung]

We are not using it at this time. On mono as an example there is a method
entered callback already

On Sat, Jul 18, 2015 at 10:46 AM, Matt Warren notifications@github.com
wrote:

Sorry that should've been private eye by @gregoryyoung

On Saturday, 18 July 2015, Matt Warren matt.warren@live.co.uk wrote:

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post

http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says
there is no way to rewrite a method (I just assumed the profilers in the
Open Source space had never needed to and so no obvious code sample)
then
that is a bit of a downer.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-122498114.

—
Reply to this email directly or view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-122514426.

Studying for the Turing test

[noahfalk]

@gregoryyoung when you mentioned the Mono model was easier, are you referring specifically to the fact that it has a flat C style convention vs. COM interfaces, or other aspects of the API? Depending on what issues you are trying to address a small adapter might be able to expose a mono-like API on top of the existing COM API? I agree with @david-mitchell that the COM API feels like a good place to start the port.

I can say that we've got some existing non-open source folks already using ReJIT / interested in using ReJIT soon. Short of some significant technical obstacle I've always imagined it on our .Net Core roadmap. The jump-stamp implementation that it relies on might not be trivial to port cross platform so I wouldn't be surprised if it was enabled a little later in the overall porting process.

@david-mitchell I took a quick look at your code changes so far. Looks good, My only comment at this point is that Maoni will probably appreciate if we put the FEATURE_TRACE_EVENT ifdefs inside the implementation of the various ETW methods vs. around the call-site. I'm assuming the various platform linkers are all smart enough to eliminate calls to empty methods. I think @brianrob has been identifying a potential ETW substitute on other platforms and a lot of these methods might start looking like:

void EmitEtwEventXYZ
{
    #ifdef FEATURE_WINDOWS_ETW
    // do the right thing for ETW
    #elif FEATURE_LINUX_EVENTS
    // do the right thing for Linux
    // add more options as needed for other OS tracing technologies
    #endif
}

[amanda-mitchell]

@noahfalk, I definitely agree about keeping FEATURE_TRACE_EVENT in one place. I had just begun down that path when I saw your message.

[brianrob]

Agree with @noahfalk that we should keep the #ifdefs in the EmitEvent method. Going forward, we'll plan to create an abstraction layer that will encapsulate the platform specific pieces, so that call sites don't have to know anything about them.

[gregoryyoung]

I just want to ask as a side question how 32 bit vs 64 bit will work with
RTM. Will they use the same environment variables etc?

On Sun, Jun 12, 2016 at 2:56 AM, Jan Kotas notifications@github.com wrote:

the short term by setting the environment variable COMPLUS_ReadyToRun=0

Or return COR_PRF_DISABLE_ALL_NGEN_IMAGES from the profiler (it requires
a fix that I have made earlier today). Profilers should treat ReadyToRun
images same way as they treat NGen images on full framework.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/coreclr/issues/445#issuecomment-225401074, or mute
the thread
https://github.com/notifications/unsubscribe/AAXRWgxl6-a5XeutltAc3EpaZz5CoVXAks5qK0spgaJpZM4DuzPL
.

Studying for the Turing test

[dotnetjt]

Looks like https://github.com/dotnet/coreclr/issues/601 solved it

[gregoryyoung]

@dotnetjt 404

[ghost]

@gregoryyoung dotnet/coreclr#601

[twgraham]

Is anyone able to give an update on the status of this issue? @kspawa I notice that the status doc hasn't been updated for a while.

[lt72]

We anticipate that .Net Core profiling API will work xplat by mid-Fall. I cannot offer a more precise deadline at this time though.

[discostu105]

@lt72 Are there any known shortcomings of the profiling api under linux known so far? Or is it just untested?

[noahfalk]

PR dotnet/coreclr#7719 has been doing some known work for ELT that is Linux specific. PR dotnet/coreclr#9298 is handling some rejit related issues with the ReadyToRun format that appears OS-agnostic. Other than those two known areas I think the rest is a lack of testing.

[lt72]

Mostly just untested, David is working on completing testing already.

[ViktorHofer]

We anticipate that .Net Core profiling API will work xplat by mid-Fall. I cannot offer a more precise deadline at this time though.

Sound like we should move it to a future milestone?

[noahfalk]

Just a PSA since I know this thread has many people interested in profiler APIs, my recent PR dotnet/coreclr#12193 has some profiling changes in it.

Is there interest in creating an informal group where profiler relevant changes can be announced/discussed in the future? I'm not that wise to ways of github but that seems like something we should be able to do if there was interest?

[rfrancisco]

Can you give us a status update of this?

[lt72]

Profiler API has now been tested on Linux and Windows for x64/x86. ARM32 testing is in progress and going well, with a few known areas to be completed that are currently being worked on. For tracking ARM32 progress, see dotnet/coreclr#14526, dotnet/coreclr#13992, and dotnet/coreclr#13993.

[noahfalk]

FYI I've got a PR out to update the status page: dotnet/coreclr#14644

Separately, I haven't found a good way to create a group that could be used in @ mentions unless all the people belong to the same organization, but it does seem like there was interest in the general idea above. Does anyone have a good suggestion about how to construct something? Currently my best thought would be to create an issue that is dedicated to profiler related announcements and anyone interested could follow it. On the page we could add links to relevant issues and PRs as they appear. The sole purpose of the page would be to generate a notification to followers, all the actual discussion would occur within the issues that were linked. Is that a weird workaround for not being able to group @ mention? Good?

[jaredcnance]

Makes sense. If the goal is an announcements page, I would recommend locking the issue to prevent unofficial conversation. It's similar to the main announcements repo: https://github.com/dotnet/announcements

which may even be a better place for it, I don't know.

[lt72]

It seems we need to create a dedicated documentation page in CoreCLR repo.

[patricksuo]

@lt72

Profiler API has now been tested on Linux and Windows for x64/x86. ARM32 testing is in progress and going well, with a few known areas to be completed that are currently being worked on. For tracking ARM32 progress, see dotnet/coreclr#14526, dotnet/coreclr#13992, and dotnet/coreclr#13993.

would you like to point me to a minimal Linux x64 profiler, or point me to your test program (for Linux x64)?

is this profiler still compatible with latest dotnet core CLR?

[patricksuo]

We have this profiler with dotnet daily build 2.1.0-preview1-25907-02, recevied the following random segment fault:

(lldb) register read
General Purpose Registers:
       rax = 0x0000000000000003
       rbx = 0x000000000093db50
       rcx = 0x0000000000000000
       rdx = 0x0000000000940390
       rdi = 0x0000000000000000
       rsi = 0x0000000000940390
       rbp = 0x00007fffef072310
       rsp = 0x00007fffef0722e0
        r8 = 0x0000000000940390
        r9 = 0x00007fff480cc950
       r10 = 0x0000000000000000
       r11 = 0x00007ffff6f5cf10
       r12 = 0x00007fff7d6cf548
       r13 = 0x0000000000674ab0
       r14 = 0x0000000000000001
       r15 = 0x0000000000000000
       rip = 0x00007ffff621f99d  libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93
    rflags = 0x0000000000010206
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000

* thread dotnet/coreclr#8: tid = 6263, 0x00007ffff621f99d libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x00007ffff621f99d libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93
    frame dotnet/coreclr#1: 0x00007ffff63574d6 libcoreclr.so`MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) + 710
    frame dotnet/coreclr#2: 0x00007ffff6356c90 libcoreclr.so`MethodDesc::JitCompileCode(PrepareCodeConfig*) + 416
    frame dotnet/coreclr#3: 0x00007ffff6356a22 libcoreclr.so`MethodDesc::PrepareILBasedCode(PrepareCodeConfig*) + 162
    frame dotnet/coreclr#4: 0x00007ffff6358ed1 libcoreclr.so`MethodDesc::DoPrestub(MethodTable*) + 993
    frame dotnet/coreclr#5: 0x00007ffff635888d libcoreclr.so`PreStubWorker + 445
    frame dotnet/coreclr#6: 0x00007ffff62da3c4 libcoreclr.so`ThePreStub + 92
    frame dotnet/coreclr#7: 0x00007fff8061478b
    frame dotnet/coreclr#8: 0x00007fff8061446f
    frame dotnet/coreclr#9: 0x00007fff7f2ee0b1
    frame dotnet/coreclr#10: 0x00007fff80613141
    frame dotnet/coreclr#11: 0x00007fff80612fd8
    frame dotnet/coreclr#12: 0x00007fff80612ca3
    frame dotnet/coreclr#13: 0x00007fff7ddb097d
    frame dotnet/coreclr#14: 0x00007fff806125ea
    frame dotnet/coreclr#15: 0x00007ffff62d96df libcoreclr.so`CallDescrWorkerInternal + 124
    frame dotnet/coreclr#16: 0x00007ffff61fce92 libcoreclr.so`DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 242
    frame dotnet/coreclr#17: 0x00007ffff637a009 libcoreclr.so`RegisterWaitForSingleObjectCallback_Worker(void*) + 201
    frame dotnet/coreclr#18: 0x00007ffff61cf19d libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 413
    frame dotnet/coreclr#19: 0x00007ffff61cf900 libcoreclr.so`ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) + 64
    frame dotnet/coreclr#20: 0x00007ffff6379e6a libcoreclr.so`RegisterWaitForSingleObjectCallback(void*, unsigned char) + 266
    frame dotnet/coreclr#21: 0x00007ffff61eceb4 libcoreclr.so`ThreadpoolMgr::AsyncCallbackCompletion(void*) + 356
    frame dotnet/coreclr#22: 0x00007ffff635e756 libcoreclr.so`UnManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 470
    frame dotnet/coreclr#23: 0x00007ffff61edcfb libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(void*) + 1211
    frame dotnet/coreclr#24: 0x00007ffff6567652 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 306
    frame dotnet/runtime#3858: 0x00007ffff79c4dc5 libpthread.so.0`start_thread + 197
    frame dotnet/runtime#3859: 0x00007ffff6ed273d libc.so.6`__clone + 109

(lldb) sos ClrStack
OS Thread Id: 0x1877 (8)
        Child SP               IP Call Site
00007FFFEF072550 00007ffff621f99d [PrestubMethodFrame: 00007fffef072550] System.Threading.WaitHandle.Dispose(Boolean)
00007FFFEF0726A0 00007FFF8061478B System.Diagnostics.ProcessWaitHandle.Dispose(Boolean)
00007FFFEF072720 00007FFF8061446F System.Threading.WaitHandle.Dispose()
00007FFFEF0727A0 00007FFF7F2EE0B1 System.Diagnostics.Process.StopWatchingForExit()
00007FFFEF072830 00007FFF80613141 System.Diagnostics.Process.CompletionCallback(System.Object, Boolean)
00007FFFEF0728B0 00007FFF80612FD8 System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context(System.Object, Boolean)
00007FFFEF072930 00007FFF80612CA3 System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context_f(System.Object)
00007FFFEF0729B0 00007FFF7DDB097D System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007FFFEF072A60 00007FFF806125EA System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)
00007FFFEF072BA0 00007ffff62d96df [GCFrame: 00007fffef072ba0]
00007FFFEF072C80 00007ffff62d96df [DebuggerU2MCatchHandlerFrame: 00007fffef072c80]

[patricksuo]

and cc @noahfalk . Maybe you can give me some suggestion.

[patricksuo]

Received segfault with dotnet run --no-build.
But run thousands of rounds dotnet bin/Debug/netcoreapp2.1/Foooo.dll without a problem.

[noahfalk]

@sillyousu - I opened a separate issue so we avoid enlarging this already very meandering thread.

[noahfalk]

I think the original issues that opened this thread have been resolved so I'm going to close it. If that is not the case we should probably open a more specific issue on any outstanding questions given how much this thread has meandered.

Above I mentioned the idea of creating a profiler group, I think a number of you liked that idea, and the best I've been able to find is creating an issue similar to the announcement repo. I spent a while chatting with a few our PMs whether I should actually use the announcement repo but ultimately came to the conclusion that it wasn't the best choice. Expectations there a bit more formal and most of the people watching it probably don't care about the low level stuff that a profiler writer is interested in. Instead I created issue dotnet/coreclr#15136 and anyone who is interested is welcome to follow it. I hope this works out, but if you have a better suggestion I'm always open to hear them. Thanks!