Guidance on How to Capture Traces in Kubernetes Pods

[brianrob]

Hi @brianrob. I'm currently running into a similar issue trying to run PerfView in a k8s pod. For those less knowledgeable with this tool is it possible to provide more context of what to do in this scenario?

Is the expectation on the Host node a user manually starts the PerfView process to collect a trace using the /EnbleEventsInContainer parameter. After they are done collecting the trace you then copy the zip into the container and perform a PerfView merge -ImageIDsOnly NameOfTrace.etl.zip? What does -ImageIDsOnly do exactly? I run that command and can't really tell if it worked.

If you don't run -ImageIDsOnly is it still possible to use that trace to profile what occurred in the k8s pod?

If this should be a new issue or discussion please let me know. Thanks!

Originally posted by @Cjewett in #1790

[brianrob]

There are some strong limitations in how we can best collect ETW traces on Windows when containers are involved. The happy path is the following:

1. Capture an ETW trace on the host node. Specify /EnableEventsInContainers when collecting to ensure that user-mode events flow from processes inside of containers back to the ETW session. If you don't specify this, then you'll get all kernel events but only user-mode events from processes running directly on the host and not inside of a container.
2. At this point, as long as the container(s) containing the process(es) of interest are still running, you can open and analyze the trace on the host node. PerfView will be able to find any binaries that it needs both on the host and inside the containers.
3. If you want to shutdown a container of interest, or copy the trace to another machine for analysis where the container is inaccessible, then you'll need to copy the trace into the container, and then inside the container, run PerfViewCollect merge -ImageIDsOnly <path/to/trace>. Once it completes, copy the resulting file (overwrites the original) out of the container. This command is responsible for opening all of the accessible binaries that have symbols in the trace, and injects PDB information into the trace, so that we can download the PDBs from the symbol server later on during analysis. If you don't do this, then some symbols just won't be resolvable later. The data is still valid, but might be less useful.

[brianrob]

@Cjewett, let me know if you have any questions on this. I'm also going to use this issue to get copilot to update the PerfView users guide with this information.

[Cjewett]

Thank you for the quick turnaround @brianrob! Will be using this today to generate some traces to help us profile MSBuild restore and dotnet test performance issues in containerized environments. Will provide feedback as I go through this.

[brianrob]

Sounds good, thanks @Cjewett!

[Cjewett]

I was able to go through all of the steps.

The only issue I ran into was this one: https://github.com/microsoft/perfview/pull/2344/files#diff-cc45d1cf5739affb5c8eb9f467e1175dd420f9751a7ef1137928bd3eae7f7099R6406
I had to specify /ImageIDsOnly after specifying the DataFile. PerfViewCollect merge MyContainerTrace.etl.zip /ImageIDsOnly

One thing I'm confused on is if my merged output is correct or not because when opening it PerfView reported 85.2% broken stacks or something like that. I was able to use the Lookup Symbols option in the GUI to display more details though. See this post for more detailed information on what I'm using this for: microsoft/testfx#7042 (reply in thread)

[brianrob]

Ah, that's a good point - I'll get that fixed up.

One of the tough things about this scenario is that because we're running in multiple environments (inside and outside of the container) it's hard to know if something is missing or if it's just not present in the current environment. Broken stacks can occur for a few different reasons, but they shouldn't be caused by lack of image ID information - Image IDs are really about symbolic resolution, so if you can resolve symbols for binaries both inside and outside the container, then you're doing it right.

[Cjewett]

Thanks! In that case this looks good. Appreciate the quick turnaround and improving the docs!