kubectl debug with chiseled images - cannot list processes

[MichelZ]

We would like to debug one of our chiseled images with dotnet-monitor.

The to-be-monitored container uses mcr.microsoft.com/dotnet/aspnet:8.0-jammy-chiseled-extra
it uses USER $APP_UID

We are trying to start a dotnet-monitor container with kubectl debug like this:

kubectl debug -it --namespace $Namespace -c $DebugName $PodName --image=mcr.microsoft.com/dotnet/monitor:8 --target=$TargetContainerName --env=DotnetMonitor_Egress__AzureBlobStorage__monitorBlob__accountUri=$storageAccount --env=DotnetMonitor_Egress__AzureBlobStorage__monitorBlob__containerName=dotnet-monitor --env=DotnetMonitor_Egress__AzureBlobStorage__monitorBlob__blobPrefix=artifacts --env=DotnetMonitor_Egress__AzureBlobStorage__monitorBlob__managedIdentityClientId=$managedIdentity --env=DotnetMonitor_Storage__DefaultSharedPath=$ContainerSharedPath --arguments-only -- collect --no-auth --urls http://localhost:52323 --metrics false

We have $ContainerSharedPath as a path inside the container that is writable in the to-be-monitored container:
COPY --from=publish --chown=$APP_UID:$APP_UID --chmod=u+rw /app/App_Data/Logs /app/App_Data/Logs

The dotnet-monitor container starts fine, we can port forward to 52323.
Response from /info:
{"version":"8.0.3-servicing.24352.8+6de4b5d5a5d4b2c4d8bc471455c998bfac42ab8f","runtimeVersion":"8.0.7","diagnosticPortMode":"Connect","diagnosticPortName":null}

However, when we try to start a dump using:
Invoke-WebRequest -UseBasicParsing -Uri http://localhost:52323/dump?egressProvider=monitorBlob
We get:
Invoke-WebRequest: {"status":400,"detail":"Unable to discover a target process."}

Listing processes using Invoke-WebRequest -UseBasicParsing -Uri http://localhost:52323/processes
leads to an empty 200 response:

StatusCode        : 200
RawContent        : HTTP/1.1 200 OK
                    Date: Mon, 22 Jul 2024 10:04:19 GMT
                    Server: Kestrel
                    Transfer-Encoding: chunked
                    Content-Type: application/json; charset=utf-8

                    []

Any suggestions how we can get this to work?

Thanks

[wiktork]

@MichelZ dotnet-monitor does not understand procfs based connections. You can get around this by:

• Creating an emptydir volume mount against /tmp in your app unfront.
• What kubectl debug does underneath is actually call the api to create an ephemeral container. Unfortunately kubectl debug does not expose volume mounting as a property. You'll need to use the kubernetes api directly (e.g. using kubectl proxy) and then patch. @iximiuz has some great examples and blogs on this particular topic.

[iximiuz]

Oh, indeed! cdebug jumps a few hoops for you to make the cdebug exec experience smoother than kubectl debug and like kubectl debug, it also uses ephemeral containers, so you can cdebug exec in distroless or slim containers with ease.

[MichelZ]

Funnily enough, there are new features for kubectl debug that may make this easier:

kubernetes/kubernetes#120346

Example of debug profile with volume:
https://github.com/bavarianbidi/kubectl-dpm?tab=readme-ov-file#kubectl-custom-debug-profiles

[iximiuz]

Oh, I'm afraid there is no pre-built version of cdebug for Windows. It will likely work out of the box or require some minor tweaks if compiled from the source, but I don't have any spare capacity to actually test it properly and set up a Windows build pipeline. What about WSL?

[EugeneKrapivin]

Hey @MichelZ !
I'm so happy to find this discussion, you won't believe it :)

I'm fighting this thing for some time now, I can't seem to figure out how to configure this whole party, hoping you'd be able to give me a direction.

We've got a bunch of net6 services (hopefully 8/9 soon) which run in k8s. Taking a dump from those services became a nightmare. I've got the dotnet-monitor running on some side project of mine as a sidecar, in my app container I added the environment variable

            - name: DOTNET_DiagnosticPorts
              value: /diag/dotnet-monitor.sock

and mounted the volume (I have the volume definition in the deployment.yaml for the app as well)

          volumeMounts:
          - mountPath: /diag
            name: diagvol

Did you find those configurations necessary on the app deployment.yaml as a precondition?

Were you successful in deploying the dotnet-monitor as an ephemeral container using kubectl debug and port-forwarding the 52323 port?

sorry for the barrage of questions, I'm just happy I found a lead on the matter

[MichelZ]

Hi @EugeneKrapivin , I've seen your posts in the dotnet-docker dotnet/dotnet-docker#5233 discussion :)
I did not pursue this further and was never able to get it working like I wanted to :(
Custom profiles for kubectl debug are on the way to be released as stable in v1.32 (kubernetes/enhancements#4292), so I might have a look again once that's released

[EugeneKrapivin]

What did you end up doing to get those sweet sweet dumps out of the pod?

[MichelZ]

I don‘t remember, I think I debugged it without the dump 😅
Fortunately I rarely need them