[MM-60371] Update Pyroscope to v1

[streamer45]

Summary

PR adds support for Pyroscope v1 (specifically v1.7.1). We were still using the 0.x release which wasn't yet part of the larger Grafana ecosystem. A couple of things have changed since then, especially on the scraping front which is now a task delegated to a separate service (Grafana Alloy).

We also add support for agent goroutines profiling (which we weren't exposing) and block profiles.

There's still an open question of whether we should be using godeltaprof as an improvement but that would require changes to the server code as well so I am deferring that for the time being and stick to the stock profiles.

Ticket Link

https://mattermost.atlassian.net/browse/MM-60371

Fixes https://mattermost.atlassian.net/browse/MM-57545

[agarciamontoro]

I had a deployment lying around, switched to this branch and re-run deployment create. I got this error:

Error: failed to create terraform env: error setting up metrics server: error upload alloy config: output: bash: line 1: /etc/alloy/config.alloy: No such file or directory
, error: Process exited with status 1
exit status 1

[streamer45]

I had a deployment lying around, switched to this branch and re-run deployment create. I got this error:

Error: failed to create terraform env: error setting up metrics server: error upload alloy config: output: bash: line 1: /etc/alloy/config.alloy: No such file or directory
, error: Process exited with status 1
exit status 1

@agarciamontoro Well, I suppose you didn't go through terraform provisioning (running the metrics.sh script) since it existed already.

[agarciamontoro]

@agarciamontoro Well, I suppose you didn't go through terraform provisioning (running the metrics.sh script) since it existed already.

Hmmmm, that may be the case :( I'd say that's a bug (just not from this PR). Is this something that Ansible would solve?

[agarciamontoro]

I'll destroy and recreate to make sure it works on my end :)

[streamer45]

@agarciamontoro Well, I suppose you didn't go through terraform provisioning (running the metrics.sh script) since it existed already.

Hmmmm, that may be the case :( I'd say that's a bug (just not from this PR). Is this something that Ansible would solve?

Well, even terraform could solve but we'd have to taint the instance and probably cause a full recreation. But yeah, something like Ansible would likely check the diff in the configuration state and apply the necessary changes.

[agarciamontoro]

Well, even terraform could solve but we'd have to taint the instance and probably cause a full recreation. But yeah, something like Ansible would likely check the diff in the configuration state and apply the necessary changes.

Yeah, I'd love for the system to do the minimal changes needed; i.e., use the existing instance, apply just the config changes.

[streamer45]

Well, even terraform could solve but we'd have to taint the instance and probably cause a full recreation. But yeah, something like Ansible would likely check the diff in the configuration state and apply the necessary changes.

Yeah, I'd love for the system to do the minimal changes needed; i.e., use the existing instance, apply just the config changes.

You need to design for it then :p

[agarciamontoro]

It worked! But do we need both the instance and the service_name tags?


Can we keep just one?

[agarciamontoro]

Those are also duplicated in the application side of things, right?

[streamer45]

@agarciamontoro Not sure, to be honest, as it sounds both are required at config level:

https://grafana.com/docs/alloy/latest/reference/components/pyroscope/pyroscope.scrape/#targets-argument

May need to check if we can "hide" one.

[agarciamontoro]

@streamer45, I guess that what would be better would be to have just agents and mattermost applications, and then filter by their tags. But do we need an application for each instance? I think that's the main difference with what we had before.

[streamer45]

@streamer45, I guess that what would be better would be to have just agents and mattermost applications, and then filter by their tags. But do we need an application for each instance? I think that's the main difference with what we had before.

Okay, let me try to unify the service name at least since we probably don't need a separate one per instance.

[streamer45]

@agarciamontoro Done. This means you now need to select the additional instance tag to get profiles from a specific instance.

To keep it simple I am using the prefixes to generate the service name. I think it's clear enough but let me know if you'd rather have a different mapping (e.g. mattermost instead of app).

[agarciamontoro]

Looks good! It seems one of the new tests is failing, but approving to unblock :)