[target allocator] Rebuild targets on scrape config regex-only changes

[swiatekm]

Fixes #1358 and #1926. The fix simply serializes the scrape configs to yaml before calculating the hash, instead of using hashstructure. This is slower by about 40%, but still a little over 1ms for 10000 configs on my machine.

[swiatekm]

@matej-g noted high resource consumption from this in #1544, which this change could potentially exacerbate, but that seems strange in general considering this code only runs on changes to ServiceMonitors and the TA config file.

If there is a performance problem here, I'd prefer to solve it by setting a rate limit on these updates instead of trying to micro-optimize the hash calculation.

[matej-g]

Thanks @swiatekm-sumo, I responded in #1544 (comment), so TL;DR if there weren't performance issues reported related to this code by someone else than me, I would not worry.

If this does side-step the unexported JSON fields issue, I think it's a good way to finally address this 👍

[jaronoff97]

I'd be interested in understanding more about why we would re-calculate this so often, i would expect it only once every 15mins which is the timeout for the watcher i think. As a result, i'm less concerned with this being a problem and more concerned with the source of why we calculate so often.

[swiatekm]

I'd be interested in understanding more about why we would re-calculate this so often, i would expect it only once every 15mins which is the timeout for the watcher i think. As a result, i'm less concerned with this being a problem and more concerned with the source of why we calculate so often.

We re-calculate this on every event that changes the scrape configs, which in principle is any change to a ServiceMonitor or PodMonitor if Prometheus CRs are enabled. Otherwise, it should only change on reloading config files - which should be never after we deprecate and remove it.

[jaronoff97]

yep, but I wouldn't expect servicemonitors or podmonitors to be changed as frequently as we see for some of these reports.

[rashmichandrashekar]

@swiatekm-sumo - Sorry for the delay in responding to this PR as this seems to fix the issue I created #1926.
But, the fix is only partial because, if the same fix isn't made to the otel collector's prometheusreceiver, even though the hash computation here fixes the relabeling for targets, the collector fails to get the updated config and will not work e2e as expected for metric relabeling. The same fix needs to be made here
Are there any plans to make this change in the otel collector repo?