[chore] Refactor allocation strategies #2928

swiatekm · 2024-05-06T10:31:04Z

Allocation strategies are currently messy and duplicate a lot of logic. This is an attempt at making them more DRY.

Factored out the bookkeeping into a new Allocator struct. It keeps track of targets, collectors, allocation, metrics, etc.
Added a new Strategy interface that strategies need to implement. Right now, the minimum needed is just a method that takes a collector map and a target, and returns the assigned collector.
Added some generic allocator tests that run for all strategies.

I'm not entirely happy with the end result, and I've left some TODOs in the code I'd like to address in a follow-up. I haven't included those changes in this PR in the interest of keeping it simpler.

jaronoff97 · 2024-05-06T13:03:34Z

yeah this is super close to what i tried to do a year or two ago, and am definitely in favor of this change. The trouble I ran into back then was how you give the right context to the strategy without duplicating memory. i.e. it's not great if the strategy has to keep its own collector map and the allocator struct does too. Overall, I think the approach makes sense. I'll give it a better read later today!

swiatekm · 2024-05-06T13:20:05Z

yeah this is super close to what i tried to do a year or two ago, and am definitely in favor of this change. The trouble I ran into back then was how you give the right context to the strategy without duplicating memory. i.e. it's not great if the strategy has to keep its own collector map and the allocator struct does too. Overall, I think the approach makes sense. I'll give it a better read later today!

Is that a realistic problem? The collector count is going to be less than 100 in almost all practical circumstances, and we're only storing pointers anyway, so the memory cost is small. None of my strategy implementations need to store collectors anyway, the per-node strategy only does it for performance reasons.

In a follow-up, I'd like to restrict what data actually gets passed to the strategy in the SetCollectors call. Right now, the strategy can assume some of this data is immutable until the next call (like the name or node), but not all of it (NumTargets can change). I'd like to express this more directly in the API instead of relying on convention.

jaronoff97 · 2024-05-06T14:33:09Z

I was worried about the labelsets cardinality, but yeah if we're just using some pointers for only the collectors we should be alright. Either way, I would love to see some benchmarks to ensure that's the case. It feels like we've had some brush ups with memory and the TA recently.

swiatekm · 2024-05-06T15:39:41Z

I was worried about the labelsets cardinality, but yeah if we're just using some pointers for only the collectors we should be alright. Either way, I would love to see some benchmarks to ensure that's the case. It feels like we've had some brush ups with memory and the TA recently.

Ran the Benchmark_Setting benchmark, which looks the most appropriate for my changes. Tried others and saw no change.

                                                             │    old.txt    │               new.txt                │
                                                             │    sec/op     │    sec/op     vs base                │
_Setting/consistent-hashing_num_cols_100_num_jobs_100-32        6.042µ ±  3%   6.054µ ±  5%        ~ (p=0.481 n=10)
_Setting/consistent-hashing_num_cols_100_num_jobs_1000-32       51.12µ ±  4%   50.71µ ±  6%        ~ (p=0.529 n=10)
_Setting/consistent-hashing_num_cols_100_num_jobs_10000-32      723.1µ ±  5%   726.4µ ±  1%        ~ (p=0.579 n=10)
_Setting/consistent-hashing_num_cols_100_num_jobs_100000-32     8.688m ± 12%   8.763m ±  3%        ~ (p=0.529 n=10)
_Setting/consistent-hashing_num_cols_1000_num_jobs_100-32       48.88µ ±  7%   49.89µ ±  6%        ~ (p=0.218 n=10)
_Setting/consistent-hashing_num_cols_1000_num_jobs_1000-32      103.6µ ±  3%   105.2µ ±  2%        ~ (p=0.684 n=10)
_Setting/consistent-hashing_num_cols_1000_num_jobs_10000-32     781.2µ ±  5%   781.5µ ±  2%        ~ (p=0.853 n=10)
_Setting/consistent-hashing_num_cols_1000_num_jobs_100000-32    8.701m ±  8%   9.831m ± 12%  +12.99% (p=0.007 n=10)
_Setting/per-node_num_cols_100_num_jobs_100-32                 42.934µ ±  1%   5.933µ ±  5%  -86.18% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_1000-32                175.94µ ±  1%   50.84µ ±  6%  -71.10% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_10000-32               1723.0µ ±  3%   726.5µ ±  6%  -57.84% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_100000-32              28.073m ± 20%   8.726m ±  4%  -68.92% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_100-32                346.49µ ±  1%   48.89µ ±  7%  -85.89% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_1000-32                496.6µ ±  1%   105.4µ ±  3%  -78.77% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_10000-32              2065.8µ ±  2%   768.4µ ±  3%  -62.80% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_100000-32             27.328m ± 15%   8.874m ± 10%  -67.53% (p=0.000 n=10)
_Setting/least-weighted_num_cols_100_num_jobs_100-32            5.885µ ±  4%   5.971µ ±  4%        ~ (p=0.089 n=10)
_Setting/least-weighted_num_cols_100_num_jobs_1000-32           51.27µ ±  6%   50.84µ ±  8%        ~ (p=0.579 n=10)
_Setting/least-weighted_num_cols_100_num_jobs_10000-32          701.0µ ±  5%   703.9µ ±  3%        ~ (p=0.190 n=10)
_Setting/least-weighted_num_cols_100_num_jobs_100000-32         8.838m ± 10%   8.635m ± 16%        ~ (p=0.063 n=10)
_Setting/least-weighted_num_cols_1000_num_jobs_100-32           49.83µ ±  4%   49.19µ ±  3%        ~ (p=0.469 n=10)
_Setting/least-weighted_num_cols_1000_num_jobs_1000-32          105.2µ ±  2%   105.1µ ±  2%        ~ (p=0.579 n=10)
_Setting/least-weighted_num_cols_1000_num_jobs_10000-32         761.8µ ±  5%   783.8µ ±  5%        ~ (p=0.739 n=10)
_Setting/least-weighted_num_cols_1000_num_jobs_100000-32        8.951m ±  8%   8.805m ±  7%        ~ (p=0.315 n=10)
geomean                                                         473.1µ         302.6µ        -36.04%

                                                             │    old.txt     │                new.txt                │
                                                             │      B/op      │    B/op     vs base                   │
_Setting/consistent-hashing_num_cols_100_num_jobs_100-32           192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_100_num_jobs_1000-32          192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_100_num_jobs_10000-32         192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_100_num_jobs_100000-32        192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_100-32          192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_1000-32         192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_10000-32        192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_100000-32       192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/per-node_num_cols_100_num_jobs_100-32                   27842.5 ± 0%   192.0 ± 0%   -99.31% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_1000-32                  85440.0 ± 0%   192.0 ± 0%   -99.78% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_10000-32                661422.0 ± 0%   192.0 ± 0%   -99.97% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_100000-32              6421405.5 ± 0%   192.0 ± 0%  -100.00% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_100-32                 299523.5 ± 0%   192.0 ± 0%   -99.94% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_1000-32                357104.0 ± 0%   192.0 ± 0%   -99.95% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_10000-32               933133.0 ± 0%   192.0 ± 0%   -99.98% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_100000-32             6693000.0 ± 0%   192.0 ± 0%  -100.00% (p=0.000 n=10)
_Setting/least-weighted_num_cols_100_num_jobs_100-32               192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_100_num_jobs_1000-32              192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_100_num_jobs_10000-32             192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_100_num_jobs_100000-32            192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_100-32              192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_1000-32             192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_10000-32            192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_100000-32           192.0 ± 0%   192.0 ± 0%         ~ (p=1.000 n=10) ¹
geomean                                                          2.639Ki        192.0        -92.90%
¹ all samples are equal

                                                             │     old.txt     │                new.txt                │
                                                             │    allocs/op    │ allocs/op   vs base                   │
_Setting/consistent-hashing_num_cols_100_num_jobs_100-32            4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_100_num_jobs_1000-32           4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_100_num_jobs_10000-32          4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_100_num_jobs_100000-32         4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_100-32           4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_1000-32          4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_10000-32         4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/consistent-hashing_num_cols_1000_num_jobs_100000-32        4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/per-node_num_cols_100_num_jobs_100-32                    222.000 ± 0%   4.000 ± 0%   -98.20% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_1000-32                  1122.000 ± 0%   4.000 ± 0%   -99.64% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_10000-32                10122.000 ± 0%   4.000 ± 0%   -99.96% (p=0.000 n=10)
_Setting/per-node_num_cols_100_num_jobs_100000-32              100122.000 ± 0%   4.000 ± 0%  -100.00% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_100-32                  1185.000 ± 0%   4.000 ± 0%   -99.66% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_1000-32                 2085.000 ± 0%   4.000 ± 0%   -99.81% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_10000-32               11085.000 ± 0%   4.000 ± 0%   -99.96% (p=0.000 n=10)
_Setting/per-node_num_cols_1000_num_jobs_100000-32             101085.000 ± 0%   4.000 ± 0%  -100.00% (p=0.000 n=10)
_Setting/least-weighted_num_cols_100_num_jobs_100-32                4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_100_num_jobs_1000-32               4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_100_num_jobs_10000-32              4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_100_num_jobs_100000-32             4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_100-32               4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_1000-32              4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_10000-32             4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
_Setting/least-weighted_num_cols_1000_num_jobs_100000-32            4.000 ± 0%   4.000 ± 0%         ~ (p=1.000 n=10) ¹
geomean                                                             44.15        4.000        -90.94%

So consistent-hashing is slightly slower and per-node is much faster and uses much less memory.

jaronoff97 · 2024-05-06T17:40:02Z

WOW! yeah that per-node allocation is 🔥 great stuff.

jaronoff97

I'm definitely in favor of these, but I would love a second look from @kristinapathak or @matej-g

kristinapathak

This is awesome! 🥳 Left a few very minor comments.

cmd/otel-allocator/allocation/allocator.go

kristinapathak · 2024-05-06T21:46:25Z

cmd/otel-allocator/allocation/allocator.go

+				TargetsPerCollector.WithLabelValues(item.CollectorName, a.strategy.GetName()).Set(float64(c.NumTargets))
+			}
+			delete(a.targetItems, k)
+			delete(a.targetItemsPerJobPerCollector[item.CollectorName][item.JobName], item.Hash())


If this is the last value in a.targetItemsPerJobPerCollector[item.CollectorName] for the job, we could probably also delete the job entry:

if len(a.targetItemsPerJobPerCollector[item.CollectorName][item.JobName]) == 0 { delete(a.targetItemsPerJobPerCollector[item.CollectorName], item.JobName) }

but I don't think this has to be done in this PR.

I ended up refactoring the deletion and unassignment routines a bit and did this as well.

cmd/otel-allocator/allocation/allocator_test.go

kristinapathak · 2024-05-06T22:12:24Z

cmd/otel-allocator/allocation/least_weighted.go

-	for k, v := range allocator.collectors {
-		collectorsCopy[k] = v
+func (s *leastWeightedStrategy) GetCollectorForTarget(collectors map[string]*Collector, item *target.Item) (*Collector, error) {
+	// if a collector is already assigned, do nothing


This is sneaky but perfect for maintaining this functionality 🙂

In a follow up, I want to remove the CollectorName field from target.Item and instead track this in a separate map in the allocator. Then we can pass this map as an argument to this function and be explicit about the dependency.

swiatekm · 2024-05-07T17:12:00Z

@jaronoff97 @kristinapathak I ended up making a fairly substantial change while addressing your feedback, so I re-requested reviews from both of you.

On a separate note, should I add a changelog entry for this change? It's technically a refactor, but I've changed enough that I think it'd be fair to alert users it happened. That would make it easier to attribute any bugs I might've introduced.

cmd/otel-allocator/allocation/allocator.go

jaronoff97 · 2024-05-07T17:22:03Z

@swiatekm-sumo i think it's worth adding a changelog given all that's going on here. I also think it's worth mentioning the performance win in there as that's been some feedback we've gotten in the past few weeks. (maybe this issue in particular #2916)

cmd/otel-allocator/allocation/allocator.go

kristinapathak · 2024-05-09T16:07:32Z

cmd/otel-allocator/allocation/allocator.go

+		err := a.addTargetToTargetItems(item)
+		if err != nil {
+			assignmentErrors = append(assignmentErrors, err)
+			item.CollectorName = ""


Why do we do this here?

Getting rid of dubious test failures, mostly. One of the problems with having CollectorName be an attribute of a target is that it's not clear if targets supplied in SetTargets can have that attribute set, and what that means. The real target pipeline never sets this, but some tests sneakily do. I set it here instead of fixing those tests, because I really wanted my changes to pass existing tests unmodified.

This is one of the things I want to fix in a follow-up.

linux-foundation-easycla · 2024-05-18T16:35:17Z

The committers listed above are authorized under a signed CLA.

✅ login: swiatekm-sumo / name: Mikołaj Świątek (c129ae9, 7958d85, 6aadfed, 9e5608f, 96673f4, 40ec1b4, 640094b, 59a7922, a607911)

* Refactor consistent-hashing strategy * Refactor per-node strategy * Refactor least-weighted strategy * Minor allocation strategy refactor * Add some common allocation strategy tests * Fix collector and target reassignment * Minor allocator fixes * Add changelog entry * Fix an incorrect comment

* Initial commit * Added Cert Manager CRDs & RBAC validation and management * Added relevant resources and started adding tests * Bump github.com/gin-gonic/gin from 1.9.1 to 1.10.0 (#2953) Bumps [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) from 1.9.1 to 1.10.0. - [Release notes](https://github.com/gin-gonic/gin/releases) - [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md) - [Commits](gin-gonic/gin@v1.9.1...v1.10.0) --- updated-dependencies: - dependency-name: github.com/gin-gonic/gin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/prometheus/prometheus in the prometheus group (#2951) Bumps the prometheus group with 1 update: [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus). Updates `github.com/prometheus/prometheus` from 0.51.2 to 0.52.0 - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](prometheus/prometheus@v0.51.2...v0.52.0) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-minor dependency-group: prometheus ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Support for collector readinessProbe (#2944) * enable readiness Probe for otel operator Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * generate CRD and controller changes Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Adjusted code to be similar to Liveness logic Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Generated manifests Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Add changelog Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Fix lint Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Removed readinessProbe from alpha CRD Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Generated manifests Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Fix lint Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> * Centralized probe validation Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> --------- Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> Co-authored-by: hesam.hamdarsi <hesam.hamdarsi@gmail.com> * Bump github.com/docker/docker (#2954) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.0.1+incompatible to 26.0.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.0.1...v26.0.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Added new Log Enconder Config (#2927) * Added new Log Enconder Config Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Added new Log Enconder Config Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Added new Log Enconder Config Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Added new Log Enconder Config Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Added new Log Enconder Config Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Added new Log Enconder Config Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Added new Debug doc Signed-off-by: Yuri Sa <yurimsa@gmail.com> --------- Signed-off-by: Yuri Sa <yurimsa@gmail.com> * [chore] move VineethReddy02 to emeritus (#2957) Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de> * Cleanup cluster roles and bindings (#2938) * Fix Signed-off-by: Pavol Loffay <p.loffay@gmail.com> * Fix Signed-off-by: Pavol Loffay <p.loffay@gmail.com> * Fix Signed-off-by: Pavol Loffay <p.loffay@gmail.com> * Fix Signed-off-by: Pavol Loffay <p.loffay@gmail.com> * Add test Signed-off-by: Pavol Loffay <p.loffay@gmail.com> --------- Signed-off-by: Pavol Loffay <p.loffay@gmail.com> * Fixed non-expected warnings on TA webhook. (#2962) Signed-off-by: Yuri Sa <yurimsa@gmail.com> * Verify ServiceMonitor and PodMonitor are installed in prom cr availability check (#2964) * Verify ServiceMonitor and PodMonitor are installed in prom cr availability check * Added changelog * Bump kyverno/action-install-chainsaw from 0.2.0 to 0.2.1 (#2968) Bumps [kyverno/action-install-chainsaw](https://github.com/kyverno/action-install-chainsaw) from 0.2.0 to 0.2.1. - [Release notes](https://github.com/kyverno/action-install-chainsaw/releases) - [Commits](kyverno/action-install-chainsaw@v0.2.0...v0.2.1) --- updated-dependencies: - dependency-name: kyverno/action-install-chainsaw dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix labels for Service Monitors (#2878) * Create a separate Service Monitor when the Prometheus exporter is present Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Improve changelog Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Fix prometheus-cr E2E test Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Remove unused target Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Add docstring Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Fix typo Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Change the label name Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Change changelog description Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Recover removed labels Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Add missing labels Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Remove wrong labels Signed-off-by: Israel Blancas <iblancasa@gmail.com> --------- Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Prepare release 0.100.0 (#2960) * Prepare release 0.100.0 Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com> * update the chlog * update the chlog with #2877 merge --------- Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com> * [chore] Refactor allocation strategies (#2928) * Refactor consistent-hashing strategy * Refactor per-node strategy * Refactor least-weighted strategy * Minor allocation strategy refactor * Add some common allocation strategy tests * Fix collector and target reassignment * Minor allocator fixes * Add changelog entry * Fix an incorrect comment * Bring back webhook port (#2973) * add back webhook port * chlog * patch 0.100.1 (#2974) * Update the OpenTelemetry Java agent version to 2.4.0 (#2967) * simplify deletion logic (#2971) * Update maintainers in the operator hub PR (#2977) Signed-off-by: Pavol Loffay <p.loffay@gmail.com> * Support for kubernetes 1.30 version (#2975) * Support for kubernetes 1.30 version * Update makefile * [chore] Move TargetAllocator CRD to v1alpha1 (#2918) * [featuregate] Automatically set GOMEMLIMIT and GOMAXPROCS for collector, target allocator, opamp bridge (#2933) * set things * fix kustomize shim * restore, better chlog * Fix querying OpenShift user workload monitoring stack. (#2984) * Bump alpine from 3.19 to 3.20 (#2990) Bumps alpine from 3.19 to 3.20. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump alpine from 3.19 to 3.20 in /cmd/operator-opamp-bridge (#2991) Bumps alpine from 3.19 to 3.20. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/go-logr/logr from 1.4.1 to 1.4.2 (#2987) Bumps [github.com/go-logr/logr](https://github.com/go-logr/logr) from 1.4.1 to 1.4.2. - [Release notes](https://github.com/go-logr/logr/releases) - [Changelog](https://github.com/go-logr/logr/blob/master/CHANGELOG.md) - [Commits](go-logr/logr@v1.4.1...v1.4.2) --- updated-dependencies: - dependency-name: github.com/go-logr/logr dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump kyverno/action-install-chainsaw from 0.2.1 to 0.2.2 (#2989) Bumps [kyverno/action-install-chainsaw](https://github.com/kyverno/action-install-chainsaw) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/kyverno/action-install-chainsaw/releases) - [Commits](kyverno/action-install-chainsaw@v0.2.1...v0.2.2) --- updated-dependencies: - dependency-name: kyverno/action-install-chainsaw dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump the otel group with 5 updates (#2986) Bumps the otel group with 5 updates: | Package | From | To | | --- | --- | --- | | [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) | `1.26.0` | `1.27.0` | | [go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp](https://github.com/open-telemetry/opentelemetry-go) | `1.26.0` | `1.27.0` | | [go.opentelemetry.io/otel/metric](https://github.com/open-telemetry/opentelemetry-go) | `1.26.0` | `1.27.0` | | [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) | `1.26.0` | `1.27.0` | | [go.opentelemetry.io/otel/sdk/metric](https://github.com/open-telemetry/opentelemetry-go) | `1.26.0` | `1.27.0` | Updates `go.opentelemetry.io/otel` from 1.26.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.26.0...v1.27.0) Updates `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp` from 1.26.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.26.0...v1.27.0) Updates `go.opentelemetry.io/otel/metric` from 1.26.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.26.0...v1.27.0) Updates `go.opentelemetry.io/otel/sdk` from 1.26.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.26.0...v1.27.0) Updates `go.opentelemetry.io/otel/sdk/metric` from 1.26.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.26.0...v1.27.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel dependency-type: direct:production update-type: version-update:semver-minor dependency-group: otel - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp dependency-type: direct:production update-type: version-update:semver-minor dependency-group: otel - dependency-name: go.opentelemetry.io/otel/metric dependency-type: direct:production update-type: version-update:semver-minor dependency-group: otel - dependency-name: go.opentelemetry.io/otel/sdk dependency-type: direct:production update-type: version-update:semver-minor dependency-group: otel - dependency-name: go.opentelemetry.io/otel/sdk/metric dependency-type: direct:production update-type: version-update:semver-minor dependency-group: otel ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump alpine from 3.19 to 3.20 in /cmd/otel-allocator (#2992) Bumps alpine from 3.19 to 3.20. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Keep multiple versions of Collector Config (#2946) * Prepare v0.101.0 release (#2994) * Prepare v0.101.0 release * Undo kustomize stuff * Undo kustomize stuff again * Undo kustomize stuff again * Apply feedback * Add crd metrics usage information (#2825) * Add crd metrics usage information Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Add mode metric Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Refactor CR metrics Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Add annotation to avoid generate Metrics Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Add unit tests Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * remove space Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * remove global provider Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update main.go Co-authored-by: Israel Blancas <iblancasa@gmail.com> * revert kusttomization.yaml Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * rename some constants Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Add connectors metrics Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Update chlog Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * merge new with init, rename some functions, improve changelog entry Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * improve todo comment Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * fix tests Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * set flag to default false Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * fix lint issues Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * breaking line Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Use api reader to avoid cache issues Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> * Add info metric to changelog entry Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> --------- Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> Co-authored-by: Israel Blancas <iblancasa@gmail.com> * Update selector documentation for Target Allocator (#3001) * Bump github.com/prometheus/prometheus in the prometheus group (#3004) Bumps the prometheus group with 1 update: [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus). Updates `github.com/prometheus/prometheus` from 0.52.0 to 0.52.1 - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](prometheus/prometheus@v0.52.0...v0.52.1) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-patch dependency-group: prometheus ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump kyverno/action-install-chainsaw from 0.2.2 to 0.2.3 (#3003) Bumps [kyverno/action-install-chainsaw](https://github.com/kyverno/action-install-chainsaw) from 0.2.2 to 0.2.3. - [Release notes](https://github.com/kyverno/action-install-chainsaw/releases) - [Commits](kyverno/action-install-chainsaw@v0.2.2...v0.2.3) --- updated-dependencies: - dependency-name: kyverno/action-install-chainsaw dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Introduce simplified parsers (#2972) * Bump go.opentelemetry.io/otel/exporters/prometheus in the otel group (#3005) Bumps the otel group with 1 update: [go.opentelemetry.io/otel/exporters/prometheus](https://github.com/open-telemetry/opentelemetry-go). Updates `go.opentelemetry.io/otel/exporters/prometheus` from 0.48.0 to 0.49.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@example/prometheus/v0.48.0...example/prometheus/v0.49.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/prometheus dependency-type: direct:production update-type: version-update:semver-minor dependency-group: otel ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump go.uber.org/zap from 1.26.0 to 1.27.0 (#3006) Bumps [go.uber.org/zap](https://github.com/uber-go/zap) from 1.26.0 to 1.27.0. - [Release notes](https://github.com/uber-go/zap/releases) - [Changelog](https://github.com/uber-go/zap/blob/master/CHANGELOG.md) - [Commits](uber-go/zap@v1.26.0...v1.27.0) --- updated-dependencies: - dependency-name: go.uber.org/zap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update Kafka version in e2e test (#3009) * [chore] Bump opentelemetry-autoinstrumentation-python to 0.45b0 (#3000) * chore: Bump opentelemetry-autoinstrumentation-python to 0.45b0 * [chore] add psycopg==0.45b0 * Fix annotation/label filter setting (#3008) * fix how options are loaded by removing special casing * oop * chlog * update to specific test * oop * Added Cert Manager CRDs & RBAC validation and management * Added relevant resources and started adding tests * minor change * Minor change * minor change * Cleanup * Cleanup, go tidy and resolved conflics * Restored local dev changes * Refactored, removed init container, minor changes * Use correct files in TLS config * Added default value to getHttpsListenAddr * Added flag to enable mTLS between the Target Allocator and the Collector. go mod cleanup * Using the enable mTLS flag * Using feature gate in place of command line flags to enable the feature * Removed flag from manager yaml * Added featuregate func description * Initial unit/e2e tests. some cleanup * Using TA params * Cleanup makefile from local changes * Added step to create cert manager RBAC for e2e mtls tests * Using Kustomize for patching certmanager permissions * Cleanup chainsaw test * Cleanup chainsaw tests * e2e test case verifying Collector got secret from TA over mTLS * Added changelog, fixed unit tests * restored makefile * Renamed fg import * Linting rules for imports * Added more tests, updated the readme * Added steps in e2e tests for new app * Ran go mod tidy * Added new variable to test TA's AddTAConfigToPromConfig * Setting otel-col-contrib 0.108.0 in e2e test until operator gets updated * Update pkg/featuregate/featuregate.go Co-authored-by: Jacob Aronoff <jaronoff97@users.noreply.github.com> * Added https, serviceMonitor and tls resources assertions to e2e tests * Using namespaced names for ClusterRoles * Cleanup * Added CertManager resources unit tests * Added unit tests and e2e assertions * Added missing assertion call * Update 00-install.yaml Removed collector image override for e2e test * Update pkg/featuregate/featuregate.go Co-authored-by: Mikołaj Świątek <mail@mikolajswiatek.com> * Minor fixes * Fixed tests referencing logging exporter * Moved mTLS file naming consts * Added missing curly bracket * Update TA-update-configs-to-enable-mtls.yaml * Update pkg/featuregate/featuregate.go Co-authored-by: Mikołaj Świątek <mail@mikolajswiatek.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Janario Oliveira <janario.oliveira@gmail.com> Signed-off-by: Yuri Sa <yurimsa@gmail.com> Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de> Signed-off-by: Pavol Loffay <p.loffay@gmail.com> Signed-off-by: Israel Blancas <iblancasa@gmail.com> Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com> Signed-off-by: Ruben Vargas <ruben.vp8510@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Janario Oliveira <janario.oliveira@gmail.com> Co-authored-by: hesam.hamdarsi <hesam.hamdarsi@gmail.com> Co-authored-by: Yuri Sa <48062171+yuriolisa@users.noreply.github.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Pavol Loffay <p.loffay@gmail.com> Co-authored-by: Aksel Skaar Leirvaag <52233080+akselleirv@users.noreply.github.com> Co-authored-by: Israel Blancas <iblancasa@gmail.com> Co-authored-by: Vineeth Pothulapati <vineethpothulapati@outlook.com> Co-authored-by: Mikołaj Świątek <mail+sumo@mikolajswiatek.com> Co-authored-by: Jacob Aronoff <jaronoff97@users.noreply.github.com> Co-authored-by: OpenTelemetry Bot <107717825+opentelemetrybot@users.noreply.github.com> Co-authored-by: Vasi Vasireddy <41936996+vasireddy99@users.noreply.github.com> Co-authored-by: Ishwar Kanse <ikanse@redhat.com> Co-authored-by: Matt Hagenbuch <hagenbuch.ml@gmail.com> Co-authored-by: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Co-authored-by: Ruben Vargas <ruben.vp8510@gmail.com> Co-authored-by: brandonkzw <3462248+brandonkzw@users.noreply.github.com> Co-authored-by: Mikołaj Świątek <mail@mikolajswiatek.com>

swiatekm marked this pull request as ready for review May 6, 2024 12:38

swiatekm requested a review from a team May 6, 2024 12:38

jaronoff97 approved these changes May 6, 2024

View reviewed changes

kristinapathak approved these changes May 6, 2024

View reviewed changes

swiatekm requested review from kristinapathak and jaronoff97 May 7, 2024 17:08

jaronoff97 reviewed May 7, 2024

View reviewed changes

cmd/otel-allocator/allocation/allocator.go Outdated Show resolved Hide resolved

cmd/otel-allocator/allocation/allocator.go Outdated Show resolved Hide resolved

cmd/otel-allocator/allocation/allocator.go Outdated Show resolved Hide resolved

swiatekm force-pushed the refactor-strategies branch from e9a4f27 to eaa90bc Compare May 7, 2024 17:43

swiatekm requested a review from a team May 7, 2024 17:53

jaronoff97 approved these changes May 7, 2024

View reviewed changes

kristinapathak approved these changes May 9, 2024

View reviewed changes

swiatekm force-pushed the refactor-strategies branch from 677ece7 to 41fd94d Compare May 18, 2024 16:35

swiatekm added 9 commits May 20, 2024 11:55

Refactor consistent-hashing strategy

7958d85

Refactor per-node strategy

6aadfed

Refactor least-weighted strategy

c129ae9

Minor allocation strategy refactor

640094b

Add some common allocation strategy tests

96673f4

Fix collector and target reassignment

40ec1b4

Minor allocator fixes

9e5608f

Add changelog entry

59a7922

Fix an incorrect comment

a607911

swiatekm force-pushed the refactor-strategies branch from 41fd94d to a607911 Compare May 20, 2024 09:56

swiatekm merged commit ddd9f34 into open-telemetry:main May 20, 2024
33 checks passed

swiatekm deleted the refactor-strategies branch May 20, 2024 11:04

alexkruc mentioned this pull request Dec 5, 2024

Merge from Upstream (#1) #3514

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore] Refactor allocation strategies #2928

[chore] Refactor allocation strategies #2928

swiatekm commented May 6, 2024

jaronoff97 commented May 6, 2024

swiatekm commented May 6, 2024

jaronoff97 commented May 6, 2024

swiatekm commented May 6, 2024

jaronoff97 commented May 6, 2024

jaronoff97 left a comment

kristinapathak left a comment

kristinapathak May 6, 2024

swiatekm May 7, 2024

kristinapathak May 6, 2024

swiatekm May 7, 2024

swiatekm commented May 7, 2024

jaronoff97 commented May 7, 2024

kristinapathak May 9, 2024

swiatekm May 9, 2024

linux-foundation-easycla bot commented May 18, 2024 •

edited

Loading

[chore] Refactor allocation strategies #2928

[chore] Refactor allocation strategies #2928

Conversation

swiatekm commented May 6, 2024

jaronoff97 commented May 6, 2024

swiatekm commented May 6, 2024

jaronoff97 commented May 6, 2024

swiatekm commented May 6, 2024

jaronoff97 commented May 6, 2024

jaronoff97 left a comment

Choose a reason for hiding this comment

kristinapathak left a comment

Choose a reason for hiding this comment

kristinapathak May 6, 2024

Choose a reason for hiding this comment

swiatekm May 7, 2024

Choose a reason for hiding this comment

kristinapathak May 6, 2024

Choose a reason for hiding this comment

swiatekm May 7, 2024

Choose a reason for hiding this comment

swiatekm commented May 7, 2024

jaronoff97 commented May 7, 2024

kristinapathak May 9, 2024

Choose a reason for hiding this comment

swiatekm May 9, 2024

Choose a reason for hiding this comment

linux-foundation-easycla bot commented May 18, 2024 • edited Loading

linux-foundation-easycla bot commented May 18, 2024 •

edited

Loading