[processor/deltatocumulative] partial linear pipeline

[sh0rez]

Description:
Partially introduces a highly decoupled, linear processing pipeline.
Implemented as a standalone struct to make review easier, will refactor this later.
Instead of overloading Map.Store() to do aggregation, staleness and
limiting, this functionality is now explcitly handled in
ConsumeMetrics.

This highly aids readability and makes understanding this processor a
lot easier, as less mental context needs to be kept.

Notes to reviewer:
See 68dc901 for the main added logic.
Compare processor.go (old, nested) to linear.go (new, linear)

Replaces #34757

Link to tracking Issue: none

Testing: This is a refactor. Existing tests were not modified and still pass

Documentation: not needed

[ArthurSens]

It looks good to me as a first glance, but it would be awesome if we had some kind of end-to-end test with generated testdata, similar to what we have with intervalprocessor.

Not sure if I'm missing something, but I don't see a test that creates a new deltatocumulative processor through the Factory, call ConsumeMetrics and checks the result. I see for the original processor, but not for linear. I can also see the Chain object you created to call two processors together, but I'm not understanding where exactly it's used xD. In summary, I think you covered the e2e tests with linear but I'm struggling to understand how exactly

[sh0rez]

@ArthurSens It looks good to me as a first glance, but it would be awesome if we had some kind of end-to-end test with generated testdata, similar to what we have with intervalprocessor.

It's there, in recently merged processor_test.go:

opentelemetry-collector-contrib/processor/deltatocumulativeprocessor/processor_test.go

Lines 39 to 44 in 52937cf

	proc, err := self.NewFactory().CreateMetricsProcessor(
	context.Background(),
	processortest.NewNopSettings(),
	cfg,
	next,
	)

---

@ArthurSens I can also see the Chain object you created to call two processors together, but I'm not understanding where exactly it's used xD

It's in factory.go. I first create both processors (Processor and Linear) as usual, then return them as Chain{}. This works, because Chain is a []processor.Metrics, which itself implements processor.Metrics on that slice

opentelemetry-collector-contrib/processor/deltatocumulativeprocessor/factory.go

Line 44 in 52937cf

return Chain{linear, proc}, nil

[ArthurSens]

But if I'm reading things correctly, we're only calling the first processor of the Chain

https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/35048/files#diff-5922c3911f502d9f26fbd71672eca70c790535c8e35d4bd61f14b10bafd82d7eR28-R33

EDIT:
ok nevermind, finally understood. One processor is the next of the other, so when we call the first processor in the chain, the first one will call the second 👍

[sh0rez]

@RichieSams can you take a look?

[RichieSams]

@RichieSams can you take a look?

Yes. Apologies. I've been meaning to take a look at this for a while now. But kept getting waylaid. I'll review this afternoon

[RichieSams]

Minor: Can we rename this to CollectStale()

[sh0rez]

in processor.go I named it the field stale staleness.Tracker, so using it reads as p.stale.Collect().
Renaming this to p.stale.CollectStale() which I very slightly like less, because it stutters.

no strong opinion

[RichieSams]

Do we still need to chain them? What isn't yet implemented in Linear? IMO it would be much simpler (for metrics, file structure, etc) to just switch wholesale, rather than trying to keep both around.

The deltatocumulative-linear branch didn't have chain. So I was confused at first when reviewing the PR.

[sh0rez]

Linear only does sums on this branch.

Making linear do everything involves some fairly advanced generics usage, which I think deserves to be reviewed properly and probably separately :)

Given the code already exists on the non-partial branch, I expect to send the next patch right after merging this one.

We should take care to merge so that we only release after merging both.

[RichieSams]

Overall, I quite like the code. I personally would vote to do the change wholesale (which I believe is what the deltatocumulative-linear branch does). Rather than one datasource at at time, which requires the chain stuff and more confusion.

[carrieedwards]

I like this refactor!