Bring pipelines transform more inline with general transform behavior

[blt]

In #10144 we've slowly come to understand why http -> pipelines -> blackhole does not perform well. The root culprit is the expansion mechanism in place for PipelinesConfig et al and the additional hops that adds to a running topology, compared to a non-pipeline configuration that achieves the same ends. We would like, then, to minimize the differences at runtime between an "unrolled" pipeline versus its pipelined peer.

Thoughts for how to minimize these differences:

• Remove the double condition execution inherent in PipelineFilterConfig expansion #11785
• Remove the need for Noop transform #11786
• Have a Pipeline transform that implements FunctionTransform #11787

The approach for this work will necessarily have to be a loop of removing one serious bottleneck, re-evaluating our progress and moving on to the next bottleneck. At the end it's entirely possible we will have removed the expansion mechanism in place for pipelines, possibly expanded the primitives available for all transforms to control branching etc.

[jszwedko]

With regards to double filtering, we have thought about adding a new named output to the filter transform in the past that would permit routing events that fail the condition. Maybe that could solve this problem as well as providing a generally useful addition to the filter transform?

[blt]

With regards to double filtering, we have thought about adding a new named output to the filter transform in the past that would permit routing events that fail the condition. Maybe that could solve this problem as well as providing a generally useful addition to the filter transform?

That makes good sense to me. The semantics of the Filter transform are "take an input, drop if not matched else pass". We almost want a splitter where the transform has two outputs where the semantics are "take an input, route high if matches else route low or drop" if there's no low. You could implement the existing Filter in terms of that or change how Filter works as a primitive.

[jdrouet]

That makes good sense to me. The semantics of the Filter transform are "take an input, drop if not matched else pass". We almost want a splitter where the transform has two outputs where the semantics are "take an input, route high if matches else route low or drop" if there's no low. You could implement the existing Filter in terms of that or change how Filter works as a primitive.

What do you think about doing a switch transform? The only difference with a router transform would be that it would only forward the event to the first route that matches and therefore wouldn't need to clone the event.

[blt]

That makes good sense to me. The semantics of the Filter transform are "take an input, drop if not matched else pass". We almost want a splitter where the transform has two outputs where the semantics are "take an input, route high if matches else route low or drop" if there's no low. You could implement the existing Filter in terms of that or change how Filter works as a primitive.

What do you think about doing a switch transform? The only difference with a router transform would be that it would only forward the event to the first route that matches and therefore wouldn't need to clone the event.

I would be interested to see how a switch performs for sure. Seems like a generally useful primitive as well. @lukesteensen's thoughts here would be valuable too.

[binarylogic]

From a UX standpoint I would like to have a single route transform with an option to switch or not. This is inline with the transform boundaries in the UX design doc. If it's not possible, that's fine, but they are close enough in behavior that a single transform would be better.

[lukesteensen]

We could add two options to the route config that would cover these:

1. An else output for when no condition matches
2. a first_match mode where we stop after the first matching condition instead of sending to all matches

I don't want to end up with too many knobs, but that seems less confusing than altering the filter semantics or adding a whole new similar-but-different transform.

[jszwedko]

We could add two options to the route config that would cover these:

1. An else output for when no condition matches
2. a first_match mode where we stop after the first matching condition instead of sending to all matches

I don't want to end up with too many knobs, but that seems less confusing than altering the filter semantics or adding a whole new similar-but-different transform.

While both of those would be required to support the semantics of the switch transform in #11797 I think, strictly for the pipelines use-case which only has two routes, that simply supporting else would be sufficient.