Have a Pipeline transform that implements FunctionTransform
#11787
Labels
transform: pipelines
Anything `pipelines` transform related
Comments
3 tasks
blt
added a commit
that referenced
this issue
May 11, 2022
This commit adjusts pipeline expansion so that they are combined, rather than, well, expanded. This means that the sub-transforms of a pipeline run in serial but that each pipeline as a whole can run multiple copies of itself at once. This also cleans up many low-priority tasks. Resolves #11787 Resolves #11784 REF #10144 Signed-off-by: Luke Steensen <luke.steensen@gmail.com> Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
blt
added a commit
that referenced
this issue
May 13, 2022
This commit adjusts pipeline expansion so that they are combined, rather than, well, expanded. This means that the sub-transforms of a pipeline run in serial but that each pipeline as a whole can run multiple copies of itself at once. This also cleans up many low-priority tasks. Resolves #11787 Resolves #11784 REF #10144 Signed-off-by: Luke Steensen <luke.steensen@gmail.com> Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
blt
added a commit
that referenced
this issue
May 16, 2022
* Combine pipeline stages This commit adjusts pipeline expansion so that they are combined, rather than, well, expanded. This means that the sub-transforms of a pipeline run in serial but that each pipeline as a whole can run multiple copies of itself at once. This also cleans up many low-priority tasks. Resolves #11787 Resolves #11784 REF #10144 Signed-off-by: Luke Steensen <luke.steensen@gmail.com> Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * Tidy up errors, fix bug in vector.toml pipeline config Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * test dings Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * temporariliy disable http_datadog_filter_blackhole Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * PR feedback Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * PR feedback with regard to outputs Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * PR feedback Signed-off-by: Brian L. Troutwine <brian@troutwine.us> * try 64 wide interior buffer Signed-off-by: Brian L. Troutwine <brian@troutwine.us> Co-authored-by: Brian L. Troutwine <brian@troutwine.us>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Today
PipelinesConfigdoes not map to aTransforminstance. It is, instead, expanded into many transforms. This imposes immediate costs in terms of metrics / logs routing but more fundamentally means that the pipeline runs as a singleton. If we were able to contrive aPipelinethat implementedFunctionTransformour topology optimizations that allow for multipleFunctionTransforminstances to run in parallel would kick in and improve CPU saturation.The text was updated successfully, but these errors were encountered: