testbed not working with transform processor in config

[swamisriman]

Component(s)

testbed

What happened?

Description

When we use a test configuration with transform processor, the test case is failing.

Steps to Reproduce

Create a test with transform processor.
Say, in tests/resource_processor_test.go/TestMetricResourceProcessor(...) we add a new transform processor.
Now it looks like this

processors := map[string]string{
				"resource":  test.resourceProcessorConfig,
				"transform": "\n  transform:\n",
			}

Run the test

Expected Result

Test should pass.
Because, the transform processor doesn't do anything (it's empty).

Actual Result

Test case failing with the following error

2024/05/22 23:48:51 Time out waiting for LoadGenerator isn't ready
    test_case.go:289: Time out waiting for LoadGenerator isn't ready
    resource_processor_test.go:153: 
                Error Trace:    /Users/<username>/Downloads/otelcol_test_temp/opentelemetry-collector-contrib/testbed/tests/resource_processor_test.go:153
                Error:          Received unexpected error:
                                rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:60360: connect: connection refused"
                Test:           TestMetricResourceProcessor/set_attribute_on_empty_resource
2024/05/22 23:48:51 Gracefully terminating Agent pid=4557, sending SIGTEM...

Collector version

v0.100.0

Environment information

Environment

OS: Mac OS Sonoma Version 14.5 (23F79)
Compiler(if manually compiled): go version go1.22.3 darwin/arm64

PS: I'm not sure what if manually compiled means.
I ran the test with the command TESTS_DIR=tests make e2e-test

OpenTelemetry Collector configuration

(taken from configStr here)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "127.0.0.1:60989"
exporters:
  otlp:
    endpoint: "127.0.0.1:60990"
    
    
    tls:
      insecure: true
    compression: "none"
processors:
  
  transform:


  resource:
    attributes:
    - key: label-key
      value: new-label-value
      action: update
    - key: resource-type
      from_attribute: opencensus.resourcetype
      action: upsert
    - key: opencensus.resourcetype
      action: delete



extensions:
  pprof:
    save_to_file: /Users/<username>/Downloads/otelcol_test_temp/opentelemetry-collector-contrib/testbed/tests/results/TestMetricResourceProcessor/update_and_rename_existing_attributes/cpu.prof
  

service:
  extensions: [pprof, ]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [transform,resource]
      exporters: [otlp]

Log output

(testbed/tests/results/TestMetricResourceProcessor/set_attribute_on_empty_resource/agent.log)

Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'processors': unknown type: "transform" for id: "transform" (valid values: [attributes resource batch memory_limiter])
2024/05/23 20:40:13 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'processors': unknown type: "transform" for id: "transform" (valid values: [attributes resource batch memory_limiter])

Additional context

We use a collector config with a series of different processors that do different operations on incoming data.
I am trying to have inputs and expected outputs and test the config if it's doing what is expected.

Can the testbed be used to test the configs with only these processors [attributes resource batch memory_limiter]?
Can't we test other processors?
If not, what is the recommended way to test a pipeline with a series of processors?

[github-actions]

Pinging code owners:

• testbed: @open-telemetry/collector-approvers

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[swamisriman]

/label help-wanted

[rogercoll]

@swamisriman I would say that this error is expected as the "transform" processor is not included within the generated collector used in the testbed tests. List of included processors: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/oteltestbedcol/builder-config.yaml#L28

[swamisriman]

@rogercoll
Is there any way we can test other processors too?
Or is it possible to include these processors in the list you shared?

Update:
When I added factory for the other processors, it seems to work: e41ef2b

Thanks @rogercoll !

[rogercoll]

Is there any way we can test other processors too?
Or is it possible to include these processors in the list you shared?

Yes, you could add them in the testbed collector factory and add a specific test file for those processors. For example, like the resource attributes one: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/tests/resource_processor_test.go

I would suggest opening an issue first for each processor you want to add, explaining the reasoning behind and what the test is going to assert.

[swamisriman]

@rogercoll
When I added factory for the other processors like this, it seems to work: e41ef2b

what the test is going to assert

I believe each end-user will have their own assertions.
But what I mean is that we should enable users to test any processor (and hence add factories of all processors to components.go)

So that, users can just test their own config using any processor and write a custom test case similar to resource_processor_test.go

[swamisriman]

Hi @rogercoll
Are you testbed maintainer?
If yes, what's your opinion on enabling this? I can raise a PR if interested.

[rogercoll]

Are you testbed maintainer?
If yes, what's your opinion on enabling this? I can raise a PR if interested.

No, I am not a maintainer, but feel free to raise a PR. I would not add all the components to the testbed binary, instead, I would just add the ones that are actually being tested in testbed.

[swamisriman]

@rogercoll
Curious, what is the intended usage of testbed?
Is that supposed to be used by end-users? or only by the component builders(receivers, processors and exporters) to make sure that they are working?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• testbed: @open-telemetry/collector-approvers

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[swamisriman]

-Stale
-help-wanted

[swamisriman]

This issue is still relevant because,
we should enable users to test any processor (and hence add factories of all processors to components.go)

So that, users can just test their own config using any processor and write a custom test case similar to resource_processor_test.go

[swamisriman]

-help wanted

[swamisriman]

I understand that we can't have all the components in testbed as per this discussion
#34849 (comment)

Closing this issue.

For future users who are facing this issue, the quickest hack to use any processor in testbed is to change the binary to otelcontribcol here:

opentelemetry-collector-contrib/testbed/testbed/test_bed.go

Line 53 in 90da89e

DefaultAgentExeRelativeFile: "../../bin/oteltestbedcol_{{.GOOS}}_{{.GOARCH}}",

Thanks everyone, for looking into this.