-
Notifications
You must be signed in to change notification settings - Fork 487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flow: otelcol.processor.attributes component #2294
Comments
I got stuck on this project because I need to create instances of "filterconfig.MatchConfig" and "attraction.Settings" but they are internal: One way to do this is to fork the contrib repo and export all the internals, as we currently do with the non-contrib one. @gouthamve suggested to try exporting those and checking if the changes get accepted in the official contrib repo before we fork. I forked the contrib repo in my private GitHub area and exported those in the pkg folder. This is on tag v0.0.900. However, then the Agent didn't compile because it was using an old version of OTEL which has a function FromRaw which does not return an error. FromRaw was changed to return an error here at version 0.65.0 of OTEL. I tried updating the version of OTEL which the Agent uses, but then there were other compilation problems. Finally, I tried rolling back both my OTEL and contrib changes, and using version 0.61.0 of otel-contrib. But it turns out that this version has the filter library in an old, awkward location which would look strange if exported. It seems to me that the only realistic way to move forward is to fork the contrib repo and export all internals. @tpaschalis mentioned that the exporting script we use for OTEL is this one, so maybe we can have some variation of this for contrib. The main thing that bothers me about the forks is that OTEL could one day remove config attributes, or even change the meaning of some attributes. I guess in those cases clients would not be able to upgrade to new versions of the Agent until they change their OTEL config, and there would be only one version of OTEL that a given version of Agent supports. By the way, while speaking with @kovrus he mentioned that a lot of OTEL components (such as the attributes processor) are implemented using the OTTL language. We could theoretically also use this language, although at the moment I guess we are just writing adaptors to already existing OTEL components. But what worries me about OTTL is that people might have a complicated OTTL config which they try to use within the Agent - then there would be a mixture of OTTL and River and that may look complicated. |
I managed to build a fork of otel-contrib based on the 0.63.0 branch where the "internal" folder is renamed to "external": I also made a lot of progress with the attribute processor code - it's on the branch indicated on this issue. It's still not finished though... I will try to finish it when I get back in January. It would be good of we can decide how we want to import otel-contrib. We could use fork where we make the internals external. But a cleaner approach could be to translate River to Yaml, and then use factories in the OTEL code which convert Yaml to the OTEL data structure. Then we can just use the data structure from there. If there are no such factories already, I think we could add them - such a change would be likely to be accepted to the OTEL repo? Another upside to this approach is that it would allow us to test the translation from River to Otel more easily. We could use configs like this in our unit tests to see if our River inputs match them. I think we were also discussing that a translation between River and Yaml might be a requirement for the Agent to become a distribution, so it may be something we want to do anyway. That said, a fork might be the only realistic short term solution. |
In general, forks should only be used as temporary workarounds. As of the latest OpenTelemetry Collector release, it's possible for us to use nothing but public APIs for the pkg/traces subsystem, which will finally help us get rid of our fork. However: some of the APIs we previously used no longer have exposed counterparts (not even if we export the internal packages), so this isn't a simple task. Similarly, we need the ability to pass a custom logger and meter provider to the collector package for us to be able to use it. I see the following path here:
|
@rfratto suggested a workaround that did in fact work for this issue (thank you!). We used the otel mapstructure to decode our river types into otel types. There is some example code here showing the implementation. As a second optional consideration, I implemented the squashed otel type sharedPolicyCfg by skipping it in the river types and using good old copy and paste in the 3 river types that use it. You have this option if you don't want to have to touch user docs after river implements squashing. In short, the downside is the repeated code in types.go and the upside is not having to change the river config/docs later when river implements squashing. |
@erikbaranowski The blocker for this task is that some of the Otel types that we need to convert are internal. Hence, we cannot instantiate them in our code. The pattern from the tail sampling code does require us to instantiate the Otel type before calling The only way we could use Another solution could be if we had a way to stream out the River config into a yaml, then maybe we could somehow convert that into Otel using some Otel/mapstructure factory. However, I still don't see how we could use |
The overall Config type from the processor is exported, so we can do mapstructure.Decode into it.
It can. mapstructure.Decode is how OpenTelemetry Collector decodes YAML configs, which is why OpenTelemetry Collector can create a config for that copmonent even though it has types from internal packages. |
Thanks @rfratto. I suspected that this works for Otel because these internal types are available to them, and they call Just now I tried this in a unit test and it built ok:
In the above unit test I tried adding this line:
The test failed with this error:
When tried this instead, it wouldn't compile because it can't find the attraction package (I suppose because it's internal to Otel):
So I basically just used a |
I took a quick look at it, and yes, that approach is the idea for configuring OpenTelemetry components that are using internal packages. |
This has been merged to |
Expose the OpenTelemtry Collector
attributes
processor as a Flow component calledotelcol.processor.attributes
. This component is from theotelcol-contrib
distribution.The text was updated successfully, but these errors were encountered: