Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource Detection processor #309

Merged

Conversation

james-bebbington
Copy link
Member

Link to tracking Issue:
open-telemetry/opentelemetry-collector#871

Description:
Adds a new processor that automatically detects the resource based on the configured set of detectors. Initial support has been added for detecting resources from an environment variable & from GCE metadata. The user can configure to override or preserve existing resource information.

Sorry this PR is a little large. I separated it into 2 commits:

  1. Add resource detection processor & "Detector" interface
  2. Add "env" & "gce" detectors

@james-bebbington james-bebbington requested a review from a team June 11, 2020 07:54
@codecov
Copy link

codecov bot commented Jun 11, 2020

Codecov Report

Merging #309 into master will increase coverage by 0.36%.
The diff coverage is 96.55%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #309      +/-   ##
==========================================
+ Coverage   83.42%   83.78%   +0.36%     
==========================================
  Files         171      178       +7     
  Lines        9243     9504     +261     
==========================================
+ Hits         7711     7963     +252     
- Misses       1202     1209       +7     
- Partials      330      332       +2     
Flag Coverage Δ
#integration 63.31% <0.00%> (-0.10%) ⬇️
#unit 83.57% <96.55%> (+0.36%) ⬆️
Impacted Files Coverage Δ
cmd/otelcontribcol/components.go 0.00% <0.00%> (ø)
processor/resourcedetectionprocessor/factory.go 90.90% <90.90%> (ø)
...resourcedetectionprocessor/internal/gcp/gce/gce.go 95.91% <95.91%> (ø)
...cedetectionprocessor/internal/resourcedetection.go 97.05% <97.05%> (ø)
...sor/resourcedetectionprocessor/internal/env/env.go 100.00% <100.00%> (ø)
...rcedetectionprocessor/internal/gcp/gce/metadata.go 100.00% <100.00%> (ø)
...r/resourcedetectionprocessor/internal/testutils.go 100.00% <100.00%> (ø)
...edetectionprocessor/resourcedetection_processor.go 100.00% <100.00%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3ee4ed...500a020. Read the comment docs.

@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 3 times, most recently from b6e8688 to 75951e4 Compare June 11, 2020 08:28
@james-bebbington james-bebbington changed the title Resourcedetection processor Resource Detection processor Jun 11, 2020
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch from bbbfc25 to 2bc57aa Compare June 11, 2020 09:00
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch from 2bc57aa to 001ea16 Compare June 11, 2020 12:52
@jrcamp jrcamp self-assigned this Jun 11, 2020
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 2 times, most recently from 9cca643 to 45e5208 Compare June 12, 2020 05:41
Copy link
Contributor

@jrcamp jrcamp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good.

processor/resourcedetectionprocessor/README.md Outdated Show resolved Hide resolved
processor/resourcedetectionprocessor/internal/env/env.go Outdated Show resolved Hide resolved
processor/resourcedetectionprocessor/internal/env/env.go Outdated Show resolved Hide resolved
processor/resourcedetectionprocessor/resource_processor.go Outdated Show resolved Hide resolved

value := match[2]
var err error
if value, err = url.QueryUnescape(value); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't quite follow from the spec, is the purpose of the query encoding so that = and , are being used as special separator values and the encoding allows those values being used in the actual resource key/values not to be confused with the separators?

Copy link
Member Author

@james-bebbington james-bebbington Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going off this PR / comment.

I believe the intention was to simplify the definition of how to encode resources in an environment variable by borrowing (i.e. referencing) the spec that is already defined for correlation context, since these are basically the same format. The reason why we need this isn't explicitly called out in that PR, but I assume it's for the reason you mentioned.

Even though that PR probably won't get merged, there seemed to be support for using this definition for resources supplied in environment variables. I included a similar statement in the resource detection OTEP.

@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 4 times, most recently from 7ad1de5 to 2df6894 Compare June 16, 2020 04:25
@flands flands added this to the 0.5.0 milestone Jun 17, 2020
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 3 times, most recently from 83be09c to 69d94cf Compare June 18, 2020 07:52
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 3 times, most recently from 37e6ba8 to e116534 Compare June 25, 2020 03:56
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 4 times, most recently from 0ded499 to bfadd02 Compare June 25, 2020 08:02
@james-bebbington james-bebbington requested a review from jrcamp June 25, 2020 08:23
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch 5 times, most recently from c784daf to 2e512ee Compare June 29, 2020 02:27
@james-bebbington james-bebbington force-pushed the resourcedetection-processor branch from 2e512ee to 500a020 Compare June 29, 2020 02:39
})

return mp
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bogdandrutu - I had to write this function to be able to print Resource Attributes to the log. It feels like it would be useful to have a helper function to make this kind of debugging easier; maybe an implemention ofString()?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to do this in a separate PR or here?

@tigrannajaryan tigrannajaryan assigned bogdandrutu and unassigned jrcamp Jun 29, 2020
@bogdandrutu bogdandrutu merged commit dfcd49a into open-telemetry:master Jun 30, 2020
wyTrivail referenced this pull request in mxiamxia/opentelemetry-collector-contrib Jul 13, 2020
* Add resourcedetection processor

* Add env and gce resource detectors

* Resource Detection Processor: increase test coverage & fix README

* Cache detected resource against processor name to avoid running resource detection code more than once

* Refactor resources detection processor to use resource provider
codeboten pushed a commit that referenced this pull request Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants