Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SetMeterProvider might miss the delegation for instruments and registries #5780

Merged
merged 7 commits into from
Sep 6, 2024

Conversation

XSAM
Copy link
Member

@XSAM XSAM commented Sep 4, 2024

Closes #5757

This PR fixes an issue where SetMeterProvider might miss the delegation for instruments and registries. This bug brings a concurrent issue that could possibly make instruments and registries unable to operate correctly, such as recording, after using the SetMeterProvider method. The data put on these instruments and registries might be lost.

Copy link

codecov bot commented Sep 4, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.5%. Comparing base (9e1b015) to head (3a6000c).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@          Coverage Diff          @@
##            main   #5780   +/-   ##
=====================================
  Coverage   84.5%   84.5%           
=====================================
  Files        272     272           
  Lines      22784   22798   +14     
=====================================
+ Hits       19268   19282   +14     
  Misses      3173    3173           
  Partials     343     343           

see 1 file with indirect coverage changes

@XSAM XSAM changed the title SetMeterProvider might miss the delegation for instruments and registry. SetMeterProvider might miss the delegation for instruments and registries Sep 4, 2024
@XSAM XSAM marked this pull request as ready for review September 4, 2024 23:33
@XSAM XSAM merged commit b37e8a9 into open-telemetry:main Sep 6, 2024
32 checks passed
@XSAM XSAM deleted the fix-meter-delegation branch September 6, 2024 23:42
Copy link
Contributor

@MrAlias MrAlias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change means that every measurement through the global API now requires at a minimum ~100ns to acquire the lock. This is on the hot-path and a performance evaluation should have been provided prior to this change being accepted.

The original design was aware of the case where a recording was performed concurrently with SetMeterProvider and it was dropped. That was intentional. There is a trade-off of performance and it was originally decided that speed on the hot-path is more important than synchronicity.

Are there panics or race conditions this addresses? If not these changes should be reverted until it can be shown that the performance impact is not significant.

@XSAM
Copy link
Member Author

XSAM commented Sep 7, 2024

The original design was aware of the case where a recording was performed concurrently with SetMeterProvider and it was dropped. That was intentional. There is a trade-off of performance and it was originally decided that speed on the hot-path is more important than synchronicity.

The drop would happen when initiating the instrument, not recording. The recording to instruments that didn't setDelegate would be ignored even after SetMeterProvider.

Imagine users initiating instruments while setting global meter provider. Some instruments were not able to record anything during the entire runtime of the application. This looks like race conditions to me.

@dashpole
Copy link
Contributor

dashpole commented Sep 9, 2024

@MrAlias it isn't every measurement, right? It is every instrument instantiation? Or am I misreading

@MrAlias
Copy link
Contributor

MrAlias commented Sep 9, 2024

It looks like I misread this change. Looks good. Please disregard.

@XSAM XSAM added this to the v1.30.0 milestone Sep 9, 2024
XSAM added a commit that referenced this pull request Sep 10, 2024
### Added

- Support `OTEL_EXPORTER_OTLP_LOGS_INSECURE` and
`OTEL_EXPORTER_OTLP_INSECURE` environments in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. (#5739)
- The `WithResource` option for `NewMeterProvider` now merges the
provided resources with the ones from environment variables. (#5773)
- The `WithResource` option for `NewLoggerProvider` now merges the
provided resources with the ones from environment variables. (#5773)
- Add UTF-8 support to `go.opentelemetry.io/otel/exporters/prometheus`.
(#5755)
### Fixed

- Fix memory leak in the global `MeterProvider` when identical
instruments are repeatedly created. (#5754)
- Fix panic instruments creation when setting meter provider. (#5758)
- Fix panic on instruments creation when setting meter provider. (#5758)
- Fix an issue where `SetMeterProvider` in `go.opentelemetry.io/otel`
might miss the delegation for instruments and registries. (#5780)

### Removed

- Drop support for [Go 1.21](https://go.dev/doc/go1.21). (#5736, #5740,
#5800)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

instruments creation may cause panic when setting meter provider
4 participants