Fix #1632 - Occasional Segfault with LongCounter instrument

[lalitb]

Fixes #1632

Changes

The problem occurs when metric collection is happening while measurements are getting recorded in different threads ( PeriodicExportingMetricReader creates separate thread for collection)

Thread1:
Instrument::Record() -> SyncMetricStorage::Record() -> AttributeHashMap::GetOrSetDefault() ->mutex_lock -> map::insert()

Thread2:
SyncMetricStorage::Collect () -> std::move(AttributeHashMap) -> ...

While thread1 is recording in attribute-hashmap, thread2 can sometime clear the attribute-hashmap (by move it to different hashmap), which cause thread1 to try recording in invalid memory.

The solution is to remove locks from attribute-hashmap, and add it to SyncMetricStorage/AsyncMetricStorage whereever it is reading/updating/moving attribute-hashmap

i.e.,

Thread1:
Instrument::Record() -> SyncMetricStorage::Record () -> mutex_lock_ -> AttributeHashMap::GetOrSetDefault() -> map::insert()

Thread2:
SyncMetricStorage::Collect () ->mutex_lock -> std::move(AttributeHashMap) -> ...

For significant contributions please make sure you have completed the following items:

• CHANGELOG.md updated for non-trivial changes
• Unit tests have been added
• Changes in public API reviewed

[codecov]

Codecov Report

Merging #1638 (78ce27f) into main (7278d83) will decrease coverage by 0.02%.
The diff coverage is 86.67%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1638      +/-   ##
==========================================
- Coverage   85.10%   85.09%   -0.01%     
==========================================
  Files         159      159              
  Lines        4999     5001       +2     
==========================================
+ Hits         4254     4255       +1     
- Misses        745      746       +1     

Impacted Files	Coverage Δ
...ntelemetry/sdk/metrics/state/sync_metric_storage.h	63.16% <50.00%> (-1.54%)	⬇️
...telemetry/sdk/metrics/state/async_metric_storage.h	86.49% <100.00%> (+1.20%)	⬆️
...entelemetry/sdk/metrics/state/attributes_hashmap.h	95.84% <100.00%> (-0.94%)	⬇️
sdk/src/metrics/state/sync_metric_storage.cc	100.00% <100.00%> (ø)
sdk/src/trace/batch_span_processor.cc	91.41% <0.00%> (+0.79%)	⬆️

[marcalff]

Good findings on the root cause.

Please see questions / comments on the fix.

[marcalff]

LGTM, thanks for the fixes.