Fix IsPermanent to account for wrapped errors

[joe-elliott]

Description:
Currently IsPermanent does not correctly test for wrapped errors. We noticed this when the collector was incorrectly retrying otlp grpc FailedPrecondition status:

level=error msg="Exporting failed. No more retries left. Dropping data." component=tempo component_kind=exporter component_type=otlp component_name=otlp error="max elapsed time expired failed to push trace data via OTLP exporter: Permanent error: fatal error sending to server" dropped_items=100

This occurs b/c the error is wrapped and the IsPermanent function test does not correctly detect wrapped errors.

Error wrapping:

https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/otlp.go#L79

Go Playground with examples:

https://play.golang.org/p/kVBm3SesDBu

Testing:
Added a simple test to detect a wrapped error.

[codecov]

Codecov Report

Merging #2455 (4cb6d71) into main (5cc0707) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #2455      +/-   ##
==========================================
- Coverage   91.76%   91.76%   -0.01%     
==========================================
  Files         266      266              
  Lines       15112    15111       -1     
==========================================
- Hits        13867    13866       -1     
  Misses        867      867              
  Partials      378      378              

Impacted Files	Coverage Δ
consumer/consumererror/permanenterror.go	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5cc0707...4cb6d71. Read the comment docs.

[mwear]

I ran into this and was preparing a fix as well. The OTLP exporter potentially rewraps permanent errors, which end up getting retried, for example:

opentelemetry-collector/exporter/otlpexporter/otlp.go

Line 91 in 9de6dd7

return md.MetricCount(), fmt.Errorf("failed to push metrics data via OTLP exporter: %w", err)

Can confirm that this fixes that issue.

[codeboten]

Thanks for fixing this!

[bogdandrutu]

@joe-elliott please fix the tests, I tried to re-run them multiple times and same error. Maybe a rebase would help.

[joe-elliott]

Wow, yeah. I was way further behind than I realized. Rebased and pushed.