fix(otlp): Write protobuf status on error (backport k230)

[loki-gh-app]

Backport 63a2442 from #15097

---

What this PR does / why we need it:

The OTLP spec states that:

If the processing of the request fails, the server MUST respond with appropriate HTTP 4xx or HTTP 5xx status code.
The response body for all HTTP 4xx and HTTP 5xx responses MUST be a Protobuf-encoded Status message that describes the problem.
This specification does not use Status.code field and the server MAY omit Status.code field.
The clients are not expected to alter their behavior based on Status.code field but MAY record it for troubleshooting purposes.
The Status.message field SHOULD contain a developer-facing error message as defined in Status message schema.

Loki currently writes the error as a string so it's lost. This PR writes the error as a protobuf Status instead.

We also ditch the otlp error interceptor and put it's logic inside the OTELError writer.

Without change Error message is lost

2024-11-25T10:48:42.662805076Z 2024-11-25T10:48:42.652Z	error	internal/queue_sender.go:92	Exporting failed. Dropping data.	{"service": "opentelemetry-collector", "kind": "exporter", "data_type": "logs", "name": "otlphttp/loki", "error": "not retryable error: Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to http://loki:3100/otlp/v1/logs responded with HTTP Status Code 400", "dropped_items": 1456}

With change Error message is displayed (set max_line_size: 1B)

2024-11-25T10:44:00.295481650Z 2024-11-25T10:44:00.281Z	error	internal/queue_sender.go:92	Exporting failed. Dropping data.	{"service": "opentelemetry-collector", "kind": "exporter", "data_type": "logs", "name": "otlphttp/loki", "error": "not retryable error: Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to http://loki:3100/otlp/v1/logs responded with HTTP Status Code 400, Message=2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '104' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '107' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '156' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '130' bytes; 52 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '213' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '103' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '140' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '108' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '139' bytes; 109 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '217' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '150' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '119' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '117' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '152' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '173' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '206' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '215' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '123' bytes; 4 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '168' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '118' bytes; 10 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '131' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '112' bytes; 206 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '110' bytes; 4 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '219' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '220' bytes; 4 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '207' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '124' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '175' bytes; 4 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '113' bytes; 30 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '208' bytes; 24 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '109' bytes; 11 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '216' bytes; 131 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '133' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '149' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '157' bytes; 303 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '211' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '148' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '171' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '314' bytes; 4 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '111' bytes; 4 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '106' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '127' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '136' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '169' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '115' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '137' bytes; 85 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '132' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '96' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '159' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '146' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '125' bytes; 39 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '218' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '101' bytes; 5 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '114' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '100' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '214' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '178' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '129' bytes; 2 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '116' bytes; 309 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '209' bytes; 5 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '134' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '135' bytes; 1 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '183' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '120' bytes; 7 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '212' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '126' bytes; 29 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '210' bytes; 3 errors like: Max entry size '1' bytes exceeded for stream '{service_name=\"unknown_service\"}' while adding an entry with length '121' bytes, Details=[]", "dropped_items": 1456}

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR