-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for logs (and traces from access logs -aka RTR logs- ) on cloudfoundryreceiver #32671
Comments
Pinging code owners:
%s See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Ops, sorry, for some reason I did not find the old issue :-/. |
Nice!, we will implemented and create a PR. We are also interested in these features:
Regarding the point 1, probably is not needed a configuration parameter per-se. Each telemetry type can be implemented which their own class from the Factory creator which will be automatically instantiated depending on the pipeline type (logs, metrics, traces). We will need to figure out the implementation details of ingesting envelopes in this way, if is not needed, no new configuration parameter will be added. Regarding the point 2, the reason to filter at this stage instead of using a processor later in the pipeline is only because of trying to reduce resource usage. We need to deal with a lot of logs and for us makes sense to discard the logs we do not need as soon as possible, instead of letting them flow on the pipeline consuming resources. What do you guys think? |
Point 1 shouldn't be necessary, as you pointed out the same receiver configuration would just be placed in each telemetry pipeline that a user wants to ingest. Point 2 makes sense, it's good functionality for the receiver to have if necessary. 👍 |
We have created a draft PR: #33044, is still a WIP because the 2nd and 3rd point were not implemented. |
**Description:** Adding support receive logs from Cloudfoundry. **Link to tracking Issue:** #32671 **Testing:** Basic testing inline with the current tests **Documentation:** * Add new section for logs and their attributes. * Update behaviour of ShardID property (`rlp_gateway.shard_id`). cc @CemDK @m1rp --------- Co-authored-by: Cem Deniz Kabakci <CemDenizKabakci@outlook.com> Co-authored-by: Sam Clulow <sam.clulow@springernature.com> Co-authored-by: Cem Deniz Kabakci <cem.kabakci@springer.com> Co-authored-by: Tomás Mota <tomasrebelomota@gmail.com> Co-authored-by: Tomas Mota <tomas.mota@springernature.com> Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Resolved by #33044 |
Component(s)
receiver/cloudfoundry
Is your feature request related to a problem? Please describe.
Currently the Cloudfoundry receiver supports only metrics, but the Loggregator-firehose subsystem (
rlp_gateway
endpoint) also process all system and app logs. We would like to be able to read those logs with the OTEL collector (adding a tag by type, egRTR
,CELL
,SSH
...). Additionally, if the RTR logs have the tracing fields, add support to export spans.Describe the solution you'd like
Extend the converter in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/cloudfoundryreceiver/converter.go#L22 to generate log and traces from logs of type RTR (access logs) in the same way as is done for metrics. Because a log line can have different sources depending on where it was generated (app,container, gorouter, api ...) it makes sense to add flags to select the interested ones. There is also to get events like app crashes, scaling, etc ... More information: https://pkg.go.dev/code.cloudfoundry.org/go-loggregator/rpc/loggregator_v2#pkg-types
Describe alternatives you've considered
This is the best option. This component is already in charge of processing metrics and it makes sense to extend it to receive logs and traces from the same source (loggregator-firehose) . The traditional old way is receiving those logs from a component like this one and emit them using syslog (https://github.com/cloudfoundry-community/firehose-to-syslog) but it adds an unneeded jump and it does not allow to process traces.
Additional context
Loggregator-firehose subsystem is in charge of managing metrics, logs and events in Cloudfoundry.
CloudFoundry http routing layer, also known as gorouter component supports tracing, but it does not emit traces by itself. It decorates the access logs (RTR logs) with tracing information:
my.host.name - [2024-04-24T14:09:44.621845452Z] "GET /health HTTP/1.1" 200 0 15 "-" "NING/1.0" "X.X.X.X:46312" "X.X.X.X:61187" x_forwarded_for:"X.X.X.X, X.X.X.X, X.X.X.X" x_forwarded_proto:"https" vcap_request_id:"07212f87-6146-4207-6d63-84fd1e8cdbc4" response_time:0.003175 gorouter_time:0.000298 app_id:"0e761239-7c71-48e8-bd41-8314d67e74dc" app_index:"1" instance_id:"eb698264-6bc3-4168-58b8-80f6" x_cf_routererror:"-" x_forwarded_host:"my.host.name" x_b3_traceid:"07212f87614642076d6384fd1e8cdbc4" x_b3_spanid:"6d6384fd1e8cdbc4" x_b3_parentspanid:"-" b3:"07212f87614642076d6384fd1e8cdbc4-6d6384fd1e8cdbc4" traceparent:"00-07212f87614642076d6384fd1e8cdbc4-6d6384fd1e8cdbc4-01" tracestate:"gorouter=6d6384fd1e8cdbc4"
The text was updated successfully, but these errors were encountered: