Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include tracing information in the Kibana logs #112973

Merged
merged 40 commits into from
Nov 11, 2021

Conversation

mshustov
Copy link
Contributor

@mshustov mshustov commented Sep 23, 2021

Summary

closes #102704
closes #102699
I combined both steps in a single task to simplify testing. The work consists of two parts:

Changing APM defaults

APM nodejs agent needs to instrument Kibana code to track transactions and spans.
By default, the APM agent sends collected metrics to an APM server for ingestion. Not all of ours users want to store APM metrics in the ES index, so the APM agent introduced contextPropagationOnly configuration flag. From the docs:

context_propagation_only is a boolean configuration option to have an APM agent perform
trace-context propagation and log correlation only; and to explicitly not send event data to APM server.
This allows an application to get automatic context propagation and log correlation,
without having deployed an APM server for event collection.

contextPropagationOnly:true disables metrics and calculation of breakdown metrics that should minimize the performance overhead.

Thus, Kibana APM config has been changed to instrument code but it does not send data to the APM server. There is a table with different scenarios and APM config configuration for each of them:

mode kibana.yml finalized agent config
APM agent sends data to the APM server, Kibana relies on the agent to implement log correlation. active: true { active: true, contextPropagationOnly: false }
APM agent doesn't send data to the APM server, but Kibana uses it to implement log correlation. not declared { active: true, contextPropagationOnly: true }
APM agent is disabled completely. active: false, contextPropagationOnly: false { active: false, contextPropagationOnly: false }

Emit trace ids into the Kibana logs

Kibana reads span.id, transaction.id and trace.id from the APM agent and emits them into Kibana logs.
Whenever a user enables logs in ECS JSON format, trancing information is emitted as ESC-compatible Tracing Fields https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html
Logs with pattern: layout do not contain this information, we can support it later if necessary.

How to test

  • Enable ECS JSON logging:
logging:
  appenders:
    consolejson:
      type: console
      layout:
        type: json
  root:
   appenders: [consolejson, default]
   level: all
  • run Kibana
  • Make sure trace data are included in the logs. Note that not all the Log records have got them. Currently not instrumented:

Risk Matrix

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk Probability Severity Mitigation/Notes
APM agent adds performance overhead High Medium Wait for nodejs@v16 upgrade or disable instrumentation by default.

Instrumenting runtime code with APM agent with contextPropagationOnly adds a tiny performance overhead. In DemoJourney test scenario, 95th perc. of the response time deviation has the value of statistical error.

master version:
2021-11-10_12-49-24

branch version:
2021-11-10_12-49-04

Checklist

Delete any items that are not applicable to this PR.

By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server.
@mshustov mshustov added chore Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed v7.16.0 labels Sep 23, 2021
@trentm
Copy link
Member

trentm commented Sep 23, 2021

In DemoJourney test scenario, 95th perc. of response time increased from 15% to 100%.

What row(s) of the posted tables show this?

@mshustov

This comment has been minimized.

@mshustov mshustov requested review from pgayvallet and removed request for pgayvallet September 30, 2021 10:02
@mshustov mshustov changed the title change APM nodejs agent default Include tracing information in the Kibana logs Sep 30, 2021
@mistic

This comment has been minimized.

@mistic
Copy link
Member

mistic commented Nov 10, 2021

@elasticmachine merge upstream

@mshustov
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@mshustov mshustov added backport:skip This commit does not require backporting and removed auto-backport Deprecated - use backport:version if exact versions are needed labels Nov 11, 2021
@mshustov mshustov merged commit 7c16e2e into elastic:main Nov 11, 2021
@mshustov mshustov deleted the issue-102704-change-apm-default branch November 11, 2021 11:13
@mshustov mshustov added v8.0.0 and removed backport:skip This commit does not require backporting labels Nov 15, 2021
mshustov pushed a commit to mshustov/kibana that referenced this pull request Nov 15, 2021
* change APM nodejs agent default

By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server.

* emit trace IDs into the logs

* use ELASTIC_APM_DISABLE_SEND to keep APM agent active but disable send when necessary

* send data whenver active is set to "true"

* update tests

* keep APM agent active. control disableSend instead

* update snapshot tests

* add debug logging

* REMOVE me. log path to the agent

* init APM agent explicitly in test plugin. it uses another package instance

* REMOVE me. create transaction explicitly

* increase timeout setting for the test

* refactor tests

* remove debug logs

* remove explicit transaction creation

* Revert "remove explicit transaction creation"

This reverts commit cdf2d30.

* point to apm nodejs agent commit temporary until a new version is released

* migrate from disableSend to contextPropagationOnly

* TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined

* Revert "TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined"

This reverts commit 62dda4f.

* bump to version with fix

* commit using @elastic.co

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
mshustov pushed a commit that referenced this pull request Nov 15, 2021
* change APM nodejs agent default

By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server.

* emit trace IDs into the logs

* use ELASTIC_APM_DISABLE_SEND to keep APM agent active but disable send when necessary

* send data whenver active is set to "true"

* update tests

* keep APM agent active. control disableSend instead

* update snapshot tests

* add debug logging

* REMOVE me. log path to the agent

* init APM agent explicitly in test plugin. it uses another package instance

* REMOVE me. create transaction explicitly

* increase timeout setting for the test

* refactor tests

* remove debug logs

* remove explicit transaction creation

* Revert "remove explicit transaction creation"

This reverts commit cdf2d30.

* point to apm nodejs agent commit temporary until a new version is released

* migrate from disableSend to contextPropagationOnly

* TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined

* Revert "TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined"

This reverts commit 62dda4f.

* bump to version with fix

* commit using @elastic.co

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
mshustov pushed a commit to mshustov/kibana that referenced this pull request Nov 24, 2021
mshustov pushed a commit that referenced this pull request Nov 24, 2021
…9567)

* Revert "Include tracing information in the Kibana logs (#112973) (#118604)"

This reverts commit 114c690.

* Revert "Don't enable RUM agent if APM is run with contextPropagationOnly (#118685) (#118995)"

This reverts commit ca86b98.
@mshustov mshustov added backport:skip This commit does not require backporting and removed v8.0.0 labels Nov 24, 2021
dmlemeshko pushed a commit that referenced this pull request Nov 29, 2021
* change APM nodejs agent default

By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server.

* emit trace IDs into the logs

* use ELASTIC_APM_DISABLE_SEND to keep APM agent active but disable send when necessary

* send data whenver active is set to "true"

* update tests

* keep APM agent active. control disableSend instead

* update snapshot tests

* add debug logging

* REMOVE me. log path to the agent

* init APM agent explicitly in test plugin. it uses another package instance

* REMOVE me. create transaction explicitly

* increase timeout setting for the test

* refactor tests

* remove debug logs

* remove explicit transaction creation

* Revert "remove explicit transaction creation"

This reverts commit cdf2d30.

* point to apm nodejs agent commit temporary until a new version is released

* migrate from disableSend to contextPropagationOnly

* TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined

* Revert "TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined"

This reverts commit 62dda4f.

* bump to version with fix

* commit using @elastic.co

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting chore release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support configuring APM modes Include tracing information in the log records
9 participants