-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Include tracing information in the Kibana logs #112973
Merged
mshustov
merged 40 commits into
elastic:main
from
mshustov:issue-102704-change-apm-default
Nov 11, 2021
Merged
Include tracing information in the Kibana logs #112973
mshustov
merged 40 commits into
elastic:main
from
mshustov:issue-102704-change-apm-default
Nov 11, 2021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server.
mshustov
added
chore
Team:Core
Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
v8.0.0
release_note:skip
Skip the PR/issue when compiling release notes
auto-backport
Deprecated - use backport:version if exact versions are needed
v7.16.0
labels
Sep 23, 2021
What row(s) of the posted tables show this? |
This comment has been minimized.
This comment has been minimized.
mshustov
requested review from
pgayvallet
and removed request for
pgayvallet
September 30, 2021 10:02
mshustov
changed the title
change APM nodejs agent default
Include tracing information in the Kibana logs
Sep 30, 2021
mshustov
force-pushed
the
issue-102704-change-apm-default
branch
from
October 11, 2021 14:19
581428e
to
861efdb
Compare
This comment has been minimized.
This comment has been minimized.
@elasticmachine merge upstream |
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
mshustov
added
backport:skip
This commit does not require backporting
and removed
auto-backport
Deprecated - use backport:version if exact versions are needed
labels
Nov 11, 2021
mshustov
added
v8.0.0
and removed
backport:skip
This commit does not require backporting
labels
Nov 15, 2021
mshustov
pushed a commit
to mshustov/kibana
that referenced
this pull request
Nov 15, 2021
* change APM nodejs agent default By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server. * emit trace IDs into the logs * use ELASTIC_APM_DISABLE_SEND to keep APM agent active but disable send when necessary * send data whenver active is set to "true" * update tests * keep APM agent active. control disableSend instead * update snapshot tests * add debug logging * REMOVE me. log path to the agent * init APM agent explicitly in test plugin. it uses another package instance * REMOVE me. create transaction explicitly * increase timeout setting for the test * refactor tests * remove debug logs * remove explicit transaction creation * Revert "remove explicit transaction creation" This reverts commit cdf2d30. * point to apm nodejs agent commit temporary until a new version is released * migrate from disableSend to contextPropagationOnly * TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined * Revert "TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined" This reverts commit 62dda4f. * bump to version with fix * commit using @elastic.co Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
mshustov
pushed a commit
that referenced
this pull request
Nov 15, 2021
* change APM nodejs agent default By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server. * emit trace IDs into the logs * use ELASTIC_APM_DISABLE_SEND to keep APM agent active but disable send when necessary * send data whenver active is set to "true" * update tests * keep APM agent active. control disableSend instead * update snapshot tests * add debug logging * REMOVE me. log path to the agent * init APM agent explicitly in test plugin. it uses another package instance * REMOVE me. create transaction explicitly * increase timeout setting for the test * refactor tests * remove debug logs * remove explicit transaction creation * Revert "remove explicit transaction creation" This reverts commit cdf2d30. * point to apm nodejs agent commit temporary until a new version is released * migrate from disableSend to contextPropagationOnly * TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined * Revert "TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined" This reverts commit 62dda4f. * bump to version with fix * commit using @elastic.co Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
mshustov
pushed a commit
to mshustov/kibana
that referenced
this pull request
Nov 24, 2021
… (elastic#118604)" This reverts commit 114c690.
mshustov
added
backport:skip
This commit does not require backporting
and removed
v8.0.0
labels
Nov 24, 2021
dmlemeshko
pushed a commit
that referenced
this pull request
Nov 29, 2021
* change APM nodejs agent default By default APM agent instruments the code to be a base for log correlation. But it doesn't send transactions to the APM server. * emit trace IDs into the logs * use ELASTIC_APM_DISABLE_SEND to keep APM agent active but disable send when necessary * send data whenver active is set to "true" * update tests * keep APM agent active. control disableSend instead * update snapshot tests * add debug logging * REMOVE me. log path to the agent * init APM agent explicitly in test plugin. it uses another package instance * REMOVE me. create transaction explicitly * increase timeout setting for the test * refactor tests * remove debug logs * remove explicit transaction creation * Revert "remove explicit transaction creation" This reverts commit cdf2d30. * point to apm nodejs agent commit temporary until a new version is released * migrate from disableSend to contextPropagationOnly * TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined * Revert "TO DISCUSS. what if we enforce contextPropagationOnly to be configured when active is defined" This reverts commit 62dda4f. * bump to version with fix * commit using @elastic.co Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
backport:skip
This commit does not require backporting
chore
release_note:skip
Skip the PR/issue when compiling release notes
Team:Core
Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
v8.1.0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
closes #102704
closes #102699
I combined both steps in a single task to simplify testing. The work consists of two parts:
Changing APM defaults
APM nodejs agent needs to instrument Kibana code to track transactions and spans.
By default, the APM agent sends collected metrics to an APM server for ingestion. Not all of ours users want to store APM metrics in the ES index, so the APM agent introduced
contextPropagationOnly
configuration flag. From the docs:contextPropagationOnly:true
disables metrics and calculation of breakdown metrics that should minimize the performance overhead.Thus, Kibana APM config has been changed to instrument code but it does not send data to the APM server. There is a table with different scenarios and APM config configuration for each of them:
active: true
active: false, contextPropagationOnly: false
Emit trace ids into the Kibana logs
Kibana reads
span.id
,transaction.id
andtrace.id
from the APM agent and emits them into Kibana logs.Whenever a user enables logs in ECS JSON format, trancing information is emitted as ESC-compatible
Tracing Fields
https://www.elastic.co/guide/en/ecs/current/ecs-tracing.htmlLogs with
pattern: layout
do not contain this information, we can support it later if necessary.How to test
bsearch
implementation?)Running task...
)Risk Matrix
When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:
nodejs@v16
upgrade or disable instrumentation by default.Instrumenting runtime code with APM agent with
contextPropagationOnly
adds a tiny performance overhead. InDemoJourney
test scenario, 95th perc. of the response time deviation has the value of statistical error.master version:
branch version:
Checklist
Delete any items that are not applicable to this PR.