[APM] Use observer.hostname instead of observer.name #76074

dgieselaar · 2020-08-27T11:43:09Z

observer.name is not guaranteed to be set. Using observer.hostname should allow us to
approximate the amount of APM Server instances.

Additionally, some of the logic was rewritten to A) reflect updates to APM Server's
implementation, and B) make it more robust by querying a static date range.

This means that the telemetry we are currently collecting for aggregated transactions is broken.

I've also added some telemetry about service.environment which can be useful in determining how we should deal with the environment selector.

`observer.name` is not guaranteed to be set. Using `observer.hostname` _should_ allow us to approximate the amount of APM Server instances. Additionally, some of the logic was rewritten to A) reflect updates to APM Server's implementation, and B) make it more robust by querying a static date range.

…ice environments telemetry

elasticmachine · 2020-08-27T11:43:11Z

Pinging @elastic/apm-ui (Team:apm)

smith · 2020-08-27T20:56:59Z

x-pack/plugins/apm/common/__snapshots__/apm_telemetry.test.ts.snap

@@ -536,61 +536,52 @@ exports[`APM telemetry helpers getApmTelemetry generates a JSON object with the
                            },
                            "transaction_count": {
                              "type": "long"
-                            }


@elastic/telemetry changes to mapping in this diff.

smith · 2020-08-27T20:57:36Z

Added @elastic/telemetry as a reviewer so they're aware of mapping changes.

smith

LGTM, once tests are passing and if upload script is working.

dgieselaar · 2020-08-28T09:50:53Z

@elasticmachine merge upstream

dgieselaar · 2020-08-31T08:42:00Z

@elasticmachine merge upstream

afharo

@dgieselaar do you mind issuing a PR with the mapping changes in the telemetry repo? I've created an issue in that repo and mentioned you in it :)

afharo · 2020-09-01T11:15:47Z

x-pack/plugins/apm/common/apm_telemetry.ts

+        services_without_environment: long,
+        services_with_multiple_environments: long,
+        top_enviroments: keyword,


should these 3 keys be nested in a properties object?

yes! thanks. good catch

…name

dgieselaar · 2020-09-04T09:42:38Z

@afharo do you want to have a final look at this? or OK to merge?

afharo

LGTM! Just added 1 NIT because it sounds strange the test is returning undefined now. But if that's intended, all good on my end 👍

afharo · 2020-09-04T15:01:31Z

x-pack/plugins/apm/common/__snapshots__/elasticsearch_fieldnames.test.ts.snap

@@ -220,9 +220,9 @@ exports[`Span METRIC_SYSTEM_FREE_MEMORY 1`] = `undefined`;

 exports[`Span METRIC_SYSTEM_TOTAL_MEMORY 1`] = `undefined`;

-exports[`Span OBSERVER_LISTENING 1`] = `undefined`;
+exports[`Span OBSERVER_HOSTNAME 1`] = `undefined`;


NIT: Is it OK for this to be now undefined?
Mind that LISTENING and ~~NAME~~ HOSTNAME have been swapped, but ~~NAME~~ HOSTNAME is now returning undefined in the tests

Yeah, that's ok. hostname is a different field, and it's not in our test data.

…name

kibanamachine · 2020-09-07T09:03:29Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: f1898c8

Build metrics

async chunks size

id	value	diff	baseline
apm	4.9MB	+8.0B	4.9MB

History

💚 Build #72544 succeeded 2938c5a
💚 Build #71868 succeeded 1c5e9e3
💔 Build #70814 failed e8d5483

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: [APM] Use observer.hostname instead of observer.name (elastic#76074) Legacy logging: fix remoteAddress being duplicated in userAgent field (elastic#76751) Remove legacy deprecation adapter (elastic#76753) [Security Solution][Detections] Rule forms cleanup (elastic#76138) Adds back in custom images for reporting + tests (elastic#76810) [APM] Fix overlapping transaction names (elastic#76083) [Ingest Pipelines] Add descriptions for ingest processors A-D (elastic#75975)

# Conflicts: # x-pack/plugins/apm/server/lib/apm_telemetry/index.ts # x-pack/plugins/apm/typings/elasticsearch/aggregations.ts

…' of github.com:jloleysens/kibana into ilm/fix/pre-existing-policy-with-no-existing-repository * 'ilm/fix/pre-existing-policy-with-no-existing-repository' of github.com:jloleysens/kibana: fix empty string in selected indices (elastic#76855) [Security Solution] Refactor OverviewHost and OverviewNetwork to use Search Strategy (elastic#76409) Use Search API in Timelion (sync) (elastic#75115) [telemetry] expose getIsOptedIn function in plugin start contract (elastic#75143) [ILM] Clean up remaining js files and any typings (elastic#76803) [Logs UI] Shared `<LogStream />` component (elastic#76262) [Security Solution] Add unit test for all hosts (elastic#76752) [Security Solution] Add unit test for authentications search strategy (elastic#76665) Do not apply search source data for tsvb (elastic#75137) [Security Solution] Refactor NetworkDns to use Search Strategy (elastic#76250) [SECURITY SOLUTION] Adds 'cypress:open-as-ci' command (elastic#76125) [Logs UI] Update alert executor tests (elastic#75764) [Functional] Unskip vega tests and fix flakiness (elastic#76600) [Data] Query String Input accepts classname prop (elastic#76848) [ML] Swim lane pagination for viewing by job id (elastic#76847) [Security Solution] Refactor MatrixHistogram to use Search Strategy (elastic#76603) [APM] Use the outcome field to calculate the transaction error rate chart (elastic#75528) [APM] Use observer.hostname instead of observer.name (elastic#76074) Legacy logging: fix remoteAddress being duplicated in userAgent field (elastic#76751)

…s-for-710 * 'master' of github.com:elastic/kibana: [Search] Use async es client endpoints (elastic#76872) fix empty string in selected indices (elastic#76855) [Security Solution] Refactor OverviewHost and OverviewNetwork to use Search Strategy (elastic#76409) Use Search API in Timelion (sync) (elastic#75115) [telemetry] expose getIsOptedIn function in plugin start contract (elastic#75143) [ILM] Clean up remaining js files and any typings (elastic#76803) [Logs UI] Shared `<LogStream />` component (elastic#76262) [Security Solution] Add unit test for all hosts (elastic#76752) [Security Solution] Add unit test for authentications search strategy (elastic#76665) Do not apply search source data for tsvb (elastic#75137) [Security Solution] Refactor NetworkDns to use Search Strategy (elastic#76250) [SECURITY SOLUTION] Adds 'cypress:open-as-ci' command (elastic#76125) [Logs UI] Update alert executor tests (elastic#75764) [Functional] Unskip vega tests and fix flakiness (elastic#76600) [Data] Query String Input accepts classname prop (elastic#76848) [ML] Swim lane pagination for viewing by job id (elastic#76847) [Security Solution] Refactor MatrixHistogram to use Search Strategy (elastic#76603) [APM] Use the outcome field to calculate the transaction error rate chart (elastic#75528) [APM] Use observer.hostname instead of observer.name (elastic#76074) Legacy logging: fix remoteAddress being duplicated in userAgent field (elastic#76751) # Conflicts: # x-pack/plugins/index_lifecycle_management/__jest__/components/edit_policy.test.tsx # x-pack/plugins/index_lifecycle_management/common/types/index.ts # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/node_allocation.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/edit_policy.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/phases/warm_phase.tsx # x-pack/plugins/index_lifecycle_management/public/application/services/api.ts

kibanamachine · 2020-09-09T09:18:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Joel Griffith <joel.griffith@elastic.co> Co-authored-by: John Schulz <john.schulz@elastic.co> Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>

dgieselaar added 2 commits August 27, 2020 09:55

Run telemetry task immediately if versions are incompatible; add serv…

e8d5483

…ice environments telemetry

dgieselaar added release_note:fix Team:APM - DEPRECATED Use Team:obs-ux-infra_services. release_note:skip Skip the PR/issue when compiling release notes v7.10.0 v7.9.1 labels Aug 27, 2020

dgieselaar requested a review from a team as a code owner August 27, 2020 11:43

smith reviewed Aug 27, 2020

View reviewed changes

smith requested a review from a team August 27, 2020 20:57

smith approved these changes Aug 27, 2020

View reviewed changes

afharo reviewed Sep 1, 2020

View reviewed changes

dgieselaar added 2 commits September 2, 2020 13:53

Merge branch 'master' of github.com:elastic/kibana into observer-host…

daff420

…name

Correct structure for environment properties

1c5e9e3

dgieselaar added v7.9.2 and removed v7.9.1 labels Sep 2, 2020

Merge branch 'master' of github.com:elastic/kibana into observer-host…

2938c5a

…name

afharo approved these changes Sep 4, 2020

View reviewed changes

Merge branch 'master' of github.com:elastic/kibana into observer-host…

f1898c8

…name

dgieselaar merged commit 113936a into elastic:master Sep 7, 2020

dgieselaar deleted the observer-hostname branch September 7, 2020 09:11

dgieselaar mentioned this pull request Sep 7, 2020

[7.x] [APM] Use observer.hostname (#76074) #76853

Merged

dgieselaar mentioned this pull request Sep 7, 2020

[7.9] [APM] Use observer.hostname (#76074) #76857

Merged

dgieselaar added a commit that referenced this pull request Sep 8, 2020

[7.x] [APM] Use observer.hostname (#76074) (#76853)

14ce8a3

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 9, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Use observer.hostname instead of observer.name #76074

[APM] Use observer.hostname instead of observer.name #76074

dgieselaar commented Aug 27, 2020

elasticmachine commented Aug 27, 2020

smith Aug 27, 2020

smith commented Aug 27, 2020

smith left a comment

dgieselaar commented Aug 28, 2020

dgieselaar commented Aug 31, 2020

afharo left a comment

afharo Sep 1, 2020

dgieselaar Sep 2, 2020

dgieselaar commented Sep 4, 2020

afharo left a comment

afharo Sep 4, 2020

dgieselaar Sep 7, 2020

kibanamachine commented Sep 7, 2020

kibanamachine commented Sep 9, 2020

[APM] Use observer.hostname instead of observer.name #76074

[APM] Use observer.hostname instead of observer.name #76074

Conversation

dgieselaar commented Aug 27, 2020

elasticmachine commented Aug 27, 2020

smith Aug 27, 2020

Choose a reason for hiding this comment

smith commented Aug 27, 2020

smith left a comment

Choose a reason for hiding this comment

dgieselaar commented Aug 28, 2020

dgieselaar commented Aug 31, 2020

afharo left a comment

Choose a reason for hiding this comment

afharo Sep 1, 2020

Choose a reason for hiding this comment

dgieselaar Sep 2, 2020

Choose a reason for hiding this comment

dgieselaar commented Sep 4, 2020

afharo left a comment

Choose a reason for hiding this comment

afharo Sep 4, 2020

Choose a reason for hiding this comment

dgieselaar Sep 7, 2020

Choose a reason for hiding this comment

kibanamachine commented Sep 7, 2020

💚 Build Succeeded

Build metrics

async chunks size

History

kibanamachine commented Sep 9, 2020