Add ECS schema for user-agent ingest processor #37727

dakrone · 2019-01-22T21:22:18Z

This switches the format of the user agent processor to use the schema from ECS.
So rather than something like this:

{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}

The structure is now like this:

{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}

This new can be configured by setting "ecs": true in the processor
configuration, and will be the default for 7.0. Leaving ecs unset or set as
false is deprecated.

Resolves #37329

This PR is against the 6.x branch and will be forward-ported with the deprecated parts removed (I will open a subsequent PR for that).

This switches the format of the user agent processor to use the schema from ECS. So rather than something like this: ``` { "patch" : "3538", "major" : "70", "minor" : "0", "os" : "Mac OS X 10.14.1", "os_minor" : "14", "os_major" : "10", "name" : "Chrome", "os_name" : "Mac OS X", "device" : "Other" } ``` The structure is now like this: ``` { "name" : "Chrome", "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36", "os" : { "name" : "Mac OS X", "version" : "10.14.1", "full" : "Mac OS X 10.14.1" }, "device" : "Other", "version" : "70.0.3538.102" } ``` This new can be configured by setting `"ecs": true` in the processor configuration, and will be the default for 7.0. Leaving `ecs` unset or set as `false` is deprecated. Resolves elastic#37329

elasticmachine · 2019-01-22T21:22:19Z

Pinging @elastic/es-core-features

webmat

I'm not qualified to vet the ES code, but this looks great!

Thanks @dakrone. And once again, sorry for making the request so late in the process.

Noted two things below. Also, should you be introducing new options for properties, such as original?

webmat · 2019-01-23T01:08:35Z

...r-agent/src/test/java/org/elasticsearch/ingest/useragent/UserAgentProcessorFactoryTests.java

@@ -179,7 +185,7 @@ public void testInvalidProperty() throws Exception {

        ElasticsearchParseException e = expectThrows(ElasticsearchParseException.class, () -> factory.create(null, null, config));
        assertThat(e.getMessage(), equalTo("[properties] illegal property value [invalid]. valid values are [NAME, MAJOR, MINOR, "
-                + "PATCH, OS, OS_NAME, OS_MAJOR, OS_MINOR, DEVICE, BUILD]"));
+                + "PATCH, OS, OS_NAME, OS_MAJOR, OS_MINOR, DEVICE, BUILD, ORIGINAL, VERSION]"));
    }


Is there also an OS_VERSION here?

For the os map, the os property now returns the entire map (sub fields aren't selectable)

...ingest-user-agent/src/test/resources/rest-api-spec/test/ingest-useragent/30_custom_regex.yml

dakrone · 2019-01-23T17:22:08Z

Also, should you be introducing new options for properties, such as original?

These new properties are for the ECS format only, they're needed since the names have changed

webmat

LGTM

martijnvg

I left one question, but other than that this LGTM.

...s/ingest-user-agent/src/main/java/org/elasticsearch/ingest/useragent/UserAgentProcessor.java

ruflin · 2019-01-28T09:09:41Z

@dakrone Thanks for getting this in. Looking forward to have this in master so we can adjust our Filebeat pipelines.

This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs). So rather than something like this: ``` { "patch" : "3538", "major" : "70", "minor" : "0", "os" : "Mac OS X 10.14.1", "os_minor" : "14", "os_major" : "10", "name" : "Chrome", "os_name" : "Mac OS X", "device" : "Other" } ``` The structure is now like this: ``` { "name" : "Chrome", "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36", "os" : { "name" : "Mac OS X", "version" : "10.14.1", "full" : "Mac OS X 10.14.1" }, "device" : "Other", "version" : "70.0.3538.102" } ``` This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not supported. Resolves elastic#37329

* Add ECS schema for user-agent ingest processor (#37727) This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs). So rather than something like this: ``` { "patch" : "3538", "major" : "70", "minor" : "0", "os" : "Mac OS X 10.14.1", "os_minor" : "14", "os_major" : "10", "name" : "Chrome", "os_name" : "Mac OS X", "device" : "Other" } ``` The structure is now like this: ``` { "name" : "Chrome", "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36", "os" : { "name" : "Mac OS X", "version" : "10.14.1", "full" : "Mac OS X 10.14.1" }, "device" : "Other", "version" : "70.0.3538.102" } ``` This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not supported. Resolves #37329 * Remove `ecs` setting from docs

* master: Remove types from watcher docs (elastic#38002) Add test coverage for Painless general casting of boolean and Boolean (elastic#37780) Fixed test bug, lastFollowTime is null if there are no follower indices. Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984) Extract TransportRequestDeduplication from ShardStateAction (elastic#37870) Expose retention leases in shard stats (elastic#37991)

* elastic/master: ILM setPriority corrections for a 0 value (elastic#38001) Temporarily disable BWC for retention lease stats (elastic#38049) Skip Shrink when numberOfShards not changed (elastic#37953) Add dispatching to `HandledTransportAction` (elastic#38050) Update httpclient for JDK 11 TLS engine (elastic#37994) Reduce flaxiness of ccr recovery timeouts test (elastic#38035) Fix ILM status to allow unknown fields (elastic#38043) Fix ILM Lifecycle Policy to allow unknown fields (elastic#38041) Update verify repository to allow unknown fields (elastic#37619) [ML] Datafeed deprecation checks (elastic#38026) Deprecate minimum_master_nodes (elastic#37868) Remove types from watcher docs (elastic#38002) Add test coverage for Painless general casting of boolean and Boolean (elastic#37780) Fixed test bug, lastFollowTime is null if there are no follower indices. Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984) Extract TransportRequestDeduplication from ShardStateAction (elastic#37870)

elastic#37984)" This reverts commit cac6b8e. Related: elastic/beats#10650 Will replace this commit with the 6.7 version

This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs). So rather than something like this: ``` { "patch" : "3538", "major" : "70", "minor" : "0", "os" : "Mac OS X 10.14.1", "os_minor" : "14", "os_major" : "10", "name" : "Chrome", "os_name" : "Mac OS X", "device" : "Other" } ``` The structure is now like this: ``` { "name" : "Chrome", "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36", "os" : { "name" : "Mac OS X", "version" : "10.14.1", "full" : "Mac OS X 10.14.1" }, "device" : "Other", "version" : "70.0.3538.102" } ``` This new can be configured by setting `"ecs": true` in the processor configuration, and will be the default for 7.0. Leaving `ecs` unset or set as `false` is deprecated. Resolves elastic#37329 This PR is against the 6.x branch and will be forward-ported with the deprecated parts removed (I will open a subsequent PR for that).

This change reverts the initial 7.0 commits and replaces them with the 6.7 variant that still allows for the ecs flag. This commit differs from the 6.7 variants in that ecs flag will now default to true. 6.7: `ecs` : default `false` 7.x: `ecs` : default `true` 8.0: no option, but behaves as `true` * Revert "Ingest node - user agent, move device to an object (#38115)" This reverts commit 5b008a3. * Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)" This reverts commit cac6b8e. * cherry-pick 5dfe193 Add ECS schema for user-agent ingest processor (#37727) * cherry-pick ec8ddc8 Ingest node - user agent, move device to an object (#38115) (#38121) * cherry-pick f63cbdb (with manual merge fixes) Dep. check for ECS changes to User Agent processor (#38362) * make true the default for the ecs option, and update 7.0 references and tests

This change reverts the initial 7.0 commits and replaces them with the 6.7 variant that still allows for the ecs flag. This commit differs from the 6.7 variants in that ecs flag will now default to true. 6.7: `ecs` : default `false` 7.x: `ecs` : default `true` 8.0: no option, but behaves as `true` * Revert "Ingest node - user agent, move device to an object (elastic#38115)" This reverts commit 5b008a3. * Revert "Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)" This reverts commit cac6b8e. * cherry-pick 5dfe193 Add ECS schema for user-agent ingest processor (elastic#37727) * cherry-pick ec8ddc8 Ingest node - user agent, move device to an object (elastic#38115) (elastic#38121) * cherry-pick f63cbdb (with manual merge fixes) Dep. check for ECS changes to User Agent processor (elastic#38362) * make true the default for the ecs option, and update 7.0 references and tests

Forward port of #38757 This change reverts the initial 7.0 commits and replaces them with the 6.7 variant that still allows for the ecs flag. This commit differs from the 6.7 variants in that ecs flag will now default to true. 6.7: `ecs` : default `false` 7.x: `ecs` : default `true` 8.0: no option, but behaves as `true` * Revert "Ingest node - user agent, move device to an object (#38115)" This reverts commit 5b008a3. * Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)" This reverts commit cac6b8e. * cherry-pick 5dfe193 Add ECS schema for user-agent ingest processor (#37727) * cherry-pick ec8ddc8 Ingest node - user agent, move device to an object (#38115) (#38121) * cherry-pick f63cbdb (with manual merge fixes) Dep. check for ECS changes to User Agent processor (#38362) * make true the default for the ecs option, and update 7.0 references and tests

dakrone added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v6.7.0 labels Jan 22, 2019

dakrone added the blocker label Jan 22, 2019

dakrone changed the title ~~Add ECS schema for user-agent injest processor~~ Add ECS schema for user-agent ingest processor Jan 22, 2019

Fix/add documentation

5ba7839

webmat reviewed Jan 23, 2019

View reviewed changes

ruflin mentioned this pull request Jan 23, 2019

Transition Beats to ECS elastic/beats#8655

Closed

simitt mentioned this pull request Jan 23, 2019

Ensure parsed user_agents is indexed. elastic/apm-server#1805

Closed

dakrone requested a review from martijnvg January 23, 2019 15:13

gwbrown mentioned this pull request Jan 23, 2019

Update Deprecation Info API for 7.0 breaking changes #36024

Closed

26 tasks

dakrone added 2 commits January 23, 2019 10:19

Remove extra trailing brace

abe6ce5

Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x

2608cea

Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x

88d78f6

webmat approved these changes Jan 23, 2019

View reviewed changes

martijnvg approved these changes Jan 24, 2019

View reviewed changes

...s/ingest-user-agent/src/main/java/org/elasticsearch/ingest/useragent/UserAgentProcessor.java Show resolved Hide resolved

dakrone added 4 commits January 24, 2019 08:24

Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x

490f438

Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x

e88ba82

Add deprecation warning for deprecated properties

f087784

Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x

9931f27

dakrone merged commit 5dfe193 into elastic:6.x Jan 25, 2019

dakrone mentioned this pull request Jan 29, 2019

Add ECS schema for user-agent ingest processor (#37727) #37984

Merged

dakrone deleted the ingest-user-agent-ecs-6x branch February 4, 2019 14:42

gwbrown mentioned this pull request Feb 4, 2019

Dep. check for ECS changes to User Agent processor #38362

Merged

jakelandis mentioned this pull request Feb 13, 2019

Make 7.x like 6.7 user agent ecs, but default to true (#38757) #38828

Merged

joegallo mentioned this pull request Oct 15, 2024

[DOCS] Fix User agent processor properties #112518

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ECS schema for user-agent ingest processor #37727

Add ECS schema for user-agent ingest processor #37727

dakrone commented Jan 22, 2019 •

edited

Loading

elasticmachine commented Jan 22, 2019

webmat left a comment

webmat Jan 23, 2019

dakrone Jan 23, 2019

dakrone commented Jan 23, 2019

webmat left a comment

martijnvg left a comment

ruflin commented Jan 28, 2019

Add ECS schema for user-agent ingest processor #37727

Add ECS schema for user-agent ingest processor #37727

Conversation

dakrone commented Jan 22, 2019 • edited Loading

elasticmachine commented Jan 22, 2019

webmat left a comment

Choose a reason for hiding this comment

webmat Jan 23, 2019

Choose a reason for hiding this comment

dakrone Jan 23, 2019

Choose a reason for hiding this comment

dakrone commented Jan 23, 2019

webmat left a comment

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

ruflin commented Jan 28, 2019

dakrone commented Jan 22, 2019 •

edited

Loading