[Telemetry] Full schema including Elasticsearch fields

[afharo]

Summary

This is a follow-up to #90273. It adds the definition of the Elasticsearch fields, being discussed as ES might provide the schema from their project instead.

I'll keep it as draft, but will rebase it from master once #90273 is merged until we agree on anything.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: a0a80e8
• Storybooks Preview
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures
• Documentation Changes

Failed CI Steps

• Jest Unit Tests
• Execute xpack-kibana-ciGroup6
• Execute xpack-kibana-ciGroup6

---

Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/telemetry/telemetry_local·ts.apis Telemetry /api/telemetry/v2/clusters/_stats with monitoring disabled should pass the schema validation

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://dryrun

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:01:34]           └-: Telemetry
[00:01:34]             └-> "before all" hook in "Telemetry"
[00:01:37]             └-: /api/telemetry/v2/clusters/_stats with monitoring disabled
[00:01:37]               └-> "before all" hook for "should pass the schema validation"
[00:01:37]               └-> "before all" hook: disable monitoring and pull local stats for "should pass the schema validation"
[00:01:39]               └-> should pass the schema validation
[00:01:39]                 └-> "before each" hook: global before each for "should pass the schema validation"
[00:01:40]                 └- ✖ fail: apis Telemetry /api/telemetry/v2/clusters/_stats with monitoring disabled should pass the schema validation
[00:01:40]                 │      Error: The telemetry schemas in 'x-pack/plugins/telemetry_collection_xpack/schema/' are out-of-date, please update it as required: [stack_stats.xpack.ml.data_frame_analytics_jobs.memory_usage]: definition for this key is missing. Received `{"peak_usage_bytes":{"total":0,"min":0,"avg":0,"max":0}}`
[00:01:40]                 │       at ObjectType.validate (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:51:19)
[00:01:40]                 │       at assertTelemetryPayload (/dev/shm/workspace/parallel/17/kibana/test/api_integration/apis/telemetry/utils/schema_to_config_schema.ts:129:34)
[00:01:40]                 │       at Context.apply (test/api_integration/apis/telemetry/telemetry_local.ts:57:9)
[00:01:40]                 │       at Object.apply (/dev/shm/workspace/parallel/17/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:30)
[00:01:40]                 │ 
[00:01:40]                 │ 

Stack Trace

ValidationError: The telemetry schemas in 'x-pack/plugins/telemetry_collection_xpack/schema/' are out-of-date, please update it as required: [stack_stats.xpack.ml.data_frame_analytics_jobs.memory_usage]: definition for this key is missing. Received `{"peak_usage_bytes":{"total":0,"min":0,"avg":0,"max":0}}`
    at ObjectType.validate (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:51:19)
    at assertTelemetryPayload (/dev/shm/workspace/parallel/17/kibana/test/api_integration/apis/telemetry/utils/schema_to_config_schema.ts:129:34)
    at Context.apply (test/api_integration/apis/telemetry/telemetry_local.ts:57:9)
    at Object.apply (/dev/shm/workspace/parallel/17/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:30) {
  cause: SchemaTypeError: definition for this key is missing
      at ObjectType.onError (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:76:20)
      at type.<anonymous> (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:41:53)
      at finish (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:529:50)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:618:24)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._validateWithOptions (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:674:29)
      at module.exports.internals.Any.root.validate (/dev/shm/workspace/kibana/node_modules/joi/lib/index.js:146:23)
      at ObjectType.validate (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:46:72)
      at assertTelemetryPayload (/dev/shm/workspace/parallel/17/kibana/test/api_integration/apis/telemetry/utils/schema_to_config_schema.ts:129:34)
      at Context.apply (/dev/shm/workspace/parallel/17/kibana/x-pack/test/api_integration/apis/telemetry/telemetry_local.ts:57:9)
      at Object.apply (/dev/shm/workspace/parallel/17/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:30) {
    cause: undefined,
    path: [
      'stack_stats',
      'xpack',
      'ml',
      'data_frame_analytics_jobs',
      'memory_usage'
    ]
  }
}

---

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/telemetry/telemetry_local·ts.apis Telemetry /api/telemetry/v2/clusters/_stats with monitoring disabled should pass the schema validation

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:01:36]           └-: Telemetry
[00:01:36]             └-> "before all" hook in "Telemetry"
[00:01:41]             └-: /api/telemetry/v2/clusters/_stats with monitoring disabled
[00:01:41]               └-> "before all" hook for "should pass the schema validation"
[00:01:41]               └-> "before all" hook: disable monitoring and pull local stats for "should pass the schema validation"
[00:01:42]               └-> should pass the schema validation
[00:01:42]                 └-> "before each" hook: global before each for "should pass the schema validation"
[00:01:43]                 └- ✖ fail: apis Telemetry /api/telemetry/v2/clusters/_stats with monitoring disabled should pass the schema validation
[00:01:43]                 │      Error: The telemetry schemas in 'x-pack/plugins/telemetry_collection_xpack/schema/' are out-of-date, please update it as required: [stack_stats.xpack.ml.data_frame_analytics_jobs.memory_usage]: definition for this key is missing. Received `{"peak_usage_bytes":{"total":0,"min":0,"avg":0,"max":0}}`
[00:01:43]                 │       at ObjectType.validate (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:51:19)
[00:01:43]                 │       at assertTelemetryPayload (/dev/shm/workspace/parallel/17/kibana/test/api_integration/apis/telemetry/utils/schema_to_config_schema.ts:129:34)
[00:01:43]                 │       at Context.apply (test/api_integration/apis/telemetry/telemetry_local.ts:57:9)
[00:01:43]                 │       at Object.apply (/dev/shm/workspace/parallel/17/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:30)
[00:01:43]                 │ 
[00:01:43]                 │ 

Stack Trace

ValidationError: The telemetry schemas in 'x-pack/plugins/telemetry_collection_xpack/schema/' are out-of-date, please update it as required: [stack_stats.xpack.ml.data_frame_analytics_jobs.memory_usage]: definition for this key is missing. Received `{"peak_usage_bytes":{"total":0,"min":0,"avg":0,"max":0}}`
    at ObjectType.validate (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:51:19)
    at assertTelemetryPayload (/dev/shm/workspace/parallel/17/kibana/test/api_integration/apis/telemetry/utils/schema_to_config_schema.ts:129:34)
    at Context.apply (test/api_integration/apis/telemetry/telemetry_local.ts:57:9)
    at Object.apply (/dev/shm/workspace/parallel/17/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:30) {
  cause: SchemaTypeError: definition for this key is missing
      at ObjectType.onError (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:76:20)
      at type.<anonymous> (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:41:53)
      at finish (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:529:50)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:618:24)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._base (/dev/shm/workspace/kibana/node_modules/joi/lib/types/object/index.js:212:45)
      at type._validate (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:614:37)
      at type._validateWithOptions (/dev/shm/workspace/kibana/node_modules/joi/lib/types/any/index.js:674:29)
      at module.exports.internals.Any.root.validate (/dev/shm/workspace/kibana/node_modules/joi/lib/index.js:146:23)
      at ObjectType.validate (/dev/shm/workspace/parallel/17/kibana/packages/kbn-config-schema/target/out/types/type.js:46:72)
      at assertTelemetryPayload (/dev/shm/workspace/parallel/17/kibana/test/api_integration/apis/telemetry/utils/schema_to_config_schema.ts:129:34)
      at Context.apply (/dev/shm/workspace/parallel/17/kibana/x-pack/test/api_integration/apis/telemetry/telemetry_local.ts:57:9)
      at Object.apply (/dev/shm/workspace/parallel/17/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:30) {
    cause: undefined,
    path: [
      'stack_stats',
      'xpack',
      'ml',
      'data_frame_analytics_jobs',
      'memory_usage'
    ]
  }
}

---

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream