Add initial support to validate the mappings in system tests

[mrodm]

Closes #2206

Add validation of the mappings when running system tests.

This process compares the mappings installed by Fleet (preview mappings) with the ones obtained after ingesting new documents into Elasticsearch.

This validation for the time being is behind an environment variable as a technical preview:

# Default value and same behavior as now (validation of fields)
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=fields elastic-package test system -v

# Test just validation of mappings 
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings elastic-package test system -v

# Test both validation of mappings and fields
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=all elastic-package test system -v

Assumptions considered while working on this PR: #2206 (comment)

As part of this PR:

• CI pipeline has been updated to remove steps related to running system tests with Elastic Agent.
  • It will be kept one step to ensure that the feature is working.
• CI pipeline has been updated to add steps to run mapping validation in system tests.

Author's checklist

• Update CI steps to remove old stack agents
• Update CI to add new steps for validating fields using mappings
• Add environment variable to validate that fields are documented with different methods
• Add new environment variables to docs.
• Remove debug/testing log statements
• Test with Elastic stack 8.x
  • Some packages are failing like prometheus failing since some fields depend on dynamic templates (not implemented yet)
• Test with Elastic stack 7.x (f5 package in integrations repository)
• Test with integration and input packages (sql_input test package in elastic-package repo).
• Test with Elastic stacks with LogsDB enabled (synthetics).
  • Tested in CI with ti_anomali_logsdb and auth0_logsdb

[mrodm]

/test

[mrodm]

New variable to hold just the Field Definitions from the local directory (package).

It is needed to check if a given mapping is related to a field definition with type array.

[jsoriano]

Why not using Schema?

[mrodm]

Added here for completeness, but it is not used in this validator.

Should we remove it from here?

[jsoriano]

If not used I would say to remove it.

[mrodm]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1f3588b

Failed CI Steps

• :go: Integration test: system

History

• 💚 Build #4412 succeeded ee755e9
• 💚 Build #4396 succeeded aa60ec4
• 💛 Build #4395 was flaky c0dbfbb
• 💚 Build #4388 succeeded 5097aee

cc @mrodm

[mrodm]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828

[mrodm]

Another package (tines) failing due to a missing mapping https://buildkite.com/elastic/integrations/builds/19362#0193b678-1607-4f12-8d46-c212e3cd4f1e

Should it be validated using dynamic templates?

Error:

test case failed: one or more errors found in mappings in logs-tines.audit_logs index template: [0] field "tines.audit_log.inputs.inputs.options" is undefined: missing definition for path (not in ECS)

Field definition in the package:

                - name: options
                  type: object
                  object_type: keyword
                  object_type_mapping_type: "*"

No mapping created in the preview, and this mapping is created in the actual mappings after ingesting some docs (no dynamic template present):

        "tines.audit_log.inputs.inputs.options": {
          "type": "keyword",
          "ignore_above": 1024
        },

cc @jsoriano

[jsoriano]

No mapping created in the preview

This is weird because the generated component template makes sense. In tines 1.13 subobjects: false` is moved from an object to the whole index, and this is the difference in the component template:

+++ /tmp/tines-subobjects-false-global/component_templates/logs-tines.audit_logs@package.json	2024-12-11 19:53:31.500743669 +0100
@@ -7,7 +7,7 @@
           "lifecycle": {
             "name": "logs"
           },
-          "default_pipeline": "logs-tines.audit_logs-1.12.2",
+          "default_pipeline": "logs-tines.audit_logs-1.13.0",
           "mapping": {
             "total_fields": {
               "limit": "1000"
@@ -113,8 +113,7 @@
                               "createFormEmptyState": {
                                 "type": "boolean"
                               }
-                            },
-                            "subobjects": false
+                            }
                           },
                           "eventName": {
                             "ignore_above": 1024,
@@ -347,7 +346,8 @@
               }
             }
           }
-        }
+        },
+        "subobjects": false
       }
     },
     "_meta": {

I can see mappings in the preview, the main difference is that when subobjects: false is used, as you mention, the properties are directly defined in the root, with dots in the names:

      ...
      "properties": {
        "@timestamp": {
          "type": "date",
          "ignore_malformed": false
        },
        "data_stream.dataset": {
          "type": "constant_keyword"
        },
        "data_stream.namespace": {
          "type": "constant_keyword"
        },
        "data_stream.type": {
          "type": "constant_keyword"
        },
        "event.agent_id_status": {
          "type": "keyword",
          "ignore_above": 1024
        },
        ...

With what version of the stack you are missing the mappings in the preview?

[jsoriano]

Ah, you mean that no mapping is generated in the preview for this dynamic mapping. Yes, this looks like some issue in Fleet, it is not generating the dynamic mappings for subobjects.

[jsoriano]

The problem is not related to subobjects, it can be also reproduced without setting subobjects: false. I think the problem is related to tines.audit_log.inputs.inputs.options not being empty, it also defines a property:

                - name: options.createFormEmptyState
                  type: boolean

This looks like an issue in Fleet, it could generate both mappings, the more specific static mapping for the boolean, and the dynamic template as catch all for the rest.

[jsoriano]

I think we can go on with this change and polish things in future PRs.

[mrodm]

Ah, you mean that no mapping is generated in the preview for this dynamic mapping.

Exactly, that specific field definition had no mapping nor dynamic mapping in the preview.

The problem is not related to subobjects, it can be also reproduced without setting subobjects: false. I think the problem is related to tines.audit_log.inputs.inputs.options not being empty, it also defines a property:

                - name: options.createFormEmptyState
                  type: boolean

This looks like an issue in Fleet, it could generate both mappings, the more specific static mapping for the boolean, and the dynamic template as catch all for the rest.

Thanks for checking this out!