Filebeat: Ensure module pipelines compatibility with previous version…

…s of Elasticsearch (backport #26737) (#27841)

* Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (#26737)

Improve the pipeline/compatibility code so that all processors in a pipeline
are scanned and acted-upon to ensure compatibility. This means:
 - Scan processors in on_failure section (both the pipeline's and each
   individual processor on-failure handler).
 - Scan the inner processor in a foreach processor.

Add a new CI stage, module-compat-7.11, to filebeat/Jenkinsfile.yml
and x-pack/filebeat/Jenkinsfile.yml, in order to run Filebeat modules tests
under ES 7.11.2 to ensure that all pipelines are functional.

This test uses a new flag, TESTING_FILEBEAT_SKIP_DIFF, to instruct the
integration test to skip the comparison between the generated documents and
the golden/expected files. The test will ensure that the pipeline loads,
there are no ingest errors, the fields in the generated documents are valid
and the number of returned documents matches the expected. This is intended
to avoid having to maintain multiple versions of the golden files due to
differences between ES versions and available processors.

Also fixes the fortinet and threatintel modules pipelines so that they pass
the new test, as some fields were left behind due to the uri_parts
processor being removed.

(cherry picked from commit 181cf69)

* Remove script.cache.max_size / script.max_compilations_rate from Filebeat tests (#19896)

These setting has been removed as per elastic/elasticsearch#59262 causing the ES container to not launch.

    elasticsearch_1  | java.lang.IllegalArgumentException: unknown setting [script.cache.max_size] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

* Configure ingest node options in env and not in FB test_modules.py

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

filebeat/Jenkinsfile.yml

@@ -44,6 +44,14 @@ stages:
     pythonIntegTest:
         mage: "mage pythonIntegTest"       ## run the ITs only if the changeset affects a specific module.
         stage: mandatory
+    module-compat-7.11:
+      mage: >-                 ## Run module integration tests under ES 7.11 to ensure ingest pipeline compatibility.
+        STACK_ENVIRONMENT=7.11
+        TESTING_FILEBEAT_SKIP_DIFF=1
+        PYTEST_ADDOPTS='-k test_modules'
+        mage pythonIntegTest
+      withModule: true
+      stage: mandatory
     macos:
         mage: "mage build unitTest"
         platforms:             ## override default label in this specific stage.

filebeat/docker-compose.yml

@@ -40,8 +40,6 @@ services:
     extends:
       file: ${ES_BEATS}/testing/environments/${TESTING_ENVIRONMENT}.yml
       service: elasticsearch
-    environment:
-      script.cache.max_size: "500"
 
   kafka:
     build: ${ES_BEATS}/testing/environments/docker/kafka