qualys_vmdr.knowledge_base: Handle *_LIST fields containing multiple values.

[kcreddy]

Proposed commit message

Current ingest pipeline only handles *_LIST response elements such as SOFTWARE_LIST as a single value.
When *_LIST is an array with objects, the respective fields remain unpopulated.

This PR:

• Handles the case when *_LIST fields are an array by extracting them into relevant fields inside ingest pipeline.
• Updates mapping for diagnosis and solution custom fields to match_only_text to avoid errors in system test. match_only_text correctly defines these fields instead of existing keyword type.

Note

Following error occurs when mappings for diagnosis and solution are keyword:
test case failed: found ignored fields in data stream: found ignored fields in data stream logs-qualys_vmdr.knowledge_base-84257: [qualys_vmdr.knowledge_base.diagnosis.value qualys_vmdr.knowledge_base.solution.value]. Affected documents:....

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

1. Run pipeline tests (added new log line):
   Command: $ eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=knowledge_base
   Result:

   --- Test results for package: qualys_vmdr - START ---
   ╭─────────────┬────────────────┬───────────┬────────────────────────────────────────────────────┬────────┬──────────────╮
   │ PACKAGE     │ DATA STREAM    │ TEST TYPE │ TEST NAME                                          │ RESULT │ TIME ELAPSED │
   ├─────────────┼────────────────┼───────────┼────────────────────────────────────────────────────┼────────┼──────────────┤
   │ qualys_vmdr │ knowledge_base │ pipeline  │ (ingest pipeline warnings test-knowledge-base.log) │ PASS   │ 348.510083ms │
   │ qualys_vmdr │ knowledge_base │ pipeline  │ test-knowledge-base.log                            │ PASS   │  70.199583ms │
   ╰─────────────┴────────────────┴───────────┴────────────────────────────────────────────────────┴────────┴──────────────╯
   --- Test results for package: qualys_vmdr - END   ---
   Done

2. Run system tests with config file option (added new config with single, multiple element responses)
   Command: cd packages/qualys_vmdr && elastic-package test system --generate -v --setup --config-file ./data_stream/knowledge_base/_dev/test/system/test-0_valid-mixed-list-config.yml
   Result:

   --- Test results for package: qualys_vmdr - START ---
   ╭─────────────┬────────────────┬───────────┬────────────────────────────┬────────┬───────────────╮
   │ PACKAGE     │ DATA STREAM    │ TEST TYPE │ TEST NAME                  │ RESULT │  TIME ELAPSED │
   ├─────────────┼────────────────┼───────────┼────────────────────────────┼────────┼───────────────┤
   │ qualys_vmdr │ knowledge_base │ system    │ setup - 0_valid-mixed-list │ PASS   │ 39.792048875s │
   ╰─────────────┴────────────────┴───────────┴────────────────────────────┴────────┴───────────────╯
   --- Test results for package: qualys_vmdr - END   ---
   Done

3. Teardown system test changes:

   elastic-package test system -v --no-provision
   elastic-package test system -v --tear-down

Related issues

• Relates [meta] Qualys VMDR: Enhancement to leverage cloud workflows #11673

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f56c105

History

• 💔 Build #18706 failed 93830fd
• 💔 Build #18703 failed fc7954a

cc @kcreddy

[elastic-sonarqube]

Quality Gate failed

Failed conditions
72.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package qualys_vmdr - 5.2.2 containing this change is available at https://epr.elastic.co/package/qualys_vmdr/5.2.2/