[microsoft_defender_cloud] Add assessment datastream support for the Cloud Detection and Response (CDR) workflow

[brijesh-elastic]

Proposed commit message

microsoft_defender_cloud: Add assessment data stream to support
Cloud Detection and Response (CDR) workflow

The new data stream, 'assessment,' retrieves all available assessments for the provided scope.
For each assessment, if sub-assessments are available, we will make another call to collect
them. We will then aggregate the results from both calls and publish them.

ECS mapping and transforms have also been added to facilitate with the
Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflow.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html
[2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management

Note

To Reviewers:

• Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/microsoft_defender_cloud directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Closes Microsoft Defender for Cloud: Implement mappings for Cloud Security Workflows #14786
• Closes Microsoft Defender for Cloud: Implement transform for Cloud Security Workflows #14788
• Relates [meta] Update Microsoft Defender for Cloud integration to Leverage Native Cloud Security Workflows #14785

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[brijesh-elastic]

Currently, we ask users to provide an interval for data fetching.

@maxcold, should we hardcode it to 24h? Because it's a full sync approach and we've set the retention period to 24h in the transform.

[maxcold]

The easiest would be to set to 24h. Smaller periods (eg. 12h) would work with the downside that deleted resources will still show up for 24h anyway. Larger period won't really work. So if we can't implement simple validation on the input, I'd say hardcoding to 24h makes sense

[brijesh-elastic]

Smaller periods (eg. 12h) would work with the downside that deleted resources will still show up for 24h anyway

Right.

Hardcoding it to 24 hours makes sense, instead of keeping it user-configurable.

[brijesh-elastic]

The interval for fetching data and the retention period in the transform are both set to 24 hours.

Since we have limited data in the instance, it still takes 5-6 minutes to fetch all of it.

@maxcold, Do you think there's a chance that sometimes the destination indices might not have any data, or might have missing or incomplete data? Should we include a grace period in the transform to accommodate this?

[maxcold]

what exactly do you mean by grace period on the transform? Can you give an example?

[brijesh-elastic]

Example:
On the first day, data collection starts (let's say at 12:00) and completes after 10 minutes, with the transformed data populated accordingly. Now, on the second day, data collection will start at 12:10. At that time, in the destination indices, due to the retention period surpassing 24 hours in the transform, we might have missing data for those few minutes. To cover that gap, we can extend the retention period from 24 hours to 25 hours (so, grace period of 1hr).

[maxcold]

got it, we can change to 26h, I just realised that what we have for our native integration, probably due to the same reason

[maxcold]

tested in the qa cluster, findings are being displayed correctly

[efd6]

LGTM pending resolution of @kcreddy's concerns.

[kcreddy]

LGTM. Will approve after the discussion: #15290 (comment)

[elastic-sonarqube]

Quality Gate failed

Failed conditions
62.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2084376

History

• 💚 Build #31429 succeeded b544a61
• 💚 Build #31319 succeeded 9c60a9d
• 💚 Build #31189 succeeded 87f270b

cc @brijesh-elastic

[elastic-vault-github-plugin-prod]

Package microsoft_defender_cloud - 3.0.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_cloud/3.0.0/