Update windows integration package spec to v3 #10781

belimawr · 2024-08-13T15:20:44Z

Proposed commit message

This commit updates the windows integration to package spec v3.2.1 and makes all necessary changes to the dashboards and fields definitons.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

~~## Author's Checklist~~

How to test this PR locally

Build the windows package
Start a stack using elastic-package
Create a new policy
Add the Windows integration
Deploy the Elastic-Agent to a Windows host
Check the following dashboards are working:
- [Windows powershell] Overview
- [Metrics Windows] Services
- [Windows AppLocker] Audited and Blocked Applications

Related issues

Closes Migrate windows integration to package spec v3 #10274

~~## Screenshots~~

Update fields definitions, manifest and changelog

elasticmachine · 2024-08-13T15:28:12Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2024-08-13T15:28:13Z

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

This commit sets typeMigrationVersion to the version that was required before the update to package spec v3

packages/windows/changelog.yml

packages/windows/data_stream/forwarded/fields/fields.yml

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

andrewkroh

Field updates and manifest LGTM. I did not review the dashboards.

packages/windows/data_stream/powershell/fields/fields.yml

leehinman · 2024-08-21T19:31:35Z

packages/windows/manifest.yml

 conditions:
  kibana:
-    version: "^8.8.0"
+    version: "^8.14.0"


any chance we can go earlier?

I tried to back to 8.8, but other things broke, not even 8.10.1 worked :/

So I just let the one that was working.

leehinman · 2024-08-21T19:33:52Z

packages/windows/data_stream/sysmon_operational/fields/ecs.yml

@@ -6,6 +6,8 @@
  name: destination.port
 - external: ecs
  name: dns.answers
+  type: object


can we get rid of the whole ecs.yml file and use the ecs dynamic mappings?

I'm not sure.

I prefer to have explicit mapping to avoid any malformed data from creating the wrong mapping. Anyways I'm open to look into the dynamic mappings.

Update the ingest pipelines so the fields comply with ECS. Most of the updates are transforming strings into arrays and some updating the fields added to comply with ECS.

Updating the sample_event.json files fixes the static tests, the changes in the README are made by the build process.

elasticmachine · 2024-08-27T19:30:16Z

🚀 Benchmarks report

Package `windows` 👍(4) 💚(2) 💔(3)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`applocker_exe_and_dll`	5076.14	4255.32	-820.82 (-16.17%)	💔
`applocker_packaged_app_execution`	21739.13	6944.44	-14794.69 (-68.06%)	💔
`sysmon_operational`	3802.28	3125	-677.28 (-17.81%)	💔

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-08-27T19:30:42Z

💚 Build Succeeded

Buildkite Build
Commit: 51278d4

History

💔 Build #14985 failed 77b17f0
💔 Build #14693 failed 23a70dd
💔 Build #14689 failed e4ea22f
💔 Build #14673 failed 685ab2e
💔 Build #14626 failed f261d3d

elastic-sonarqube · 2024-08-27T19:30:48Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-09-03T20:00:11Z

Package windows - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=windows

maggieghamry · 2024-10-08T17:07:07Z

Just as a head's up, it appears that as of this PR (confirmed in ESS and On-Prem), all Powershell event dashboards are now empty) after upgrading to 2.0.0 and above. Is there already an issue to capture this?

belimawr · 2024-10-09T13:23:30Z

Just as a head's up, it appears that as of this PR (confirmed in ESS and On-Prem), all Powershell event dashboards are now empty) after upgrading to 2.0.0 and above. Is there already an issue to capture this?

I don't think so, at least I could not find one. This could have been caused by this PR, could you open an issue with details about the environment and how reproduce and test?

pierrehilbert · 2024-10-09T14:36:21Z

Looks like we have an issue: elastic/elastic-agent#5746 (comment)

belimawr · 2024-10-10T13:17:49Z

Looks like we have an issue: elastic/elastic-agent#5746 (comment)

Are you sure this is the same issue @maggieghamry described? Maybe you pasted the wrong link?

My understanding of the issue described by @maggieghamry is that the dashboard is not showing any data, but the agent is healthy.

@maggieghamry do you see any errors/unhealthy agents? Is new data not appearing in the dashboard or does the dashboard just stopped working even with data from the previous version of the agent present in the cluster?

maggieghamry · 2024-10-10T13:34:58Z

@belimawr I'm not sure it is the same issue, but we confirmed by testing on the last functional version (1.47.0) - after upgrading (you can test on-prem or ESS) to 2.0.0 or above, attempting to access the Powershell Dashboards, they are all empty with only workaround at this time being to downgrade or rollback for now.

--------- Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

belimawr added 10 commits August 12, 2024 18:28

Update fields definitions and manifest

f1f896a

Update fields definitions, manifest and changelog

Inline references

09b58f2

Add missing files after inlining visualisations

d2afc39

Convert visualisations to Lens

42c6f12

Convert more visualisations to lens

878c982

Remove legacy visualisations

2cdf70f

Fix names

0805290

Add filter

db0e3ab

Add filter to dashboard

0072cca

Remove type for metrics.*.*

f261d3d

belimawr requested review from a team as code owners August 13, 2024 15:20

belimawr requested review from rdner and mauri870 August 13, 2024 15:20

andrewkroh added Integration:windows Windows Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team [elastic/elastic-agent-data-plane] Team:Security-Windows Platform Security Windows Platform Team [elastic/sec-windows-platform] labels Aug 13, 2024

pierrehilbert requested review from leehinman and removed request for mauri870 August 13, 2024 15:33

belimawr added 2 commits August 14, 2024 17:27

Fix typeMigrationVersion

685ab2e

This commit sets typeMigrationVersion to the version that was required before the update to package spec v3

Update Kibana version

e4ea22f

andrewkroh reviewed Aug 15, 2024

View reviewed changes

packages/windows/changelog.yml Outdated Show resolved Hide resolved

packages/windows/data_stream/forwarded/fields/fields.yml Outdated Show resolved Hide resolved

belimawr and others added 2 commits August 15, 2024 09:31

Update packages/windows/changelog.yml

707ce28

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

fix invocation_details type

23a70dd

andrewkroh approved these changes Aug 15, 2024

View reviewed changes

andrewkroh added the enhancement New feature or request label Aug 19, 2024

leehinman reviewed Aug 21, 2024

View reviewed changes

Fix ingest pipeline errors

3c7acbc

Update the ingest pipelines so the fields comply with ECS. Most of the updates are transforming strings into arrays and some updating the fields added to comply with ECS.

belimawr added 2 commits August 23, 2024 13:06

Update the pipelines to comply with ECS

77b17f0

Update sample events and README

51278d4

Updating the sample_event.json files fixes the static tests, the changes in the README are made by the build process.

belimawr requested a review from leehinman August 30, 2024 18:55

leehinman approved these changes Aug 30, 2024

View reviewed changes

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Aug 30, 2024

belimawr merged commit 930efbc into elastic:main Sep 3, 2024
5 checks passed

belimawr deleted the update-windows-integration-package-spec-v3 branch September 3, 2024 19:45

w0rk3r mentioned this pull request Oct 15, 2024

[Rule Tuning] Fix double bumps caused by Windows Integration Update elastic/detection-rules#4156

Merged

harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this pull request Feb 4, 2025

Update windows integration package spec to v3 (elastic#10781)

8e10d28

--------- Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update windows integration package spec to v3 #10781

Update windows integration package spec to v3 #10781

belimawr commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

andrewkroh left a comment •

edited

Loading

leehinman Aug 21, 2024

belimawr Aug 28, 2024

leehinman Aug 21, 2024

belimawr Aug 22, 2024

elasticmachine commented Aug 27, 2024

elasticmachine commented Aug 27, 2024

elastic-sonarqube bot commented Aug 27, 2024

elasticmachine commented Sep 3, 2024

maggieghamry commented Oct 8, 2024

belimawr commented Oct 9, 2024

pierrehilbert commented Oct 9, 2024

belimawr commented Oct 10, 2024

maggieghamry commented Oct 10, 2024 •

edited

Loading

Update windows integration package spec to v3 #10781

Update windows integration package spec to v3 #10781

Conversation

belimawr commented Aug 13, 2024

Proposed commit message

Checklist

How to test this PR locally

Related issues

elasticmachine commented Aug 13, 2024

elasticmachine commented Aug 13, 2024

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

leehinman Aug 21, 2024

Choose a reason for hiding this comment

belimawr Aug 28, 2024

Choose a reason for hiding this comment

leehinman Aug 21, 2024

Choose a reason for hiding this comment

belimawr Aug 22, 2024

Choose a reason for hiding this comment

elasticmachine commented Aug 27, 2024

🚀 Benchmarks report

Package windows 👍(4) 💚(2) 💔(3)

elasticmachine commented Aug 27, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Aug 27, 2024

Quality Gate passed

elasticmachine commented Sep 3, 2024

maggieghamry commented Oct 8, 2024

belimawr commented Oct 9, 2024

pierrehilbert commented Oct 9, 2024

belimawr commented Oct 10, 2024

maggieghamry commented Oct 10, 2024 • edited Loading

andrewkroh left a comment •

edited

Loading

Package `windows` 👍(4) 💚(2) 💔(3)

maggieghamry commented Oct 10, 2024 •

edited

Loading