[Security solution][Endpoint] Removes zip compression when creating artifacts #101379

dasansol92 · 2021-06-04T12:37:43Z

Summary

Description:

As a result of moving Artifact storage to fleet (v7.13) the endpoint ManifestManager process should be refactored to no longer compress (zip) the artifacts content

Also:

It fixes related unit tests.
Removes old unused code.

For maintainers

This was checked for breaking API changes and was labeled appropriately

…unit tests and removes old code

elasticmachine · 2021-06-04T14:59:18Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

dasansol92 · 2021-06-07T07:50:21Z

@elasticmachine merge upstream

…t_processing-965

ashokaditya

🚀 LGTM.

...ns/security_solution/server/endpoint/services/artifacts/manifest_manager/manifest_manager.ts

ashokaditya · 2021-06-08T09:06:26Z

x-pack/plugins/security_solution/server/endpoint/schemas/artifacts/saved_objects.mock.ts

-  InternalArtifactCompleteSchema,
-} from './saved_objects';
-
-const compressArtifact = async (artifact: InternalArtifactCompleteSchema) => {


paul-tavares

Just some questions for now, but looks good. Going to run a test locally now and will approve after that

paul-tavares · 2021-06-08T13:59:49Z

x-pack/plugins/security_solution/server/endpoint/services/artifacts/artifact_client.ts

    const createdArtifact = await this.fleetArtifacts.createArtifact({
-      content: artifactContent.toString(),
+      content: Buffer.from(artifact.body, 'base64').toString(),


This is the part I was questioning in my mind as to how we would handle it. I think the returned records from fleetArtifacts.createArtifact() needs to be "merged" in with the interal record manifest manager keeps, so that the copreesed data is added in. maybe? not? :)

Not sure about it, I just removed the zip compression step, so not needed to uncompress the content here, just decode from base64

Looks good.

(FYI: Now I'm also thinking in the future, we might want to remove this redundant base64 encoding 🤔 )

paul-tavares

Did a local test (without Endpoint) and looks like we're populating the Policy manifest with information about artifacts that does not include the compression info. (which still happens, but is done by the Fleet artifct exposed client).

You can see this by looking at the Policy record (endpoint integration policy) and noticed that the compression_algorithm' is set to none` and that body decoded and encoded data is the same

I think the challange here will be to figure out how we can get the data returned by from fleet artifact client when the artifact is created back into the internal manifest entry that Manifest Manager keeps. I belive the internal record is written/saved before the artifact is actuallly created, so we might have to chagne the flow there.

…th zlip compression

…965' of github.com:dasansol92/kibana into feature/olm-no_compress_artifacts_during_it_processing-965

dasansol92 · 2021-06-09T15:39:32Z

@elasticmachine merge upstream

…t_processing-965

…est. Also fixes unit tests

paul-tavares

I did not check out the code locally this time, but changes looks good. Maybe just add a screen capture to the Description of the PR that shows the Policy manifest entries (json), so that we can see that zlib is still being set for compression and that the size/hashes are also set accordingly.

Other than that - 🚢

Thanks for taking this on

dasansol92 · 2021-06-14T08:12:04Z

@elasticmachine merge upstream

…t_processing-965

kibanamachine · 2021-06-14T10:17:06Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #130414 succeeded 14d2186
💔 Build #130183 failed 101775a
💚 Build #129389 succeeded 53783a5
💚 Build #129158 succeeded e07c1bd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…rtifacts (elastic#101379) * Removes zlib compression when creating artifacts. Also fixes related unit tests and removes old code * Replaces artifact in new manifest using the ones from fleet client with zlip compression * Fixes create_policy_artifact_manifest pushArtifacts missing new manifest. Also fixes unit tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-06-14T10:28:04Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* master: [Security solution][Endpoint] Removes zip compression when creating artifacts (elastic#101379) [Dashboard]: Fixes disabled viz filter is applied (elastic#101859) [Discover] Deangularization of search embeddable (elastic#100552)

…rtifacts (#101379) (#102033) * Removes zlib compression when creating artifacts. Also fixes related unit tests and removes old code * Replaces artifact in new manifest using the ones from fleet client with zlip compression * Fixes create_policy_artifact_manifest pushArtifacts missing new manifest. Also fixes unit tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: David Sánchez <davidsansol92@gmail.com>

…rtifacts (elastic#101379) * Removes zlib compression when creating artifacts. Also fixes related unit tests and removes old code * Replaces artifact in new manifest using the ones from fleet client with zlip compression * Fixes create_policy_artifact_manifest pushArtifacts missing new manifest. Also fixes unit tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Removes zlib compression when creating artifacts. Also fixes related …

e07c1bd

…unit tests and removes old code

dasansol92 added auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v7.14.0 v8.0.0 labels Jun 4, 2021

dasansol92 requested a review from paul-tavares June 4, 2021 14:51

dasansol92 marked this pull request as ready for review June 4, 2021 14:59

dasansol92 requested a review from a team as a code owner June 4, 2021 14:59

Merge branch 'master' into feature/olm-no_compress_artifacts_during_i…

53783a5

…t_processing-965

ashokaditya approved these changes Jun 8, 2021

View reviewed changes

paul-tavares reviewed Jun 8, 2021

View reviewed changes

paul-tavares requested changes Jun 8, 2021

View reviewed changes

dasansol92 added 2 commits June 9, 2021 17:31

Replaces artifact in new manifest using the ones from fleet client wi…

ca0f2ff

…th zlip compression

Merge branch 'feature/olm-no_compress_artifacts_during_it_processing-…

2e3e147

…965' of github.com:dasansol92/kibana into feature/olm-no_compress_artifacts_during_it_processing-965

kibanamachine and others added 2 commits June 9, 2021 11:39

Merge branch 'master' into feature/olm-no_compress_artifacts_during_i…

101775a

…t_processing-965

Fixes create_policy_artifact_manifest pushArtifacts missing new manif…

14d2186

…est. Also fixes unit tests

dasansol92 requested a review from paul-tavares June 10, 2021 12:28

paul-tavares approved these changes Jun 10, 2021

View reviewed changes

Merge branch 'master' into feature/olm-no_compress_artifacts_during_i…

147901b

…t_processing-965

dasansol92 merged commit ec2ec6a into elastic:master Jun 14, 2021

kibanamachine mentioned this pull request Jun 14, 2021

[7.x] [Security solution][Endpoint] Removes zip compression when creating artifacts (#101379) #102033

Merged

dasansol92 mentioned this pull request Jun 21, 2021

[Security solution][Endpoint] Removes 'none' compression as it not used anymore #102767

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security solution][Endpoint] Removes zip compression when creating artifacts #101379

[Security solution][Endpoint] Removes zip compression when creating artifacts #101379

dasansol92 commented Jun 4, 2021 •

edited

Loading

elasticmachine commented Jun 4, 2021

dasansol92 commented Jun 7, 2021

ashokaditya left a comment

ashokaditya Jun 8, 2021

paul-tavares left a comment

paul-tavares Jun 8, 2021

dasansol92 Jun 8, 2021

paul-tavares Jun 10, 2021

paul-tavares left a comment •

edited

Loading

dasansol92 commented Jun 9, 2021

paul-tavares left a comment

dasansol92 commented Jun 14, 2021

kibanamachine commented Jun 14, 2021

kibanamachine commented Jun 14, 2021

[Security solution][Endpoint] Removes zip compression when creating artifacts #101379

[Security solution][Endpoint] Removes zip compression when creating artifacts #101379

Conversation

dasansol92 commented Jun 4, 2021 • edited Loading

Summary

Description:

For maintainers

elasticmachine commented Jun 4, 2021

dasansol92 commented Jun 7, 2021

ashokaditya left a comment

Choose a reason for hiding this comment

ashokaditya Jun 8, 2021

Choose a reason for hiding this comment

paul-tavares left a comment

Choose a reason for hiding this comment

paul-tavares Jun 8, 2021

Choose a reason for hiding this comment

dasansol92 Jun 8, 2021

Choose a reason for hiding this comment

paul-tavares Jun 10, 2021

Choose a reason for hiding this comment

paul-tavares left a comment • edited Loading

Choose a reason for hiding this comment

dasansol92 commented Jun 9, 2021

paul-tavares left a comment

Choose a reason for hiding this comment

dasansol92 commented Jun 14, 2021

kibanamachine commented Jun 14, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Jun 14, 2021

💚 Backport successful

dasansol92 commented Jun 4, 2021 •

edited

Loading

paul-tavares left a comment •

edited

Loading