[Security Solution] add an excess validation instead of the exact match #76472

XavierM · 2020-09-02T02:25:45Z

Summary

Instead of using the exactCheck and the io-ts exact function to validate your body or query from a route, I created an excess function that will do the same as the exactCheck in combination with the exact function from io-ts by telling the user if you add extra attributes to your body/query (please check the unit test) and of course validating the object. The good thing with this excess function, you do not need to wrap your type inside of an exact function anymore.

With the help of @dhurley14, we did some test on the speed between these two different approaches, with the exact from io-ts, it will take around 9 minutes for validating 203 rules. With the excess function, it will only take a few milliseconds.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2020-09-02T02:26:17Z

Pinging @elastic/siem (Team:SIEM)

XavierM · 2020-09-02T02:27:15Z

@elasticmachine merge upstream

…rt-validation

XavierM · 2020-09-02T02:31:31Z

x-pack/plugins/security_solution/server/utils/runtime_types.ts

+
+type ErrorFactory = (message: string) => Error;
+
+export type GenericIntersectionC =


I know but I was not able to win this fight with typescript if you have a better idea, I am here to hear it.

…rt-validation

x-pack/plugins/security_solution/server/utils/runtime_types.ts

dhurley14 · 2020-09-02T16:23:04Z

x-pack/plugins/security_solution/server/utils/build_validation/route_validation.ts

+  validationResult: RouteValidationResultFactory
+) =>
+  pipe(
+    excess(schema).decode(inputValue),


I think there is a better name we could use other than just excess. Maybe something like excessPropsValidation or something that just describes what the excess function is really doing behind the scenes which could help make it clearer at first glance without diving into the code.

…rt-validation

dhurley14

Thanks for looking into this further! LGTM!

…rt-validation

kibanamachine · 2020-09-03T04:22:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: eae333f

Build metrics

distributable file count

id	value	diff	baseline
total	45382	+1	45381

History

💚 Build #71984 succeeded f2a6874
💔 Build #71910 failed cbf3b04
💔 Build #71775 failed 56ec9a5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…ch (elastic#76472) * add an excess validation instead of the exact match * fix readble type + unit test * review I * remove buildRouteValidation to use buildRouteValidationWithExcess

…nes/processors-forms-k-s * 'master' of github.com:elastic/kibana: (216 commits) [Ingest Manager] Split Registry errors into Connection & Response (elastic#76558) [Security Solution] add an excess validation instead of the exact match (elastic#76472) Introduce TS incremental builds & move src/test_utils to TS project (elastic#76082) fix bad merge (elastic#76629) [Newsfeed] Ensure the version format when calling the API (elastic#76381) remove server_extensions mixin (elastic#76606) Remove legacy applications and legacy mode (elastic#75987) [Discover] Fix sidebar element focus behavior when adding / removing columns (elastic#75749) Replace FetchOptions with ISearchOptions (elastic#76538) Move streams utils to the core (elastic#76397) [Ingest Manager] Wording & linking improvements (elastic#76284) remove legacy kibana plugin (elastic#76064) [Form lib] Fix regression on field not being validated after reset to its default value. (elastic#76379) Legacy SO import: Fix bug causing multiple overrides to only show the last confirm modal (elastic#76482) [APM] Service maps layout enhancements (elastic#76481) [Security Solution][Detection Engine] Remove RuleTypeSchema in favor of Type for TypeScript (elastic#76586) [Security Solution][Exceptions] - Updates enum schema and tests (elastic#76544) Index Pattern class - remove toJSON and toString (elastic#76246) skip failing suite (elastic#76581) Split up and clarify Enterprise Search codeowners (elastic#76580) ... # Conflicts: # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/processor_settings_fields.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/processors/foreach.tsx

…oleysens/kibana into ingest-pipelines/processors-forms-k-s * 'ingest-pipelines/processors-forms-k-s' of github.com:jloleysens/kibana: (216 commits) [Ingest Manager] Split Registry errors into Connection & Response (elastic#76558) [Security Solution] add an excess validation instead of the exact match (elastic#76472) Introduce TS incremental builds & move src/test_utils to TS project (elastic#76082) fix bad merge (elastic#76629) [Newsfeed] Ensure the version format when calling the API (elastic#76381) remove server_extensions mixin (elastic#76606) Remove legacy applications and legacy mode (elastic#75987) [Discover] Fix sidebar element focus behavior when adding / removing columns (elastic#75749) Replace FetchOptions with ISearchOptions (elastic#76538) Move streams utils to the core (elastic#76397) [Ingest Manager] Wording & linking improvements (elastic#76284) remove legacy kibana plugin (elastic#76064) [Form lib] Fix regression on field not being validated after reset to its default value. (elastic#76379) Legacy SO import: Fix bug causing multiple overrides to only show the last confirm modal (elastic#76482) [APM] Service maps layout enhancements (elastic#76481) [Security Solution][Detection Engine] Remove RuleTypeSchema in favor of Type for TypeScript (elastic#76586) [Security Solution][Exceptions] - Updates enum schema and tests (elastic#76544) Index Pattern class - remove toJSON and toString (elastic#76246) skip failing suite (elastic#76581) Split up and clarify Enterprise Search codeowners (elastic#76580) ...

…ch (#76472) (#76635) * add an excess validation instead of the exact match * fix readble type + unit test * review I * remove buildRouteValidation to use buildRouteValidationWithExcess

* master: (340 commits) [data.search.SearchSource] Remove legacy ES client APIs. (elastic#75943) [release notes] automatically retry on Github API 5xx errors (elastic#76447) [es_ui_shared] Fix eslint exhaustive deps rule (elastic#76392) [i18n] Integrate 7.9.1 Translations (elastic#76391) [APM] Update aggregations to support script sources (elastic#76429) [Security Solution] Refactor Network Top Countries to use Search Strategy (elastic#76244) Document security settings available on ESS (elastic#76513) [Ingest Manager] Add input revision to the config send to the agent (elastic#76327) [DOCS] Identifies cloud settings for Monitoring (elastic#76579) [DOCS] Identifies Cloud settings in Dev Tools (elastic#76583) [Ingest Manager] Better default value for fleet long polling timeout (elastic#76393) [data.indexPatterns] Fix broken rollup index pattern creation (elastic#76593) [Ingest Manager] Split Registry errors into Connection & Response (elastic#76558) [Security Solution] add an excess validation instead of the exact match (elastic#76472) Introduce TS incremental builds & move src/test_utils to TS project (elastic#76082) fix bad merge (elastic#76629) [Newsfeed] Ensure the version format when calling the API (elastic#76381) remove server_extensions mixin (elastic#76606) Remove legacy applications and legacy mode (elastic#75987) [Discover] Fix sidebar element focus behavior when adding / removing columns (elastic#75749) ...

kibanamachine · 2020-09-07T13:18:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine · 2020-09-08T13:28:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine · 2020-09-09T13:38:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

…ct match (#76472) (#76637) * [Security Solution] add an excess validation instead of the exact match (#76472) * add an excess validation instead of the exact match * fix readble type + unit test * review I * remove buildRouteValidation to use buildRouteValidationWithExcess * fix test

ghost · 2020-09-16T13:17:39Z

Hi @XavierM

We have gone through the ticket and observed it required DEV_Validation to test the ticket .
Please let us know if anything is missing from our end.

thanks !!

elasticmachine · 2021-09-23T14:26:26Z

Pinging @elastic/security-solution (Team: SecuritySolution)

add an excess validation instead of the exact match

c85179f

XavierM requested review from dhurley14, FrankHassanabad and angorayc September 2, 2020 02:25

XavierM requested review from a team as code owners September 2, 2020 02:25

XavierM self-assigned this Sep 2, 2020

XavierM added Team:SIEM v7.10.0 v7.9.2 v8.0.0 labels Sep 2, 2020

XavierM added the release_note:skip Skip the PR/issue when compiling release notes label Sep 2, 2020

Merge branch 'master' of github.com:elastic/kibana into timeline-impo…

56ec9a5

…rt-validation

XavierM commented Sep 2, 2020

View reviewed changes

XavierM added 2 commits September 2, 2020 10:45

fix readble type + unit test

f2b0c85

Merge branch 'master' of github.com:elastic/kibana into timeline-impo…

cbf3b04

…rt-validation

dhurley14 reviewed Sep 2, 2020

View reviewed changes

x-pack/plugins/security_solution/server/utils/runtime_types.ts Outdated Show resolved Hide resolved

dhurley14 reviewed Sep 2, 2020

View reviewed changes

x-pack/plugins/security_solution/server/utils/runtime_types.ts Outdated Show resolved Hide resolved

dhurley14 reviewed Sep 2, 2020

View reviewed changes

XavierM added 2 commits September 2, 2020 13:27

review I

8badaec

Merge branch 'master' of github.com:elastic/kibana into timeline-impo…

f2a6874

…rt-validation

dhurley14 approved these changes Sep 2, 2020

View reviewed changes

angorayc approved these changes Sep 2, 2020

View reviewed changes

XavierM added 2 commits September 2, 2020 21:47

remove buildRouteValidation to use buildRouteValidationWithExcess

61739fd

Merge branch 'master' of github.com:elastic/kibana into timeline-impo…

eae333f

…rt-validation

XavierM merged commit 299c3f5 into elastic:master Sep 3, 2020

XavierM mentioned this pull request Sep 3, 2020

[7.x] [Security Solution] add an excess validation instead of the exact match (#76472) #76635

Merged

XavierM mentioned this pull request Sep 3, 2020

[7.9] [Security Solution] add an excess validation instead of the exact match (#76472) #76637

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 7, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 9, 2020

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] add an excess validation instead of the exact match #76472

[Security Solution] add an excess validation instead of the exact match #76472

XavierM commented Sep 2, 2020 •

edited

Loading

elasticmachine commented Sep 2, 2020

XavierM commented Sep 2, 2020

XavierM Sep 2, 2020

dhurley14 Sep 2, 2020

dhurley14 left a comment

kibanamachine commented Sep 3, 2020

kibanamachine commented Sep 7, 2020

kibanamachine commented Sep 8, 2020

kibanamachine commented Sep 9, 2020

ghost commented Sep 16, 2020

elasticmachine commented Sep 23, 2021


		type ErrorFactory = (message: string) => Error;

		export type GenericIntersectionC =

[Security Solution] add an excess validation instead of the exact match #76472

[Security Solution] add an excess validation instead of the exact match #76472

Conversation

XavierM commented Sep 2, 2020 • edited Loading

Summary

Checklist

elasticmachine commented Sep 2, 2020

XavierM commented Sep 2, 2020

XavierM Sep 2, 2020

Choose a reason for hiding this comment

dhurley14 Sep 2, 2020

Choose a reason for hiding this comment

dhurley14 left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 3, 2020

💚 Build Succeeded

Build metrics

distributable file count

History

kibanamachine commented Sep 7, 2020

kibanamachine commented Sep 8, 2020

kibanamachine commented Sep 9, 2020

ghost commented Sep 16, 2020

elasticmachine commented Sep 23, 2021

XavierM commented Sep 2, 2020 •

edited

Loading