Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Manually build Fleet kuery with Node arguments #76589

Merged
merged 3 commits into from
Sep 14, 2020

Conversation

nchaulet
Copy link
Member

@nchaulet nchaulet commented Sep 3, 2020

Summary

This PR #75693 introduced a way to manually build filter instead of parsing the KQL string to improve perfomance. While improving a lot the performance there is still room for improvment if we use buildNodeWithArgumentNodes.

Benchmarking the potential improvment

  const suite = new Benchmark.Suite();

  suite
    .add('parse KQL', function () {
      esKuery.fromKueryExpression(
        'not fleet-agent-actions.attributes.sent_at: * and fleet-agent-actions.attributes.agent_id:1234567'
      );
    })
    .add('buildNode', function () {
      esKuery.nodeTypes.function.buildNode('and', [
        esKuery.nodeTypes.function.buildNode(
          'not',
          esKuery.nodeTypes.function.buildNode('is', 'fleet-agent-actions.attributes.sent_at', '*')
        ),
        esKuery.nodeTypes.function.buildNode(
          'is',
          'fleet-agent-actions.attributes.agent_id',
          '1234567'
        ),
      ]);
    })
    .add('buildNodeWithArgumentNodes', function () {
      nodeTypes.function.buildNodeWithArgumentNodes('and', [
        nodeTypes.function.buildNode(
          'not',
          nodeTypes.function.buildNodeWithArgumentNodes('is', [
            nodeTypes.literal.buildNode(`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at`),
            nodeTypes.wildcard.buildNode(nodeTypes.wildcard.wildcardSymbol),
            nodeTypes.literal.buildNode(false),
          ])
        ),
        nodeTypes.function.buildNodeWithArgumentNodes('is', [
          nodeTypes.literal.buildNode(`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.agent_id`),
          nodeTypes.literal.buildNode('123456'),
          nodeTypes.literal.buildNode(false),
        ]),
      ]);
    })
    // add listeners
    .on('cycle', function (event) {
      console.log(String(event.target));
    })
    .on('complete', function () {
      console.log('Fastest is ' + this.filter('fastest').map('name'));
    })
    // run async
    .run({ async: true, minSamples: 200 });

Give the following result

parse KQL x 1,005 ops/sec ±4.58% (81 runs sampled)
buildNode x 6,143 ops/sec ±4.75% (84 runs sampled)
buildNodeWithArgumentNodes x 905,728 ops/sec ±5.77% (84 runs sampled)
Fastest is buildNodeWithArgumentNodes

@nchaulet nchaulet added v8.0.0 v7.10.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 3, 2020
@nchaulet nchaulet requested review from kobelb and a team September 3, 2020 01:24
@nchaulet nchaulet self-assigned this Sep 3, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@nchaulet nchaulet added the release_note:skip Skip the PR/issue when compiling release notes label Sep 3, 2020
@nchaulet nchaulet force-pushed the feature-manually-build-fleet-kuery branch from d18b7a0 to a0b7c77 Compare September 3, 2020 01:34
@nchaulet nchaulet requested a review from a team as a code owner September 3, 2020 01:34
@nchaulet nchaulet force-pushed the feature-manually-build-fleet-kuery branch from a0b7c77 to 7ed80d5 Compare September 3, 2020 01:39
@kobelb
Copy link
Contributor

kobelb commented Sep 3, 2020

Very nice! I did not know this was an option. @lukasolson, you're the KQL expert, does this make sense to you?

@lukasolson
Copy link
Member

Hmm, if this performance improvement is really necessary, this is fine, but I'd generally like to avoid using buildNodeWithArgumentNodes outside of the grammar.peg file, since it leads to worse readability.

@kobelb
Copy link
Contributor

kobelb commented Sep 3, 2020

@lukasolson perhaps we should address this by providing a better interface for building these KQL queries which hide the intricacies? I believe you mentioned considering implementing this before, is there an existing GitHub issue for that?

@nchaulet
Copy link
Member Author

nchaulet commented Sep 8, 2020

@elasticmachine merge upstream

@nchaulet
Copy link
Member Author

nchaulet commented Sep 9, 2020

@lukasolson Do you think we can move forward with this PR?

@lukasolson
Copy link
Member

Yes, it's fine. I've opened #77085 to track building a better programmatic API.

@nchaulet
Copy link
Member Author

nchaulet commented Sep 9, 2020

@lukasolson thanks for the issue, a simple API like the one you put in the issue will be amazing :)

There is one change to the WildcardType in this PR if you have time for a review

@nchaulet nchaulet requested a review from jfsiii September 9, 2020 19:01
@nchaulet
Copy link
Member Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@nchaulet nchaulet merged commit 5256604 into elastic:master Sep 14, 2020
@nchaulet nchaulet deleted the feature-manually-build-fleet-kuery branch September 14, 2020 21:19
nchaulet added a commit to nchaulet/kibana that referenced this pull request Sep 14, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
* master: (25 commits)
  [Security Solution] Add unit tests for Network search strategy (elastic#77416)
  [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040)
  [Ingest Manager] Add route for package installation by upload (elastic#77044)
  [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359)
  [APM] Remove useLocation and some minor route improvements (elastic#76343)
  [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258)
  [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444)
  Move remaining uses of serviceName away from urlParams (elastic#77248)
  [Lens] Move configuration popover to flyout (elastic#76046)
  [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589)
  skip flaky suite (elastic#59975)
  Neutral-naming in reporting plugin (elastic#77371)
  [Enterprise Search] Add UserIcon styles (elastic#77385)
  [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201)
  [Ingest Manager] Fix polling for new agent action (elastic#77339)
  Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212)
  Stacked headers and navigational search (elastic#72331)
  [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326)
  [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099)
  Add Lens to Recently Accessed (elastic#77249)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
* master: (293 commits)
  Fix tsvb filter ration for table (elastic#77272)
  [Security Solution] Add unit tests for Network search strategy (elastic#77416)
  [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040)
  [Ingest Manager] Add route for package installation by upload (elastic#77044)
  [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359)
  [APM] Remove useLocation and some minor route improvements (elastic#76343)
  [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258)
  [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444)
  Move remaining uses of serviceName away from urlParams (elastic#77248)
  [Lens] Move configuration popover to flyout (elastic#76046)
  [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589)
  skip flaky suite (elastic#59975)
  Neutral-naming in reporting plugin (elastic#77371)
  [Enterprise Search] Add UserIcon styles (elastic#77385)
  [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201)
  [Ingest Manager] Fix polling for new agent action (elastic#77339)
  Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212)
  Stacked headers and navigational search (elastic#72331)
  [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326)
  [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants