update: support running multiple event sources in parallel #2182

jasondellaluce · 2022-08-30T14:29:58Z

What type of PR is this?

/kind design

/kind feature

Any specific area of the project related to this PR?

/area engine

What this PR does / why we need it:

See #2074. This is the last piece of "glue code" that uses all the preliminary work to support multiple parallel event processing loops coming from different sources.

From the UX known so far for Falco, this is a big no-op. However, this now supports loading multiple plugin with event sourcing capability and running all of them in parallel along with the traditional syscall event source. Each event source runs in its dedicated thread and does not influence the others. For example, this re-enables having both the k8s_audit and the syscall event sources on the same Falco instance.

By default, Falco enables the syscall event source and all the event sources implemented by the loaded plugin. The set of event sources enabled for event processing can be influenced with the --enable-source and --disable-source CLI options.

Which issue(s) this PR fixes:

Special notes for your reviewer:

There is a user-facing change that needed to be introduced in the stats writer (-s and --stats-interval CLI options). Considering that multiple event source can now run in parallel, the old stats formatting didn't make sense anymore. As such, the periodic sample is now partitioned by event source. Here's an example of the expected output:

{
  "sample": 71,
  "k8s_audit": {
    "cur": {
      "events": 1
    },
    "delta": {
      "events": 1
    }
  },
  "syscall": {
    "cur": {
      "drop_pct": 0,
      "drops": 0,
      "events": 9525,
      "preemptions": 0
    },
    "delta": {
      "drop_pct": 0,
      "drops": 0,
      "events": 137,
      "preemptions": 0
    }
  }
}

Does this PR introduce a user-facing change?:

new: support running multiple event sources in parallel
update(userspace/falco)!: adapt stats writer for multiple parallel event sources

FedeDP

First-pass light review; i will surely need a second deeper review.

This is just useful for me to clear up some notions about the impl.

FedeDP · 2022-08-30T14:46:54Z

userspace/falco/app_actions/init_falco_engine.cpp

-
-	if(m_state->config->m_json_output)
+	// add syscall as last source
+	add_source_to_engine(falco_common::syscall_source);


Do we want to always add syscall_source unconditionally, without checking if it is loaded?

Good point I'm gonna try to respond to this point and some of the other questions below. So, for event sources we maintain in the app state three things:

loaded_sources: a set of names all loaded event sources. By default this contains syscall that is provided out-of-the-box by Falco, and the event sources coming from the loaded plugins.

enabled_sources: a set of names all event sources and that are enabled for event processing (either in capture/offline or in live mode). By default this is the same as loaded_sources, and can then be changed with the --enable-source and --disable-source options.

sources: a list indexed by source name that contains all information mapped to a given source. For example, we map 1 inspector for each source name, 1 filtercheck list, and so on

The action of adding an event source to the rule engine relies on the loaded_sources list, because the rule engine must be aware of all sources no matter if they are actually enabled for event processing or not, because otherwise it will not be able to load the configured rulesets. Doing otherwise will deteriorate the UX as Falco would stop with error at startup if configured with even a single rule attached to a non-enabled event source. This is also why the syscall event source is added unconditionally.

Note that the "all enabled by default" and the --disable-source options behavior are the same as when we had k8s_audit support bundled into Falco. I thought it made sense to reproduce the same UX (we didn't have --enable-source at that time, which now make things easier).

Makes sense! Thanks Jason!

userspace/falco/app_actions/load_plugins.cpp

userspace/falco/app_actions/process_events.cpp

userspace/falco/app_actions/select_event_sources.cpp

jasondellaluce · 2022-08-31T08:19:32Z

/milestone 0.33.0

FedeDP · 2022-09-09T07:34:45Z

Local build is failing with

In file included from /home/federico/Work/falco/userspace/engine/filter_evttype_resolver.cpp:17:
/home/federico/Work/falco/userspace/engine/filter_evttype_resolver.h: In static member function ‘static void falco_event_types::check_range(uint16_t)’:
/home/federico/Work/falco/userspace/engine/filter_evttype_resolver.h:36:36: error: ‘range_error’ is not a member of ‘std’
36 |                         throw std::range_error("invalid event type");

Do i miss anything?
This is a clean build!

jasondellaluce · 2022-09-09T12:28:28Z

Local build is failing with

In file included from /home/federico/Work/falco/userspace/engine/filter_evttype_resolver.cpp:17:
/home/federico/Work/falco/userspace/engine/filter_evttype_resolver.h: In static member function ‘static void falco_event_types::check_range(uint16_t)’:
/home/federico/Work/falco/userspace/engine/filter_evttype_resolver.h:36:36: error: ‘range_error’ is not a member of ‘std’
36 |                         throw std::range_error("invalid event type");

Do i miss anything? This is a clean build!

Huh, I think this happens after rebasing on top of #2151. Does this happen on master too?

FedeDP · 2022-09-09T12:36:55Z

Does this happen on master too?

Confirm! Thanks! Weird that the issue is not spotted by the CI though.

FedeDP · 2022-09-09T12:39:20Z

Fixed by #2197

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…esults Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…ector only Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…e private methods Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Now, the action is in charge of loading all plugins and initializing: - the offline inspector - the list of loaded event sources - the list of loaded plugins and their config After this action runs, plugins are loaded but not yet initialized. Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

… in the right order Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…tor action Now, the action takes care of inizializing all app inspectors (just one in capture mode, one for each evt source in live mode), and of registering and initializing all loaded plugins in the right inspector as needed. The plugin initialization logic, which also involves the filtercheck list population and checks, was moved and refactored from the previous implementation of the load_plugins action. Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…action Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

… loops Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…ctors Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…ent sources Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

jasondellaluce · 2022-09-12T13:43:39Z

Fixed by #2197

Tests are back being all-green!

FedeDP

/approve

poiana · 2022-09-12T14:13:18Z

LGTM label has been added.

Git tree hash: 4ecde754214d0334b75968c12d0a49938fb37653

leogr

This is awesome! 🤩

poiana · 2022-09-12T14:13:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FedeDP, jasondellaluce, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [FedeDP,jasondellaluce,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

poiana added kind/design release-note dco-signoff: yes kind/feature area/engine labels Aug 30, 2022

poiana requested review from Kaizhe and mfdii August 30, 2022 14:30

poiana added approved size/XXL labels Aug 30, 2022

jasondellaluce mentioned this pull request Aug 30, 2022

[Tracking] Opening multiple event sources in the same Falco instance #2074

Closed

15 tasks

FedeDP reviewed Aug 30, 2022

View reviewed changes

poiana added this to the 0.33.0 milestone Aug 31, 2022

jasondellaluce force-pushed the refactor/multi-src-glue-code branch 2 times, most recently from 8989a6e to 85c45bb Compare September 6, 2022 09:00

jasondellaluce added 11 commits September 12, 2022 12:05

refactor(userspace/falco): re-design application state and methods

213293b

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/falco): add convenience method for merging app run r…

1f73c93

…esults Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/falco): reorder actions for their new semantics

7438cb9

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

cleanup(userspace/falco): remove legacy hacks on source selection action

dfe4866

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/falco): restrict clients init action to syscall insp…

cca8a74

…ector only Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

refactor(userspace/falco): turn open inspector action into convenienc…

5c3a2a3

…e private methods Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

refactor(userspace/falco): init all event sources in falco engine and…

083547c

… in the right order Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/falco): fix plugin list access in rule file loading …

9decca9

…action Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

refactor(userspace/falco): support multiple parallel event processing…

6032bc2

… loops Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

jasondellaluce added 7 commits September 12, 2022 12:09

fix(test): adapt plugin tests to new error msgs and features

d82d902

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/falco): rename init_inspector action into init_inspe…

dba23ae

…ctors Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/falco)!: adapt stats writer for multiple parallel ev…

c3f0b8f

…ent sources Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

chore(userspace/falco): fix codespell typo

91c980a

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

chore(userspace/falco): rename sources app state list for more clarity

b29cd44

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

fix(userspace/falco): move stats collection in event success path

6e7405f

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

fix(userspace/falco): adopt stricter memory order semantics

f5e3d0b

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

jasondellaluce force-pushed the refactor/multi-src-glue-code branch from 85c45bb to f5e3d0b Compare September 12, 2022 12:39

FedeDP approved these changes Sep 12, 2022

View reviewed changes

poiana assigned FedeDP Sep 12, 2022

poiana added the lgtm label Sep 12, 2022

leogr approved these changes Sep 12, 2022

View reviewed changes

poiana assigned leogr Sep 12, 2022

poiana merged commit 9c184af into master Sep 12, 2022

poiana deleted the refactor/multi-src-glue-code branch September 12, 2022 14:14

jasondellaluce mentioned this pull request Sep 27, 2022

[New Feature]: Falco native resource utilization metrics logs support #2222

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update: support running multiple event sources in parallel #2182

update: support running multiple event sources in parallel #2182

jasondellaluce commented Aug 30, 2022 •

edited

Loading

FedeDP left a comment •

edited

Loading

FedeDP Aug 30, 2022

jasondellaluce Aug 30, 2022

jasondellaluce Aug 30, 2022 •

edited

Loading

FedeDP Aug 31, 2022

jasondellaluce commented Aug 31, 2022

FedeDP commented Sep 9, 2022

jasondellaluce commented Sep 9, 2022

FedeDP commented Sep 9, 2022

FedeDP commented Sep 9, 2022

jasondellaluce commented Sep 12, 2022

FedeDP left a comment

poiana commented Sep 12, 2022

leogr left a comment

poiana commented Sep 12, 2022

update: support running multiple event sources in parallel #2182

update: support running multiple event sources in parallel #2182

Conversation

jasondellaluce commented Aug 30, 2022 • edited Loading

FedeDP left a comment • edited Loading

Choose a reason for hiding this comment

FedeDP Aug 30, 2022

Choose a reason for hiding this comment

jasondellaluce Aug 30, 2022

Choose a reason for hiding this comment

jasondellaluce Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

FedeDP Aug 31, 2022

Choose a reason for hiding this comment

jasondellaluce commented Aug 31, 2022

FedeDP commented Sep 9, 2022

jasondellaluce commented Sep 9, 2022

FedeDP commented Sep 9, 2022

FedeDP commented Sep 9, 2022

jasondellaluce commented Sep 12, 2022

FedeDP left a comment

Choose a reason for hiding this comment

poiana commented Sep 12, 2022

leogr left a comment

Choose a reason for hiding this comment

poiana commented Sep 12, 2022

jasondellaluce commented Aug 30, 2022 •

edited

Loading

FedeDP left a comment •

edited

Loading

jasondellaluce Aug 30, 2022 •

edited

Loading