Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add plugins support #1753

Merged
merged 12 commits into from
Nov 12, 2021
Merged

Add plugins support #1753

merged 12 commits into from
Nov 12, 2021

Conversation

mstemm
Copy link
Contributor

@mstemm mstemm commented Oct 12, 2021 •
edited by leogr
Loading

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

If contributing rules or changes to rules, please make sure to also uncomment one of the following line:

/kind rule-update

/kind rule-create

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area engine

/area rules

/area tests

/area proposals

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

new: add support for plugins to extend Falco functionality to new event sources and custom fields

@poiana poiana requested review from fntlnz and Kaizhe October 12, 2021 23:02
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch 5 times, most recently from 6da25bd to 6094f58 Compare October 15, 2021 23:55
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch 2 times, most recently from a52c6a7 to 8be6eb0 Compare October 16, 2021 00:08
@mstemm
Copy link
Contributor Author

mstemm commented Oct 19, 2021

/retest

@mstemm mstemm force-pushed the new/plugin-system-api-additions branch from 8be6eb0 to a93d110 Compare October 19, 2021 20:08
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch from f32a212 to bd1d88e Compare October 19, 2021 20:42
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch from 943ca74 to 6762711 Compare October 19, 2021 21:08
@poiana
Copy link
Contributor

poiana commented Nov 12, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leogr, mstemm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@FedeDP FedeDP mentioned this pull request Nov 12, 2021
Merged
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch from 3e087e4 to 56c5d8e Compare November 12, 2021 16:20
@poiana poiana removed the lgtm label Nov 12, 2021
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch from 56c5d8e to 890c218 Compare November 12, 2021 16:22
mstemm and others added 12 commits November 12, 2021 08:24
@mstemm @leogr @ldegio
Update falcosecurity/libs version
d9c991f 
This moves up the commit to one that has plugins support.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
Falco yaml config for plugins
3e3421f 
Update config code/default falco.yaml to add support for plugins:

- Update config parsing methods to support reading plugin config
  objects in a list from yaml.

- The default config defines the cloudtrail/json plugins but does not
  give them any actual config for init config/open
  params (cloudtrail), or init config (json).

- load_plugins is empty so neither plugin is actually loaded by default.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>-
@mstemm @leogr @ldegio
Finish moving token bucket impl from falco to libs
f087ac4 
It took a while, but we remembered to finish moving the token_bucket
from falco engine to libs. There were 2 copies for a while.

This brings over one change to libs--to have an optional timer
function.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
cmake/build changes for plugins
b51011d 
Add a cmake module "plugins" that does the following:

 - Downloads/installs the plugins artifacts from a known tag
 - Copies the resulting cloudtrail/json shared libraries to
   CMAKE_CURRENT_BINARY_DIR/plugins
 - Installs them to FALCO_SHARE_DIR/plugins

The default config will define the plugins but they will be disabled
by default.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
Add initial set of Cloudtrail rules
47158fc 
These rules can be used when combined with the cloudtrail plugin.

They're installed to /etc/falco like the other rules files.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
Falco engine support for plugins
71bfbaa 
Mostly plugins are just handled as a new filter/formatter factory with
a new source based on the loaded input plugin, but there are a few
changes at the engine level:

- is_source_valid returns whether a filter/formatter factory exists
  for a given source. Will be used by rules loaded to skip rules for
  an unknown source.

- the falco engine now holds the required_plugin_version predicates
  found in rules files and a method is_plugin_compatible returns whether
  a plugin semver is compatible with the predicates in the rules

- Update the falco engine version and fields checksum for plugins

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
Rules loading changes for plugins
bdae74b 
Rules loading changes for plugins:

 - parse required_engine_versions from yaml and pass up to rules
   loader as a lua table as an additional return value from load_rules().
 - c++ rules loader converts to map: plugin -> list of required plugin
   versions
 - support is_source_valid callback from lua, calls engine method. If
   a source is not valid, skip any rules for that source and add a warning.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
Plugins support (outputs)
6097520 
The generic events support already handled most of this, with a
dedicated formatter factory for plugin sources. Just one missing
header include and change the logic slightly for json parsing.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm @leogr @ldegio
Plugins support in falco executable
afbdf62 
Update the falco binary to add support for plugins.

- Keep track of an "event source", which is initially "syscall" but
  changes to the input plugin's source if an source plugin ends up being
  loaded.

- New argument --list-plugins will return info on any loaded plugins,
  using sinsp_plugin::plugin_infos.

- Create filter/formatter factories for plugins. This ensures that
  filterchecks for syscalls are not used for plugins and vice versa.

- Use sinsp_plugin::register_plugin() to load each plugin found in
  config. The first source plugin found (if any) calls
  engine->add_source withthe source plugin's event source.

- If a second source plugin is found, exit with an error.

- Extractor plugins must be compatible with the event source (usually
  the plugin event source, but could be "syscall"). If not, exit with
  an error.

- Multiple Extractor plugins are allowed, but they can not have
  overlapping compatible event sources. This is mostly to avoid
  confusion, but we might change this later.

- After loading plugins, use engine is_plugin_compatible to ensure
  that the plugin is compatible with any required_plugin_version blocks
  in falco rules.

- Normally falco would log warnings if too many SCAP_TIMEOUT results
  were received. These are more expected when using plugins, so only
  log these warnings when using syscalls.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm
Add automated tests for plugins
5d03e0f 
Test infrastructure and sample confs/rules/traces for plugins
automated tests:

New test cases are in falco_tests_plugins.yaml and cover:
- Listing plugins and fields when plugins are loaded.
- Basic cloudtrail + json plugin on a fake cloudtrail json file and a
  sample rule that uses both plugins.
- Conflicts between source/extractor plugins
- Incompatible plugin api
- Wrong plugin path
- Checking for warnings when reading rules with unnown sources (e.g. when plugins are not loaded)

Some test-only plugins written in C are in test/plugins and built on
the fly. (They aren't included in packages of course).

The test framework needed some small changes to handle these tests:
- Add a mode to not check detection counts at all (for --list/--list-plugins)
- addl_cmdline_opts to allow specifying --list/--list-plugins
- Using DOTALL when matching stderr/stdout (allows multi-line matches more easily)

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm
Detect strlcpy on the fly (musl libc)
bd9d536 
Detect strlcpy on the fly, as was done in falcosecurity/libs#110.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm
Skip plugins list/load/tests for MUSL_OPTIMIZED_BUILD
a645011 
When MUSL_OPTIMIZED_BUILD is specified, falco is statically linked under
musl, and can't dlopen() files: see
https://inbox.vuxu.org/musl/20200423162406.GV11469@brightrain.aerifal.cx/T/

So skip listing/loading/testing plugins when MUSL_OPTIMIZED_BUILD is specified.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@mstemm mstemm force-pushed the new/plugin-system-api-additions branch from 890c218 to a645011 Compare November 12, 2021 16:24
@mstemm
Copy link
Contributor Author

mstemm commented Nov 12, 2021

Ok I also removed the leftover plugin_free_mem reference. Could you take a look again?

@poiana poiana added the lgtm label Nov 12, 2021
@poiana
Copy link
Contributor

poiana commented Nov 12, 2021

LGTM label has been added.

Git tree hash: 45f912af16a583d24210382a34ffd625e41eb4f1

krisnova
krisnova approved these changes Nov 12, 2021
View reviewed changes
Copy link
Contributor

@krisnova krisnova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@poiana poiana merged commit cb51522 into master Nov 12, 2021
@poiana poiana deleted the new/plugin-system-api-additions branch November 12, 2021 17:28
This was referenced Nov 18, 2021
Merged
Closed
@leogr leogr added this to the 0.31.0 milestone Jan 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jasondellaluce jasondellaluce jasondellaluce left review comments

@krisnova krisnova krisnova approved these changes

@leogr leogr leogr approved these changes

@fntlnz fntlnz Awaiting requested review from fntlnz

@Kaizhe Kaizhe Awaiting requested review from Kaizhe

Assignees

@leogr leogr

@krisnova krisnova

Labels
approved area/engine area/rules dco-signoff: yes kind/feature kind/rule-create lgtm release-note size/XXL
Projects
None yet
Milestone
0.31.0
Development

Successfully merging this pull request may close these issues.
5 participants
@mstemm @poiana @leogr @krisnova @jasondellaluce