[Question] How to import rules from this repository in Kibana?

[vedard]

Hi Elastic, thanks you for opening this repository to the public.

I was wondering if it was possible to fork this repository, add some of our environment specific rules and import them in our Kibana instance.

It would improve our workflow:

• We could write tests for our rules
• We would keep a history of our rules
• We could frequently update Elastic's rules (only need to pull from upstream)

I see there is a build-release command, but I'm not sure if I can import the generated package in Kibana or if it's only when you release a new version of Kibana

Thanks.

[rw-access]

💯

glad to see you've done some sleuthing! I think we have most of the pieces in place for this — you probably noticed the wrapper around the Kibana API as well.

For our initial release of the repository, we intentionally kept this out of scope. We just needed to get something released quickly and wanted to have a path for users like yourself to contribute directly to the released detection engine.

This is a near-term priority for us. @brokensound77 and I are figuring out the details but most of the pieces are there. We know there is a need for many organizations like to synchronize their rules to/from git. Or at least to pull out of the platform and push back.

It's hard to commit to a timeline, especially as we start wrapping up a release. But I think it's fair to say this is a "near-term" goal.

We'll keep you posted!

[vedard]

This is good news! Thank you for the quick response.

[vedard]

Hi rw-access,

I'm really happy to see you've added the kibana-upload command. I've been trying it out, and I think it would be really convenient:

• to be able to update a rule without having to delete the previous version.
• to be able to upload all rules at once, this one is easy, i was able to do it by allowing the toml-files arguments to be None:

  @click.argument("toml-files", nargs=-1, default=None)

Are these features you are planning to implement?

[SHolzhauer]

I am interested in trying to pick this issue up; But let's first have a look at what it is that's actually needed.

We need a way to

• Update existing, custom, rules in Kibana,
• Update existing, elastic, rules in Kibana between stack releases,
• Create new, custom, rules in Kibana
• Create new, elastic, rules in Kibana between stack releases
• Do above with multiple rules at once

This process should

• be possible to automate (CI/CD)
• validate rules (python -m detection_rules test functionality ?)
• only update/create valid rules

This FR should not

• interfere with/change existing commands

I am purposefully leaving the original first requirement from @vedard out to scope the FR down.

Looking forward to hearing what everyone thinks?

[rw-access]

@vedard does this work now for multiple upload?
The current command looks like this which should be working:

detection-rules/detection_rules/eswrap.py

Line 231 in 464d5e6

@click.argument("toml-files", nargs=-1, required=True)

@brokensound77 have you looked at this?

[vedard]

@rw-access it does if you want to specify one or multiple files, but it would be nice to have an option upload all rules at once.

I was able to do it by set None as the default value for the argument, but it could another option like --all-rules.

[brokensound77]

This is similar to how it is done for import-rules

detection-rules/detection_rules/main.py

Lines 50 to 56 in 140091e

	@root.command('import-rules')
	@click.argument('infile', type=click.Path(dir_okay=False, exists=True), nargs=-1, required=False)
	@click.option('--directory', '-d', type=click.Path(file_okay=False, exists=True), help='Load files from a directory')
	def import_rules(infile, directory):
	"""Import rules from json, toml, or Kibana exported rule file(s)."""
	rule_files = glob.glob(os.path.join(directory, '**', '*.*'), recursive=True) if directory else []
	rule_files = sorted(set(rule_files + list(infile)))

The glob pattern could be updated to *.toml, but the onus would still be on the user to ensure that a directory contains only valid rule files