Initial version of beatless #8437

ph · 2018-09-25T17:51:30Z

This PR provide the following

Plugin infra for developing providers and functions
A local stdin provider only used for testing, I will remove it in the
final version.
AWS provider and function types for:

Cloudwatch logs
SQS
Kinesis
Api web gateway proxy

License checker
Packaging of artifact
Runners
CLI infrastructure
CLI to push a cloudwatch logs function.
CLI to delete any function
Processors support.
Types to validate value from the users and the lambda function.

What it doesn't provides

ECS and full event extraction. (NOT for v1)
Specifying the AWS credentials in the configuration
CLI for SQS, Kinesis, API
Robust CLI interaction with the API, rollback on failure / versioning.
Removal of not supported outputs
Removal of seccomp check
Integration tests
Updated build task to produce containing the user executable beat and
the linux beats.
Concurrency / memory settings

Vendored

aws/aws-lambda-go
aws/aws-sdk-go-v2

Note: The PR look much bigger because of the vendored files.

Notes 2:

Make sure you add this to your config YAMl, I wan't to make the following the default options but it would have been really messy to do it in this PR.

# Disable anything that could be send to disk
path.data: /tmp
path.logs: /tmp/logs
logging.to_stderr: true
logging.to_files: false
#logging.level: debug
setup.template.enabled: true
queue.mem:
  events: 50
  flush.min_events: 1
  flush.timeout: 0.5s

related work

houndci-bot · 2018-09-25T17:51:45Z

libbeat/feature/registry.go

 // feature are grouped by namespace, a namespace is a kind of plugin like outputs, inputs, or queue.
 // The feature name must be unique.
-type registry struct {
+type FeatureRegistry struct {


type name will be used as feature.FeatureRegistry by other packages, and that stutters; consider calling this Registry

houndci-bot · 2018-09-25T17:51:45Z

libbeat/feature/registry.go

@@ -28,26 +28,26 @@ import (

 type mapper map[string]map[string]Featurable

-// Registry implements a global registry for any kind of feature in beats.
+// Registry implements a global FeatureRegistry for any kind of feature in beats.


comment on exported type FeatureRegistry should be of the form "FeatureRegistry ..." (with optional leading article)

ph · 2018-09-25T17:55:07Z

Forgot about the license check, working on fixing that.

kvch · 2018-09-28T09:51:32Z

x-pack/beatless/beater/beatless.go


-	// TODO: Add registry reference here.
+// Beatless is a beat designed to un under a serverless environment and listen to external triggers,


tyop: "to run"

kvch · 2018-09-28T10:02:20Z

x-pack/beatless/beater/beatless.go

+// Beatless is a beat designed to un under a serverless environment and listen to external triggers,
+// each invocation will generate one or more events to Elasticsearch.
+//
+// Each serverless implementation is different but beatless follows a few executions rules.


tyop: execution

kvch · 2018-09-28T10:04:21Z

x-pack/beatless/beater/beatless.go

-		return nil, fmt.Errorf("error reading config file: %v", err)
+	c := &config.DefaultConfig
+	if err := cfg.Unpack(c); err != nil {
+		return nil, fmt.Errorf("xerror reading config file: %v", err)


xerror? is this a typo?

kvch · 2018-09-28T11:10:44Z

x-pack/beatless/core/coordinator.go

+	return &Coordinator{log: log, runners: runners}
+}
+
+// Start starts each functions into an independant goroutine and wait until all the goroutine are


tyop: independent

kvch · 2018-09-28T11:13:33Z

x-pack/beatless/core/coordinator.go

+
+// Start starts each functions into an independant goroutine and wait until all the goroutine are
+// stopped to exit.
+func (r *Coordinator) Start(ctx context.Context) error {


I would rather name this function Run. Start implies there is a Stop function to call.

ruflin · 2018-09-28T11:32:39Z

@ph One idea to make this more reviewable and not have to tell Github which files it should expand and which ones not: Could you open a PR with just the dependency against the feature branch and then rebase this one on top of it when it's merged?

kvch · 2018-09-28T11:41:07Z

x-pack/beatless/provider/registry.go

+)
+
+// namespace is the namespace were providers will be registered in the global registry.
+var namespace = "beatless.provider"


This could be const.

kvch · 2018-09-28T12:06:24Z

x-pack/beatless/provider/feature.go

+// ns return the namespace for functions of a specific provider. The registry have a flat view
+// representation of the plugin world this mean we don't really have a tree, instead what we do is
+// to create a unique keys per providers that will only keep the functions of the provider.
+func ns(provider string) string {


Please rename this function to getNamespace or something more descriptive.

kvch · 2018-09-28T12:34:44Z

x-pack/beatless/provider/aws/cloudwatch_logs.go

+		}
+
+		// defensive checks
+		if len(parsedEvent.LogEvents) == 0 {


Why is it invalid to get no events?

I've looked at the SDK; this cannot happen the function should not be invoked in that case. I've removed the defensive check for all the function types.

ph · 2018-09-28T12:43:00Z

@ruflin @kvch I've created #8483 with only the vendor changes lets get that merged and I will rebase this one.

This provide the following: 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. What it doesn't provides: - ECS and full event extraction. - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. Vendored: - https://github.com/aws/aws-lambda-go - https://github.com/aws/aws-sdk-go-v2

When a project had two licenses, the script was creating two entries in the NOTICE.txt but both entries would share the same license. This commit also add the possibility to skip file that start with LICENSE but doesn't contains any license information. It also add MIT-0 license wording see https://github.com/aws/mit-0

ph · 2018-09-28T13:41:41Z

Rebased + first round of comments fixed, I am waiting on merging #8483

ph added in progress Pull request is currently in progress. Functionbeat labels Sep 25, 2018

houndci-bot reviewed Sep 25, 2018

View reviewed changes

ph requested a review from ruflin September 25, 2018 17:54

ph added review and removed in progress Pull request is currently in progress. labels Sep 26, 2018

ph requested a review from kvch September 27, 2018 12:34

kvch reviewed Sep 28, 2018

View reviewed changes

ph force-pushed the feature-beatless branch from fd0a33e to 533ba8c Compare September 28, 2018 12:40

ph added 11 commits September 28, 2018 09:28

update function

e1d8a27

Will be in another PR.

d89a9b5

bad type

98bddd4

remove trace

37c3f9c

update vendored notice file

1a78170

fix license

41bbcbf

oops

f3c41a3

wrong config in the yaml

14c2c47

first round of comments from kvch

40c4536

ph force-pushed the fix/beatless-cli branch from d33f5b7 to 40c4536 Compare September 28, 2018 13:41

ph added 4 commits September 28, 2018 10:16

pep8

d0b06a6

cleanup and fix a issue with the rebase.

d4989b1

missing file

a705172

fix failure

23d3622

kvch self-requested a review September 29, 2018 11:53

ph closed this Oct 2, 2018

ph mentioned this pull request Oct 4, 2018

Beatless has now his default list of settings #8452

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial version of beatless #8437

Initial version of beatless #8437

ph commented Sep 25, 2018 •

edited

Loading

houndci-bot Sep 25, 2018

houndci-bot Sep 25, 2018

ph commented Sep 25, 2018

kvch Sep 28, 2018

kvch Sep 28, 2018

kvch Sep 28, 2018

kvch Sep 28, 2018

kvch Sep 28, 2018

ruflin commented Sep 28, 2018

kvch Sep 28, 2018

kvch Sep 28, 2018 •

edited

Loading

kvch Sep 28, 2018

ph Sep 28, 2018

ph commented Sep 28, 2018

ph commented Sep 28, 2018


		// TODO: Add registry reference here.
		// Beatless is a beat designed to un under a serverless environment and listen to external triggers,

Initial version of beatless #8437

Initial version of beatless #8437

Conversation

ph commented Sep 25, 2018 • edited Loading

This PR provide the following

What it doesn't provides

Vendored

related work

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Sep 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Sep 28, 2018

Choose a reason for hiding this comment

kvch Sep 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Sep 28, 2018

ph commented Sep 28, 2018

ph commented Sep 25, 2018 •

edited

Loading

kvch Sep 28, 2018 •

edited

Loading