Add ability to use existing processors from script processor #11260

andrewkroh · 2019-03-15T04:13:54Z

This adds a require function to the JS runtime that scripts can call to import
"modules". The modules that are added in this PR are

processor - You can construct beat processors in JS (e.g. new processor.Dissect({...})).
console - You can write to beat's logger.
path - You can parse win32 and posix paths.

The processor module supports constructing:

AddCloudMetadata
AddDockerMetadata
AddHostMetadata
AddKubernetesMetadata
AddLocale
AddProcessMetadata
CommunityID
DecodeJSONFields
Dissect
DNS

Basic Example

This is a basic example. It's nothing you couldn't do already with the YAML format, but imagine you need to start adding in some more complex conditions and following some conditional branches.

var console = require('console');
var processor = require('processor');

var dissect = new processor.Dissect({
    tokenizer: "pid=%{process.pid}|ppid=%{process.ppid}|",
});
var processMetadata = new processor.AddProcessMetadata({
    match_pids: ["dissect.process.pid"],
});
var parentMetadata = new processor.AddProcessMetadata({
    match_pids: ["dissect.process.ppid"],
    target: "process.parent",
});

function process(event) {
    console.log("fields: %j", event.fields);
    dissect.Run(event);
    processMetadata.Run(event);
    parentMetadata.Run(event);
    event.Delete("dissect");
    event.Rename("message", "log.original");
}

ph

really minor changes, I've inspect the added goja code too.

ph · 2019-03-19T17:08:05Z

libbeat/processors/add_locale/add_locale.go

 }

-func newAddLocale(c *common.Config) (processors.Processor, error) {
+// New constructs a new add_locale processor.
+func New(c *common.Config) (processors.Processor, error) {
 config := struct {
 Format string `config:"format"`
 }{


nice little refactoring.

ph · 2019-03-20T00:59:44Z

libbeat/processors/script/javascript/module/console/console.go

+
+import (
+ "github.com/dop251/goja"
+ "github.com/dop251/goja_nodejs/require"


I've inspected the package from goja everything is clean 👍

ph · 2019-03-20T01:01:45Z

libbeat/processors/script/javascript/module/console/console.go

+
+ log := logp.L().Named("processor.javascript")
+ switch level {
+ case logp.DebugLevel:


Are we missing Info?

I'm glad you brought this up. For some reason I had it in my head that the typical console API in the browser didn't have an info. But I was wrong. https://developer.mozilla.org/en-US/docs/Web/API/Console/info

I'll add this.

ph · 2019-03-20T01:09:48Z

libbeat/processors/script/javascript/module/console/console.go

+ o.Set("log", c.makeLogFunc(logp.DebugLevel))
+ o.Set("warn", c.makeLogFunc(logp.WarnLevel))
+ o.Set("error", c.makeLogFunc(logp.ErrorLevel))
+}


I see we are trying to add a convenience debug method of console, It's bad that console doesn't completely map to the our Error Level, we might want to consider to alias info(), debug() to log().

Or by looking at the code we don't allow a user to use the info level, I am wondering if we should permit that by either allow info() to log to the info level. official doc

Another way would be to expose logp as new module with the method mapped to the right level.

ph · 2019-03-20T01:10:06Z

libbeat/processors/script/javascript/module/console/console_test.go

+}
+`
+
+ logp.TestingSetup(logp.ToObserverOutput())


ph · 2019-03-20T01:16:34Z

libbeat/processors/script/javascript/module/path/windows.go

+ return b.volAndPath[:b.volLen+b.w]
+ }
+ return b.volAndPath[:b.volLen] + string(b.buf[:b.w])
+}


We might want to add unit test for the windows path handling.

There are some tests in path_test.go for Windows.

ph · 2019-03-20T01:20:51Z

libbeat/processors/script/javascript/module/processor/processor.go

+
+func (bp *beatProcessor) Run(call goja.FunctionCall) goja.Value {
+ if len(call.Arguments) != 1 {
+ panic(bp.runtime.NewGoError(errors.Errorf("Run requires one argument")))


Run -> run?

Run is the name of the function that's exposed in the JS. That's why it's capitalized.

I should have been more precise I was talking about the string user in Errorf

errors.Errorf("Run requires one argument")

ph · 2019-03-20T01:22:02Z

libbeat/processors/script/javascript/module/processor/processor.go

+//
+// // javascript
+// var processor = require('processor');
+// var chopLog = new processor.Dissect({tokenizer: "%{key}: %{value}"});


Get to the choppa https://www.youtube.com/watch?v=Xs_OacEq2Sk

ph · 2019-03-20T01:24:50Z

@andrewkroh you will need to run make fmt to fix the CI.

This adds a `require` function to the JS runtime that scripts can call to import "modules". Some standard modules that are added in this PR are - `processor` - You can construct beat processors in JS (e.g. `new processor.Dissect({...})`). - `console` - You can write to beat's logger. - `path` - You can parse win32 and posix paths. The `processor` module supports constructing: - `AddCloudMetadata` - `AddDockerMetadata` - `AddHostMetadata` - `AddKubernetesMetadata` - `AddLocale` - `AddProcessMetadata` - `CommunityID` - `DecodeJSONFields` - `Dissect` - `DNS`

The module registry is a global so 'console' is only initialized once. But if logp is reinitialized the stored logger would have the old settings. So don't use a global logger.

andrewkroh · 2019-03-21T16:20:31Z

@ph So I added console.debug() and console.info. Then I make console.log be effectively an alias to console.info(). Plus I ran make fmt.

ph · 2019-03-21T18:46:36Z

@andrewkroh changes look good, only the Errorf message and there is still a test failling on travis.

command [go test -cover -coverprofile /tmp/gotestcover-280386040 github.com/elastic/beats/libbeat/processors/script/javascript/module/console]: exit status 1
--- FAIL: TestConsole (0.00s)
    console_test.go:60: 
        	Error Trace:	console_test.go:60
        	Error:      	Expected value not to be nil.
        	Test:       	TestConsole
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa6e066]
goroutine 20 [running]:
testing.tRunner.func1(0xc00013a200)
	/home/travis/.gimme/versions/go1.11.5.linux.amd64/

Use logp.DevelopmentSetup and not TestingSetup.

andrewkroh · 2019-03-21T19:37:03Z

I'm pretty sure I figured out the problem with the test now. I should really read the godocs that I wrote next time. 🤦‍♂️ logp.TestingSetup only initializes the logger when -v is used, but this test always needs it setup so I needed to use logp.DevelopmentSetup.

andrewkroh added review libbeat :Processors labels Mar 15, 2019

andrewkroh requested a review from a team as a code owner March 15, 2019 04:13

ph self-assigned this Mar 15, 2019

andrewkroh force-pushed the feature/libbeat/js-require branch from 2b5dd72 to deaa271 Compare March 15, 2019 14:52

ph approved these changes Mar 20, 2019

View reviewed changes

andrewkroh added 6 commits March 21, 2019 12:18

Add github.com/dop251/goja_nodejs to vendor

362d089

Refactor console test assertions

7250187

Get logger at time of use for console

0f89430

The module registry is a global so 'console' is only initialized once. But if logp is reinitialized the stored logger would have the old settings. So don't use a global logger.

Add console.info() and console.debug()

4d52a06

Run goimports

c3637a5

andrewkroh force-pushed the feature/libbeat/js-require branch from 98fdf8b to c3637a5 Compare March 21, 2019 16:19

andrewkroh added 2 commits March 21, 2019 13:39

Skip DNS test on Windows

eba6986

Troubleshooting console_test, remove later

ae4a8d3

andrewkroh added 2 commits March 21, 2019 15:24

move debugging

256e650

Fix console test for real

0f30fc3

Use logp.DevelopmentSetup and not TestingSetup.

ph approved these changes Mar 21, 2019

View reviewed changes

Use errors.New instead of errors.Errorf

ea373bb

ph approved these changes Mar 21, 2019

View reviewed changes

andrewkroh merged commit d36eb5a into elastic:master Mar 21, 2019

andrewkroh mentioned this pull request Mar 21, 2019

Register javascript modules #11384

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to use existing processors from script processor #11260

Add ability to use existing processors from script processor #11260

andrewkroh commented Mar 15, 2019 •

edited

Loading

ph left a comment

ph Mar 19, 2019

ph Mar 20, 2019

ph Mar 20, 2019

andrewkroh Mar 21, 2019

ph Mar 20, 2019

ph Mar 20, 2019

ph Mar 20, 2019

andrewkroh Mar 21, 2019

ph Mar 20, 2019

andrewkroh Mar 21, 2019

ph Mar 21, 2019

ph Mar 20, 2019

ph commented Mar 20, 2019

andrewkroh commented Mar 21, 2019

ph commented Mar 21, 2019

andrewkroh commented Mar 21, 2019

Add ability to use existing processors from script processor #11260

Add ability to use existing processors from script processor #11260

Conversation

andrewkroh commented Mar 15, 2019 • edited Loading

Basic Example

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Mar 20, 2019

andrewkroh commented Mar 21, 2019

ph commented Mar 21, 2019

andrewkroh commented Mar 21, 2019

andrewkroh commented Mar 15, 2019 •

edited

Loading