New processor: copy_fields #11303

kvch · 2019-03-19T10:23:00Z

A new processor is introduced as part of support for keeping orignal messages. Options and naming follows the convention of other processors.

`copy_fields`

This processor copies one field to another. Example configuration is below:

processors:
- copy_fields:
    fields:
      - from: message
        to: event.original
    fail_on_error: false
    ignore_missing: true

Blocks #11297

ruflin

Needs a changelog entry.

libbeat/processors/actions/copy_fields.go

libbeat/processors/actions/copy_fields_test.go

libbeat/processors/actions/copy_fields.go

ruflin · 2019-03-21T09:47:50Z

libbeat/_meta/config.reference.yml

+#        - from: message
+#          to: message_copied
+#    fail_on_error: false
+#    ignore_missing: true


Seems like this is exactly the opposite to what we have in the defaults in the code for fail_on_error and ignore_missing. Is that on purpose?

No. Thanks for finding it.

I think this is still missing?

ruflin · 2019-03-21T09:49:49Z

libbeat/processors/actions/copy_fields_test.go

+		},
+		"copy number from fieldname with dot to dotted message.copied": {
+			FromTo: fromTo{
+				From: "message.original",


What happens if you copy message.original to message? Will it fail?

I set message.original as a single key containing a dot, so it does not lead to an error when someone wants to manipulate the submaps under message. It is a different key.

If the test input was different e.g. the key message was already present

Input: common.MapStr{ "message": common.MapStr{ "original": 42, }, }

It would fail and tell the user to drop the field message first, before attempting to copy stuff.

This is a more realistic test case, so I am adding it also.

Interesting, good to know. Could you add a test case for both and one of them would fail. The reason I'm asking also for the failing one is because in case we change the behaviour of the above in the future, we will know that it has an impact.

The test with the description "copy number from hierarchical message.original to top level message which fails" will fail if the behaviour changes, as the new field would show up in the output. I have also added a new test case which demonstrates what happens when the key contains a dot and is present as a top level key in the fields. If MapStr is changed one of the tests will fail.

libbeat/processors/actions/rename.go

libbeat/processors/actions/rename_test.go

kvch · 2019-03-21T09:57:08Z

Failing test in Travis is unrelated.

ph · 2019-03-21T19:32:13Z

libbeat/processors/actions/copy_fields.go

+		err := f.copyField(field.From, field.To, event.Fields)
+		if err != nil && f.config.FailOnError {
+			errMsg := fmt.Errorf("Failed to copy fields in copy_fields processor: %s", err)
+			logp.Debug("copy_fields", errMsg.Error())


Could we have a selector for logger? I like what @andrewkroh's did in the script processor, see

beats/libbeat/processors/script/javascript/config.go

Line 28 in 6865403

Tag string `config:"tag"` // Processor ID for debug and metrics.

And

beats/libbeat/processors/script/javascript/session.go

Line 33 in 6865403

logName = "processor.javascript"

TBH in case of this simple processor, I don't see the added value of letting a user set a debug selector. I doubt that users would need to filter for a specific copy processor in their logs, or if they have to, I think messages let then identify the processor easily.

If we want to let user set a selector, it would be much better to let them configure it for all processors. There are other processors which would benefit more from having such setting than this one. What if I open an issue and implement the functionality for all processors in a follow up PR?

I am OK with a followup PR.

ph

@kvch small issue with the changelog but LGTM

ph · 2019-03-22T12:09:40Z

CHANGELOG.next.asciidoc

@@ -257,6 +257,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add ip fields to default_field in Elasticsearch template. {pull}11035[11035]
 - Gracefully shut down on SIGHUP {pull}10704[10704]
 - Add `script` processor that supports using Javascript to process events. {pull}10850[10850] {pull}11260[11260]
+- Add `script` processor that supports using Javascript to process events. {pull}10850[10850]
+- New processor: `copy_fields`. {pull}11303[11303]
+- Add error.message to events when `fail_on_error` is set in `rename` and `copy_fields` processors. {pull}11303[11303]


problems with the rebase?

andrewkroh · 2019-03-22T19:21:40Z

@kvch Can you please make sure this new processor gets added to the documentation.

A new processor is introduced as part of support for keeping orignal messages. Options and naming follows the convention of other processors. ### `copy_fields` This processor copies one field to another. Example configuration is below: ```yaml processors: - copy_fields: fields: - from: message to: event.original fail_on_error: false ignore_missing: true ```

cwurm · 2019-09-17T14:52:53Z

@kvch can you add the docs when you have a minute? I was just looking for this functionality and couldn't find it.

zez3 · 2022-09-12T06:43:04Z

@kvch or @ruflin
Does this copy_fields processor actually append to an existing field(value) as array or fails on any error?

@dedemorton
It's not clear from the documentation if that happens or not.
Can someone plz fix that? Thx

dedemorton · 2022-09-12T17:28:15Z

@zez3 Please open an issue to report Beats documentation problems. Comments on old PRs are likely to get lost. Thanks!

dedemorton · 2022-09-12T17:29:50Z

@zez3 Also...forgot to mention that you can ask questions and search for answers in the discuss forum.

zez3 · 2022-09-14T17:23:43Z

@dedemorton
I did searched the discussion forum but most of the proposed workarounds for append@beat level are using JS script or ES pipelines.
Ideally the copy_fields documentation should state that this processor does not append anything and that it actuality replaces the value(s).

I've opened a new 33088 Issue

dedemorton · 2022-09-20T23:13:24Z

Thanks for creating the issue! I've added it to our project queue.

kvch added review libbeat labels Mar 19, 2019

kvch requested review from a team as code owners March 19, 2019 10:23

kvch requested review from ruflin and ph and removed request for a team March 19, 2019 10:23

kvch force-pushed the feature-libbeat-new-copy-processor branch from a65f82e to 30cfee1 Compare March 19, 2019 10:27

kvch mentioned this pull request Mar 19, 2019

New processor: truncate_fields #11297

Merged

ruflin reviewed Mar 19, 2019

View reviewed changes

kvch force-pushed the feature-libbeat-new-copy-processor branch from c7fc1c2 to fe47eba Compare March 19, 2019 12:50

ph suggested changes Mar 20, 2019

View reviewed changes

libbeat/processors/actions/copy_fields_test.go Outdated Show resolved Hide resolved

ruflin reviewed Mar 21, 2019

View reviewed changes

ph reviewed Mar 21, 2019

View reviewed changes

kvch added 12 commits March 22, 2019 10:03

add copy_fields processor

7c56114

add changelog entry

4e4da1f

address review notes

215ebcb

add error under key error.message when fail_on_error is set

69f555f

address more review notes

ec97c65

add changelog entry

207273e

update config references

c3cd832

update error messages

00dd7eb

rm debug prints

56d468a

add one more realistic test case

c19dc47

use fmt.Errorf

bba6555

add more test cases

4d3c2f5

kvch force-pushed the feature-libbeat-new-copy-processor branch from f4579e3 to 4d3c2f5 Compare March 22, 2019 09:03

ph approved these changes Mar 22, 2019

View reviewed changes

fix changelog

185f8cc

kvch merged commit 4b9f945 into elastic:master Mar 22, 2019

kvch added the needs_docs label Mar 24, 2019

This was referenced Apr 2, 2019

Keep original messages in case of Filebeat modules #8448

Closed

Filebeat modules: keep raw message #8083

Closed

Add configuration option to keep raw JSON messages #9172

Closed

kvch mentioned this pull request Apr 11, 2019

Feature Request: Copy Processor #7987

Closed

andrewkroh mentioned this pull request Jun 21, 2019

Missing docs for copy_fields and truncate_fields processors #12633

Closed

This was referenced Sep 10, 2019

keep_original is not documented #13574

Closed

Keep message field intact when using modules #8950

Closed

dedemorton removed the needs_docs label Nov 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New processor: copy_fields #11303

New processor: copy_fields #11303

kvch commented Mar 19, 2019

ruflin left a comment

ruflin Mar 21, 2019

kvch Mar 21, 2019

ruflin Mar 22, 2019

ruflin Mar 21, 2019

kvch Mar 21, 2019

ruflin Mar 22, 2019

kvch Mar 22, 2019

kvch commented Mar 21, 2019

ph Mar 21, 2019

kvch Mar 22, 2019 •

edited

Loading

ph Mar 22, 2019

ph left a comment

ph Mar 22, 2019

andrewkroh commented Mar 22, 2019

cwurm commented Sep 17, 2019

zez3 commented Sep 12, 2022

dedemorton commented Sep 12, 2022

dedemorton commented Sep 12, 2022

zez3 commented Sep 14, 2022 •

edited

Loading

dedemorton commented Sep 20, 2022

New processor: copy_fields #11303

New processor: copy_fields #11303

Conversation

kvch commented Mar 19, 2019

copy_fields

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Mar 21, 2019

Choose a reason for hiding this comment

kvch Mar 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Mar 22, 2019

cwurm commented Sep 17, 2019

zez3 commented Sep 12, 2022

dedemorton commented Sep 12, 2022

dedemorton commented Sep 12, 2022

zez3 commented Sep 14, 2022 • edited Loading

dedemorton commented Sep 20, 2022

`copy_fields`

kvch Mar 22, 2019 •

edited

Loading

zez3 commented Sep 14, 2022 •

edited

Loading