flatten processor: option for keys wihout brackets

[timo-mue]

Is your feature request related to a problem? Please describe.
Currently the flatten processor adds empty squared brackets to the keys of list elements, even when the remove_list_indices option is set to true. So something like

{
  "key1": {
    "key2": [
      {
        "key3": "value1",
        "key4": "value2"
      },
      {
        "key3": "value3",
        "key4": "value4"
      }
    ]
  }
}

turns into

{
  "key1.key2[].key3": ["value1", "value3"],
  "key1.key2[].key4": ["value2", "value4"]
}

When the mentioned option is set to false the brackets are needed to reference list indices but in this case they could easily be ommited. It would be nice if the keys would keep their original names and the output would be this instead:

{
  "key1.key2.key3": ["value1", "value3"],
  "key1.key2.key4": ["value2", "value4"]
}

Describe the solution you'd like
Either remove the brackets from keys when remove_list_indices is set to true, or add an option (original_key or remove_brackets) that has the same effect when used together with remove_list_indices set to true.
Additional context
For some automated processes (e.g. creating mappings via index templates before writing documents) it could be valuable to retain the original key names.

[oeyh]

This is a reasonable ask and should be a small change.

[oeyh]

@timo-mue would you like contribute to Data Prepper and make the changes for this issue? I can also help.

The related code is here:

data-prepper/data-prepper-plugins/flatten-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/flatten/FlattenProcessor.java

Line 118 in ad92aa2

private Map<String, Object> removeListIndicesInKeys(final Map<String, Object> inputMap) {

The configs are set up in FlattenProcessorConfig.java

[oeyh]

There is a use case (#3965) to keep the brackets after removing list indices, so we want to retain that option.

We can add an option for removing the indices along with the brackets. Either through:

remove_list_indices: true
remove_brackets: true

or just:

remove_list_indices_and_brackets: true

[timo-mue]

@timo-mue would you like contribute to Data Prepper and make the changes for this issue? I can also help.

Sure, I will try to make the changes.

We can add an option for removing the indices along with the brackets. Either through:

remove_list_indices: true
remove_brackets: true

What would be the behavior for using remove_list_indices: false together with remove_brackets: true? Should that turn something like "key1.key2[0].key3": "value1" into "key1.key20.key3": "value1" or just get ignored?

[oeyh]

Thanks @timo-mue!

We should probably disallow that case by adding a validation (see an example here) so that when remove_list_indices is false, remove_brackets cannot be true.

Or, we add the option named remove_list_indices_and_brackets to bind them together.