[Functionbeat] Index multiple individual events from Kinesis events array

[bczifra]

Describe the enhancement:
Kinesis batches events together into a single array of events. These might look like this:

{
    "owner": "111111111111",
    "logGroup": "CloudTrail",
    "logStream": "111111111111_CloudTrail_us-east-1",
    "subscriptionFilters": [
        "RecipientStream"
    ],
    "messageType": "DATA_MESSAGE",
    "logEvents": [
        {
            "id": "3195310660696698337880902507980421114328961542429EXAMPLE",
            "timestamp": 1432826855000,
            "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}"
        },
        {
            "id": "3195310660696698337880902507980421114328961542429EXAMPLE",
            "timestamp": 1432826855000,
            "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}"
        },
        {
            "id": "3195310660696698337880902507980421114328961542429EXAMPLE",
            "timestamp": 1432826855000,
            "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}"
        }
    ]
}

This enhancement request is to introduce functionality to allow Functionbeat to output each of the entries in the array as individual, separate events. That is, this single source event from Kinesis should generate three separate events that could then be JSON parsed and sent to Elasticsearch for processing.

Describe a specific use case for the enhancement or feature:
One example of when this would be useful is for parsing Cloudwatch logs, as shown in the example JSON above. Though Functionbeat does have a cloudwatch function, there may be times where it doesn't have direct access to Cloudwatch but does have access to a Kinesis stream.

[acchen97]

@bczifra thanks for filing! I'd like to confirm here, is this Cloudwatch or Cloudtrail data?

[bczifra]

@acchen97 Cloudwatch

[Inbloo]

Same problem here. Is someone working on this? Any workarround?

[ppf2]

This should be resolved in #13317 (thx!). However, we still have an outstanding task to update the documentation, and backport to the version when we first added this function to include information on this new function and to mark it as experimental (for now). @dedemorton

[dedemorton]

Hi my apologies for missing this notification (it's been awhile). I've created an issue here to track the doc addition: #20042

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.