[meta] Implement the kafka output

[jlind23]

Describe the enhancement:
Bringing the Elastic Agent more in line with outputs supported by Beats.

Describe a specific use case for the enhancement or feature:
Enable customers who are using beats to send events/logs to a Kafka broker to be able to create the same environment and functionality using the Elastic Agent. Lack of this capability may be an inhibitor for the adoption of Elastic Agent.

Tasks to complete:

• Initial implementation Kafka Output #147
• Kafka output should support codecs #165
• Add an integration test for the Kafka output #166
• Kafka output topic selection should match that from beats output #167

Further related tasks

• Make Kerberos support GA for Kafka output #168
• Add "real" selector support for kafka output #203
• The shipper should report its output unit as DEGRADED when it cannot publish events to Kafka #230
• Performance test the shipper Kafka output #232

[amitkanfer]

@jlind23 what's left for this issue?

[jlind23]

Please @cmacknz @leehinman keep me honest here but the remaining work is related to the two issues mentioned + the UI work that needs to be done: elastic/kibana#143324

[amitkanfer]

@jlind23 if indeed this is the case and these 2 issues are blockers - can we assign them to a sprint? (The UI work is scheduled for sprint 9)

[jlind23]

@amitkanfer done, thanks for the heads up.

[pjaak]

Is there any ETA on the release of the Kafka output?

[jlind23]

@pjaak No concrete ETA for now but probably by the end of the summer.

[chais0n]

In an effort to further validate the requirement for this..

We have a large premium customer that is waiting for this functionality to be available in order to adopt Elastic Agent. All existing data collected with Beats are sent to Kafka. Without that functionality they are unable to move fully to Agent.

[cmacknz]

Thanks for confirming, we are actively working to support this soon. It's a very frequent ask and a known blocker for adopting agent.

[mbarretta]

Nudge to +1 (more like +10) the urgency of this feature

[cmacknz]

It is being worked on, but has been decoupled from the development of the shipper. Follow elastic/kibana#143324 instead.

[dijkstrajesse]

I can see the UI part has been finished which already is availabe in the 8.10 SNAPSHOT in staging, but when is the actual output work going to be finished? Adding my +1 for many customers as well.

[jlind23]

@dijkstrajesse The output and the UI part are going to be available all together.

[dijkstrajesse]

@jlind23 Okay thanks for confirming. Is the Elastic Agent 8.10 (SNAPSHOT) binary available in the artifacts repo? It doesn't seem so?

[dijkstrajesse]

Apologies, accidentally hit close.

[jlind23]

@dijkstrajesse not in the public repo but should be available in the snapshot one.

[essodjolo]

Hi @jlind23. Does #116 (comment) mean that we can expect Kafka output in 8.10?

[dijkstrajesse]

@essodjolo Yes. However, I tried and got the following error with the following Elastic Agent version: https://www.docker.elastic.co/r/beats/elastic-agent:8.10.0-SNAPSHOT.

failed to prepare output \“default\“:: encountered unexpected output type while preparing outputs: kafka

Am I missing something or can you confirm this is a bug?

[jlind23]

@criamico @kevinlog do you know if the kafka tests already began?
Cc @cmacknz for awareness

[kevinlog]

@jlind23

do you know if the kafka tests already began?

@criamico and @szwarckonrad did some API level testing with real Agents.

failed to prepare output \“default\“:: encountered unexpected output type while preparing outputs: kafka

Does the above error look familiar? It looks like the Agent doesn't recognize the the output at all.

[jlind23]

Which is weird if you tell me that @criamico and @szwarckonrad were able to conduct tests with real agents. Might be a setup problems on @dijkstrajesse's end.
What's the kibana version you used to prepare this? Your entire stack was running 8.10.0-SNAPSHOT?

[kevinlog]

@jlind23 - I followed back up with @szwarckonrad - I was incorrect. The testing that was done was at the API integration level, but not the actual Agent, so we may have a bug here.

here is a gist which shows an example Agent policy with Kafka output. Is this what Agent would expect?

We will set up a cloud instance we can use for integration testing. I think it will be best for us to work with someone from Agent team to stand up the Agent and connect with the kafka instance and we can see what the problem is. cc @cmacknz

failed to prepare output \“default\“:: encountered unexpected output type while preparing outputs: kafka

Again, this error makes it seem like the Agent doesn't recognize the output at all, but I'm not sure.

Also reopening elastic/kibana#143324 so that we can perform the e2e. Apologies for the miscommunication on my end.

[faec]

Hi from the Agent team :-)

That particular error, "encountered unexpected output type while preparing outputs: kafka", is actually from Fleet Server not Agent (see https://github.com/elastic/fleet-server/blob/87d618e58d390bc272ed7bb6605488f41a59cc28/internal/pkg/policy/policy_output.go#L59), it looks like a new output type case needs to be added there.

Once Fleet lets the policy go through you will probably see a similar error on the Agent side until you update the appropriate spec files to list kafka as a valid output type. (Here are spec file docs but in this case it is likely literally as simple as adding "kafka" to the outputs field on any inputs that need to support it -- it looks like this has already been done for some inputs.)

I've worked with the Kafka output and also with Agent spec files / output configuration, feel free to message me with questions (and feel free to ping on Slack since my github notifications latency is higher).

[nimarezainia]

@essodjolo @dijkstrajesse (and others) I do appreciate that there is a lot of interest in this feature for agent (we also have a list of beta customers to reach out to). Could you please hold off on communicating 8.10 to any of our users until we have enough testing under our belts. Appreciate it.