Feature request: Push to pubsub topic

[mari0d]

Would it be possible to add support for pushing log lines to a Google Pub/Sub topic?

[ruflin]

We are currently not planning to add more outputs (see discussion #581). For now you have to put Logstash in the middle.

[mari0d]

Ah, that is unfortunate - logstash doesn't write to pubsub. Will we need to fork libbeat if we add in pubsub support?

[urso]

There is no need to fork beats or libbeat for adding an output. Internally outputs/processors/... are plugins. One has two options for adding an output without forking:

1. use go plugins support. go is somewhat 'restrictive' on plugin loading. Best is to compile beats and the plugin from same workspace. The plugin can be loaded via CLI flags when the beat starts.
   See beats-sample-plugins by me. These still use the 5.x API, but for 6.x there is no big difference. The interface of an output has changed, but the 'symbol' exporting hasn't changed.

2. The single beats act mostly as framework. You can create your own beat based on the original beat and import your own output. The trick is to copy/adapt the original main.go and add an import statement to your own output like: _ "github.com/my/output/pubsub.

[jsvd]

Logstash doesn't currently have any supported plugin to write to pubsub, but there's a community made plugin https://github.com/NirKamara/logstash-output-google-cloud-pubsub

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.