Contributions and ideas

[vidosits]

Hey there!

I was wondering if you take contributions to the repo?

I tried this package in lieu of Debezium for the exact reasons mentioned in the README and I've found that it's quite great.
In our case we would want to stream the changes from our TimescaleDB instance to an external web service. We would like to avoid having to do any kind of stream processing (at least not downstream) and would want to avoid having to introduce additional dependencies to the stack of the project like redis or kafka just for being able to receive the changes.

For this reason I've extended our fork with an HttpSink type Sink which takes the envelope of the Event and makes a POST request when emitting. It's quite basic, currently it only supports POST-ing to an endpoint with either no auth, basic auth or with a configurable header and a value for the header.

I can make a PR with code + tests if that's something that would be interesting for you.

I've also found a handful of typos here and there, mostly benign, some of them just from copy-pasting elsewhere. I can make a PR of those as well, if you want of course.

I've also had another idea, I wonder what you think of it.

Right now the package emits the key/envelope of the event to the selected sink. If the payload format is not appropriate for a downstream system then you have to persist the emitted messages, transform them and then stream them further downstream.
For a stack that's using kafka/redis/etc. this may not be an issue at all, but for us for example we don't want to persist the "raw" messages emitted by the library just so we can iterate them again, transform and send them further downstream. So the idea would be to use the Go Plugin package so that plugins adhering to a well defined interface can transform the envelope before it gets sent to the sink for emitting.

I'm not sure what the best approach would be, but I was thinking of introducing plugins somewhere around here so that the user can either supply additional payloadFactories or somewhere around here so that a user may be able to transform the key/envelope before it being sent to a sink for emitting.

I'd be happy to make a prototype and PR for the above as well.

Thanks, great package!

[noctarius]

Hey @vidosits!

I absolutely take in contributions! I'm actually be honored to see people find it useful and want to see more use cases being available!

Please send a PR for all the small fixes and misspellings you've found and fixed. If you've written code, you won't find them yourself, ever 😅

For the HTTP Sink, you could either send a PR or you can already use the plugin system (https://github.com/noctarius/timescaledb-event-streamer/tree/main/spi/plugins). Same for the payload transformers. The latter ones are actually on my bucket list for a while, but since all downstream sinks use JSON for now, I never got around implementing those. However, they are planned anyways to support AVRO, Protobuf, or whatever people are looking for (for myself I'm a massive fan of CBOR!).

Feel free to start with an implementation. We can always cooperate building it out 💪

Finally, I'm glad you find it useful and even want to contribute 🙏

[vidosits]

Hey @noctarius!

Please send a PR for all the small fixes and misspellings you've found and fixed. If you've written code, you won't find them yourself, ever 😅

Don't worry, I know the feeling!

For the HTTP Sink, you could either send a PR or you can already use the plugin system

Thanks! I don't understand how I did not find the plugin system, I must be blind, but the important thing is is that You had the same idea and you've already implemented it.

I'm working on finishing the integration tests for the HTTP Sink in the meanwhile I've created a draft PR #67 , so that if I'm really messing up you can stop the whole thing :)

Same for the payload transformers

That's exactly what we need and I'm going to look into using the established plugin system.

Finally, I'm glad you find it useful and even want to contribute 🙏

Thanks so much for starting this project!