Iceberg sink for Apache Flink

[bowenli86]

Users should be able to have their Flink apps write into Iceberg directly via a sink to leverage Iceberg as the table format for their near-real-time or even real time data lake/warehouse.

Background: Apache Flink is the most popular and powerful stream processing engine nowadays, and has been widely adopted in companies like Uber, Lyft, Netflix, etc.

[bowenli86]

Hi @rdblue , I wonder if this is something that the iceberg community will consider?

[rdblue]

Yeah, we (Netflix) have a prototype sink from Flink to Iceberg that we use. I think it's a great idea to add an open one; we've been meaning to clean up ours if we can to get it upstream. Would you want to add a sink in Flink or in Iceberg?

[bowenli86]

I'd like to do it in Iceberg first. When the Iceberg project graduates, I can help evaluate merging it back to Flink. What do you think?

Would you prefer to use your internal one, or develop a new one from scratch (which I can help with)? Either way, it would be great if you can share some code snippets of the prototype sink early to gather some feedback and ideas.

[rdblue]

I don't think that it makes sense to keep code in one repository or another based on incubation status. I'm wondering how difficult it would be to maintain the code here vs in Flink. Does Flink have a stable connector API for plugging in sinks?

[bowenli86]

Sorry for the confusion. I was not suggesting incubating/graduating as an argument, but only an arbitrary time in the future to reevaluate the situation.

I think we better do it in iceberg at least for now because 1) lt potentially can take longer time (google pubsub connector took ~a year, pulsar connector took ~1.5 y and hasn't get in yet) in Flink to reach consensus, write FLIP (Flink improvement proposal) and get it reviewed, etc, and I don't know if someone wants to drive that 2) AFAIU, Flink tends to have less connectors in its repo (but show them on Flink website) to reduce build/test time 3) iceberg itself can potentially evolve faster as an emerging tech and it can be easier to make frequent changes in itself

In a nutshell, I feel it's much easier and more efficient if we do it in iceberg, similar to iceberg-spark, and help the community to grow.

[jerryshao]

Just curious about the status of Flink connector. Iceberg requires lots of things exposed from connector, so that Iceberg could leverage it.

[stevenzwu]

As for open sourcing Flink Iceberg sink that @rdblue mentioned, it seems that we probably can only get to it on later Q1 or early Q2 due to resource constrains. Need to figure out how to untangle the Netflix specific logics. Goal is to make the open-source Flink sink connector generic and extensible

[aokolnychyi]

Here is a relevant thread on the dev list for anybody interested in this work.

[bowenli86]

@stevenzwu look forward to that. Thanks!

[waterlx]

Working on porting/moving the code provided as nfflink-connector-iceberg to master. Hopefully could make it done and upload a PR in the middle of next week. It might be useful if anyone would like to get a simple working version of Flink Iceberg sink for PoC. @stevenzwu @bowenli86 @rdblue FYI

[bowenli86]

@waterlx awesome, thanks! Let us know if you need any help.

[openinx]

Thanks @waterlx for your work, would take a look if the PR is prepared.

[waterlx]

Just would like to share some progress:

I have moved the source code part of nfflink-connector-iceberg into a dedicate project (iceberg-flink) under Iceberg, also made some corrections and improvements, with most of the checkstyle issues addressed.

It works (DataFiles generated and commit made as expected) but is still an immature version.
We also found some bugs (such as duplicate records) as well as some enhancments good to have(such as if we could remove the usage of Avro IndexedRecord as the intermediate format).

Plan to upload a WIP PR in the next few days (hopefully before the end of this week) to show the current workable version.
FYI @stevenzwu @bowenli86 @openinx @aokolnychyi @chenjunjiedada @rdblue