-
Notifications
You must be signed in to change notification settings - Fork 3k
Closed
Description
We have upgraded the flink version to 1.11, and flink 1.11 have turned its Row data type to RowData. While the previous parquet/avro readers writers we developed were based on Row type, now @JingsongLi have contributed the RowData avro reader and writer (#1232), @chenjunjiedada is helping to contribute the RowData parquet reader(#1266) and writer (#1272), and I've pushed a RowData orc reader and writer (#1255) for reviewing.
IMO, we'd better to replace the Row with RowData in the flink module as soon as possible, so that we could unify all the path and put all the resources (both developing and reviewing resources) on RowData path. My plan is:
- As the patch (Implement the flink stream writer to accept the row data and emit the complete data files event to downstream #1145) about flink IcebergStreamWriter has been reviewed and is ready to merge now, so we let this patch get into master branch firstly.
- The flink TaskWriter unit tests are running based on
Rowpartition key, before turning toRowDatawe need to implementRowDatapartition key firstly. So I prepared the patchRowDataWrapper(Flink: support to RowData partition. #1299). Get this patch merged is the second step. - We will need an extra patch doing the refactor to replace all the
Rowtype withRowData(I have implemented one in my own branch 2af37c5), and make sure all the unit tests could pass. From this point in time, all flink development and unit tests will useRowData. - The future RowData parquet/orc reader and writer will be added in the
TaskWritertests.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels