[Potential Bug] insertedRowMap too large could cause flink application failure?

[puchengy]

Apache Iceberg version

main (development)

Query engine

Flink

Please describe the bug 🐞

iceberg/core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java

Line 113 in 5d58750

private Map<StructLike, PathOffset> insertedRowMap;

When Flink using BaseEqualityDeltaWriter, wouldn't flink application crash if too many data in the insertedRowMap ?

[pvary]

You would like to compact the table to get rid of the delete files. RewriteDataFilesAction could help, or Spark compaction

[puchengy]

But it does not avoid the map getting too large and fail the application right?

…

On Fri, Mar 22, 2024 at 11:49 PM pvary ***@***.***> wrote: You would like to compact the table to get rid of the delete files. RewriteDataFilesAction could help, or Spark compaction — Reply to this email directly, view it on GitHub <#10030 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5S57ALE2GB2TNOBFU55I3YZUQWFAVCNFSM6AAAAABFEELEQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWGM3TSNRUGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[pvary]

Oh... sry, this is the writer side. My suggestion helps on reader side only.

On writer side you can increase your checkpointing period, migrate to a smaller primary key (introduce a new long id), or increase your task memory.

I have seen suggestions to add a new feature to store these deletes in RocksDB or state, but I think there were no decisions around it yet.

[manuzhang]

AFAIK, there are #2680 and RFC: Control flink upsert sink’s memory usage of insertedRowMap

[puchengy]

Thanks