Rate limiting feature for structured streaming

[namrathamyske]

Apache Iceberg version

main (development)

Query engine

Spark

Please describe the bug 🐞

In rate limiting for structured streaming PR - #4479.
According to line https://github.com/apache/iceberg/pull/4479/files#diff-26782bf5c27f69e5cc9cd4a9363f601a97d1c9f97fe0c1a7fb927da7c60c014fR169 unit test, it says the stream get stuck if number of records exceed SparkReadOptions.STREAMING_MAX_ROWS_PER_MICRO_BATCH. It's is a major blocker to consume this feature. If the stream is stuck, then no further advancement of stream takes place even if new snapshots comes in.

E.g.:
STREAMING_MAX_ROWS_PER_MICRO_BATCH - 2
Snapshot1 - (2 records, 1 data file) - Read fully in Microbatch-1
Snapshot2 - (3 records, 1 data file) - Can never be read as 3 records > STREAMING_MAX_ROWS_PER_MICRO_BATCH ( Stuck forever )
Snapshot3 - 3 records
Please let me know if this is intended behavior or is it expected to change.

@singhpk234 @jackye1995 @RussellSpitzer @rdblue

[RussellSpitzer]

Yes I believe the smallest unit that is allowed to be streamed is a single data file. So the rate limit must be larger than the largest possible data file

[singhpk234]

+1 to @RussellSpitzer's point, presently we don't read a file partially thus making file the smallest unit to be streamed, thus we should atleast make sure that num of records per microbatch > largest data file.

[namrathamyske]

Thanks for responding. @RussellSpitzer @singhpk234

[jackieo168]

Hi @RussellSpitzer and @singhpk234, just to follow up on this, since it's not always practical to determine/calculate the record count of the largest data file ahead of time and and adjust SparkReadOptions.STREAMING_MAX_ROWS_PER_MICRO_BATCH each time, is it not possible to instead load the full file in such scenarios?

Could you clarify the rationale behind keeping the stream forever stuck unless the SparkReadOptions.STREAMING_MAX_ROWS_PER_MICRO_BATCH is adjusted?

[wypoon]

@singhpk234 for my understanding, can you please confirm or refute the following --
Suppose streaming-max-rows-per-micro-batch = 1000 and streaming-max-files-per-micro-batch > 1. Suppose in a batch, two new files need to be read; one has 800 rows and the other has 700 rows. Then in this batch, either 800 or 700 rows will be streamed, depending on which file is read first. The rows from the other file will be streamed in the next batch. Am I correct?

[singhpk234]

yes, that true @wypoon, presently its the limitation as for the initial implementation,

I didn't wanted to block it on opening a file and reading it partially. As there were 2 options for me
1/ either open the file again read till row X and include more row till the rows in file is exhausted
2/ read the file totally once and keep the value in memory and hold it till asked

I started getting back on OSS recently, I will try address this limitation as well as add this pr Trigger.AvaliableNow() very soon !