Parquet Incremental Sync#768

Open

sapienza88 wants to merge 38 commits intoapache:mainfrom

sapienza88:parquet_incr_sync

Commits on Dec 10, 2025

given a parquet file return data from a certain modification time
Selim Soufargi
committed

Commits on Dec 13, 2025

create the path based on the partition then inject the file to append into the parquet table
Selim Soufargi
committed

Commits on Dec 14, 2025

Handle case of path construction with file partitioned over many fields, interfacing with ConversionSource
Selim Soufargi
committed
test append Parquet file into table init
Selim Soufargi
committed

Commits on Dec 15, 2025

add function to test schema equivalence before appending
Selim Soufargi
committed

Commits on Dec 16, 2025

construct path to inject to based on partitions
Selim Soufargi
committed
fix imports
Selim Soufargi
committed

Commits on Dec 17, 2025

refactoring (lombok, logs, javadocs and function and approach commenting)
Selim Soufargi
committed

Commits on Jan 1, 2026

use appendFile to append a file into a table while tracking the appends using the FileStatus' modifTime attribute
Selim Soufargi
committed
find the files that satisfy to the time condition
Selim Soufargi
committed
treat appends as separate files to add in the target partition folder
Selim Soufargi
committed

Commits on Jan 2, 2026

update approach: selective block compaction
Selim Soufargi
committed
update approach: added a basic test to check data selection using modificationTime selector
Selim Soufargi
committed
fix append based on partition value
Selim Soufargi
committed
fix test with basic example where partitions are not considered
Selim Soufargi
committed
fix test with basic example where partitions are not considered2
Selim Soufargi
committed
fix test with basic example where partitions are not considered3
Selim Soufargi
committed

Commits on Jan 3, 2026

test with time of last append is now
Selim Soufargi
committed
test appendFile with Parquet: TODO test with multiple partitions 1) append and 2) filter for sync
Selim Soufargi
committed
merge recursively one partition files
Selim Soufargi
committed
fix paths for files to append
Selim Soufargi
committed
fix bug of appending file path
Selim Soufargi
committed
fix bug of schema
Selim Soufargi
committed
make sure returned files are non empty when iterating in the partitions
Selim Soufargi
committed
make sure returned files are non empty when iterating in the partitions 2
Selim Soufargi
committed
discard empty files when appending
Selim Soufargi
committed
discard empty files when appending
Selim Soufargi
committed
one spark session for both tests
Selim Soufargi
committed
bug fix
Selim Soufargi
committed
bug fix
Selim Soufargi
committed
cleanups + TODO: partitions match and merge
Selim Soufargi
committed

Commits on Jan 4, 2026

added test for many partitions, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed
selecting data bug fix, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed
run all tests, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed
run all tests, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed
spotless:apply, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed
bug fixes, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed

Commits on Jan 5, 2026

bug fixes, TODO integrate functions into ParquetConversionSource
Selim Soufargi
committed