Error optimizing large table

[cmackenzie1]

Environment

Delta-rs version: 0.12

Binding: rust

Environment:

• Cloud provider:
• OS:
• Other:

---

Bug

What happened:

Issuing optimize on a large table failed with

called `Result::unwrap()` on an `Err` value: Generic("Expected writer to return only one add action")
thread 'tests::test_optimize' panicked at 'called `Result::unwrap()` on an `Err` value: Generic("Expected writer to return only one add action")', src/main.rs:312:71

What you expected to happen:

Optimize to succeed

How to reproduce it:

Call DeltaOps::from(table).optimize().await.unwrap(); on a table where the writer buffer would be filled when reading part of a source file.

More details:

Table consists of 37040 commits and ~75k files.

[cmackenzie1]

delta-rs/rust/src/operations/optimize.rs

Lines 325 to 330 in b17f286

	if add_actions.len() != 1 {
	// Ensure we don't deviate from the merge plan which may result in idempotency being violated
	return Err(DeltaTableError::Generic(
	"Expected writer to return only one add action".to_owned(),
	));
	}

[cmackenzie1]

From first glance, if the writer buffer exceeds the target_file_size, it gets flushed to storage, increasing the number of Add operations, which is throwing this error.

Is the check add_actions.len() != 1 actually helping here or is that not actually needed?

[roeap]

Without having checked the code, in principle the bin packing should bin files such that they yield a single resulting file. My guess would be that either we have a bug in in the binning logic, or written file size is not as predictable as assumed..

Are we e.g. using the same compression during initial write and optimization?

[wjones127]

I think the problem is we don't set compression by default when optimizing. Also written file size can be unpredictable.

[wjones127]

I'll try to address this in my PR #1383.

[cmackenzie1]

@wjones127 were you able to address this in the aforementioned PR or is it still outstanding?

[wjones127]

It will be addressed in that PR.

[cmackenzie1]

Closing this as it seems to be resolved in #1383