fix(rust): serialize MetricDetails from compaction runs to a string

[liamphmurphy]

I am by no means a Rust developer and haven't touched it in years; so please let me know if there's a better way to go about this. The Rust z_order and optimize.compact already serializes the metrics before it is passed back to Python, which then deserializes it back, so the Python behavior in terms of expecting this as a Dict has not changed which I think is what we want.

Description

Adds a custom serialzer and Display implementation for the MetricDetails fields, namely filesAdded and filesRemoved so that those fields are written as strings instead of a struct to the commit log. Query engines expect these fields to be strings on reads.

I had trouble getting the pyspark tests running locally, but here is an example optimize commit log that gets written with these changes:

{"commitInfo":{"timestamp":1711125995487,"operation":"OPTIMIZE","operationParameters":{"targetSize":"104857600","predicate":"[]"},"clientVersion":"delta-rs.0.17.1","readVersion":10,"operationMetrics":{"filesAdded":"{\"avg\":19956.0,\"max\":19956,\"min\":19956,\"totalFiles\":1,\"totalSize\":19956}","filesRemoved":"{\"avg\":4851.833333333333,\"max\":10358,\"min\":3734,\"totalFiles\":6,\"totalSize\":29111}","numBatches":6,"numFilesAdded":1,"numFilesRemoved":6,"partitionsOptimized":1,"preserveInsertionOrder":true,"totalConsideredFiles":6,"totalFilesSkipped":0}}}

Related Issue(s)

• After dt.optimize.compact(), commitInfo.operationMetrics.filesAdded is a JSON map when other readers (e.g. Databricks) expect a string #2087

Documentation

N/A

[ion-elgreco]

@liamphmurphy you could add a python test to test that DESCRIBE HISTORY now works in pyspark on a delta-rs compacted delta table

[liamphmurphy]

@liamphmurphy you could add a python test to test that DESCRIBE HISTORY now works in pyspark on a delta-rs compacted delta table

Good idea 👍 , just need to figure out why the spark tests aren't working locally.

[ion-elgreco]

@liamphmurphy are you adding the markers in pytest?

[liamphmurphy]

@liamphmurphy are you adding the markers in pytest?

Was a numpy error, had to reinstall it manually through pip

[liamphmurphy]

@ion-elgreco Pushed up some changes for the lint / cargo fmt errors. The pyspark integration tests pass locally after importing the DeltaTable so hopefully all good there. The last big one seems to be the benchmark showing a regression, will be slow to respond until the week.

[ion-elgreco]

Thanks @liamphmurphy!

In ideal world delta-spark would just be able to parse any commit info value to string, but this also works :D