feat: arrow based table state and checkpoint handling

[roeap]

this is based on #1829 which is based on #1807

Description

Allows for reading and writing checkpoints more in line with current spec and updates the internal representation of the table state to use Arrow record batches. This is mainly in pursuit of trying to be more aligned with the ongoing work in kernel as well as moving our protocol support forward.

Still very much a work in progress, but opening it for feedback and discussion.

As a follow up I am planning on moving some of the updated code back to kernel.

Related Issue(s)

supersedes: #454
closes: #1776
closes: #425 (should also be addressed in the current implementation)
closes: #288 (multi-part checkpoints are deprecated)
related: #1598 (maybe we can even close this)
related: #435

we may also want to consider closing: #125

Documentation

On order to retain the parquet2 functionality and ease the transition I defined a Snapshot trait, that looks similar to a kernel snapshot. The new arrow backed state as well as the existing DeltaTableState implement this trait.

/// A snapshot of a Delta table at a given version.
pub trait Snapshot: std::fmt::Display + Send + Sync + std::fmt::Debug + 'static {
    /// The version of the table at this [`Snapshot`].
    fn version(&self) -> i64;

    /// Table [`Schema`](crate::kernel::schema::StructType) at this [`Snapshot`]'s version.
    fn schema(&self) -> Option<&StructType>;

    /// Table [`Metadata`] at this [`Snapshot`]'s version.
    fn metadata(&self) -> DeltaResult<Metadata>;

    /// Table [`Protocol`] at this [`Snapshot`]'s version.
    fn protocol(&self) -> DeltaResult<Protocol>;

    /// Iterator over the [`Add`] actions at this [`Snapshot`]'s version.
    fn files(&self) -> DeltaResult<Box<dyn Iterator<Item = Add> + '_>>;

    /// Well known table [configuration](crate::table::config::TableConfig).
    fn table_config(&self) -> TableConfig<'_>;
}

log replay / v2 checkpoints

• handle compaction actions
• parse checkpoints with v2 metadata (CheckpointMetadata and Sidecar actions)
• add a new log path abstraction to set us on a path for supporting urls in delta log.

pub enum LogPath {
    ObjectStore(Path),
    Url(Url),
}

Usage is still fairly contained to the inner parts of delta-rs. Main usage right now is providing helpers for the various kinds of files in the log and using URLs will mostly just error. Nonetheless i felt it advantageous to start modelling this.

timestamps (moved to follow-up PR)

non surprisingly, changes to timestamp handling propagate quite a bit, so better to do a follow up. Keeping the notes for reference.

The delta protocol states, that timestamp types should be written to parquet with the isAdjustedToUtc flag set to true. Right now delta-rs does not honour this. Right now I opted for:

• converting delta timestamps to arrow timestamps with tz="UTC"
• allow for converting arrow timestamps with utc to delta timestamps (previously only timestamps with tz = None were parsed)
• maintaining the behaviour of converting arrow timestamps with tz = None to delta timestamps until we implement timestampNtz.

[houqp]

Really exciting work, glad to see this finally happen! Any early memory consumption benchmark results to share?

[roeap]

superseded by #2037