feat: support CREATE OR REPLACE

[universalmind303]

Description

add support for SaveMode::Overwrite (CREATE OR REPLACE)

Related Issue(s)

closes #1689

Documentation

full working example:

use deltalake::{
    operations::create::CreateBuilder, protocol::SaveMode, DeltaOps, SchemaDataType, SchemaField,
};

fn get_table_columns() -> Vec<SchemaField> {
    vec![SchemaField::new(
        String::from("date"),
        SchemaDataType::primitive(String::from("date")),
        false,
        Default::default(),
    )]
}

fn get_table_columns_2() -> Vec<SchemaField> {
    vec![
        SchemaField::new(
            String::from("date"),
            SchemaDataType::primitive(String::from("date")),
            false,
            Default::default(),
        ),
        SchemaField::new(
            String::from("utf8"),
            SchemaDataType::primitive(String::from("utf8")),
            false,
            Default::default(),
        ),
    ]
}

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), deltalake::errors::DeltaTableError> {
    let ops = DeltaOps::new_in_memory();
    let object_store = ops.0.object_store();

    CreateBuilder::new()
        .with_object_store(object_store.clone())
        .with_columns(get_table_columns())
        .with_table_name("my_table")
        .with_comment("A table to show how delta-rs works")
        .await?;

    let table = CreateBuilder::new()
        .with_object_store(object_store.clone())
        .with_columns(get_table_columns_2())
        .with_save_mode(SaveMode::Overwrite)
        .with_table_name("my_table")
        .with_comment("A table to show how delta-rs works")
        .await?;

    let schema = table.get_schema().unwrap();
    assert!(schema.get_field_with_name("utf8").is_ok());
    Ok(())
}

[rtyler]

It looks like this pull request was opened in such a way that disallows edits by maintainers of the delta-rs repo, which means I cannot merge HEAD into the branch to merge it. You'll need to update the branch so we can merge it

[universalmind303]

It looks like this pull request was opened in such a way that disallows edits by maintainers of the delta-rs repo, which means I cannot merge HEAD into the branch to merge it. You'll need to update the branch so we can merge it

Ill take a look & make sure it doesn't happen on future PR's

[roeap]

Based on @ion-elgreco's question above if we can close that issue, I had a quick look at this PR and am not entirely sure it does what we want it to.

If I read that correctly, we are essentially using the current snapshot to pass to commit and are still committing the regular actions for create. This means we alter the table metadata / protocol actions. This however would not impact the files currently tracked in the table. So we still track the same data as before, just now it might be inconsistent with the new schema and we may also not be able to read it anymore due to config / table feature changes.

Not entirely sure what overwrite should look like, but we should either add Remove actions for all currently tracked files, which would increment the table version, or purge the entire log and start fresh at version 1?