Does not work with column mapping

[jtanx]

I don't actually know if this is a bug with this extension or in delta-kernel-rs (or maybe I'm doing something wrong?)

Test table created with pyspark:

import pyspark
from delta import *
import os


builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

spark = configure_spark_with_delta_pip(builder).getOrCreate()

spark.sql("""CREATE TABLE delta.`/tmp/delta_table` USING DELTA AS SELECT col1 as id FROM VALUES 0,1,2,3,4;""")
spark.sql("""
ALTER TABLE delta.`/tmp/delta_table` SET TBLPROPERTIES (
    'delta.minReaderVersion' = '2',
    'delta.minWriterVersion' = '5',
    'delta.columnMapping.mode' = 'name'
)
""")

spark.sql("""
ALTER TABLE delta.`/tmp/delta_table` RENAME COLUMN id TO id2
""")

Querying with duckdb

import duckdb
con = duckdb.connect()
print(con.query("select * from delta_scan('/tmp/delta_table')"))

returns

┌───────┐
│  id2  │
│ int32 │
├───────┤
│  NULL │
│  NULL │
│  NULL │
│  NULL │
│  NULL │
└───────┘

i.e. returns all nulls instead of the expected values

[jtanx]

From my brief look, it seems like this is missing the column mapping:

duckdb_delta/src/functions/delta_scan.cpp

Lines 727 to 731 in b3def68

	// FIXME: this override is pretty hacky: for missing columns we just insert NULL constants
	auto &global_type = global_types[global_id];
	Value val (global_type);
	reader_data.constant_map.push_back({i, val});
	continue;

(see also: delta-incubator/delta-kernel-rs#205)

I don't know if there's a better way, but the struct schema visitor also does not pass the mapping through:
https://github.com/delta-incubator/delta-kernel-rs/blob/d19b125519a5834377e6442f1ed73bdb344d0101/ffi/src/lib.rs#L865

(I think you can do field.physical_name(column_mapping_type))

[samansmink]

Thanks for reporting @jtanx! I will do some debugging and forward the bug report to the delta-kernel-rs repo if necessary

[jsum]

Hello, I am facing the same issue with delta_scan as well. If I use read_parquet, I get some random column names like "col-xxxxx" instead of actual columns names when the 'delta.columnMapping.mode' is set to 'name'. Is there a work around for this. Thanks for you help in advance.

[aersam]

Hello, I am facing the same issue with delta_scan as well. If I use read_parquet, I get some random column names like "col-xxxxx" instead of actual columns names when the 'delta.columnMapping.mode' is set to 'name'. Is there a work around for this. Thanks for you help in advance.

Well, this is by design. You could use our deltalake2db package as a workaround 😉

[samansmink]

We are still working on implementing column mapping, that currently does not work yet with duckdb_delta

[jsum]

Thanks @samansmink and @aersam.

[kmatt]

We are still working on implementing column mapping, that currently does not work yet with duckdb_delta

Is the root issue with delta-kernel-rs or DuckDB? Wondering if this is a solution, or changes to DuckDB will be need to incorporate it: delta-incubator/delta-kernel-rs#205

Similar to #11

[samansmink]

@kmatt The pr you link is currently not yet exposed in the ffi in a way DuckDB can use it. This is something we are working on though!