Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large Utf8 not supported in DuckDB data accelerator #1481

Closed
Sevenannn opened this issue May 22, 2024 · 1 comment
Closed

Large Utf8 not supported in DuckDB data accelerator #1481

Sevenannn opened this issue May 22, 2024 · 1 comment
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@Sevenannn
Copy link
Contributor

Describe the bug
Large Utf8 not supported in DuckDB data accelerator due to the lack of support in duckdb-rs, will require upstream fix to

To Reproduce
Use the folllowing spicepod.yaml config:

from: s3://spiceai-demo-datasets/taxi_trips/2024/
name: taxi_trips
description: taxi trips in s3
acceleration:
  engine: duckdb
  enabled: true
  refresh_check_interval: 10s

Error log shown:
2024-05-22T00:06:46.964431Z WARN runtime: Unable to attach data connector s3: Unable to create dataset acceleration: Acceleration creation failed: Unable to create table: External error: Unable to create duckdb table: Invalid Input Error: column Field { name: "store_and_fwd_flag", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } is not supported yet, please file an issue https://github.com/wangfenjin/duckdb-rs

Expected behavior
Large Utf8 is supported in duckdb-rs and spice duckdb data connector

@Sevenannn Sevenannn added the kind/bug Something isn't working label May 22, 2024
@digadeesh digadeesh added this to the v0.14.1-alpha milestone Jun 17, 2024
@Jeadie Jeadie assigned Jeadie and unassigned phillipleblanc Jun 19, 2024
@Jeadie
Copy link
Contributor

Jeadie commented Jun 19, 2024

Possible fix would be spiceai/duckdb-rs#1

Issue is that we then create possible type incompatibilities. i.e. when we return it from duckdb, we have no way of figuring out if we should

  1. Text -> UTF8
  2. Text -> LargeUTF8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants