Skip to content

Conversation

YaroslavLitvinov
Copy link
Contributor

@YaroslavLitvinov YaroslavLitvinov commented Feb 10, 2025

Closes #115

It has no breaking changes as "json" format is not replaced by "arrow" yet, as of known issues in arrow support.
This PR adds partial arrow support, that can be switched on using SERIALIZATION_FORMAT const.
In this PR UInt64 datatype replaced by Int64 in count batch of output schema.
Following queries return count batch that being fixed:

CREATE TABLE
CREATE SCHEMA
CREATE DATABASE
INSERT, UPDATE, DELETE

Datafusion repo updated as a part of fix:
Embucket/datafusion#17

Hash commit of change in datafusion propagated in repos too:
Embucket/datafusion-functions-json#4
Embucket/iceberg-rust#14

@YaroslavLitvinov YaroslavLitvinov added the enhancement New feature or request label Feb 10, 2025
@YaroslavLitvinov YaroslavLitvinov self-assigned this Feb 10, 2025
Copy link
Contributor

@DanCodedThis DanCodedThis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we go from UInt64 to Int128? Otherwise, could be potentially unsafe, since the maximum number of UInt64 is not the maximum number of Int64

@YaroslavLitvinov
Copy link
Contributor Author

Shouldn't we go from UInt64 to Int128? Otherwise, could be potentially unsafe, since the maximum number of UInt64 is not the maximum number of Int64

For result count it makes no sence.

@YaroslavLitvinov YaroslavLitvinov force-pushed the issues/115_switch_to_arrow_ipc branch from bea7a8f to afaa30a Compare February 11, 2025 18:44
@YaroslavLitvinov YaroslavLitvinov changed the title Enables arrow format for dbt/snowflake endpoint Add partial support of arrow format for dbt/snowflake endpoint Feb 12, 2025
@YaroslavLitvinov YaroslavLitvinov merged commit fc752a7 into main Feb 12, 2025
5 checks passed
@YaroslavLitvinov YaroslavLitvinov deleted the issues/115_switch_to_arrow_ipc branch February 12, 2025 00:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api: switch from JSON data type to ARROW IPC

3 participants