Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more type support in arrow #6

Closed
wants to merge 25 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4f6535c
Organise workspace crates and consolidate Cargo.toml manifest propert…
Swoorup Jun 4, 2024
74fce0f
Update README.md (#242)
yoonghm Jun 4, 2024
f628e5a
Add support for DuckDB arrays when using Arrow's FixedSizeList (#323)
Jeadie Jun 4, 2024
4f772b3
Add support for BinaryArray in arrow-vtab (#324)
phillipleblanc Jun 5, 2024
f5da417
use openssl from runner (#327)
Maxxen Jun 6, 2024
3bd503c
Add ability to pass vendored feature to openssl in libduckdb-sys (#321)
muralisoundararajan Jun 6, 2024
f48a4e3
Implement appender for date/time types (#313)
rijkvp Jun 6, 2024
2fea269
Support decimal128 without casting to double (#328)
Maxxen Jun 6, 2024
fe9a3ab
feat: experimental support for further nested types (#318)
Mause Jun 8, 2024
c175a8a
upgrade arrow-rs to 52 (#332)
Mause Jun 8, 2024
c56e458
expose underlying schema type of statement (#333)
abhiaagarwal Jun 9, 2024
1c5e7cd
Remove mut from raw_query (#339)
rijkvp Jul 5, 2024
88dd455
Support Arrow type `LargeUtf8`. (#341)
Jeadie Jul 10, 2024
15ead11
Update duckdb-rs to DuckDB v1.0.0 (#336)
Mytherin Jul 10, 2024
983e02a
fix: don't write files outside of `OUT_DIR` (#347)
frectonz Jul 10, 2024
4d227f7
Remove built-in HTTPFS extension and fix Windows CI (#353)
Mytherin Jul 11, 2024
5996c3e
Upgrade duckdb-loadable-macros (#354)
Mytherin Jul 11, 2024
1c73aef
Move files out of vtab to core module (#359)
Swoorup Jul 20, 2024
f40058b
feat: set duckdb api to rust and add custom_user_agent config (#360)
hrl20 Aug 1, 2024
201a3d0
Rename `LogicalType` and `DataChunk` to have `Handle` suffix (#361)
Swoorup Aug 1, 2024
d12a5a2
add support of fixedsizebinary, duration, interval support in arrow
y-f-u Aug 23, 2024
44e0ff1
Fix CI (#375)
samansmink Aug 27, 2024
c5cbd6c
Merge branch 'duckdb:main' into add-more-type-support-in-arrow
peasee Aug 28, 2024
c72c2c2
chore: Cargo fmt
peasee Aug 28, 2024
52f04f0
fix: clippy::expect-fun-call
peasee Aug 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/.codecov.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
ignore:
- "libduckdb-sys/duckdb"
- "libduckdb-sys/src/bindgen_bundled_version.rs"
- "crates/libduckdb-sys/duckdb"
- "crates/libduckdb-sys/src/bindgen_bundled_version.rs"
coverage:
status:
project:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ jobs:
components: 'rustfmt, clippy'

# cargo publish
- uses: wangfenjin/publish-crates@main
name: cargo publish
- name: publish crates
uses: katyo/publish-crates@v2
with:
path: './'
args: --no-verify --allow-dirty --all-features
registry-token: ${{ secrets.CARGO_REGISTRY_TOKEN }}
ignore-published: true
ignore-unpublished-changes: true

# create release
- name: "Build Changelog"
Expand Down
12 changes: 4 additions & 8 deletions .github/workflows/rust.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
name: Download duckdb
with:
repository: "duckdb/duckdb"
tag: "v0.10.1"
tag: "v1.0.0"
fileName: ${{ matrix.duckdb }}
out-file-path: .

Expand Down Expand Up @@ -107,10 +107,6 @@ jobs:
runs-on: windows-latest
steps:
- uses: actions/checkout@v2
# - run: echo "VCPKG_ROOT=$env:VCPKG_INSTALLATION_ROOT" | Out-File -FilePath $env:GITHUB_ENV -Append
# - run: vcpkg install openssl:x64-windows-static-md
- run: choco install openssl --execution-timeout 5400
- run: echo 'OPENSSL_DIR=C:\Program Files\OpenSSL' | Out-File -FilePath $env:GITHUB_ENV -Append
- uses: actions/cache@v3
with:
path: ~/.cargo/registry/index
Expand Down Expand Up @@ -145,10 +141,10 @@ jobs:
# as the other tests have them.
RUST_BACKTRACE: "0"
run: cargo -Z build-std test --features "modern-full extensions-full" --target x86_64-unknown-linux-gnu
- uses: wangfenjin/publish-crates@main
name: cargo publish --dry-run
- name: publish crates --dry-run
uses: katyo/publish-crates@v2
with:
path: './'
args: --allow-dirty --all-features
dry-run: true
ignore-published: true
ignore-unpublished-changes: true
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,6 @@ Cargo.lock

*.db

libduckdb-sys/duckdb-sources/
libduckdb-sys/duckdb/
libduckdb-sys/._duckdb
crates/libduckdb-sys/duckdb-sources/
crates/libduckdb-sys/duckdb/
crates/libduckdb-sys/._duckdb
4 changes: 2 additions & 2 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[submodule "libduckdb-sys/duckdb-sources"]
path = libduckdb-sys/duckdb-sources
[submodule "crates/libduckdb-sys/duckdb-sources"]
path = crates/libduckdb-sys/duckdb-sources
url = https://github.com/duckdb/duckdb
update = none
7 changes: 6 additions & 1 deletion .rustfmt.toml
Original file line number Diff line number Diff line change
@@ -1,2 +1,7 @@
max_width = 120
imports_granularity = "Crate"
imports_granularity = "Crate"
reorder_imports = true
fn_call_width = 72
# indent_style = "Block"
# tab_spaces = 2
# group_imports="StdExternalCrate"
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,24 +43,24 @@ export DUCKDB_INCLUDE_DIR=~/duckdb-lib
Use the exported library and header:

```shell
cd ~/github/duckdb-rs/libduckdb-sys
cd ~/github/duckdb-rs/crates/libduckdb-sys
cargo test --features buildtime_bindgen
```

Use the bundled header file:
```shell
cd ~/github/duckdb-rs/libduckdb-sys
cd ~/github/duckdb-rs/crates/libduckdb-sys
cargo test --features bundled
```

Currently in [github actions](https://github.com/wangfenjin/duckdb-rs/actions), we always use the bundled file for testing. So if you change the header in duckdb-cpp repo, you need to make the PR merged and updated the [bundled-file](https://github.com/wangfenjin/duckdb-rs/tree/main/libduckdb-sys/duckdb).
Currently in [github actions](https://github.com/wangfenjin/duckdb-rs/actions), we always use the bundled file for testing. So if you change the header in duckdb-cpp repo, you need to make the PR merged and updated the [bundled-file](https://github.com/wangfenjin/duckdb-rs/tree/main/crates/libduckdb-sys/duckdb).
You can generated the amalgamated file by:

```shell
cd ~/github/duckdb
mkdir -p build/amaldebug
python scripts/amalgamation.py
cp src/amalgamation/duckdb.cpp src/include/duckdb.h src/amalgamation/duckdb.hpp ../duckdb-rs/libduckdb-sys/duckdb/
cp src/amalgamation/duckdb.cpp src/include/duckdb.h src/amalgamation/duckdb.hpp ../duckdb-rs/crates/libduckdb-sys/duckdb/
```

### duckdb-rs
Expand Down
137 changes: 50 additions & 87 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,101 +1,64 @@
[package]
name = "duckdb"
version = "0.10.2"
[workspace]
resolver = "2"
members = [
"crates/duckdb",
"crates/libduckdb-sys",
"crates/duckdb-loadable-macros"
]

[workspace.package]
version = "1.0.0"
authors = ["wangfenjin <wangfenj@gmail.com>"]
edition = "2021"
description = "Ergonomic wrapper for DuckDB"
repository = "https://github.com/wangfenjin/duckdb-rs"
homepage = "https://github.com/wangfenjin/duckdb-rs"
repository = "https://github.com/duckdb/duckdb-rs"
homepage = "https://github.com/duckdb/duckdb-rs"
documentation = "http://docs.rs/duckdb/"
readme = "README.md"
keywords = ["duckdb", "database", "ffi"]
license = "MIT"
categories = ["database"]

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
name = "duckdb"

[workspace]
members = ["libduckdb-sys", "duckdb-loadable-macros"]

[features]
default = []
bundled = ["libduckdb-sys/bundled"]
httpfs = ["libduckdb-sys/httpfs", "bundled"]
json = ["libduckdb-sys/json", "bundled"]
parquet = ["libduckdb-sys/parquet", "bundled"]
vtab = []
vtab-loadable = ["vtab", "duckdb-loadable-macros"]
vtab-excel = ["vtab", "calamine"]
vtab-arrow = ["vtab", "num"]
appender-arrow = ["vtab-arrow"]
vtab-full = ["vtab-excel", "vtab-arrow", "appender-arrow"]
extensions-full = ["httpfs", "json", "parquet", "vtab-full"]
buildtime_bindgen = ["libduckdb-sys/buildtime_bindgen"]
modern-full = ["chrono", "serde_json", "url", "r2d2", "uuid", "polars"]
polars = ["dep:polars"]
chrono = ["dep:chrono", "num-integer"]

[dependencies]
# time = { version = "0.3.2", features = ["formatting", "parsing"], optional = true }
hashlink = "0.8"
chrono = { version = "0.4.22", optional = true }
serde_json = { version = "1.0", optional = true }
csv = { version = "1.1", optional = true }
url = { version = "2.1", optional = true }
lazy_static = { version = "1.4", optional = true }
byteorder = { version = "1.3", features = ["i128"], optional = true }
[workspace.dependencies]
duckdb = { version = "1.0.0", path = "crates/duckdb" }
libduckdb-sys = { version = "1.0.0", path = "crates/libduckdb-sys" }
duckdb-loadable-macros = { version = "0.1.2", path = "crates/duckdb-loadable-macros" }
autocfg = "1.0"
bindgen = { version = "0.69", default-features = false }
byteorder = "1.3"
calamine = "0.22.0"
cast = "0.3"
cc = "1.0"
chrono = "0.4.22"
csv = "1.1"
doc-comment = "0.3"
fallible-iterator = "0.3"
fallible-streaming-iterator = "0.1"
flate2 = "1.0"
hashlink = "0.9"
lazy_static = "1.4"
memchr = "2.3"
uuid = { version = "1.0", optional = true }
smallvec = "1.6.1"
cast = { version = "0.3", features = ["std"] }
arrow = { version = "51", default-features = false, features = ["prettyprint", "ffi"] }
num = { version = "0.4", default-features = false }
num-integer = "0.1.46"
pkg-config = "0.3.24"
polars = "0.35.4"
polars-core = "0.35.4"
pretty_assertions = "1.4.0"
proc-macro2 = "1.0.56"
quote = "1.0.21"
r2d2 = "0.8.9"
rand = "0.8.3"
regex = "1.6"
rust_decimal = "1.14"
strum = { version = "0.25", features = ["derive"] }
r2d2 = { version = "0.8.9", optional = true }
calamine = { version = "0.22.0", optional = true }
num = { version = "0.4", optional = true, default-features = false, features = ["std"] }
duckdb-loadable-macros = { version = "0.1.1", path="./duckdb-loadable-macros", optional = true }
polars = { version = "0.35.4", features = ["dtype-full"], optional = true}
num-integer = {version = "0.1.46", optional = true}

[dev-dependencies]
doc-comment = "0.3"
serde = "1.0"
serde_json = "1.0"
smallvec = "1.6.1"
strum = "0.25"
syn = "2.0.15"
tar = "0.4.38"
tempdir = "0.3.7"
tempfile = "3.1.0"
lazy_static = "1.4"
regex = "1.6"
uuid = { version = "1.0", features = ["v4"] }
unicase = "2.6.0"
rand = "0.8.3"
tempdir = "0.3.7"
polars-core = "0.35.4"
pretty_assertions = "1.4.0"
# criterion = "0.3"

# [[bench]]
# name = "data_types"
# harness = false

[dependencies.libduckdb-sys]
path = "libduckdb-sys"
version = "0.10.2"


[package.metadata.docs.rs]
features = ['vtab', 'chrono']
all-features = false
no-default-features = true
default-target = "x86_64-unknown-linux-gnu"

[package.metadata.playground]
features = []
all-features = false

[[example]]
name = "hello-ext"
crate-type = ["cdylib"]
required-features = ["vtab-loadable"]
url = "2.1"
uuid = "1.0"
vcpkg = "0.2"
arrow = { version = "52", default-features = false }
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@ You can adjust this behavior in a number of ways:
as we are still in development, we will update it regularly. After we are more stable,
we will use the stable released version from [duckdb](https://github.com/duckdb/duckdb/releases).
This is probably the simplest solution to any build problems. You can enable this by adding the following in your `Cargo.toml` file:

```bash
cargo add duckdb --features bundled
```

`Cargo.toml` will be updated.

```toml
[dependencies]
# Assume that version DuckDB version 0.9.2 is used.
Expand Down
22 changes: 22 additions & 0 deletions crates/duckdb-loadable-macros/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[package]
name = "duckdb-loadable-macros"
version = "0.1.2"
authors.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
keywords.workspace = true
readme.workspace = true
categories = ["external-ffi-bindings", "database"]
description = "Native bindings to the libduckdb library, C API; build loadable extensions"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
proc-macro2 = { workspace = true }
quote = { workspace = true }
syn = { workspace = true, features = ["extra-traits", "full", "fold", "parsing"] }

[lib]
proc-macro = true
1 change: 1 addition & 0 deletions crates/duckdb-loadable-macros/LICENSE
1 change: 1 addition & 0 deletions crates/duckdb-loadable-macros/README.md
Loading
Loading