-
Notifications
You must be signed in to change notification settings - Fork 434
/
Copy pathREADME.adoc
104 lines (71 loc) · 3.06 KB
/
README.adoc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
:toc: macro
= delta-rs
image:https://github.com/delta-io/delta-rs/workflows/build/badge.svg[Build Status,link=https://github.com/delta-io/delta-rs/actions]
image:https://img.shields.io/crates/v/deltalake.svg?style=flat-square[Crate,link=https://crates.io/crates/deltalake]
image:https://img.shields.io/badge/docs-rust-blue.svg?style=flat-square[Docs,link=https://docs.rs/deltalake]
image:https://img.shields.io/pypi/v/deltalake.svg?style=flat-square[Python binding,link=https://pypi.org/project/deltalake]
image:https://img.shields.io/pypi/dm/deltalake?style=flat-square[PyPI - Downloads,link=https://pypi.org/project/deltalake]
image:https://img.shields.io/badge/docs-python-blue.svg?style=flat-square[Docs,link=https://delta-io.github.io/delta-rs/python]
image::logo.png[Delta-rs logo]
A native interface to
link:https://delta.io[Delta Lake].
toc::[]
== About
This library provides low level access to Delta tables in Rust, which can be
used with data processing frameworks like
link:https://github.com/apache/arrow-datafusion[datafusion],
link:https://github.com/apache/arrow-datafusion/tree/master/ballista[ballista],
link:https://github.com/pola-rs/polars[polars],
link:https://github.com/rajasekarv/vega[vega], etc. It also provides bindings to other higher level language link:https://delta-io.github.io/delta-rs/python/[Python].
=== Features
**Supported backends:**
* Local file system
* AWS S3
* Azure Blob Storage / Azure Datalake Storage Gen2
* Google Cloud Storage
* HDFS
.Support features
|===
| Operation/Feature | Rust | Python
| Read table
| :heavy_check_mark:
| :heavy_check_mark:
| Stream table update
| :heavy_check_mark:
| :heavy_check_mark:
| Filter files with partitions
| :heavy_check_mark:
| :heavy_check_mark:
| Vacuum (delete stale files)
| :heavy_check_mark:
| :heavy_check_mark:
| History
| :heavy_check_mark:
| :heavy_check_mark:
| Write transactions
| :heavy_check_mark:
|
| Checkpoint creation
| :heavy_check_mark:
| :heavy_check_mark:
| High-level file writer
|
| :heavy_check_mark:
| Optimize
| :heavy_check_mark:
| :heavy_check_mark:
|===
== Get Involved
Join link:https://join.slack.com/t/delta-users/shared_invite/zt-1t1ahwrpx-hIEXnP20PCDzcBSf5V3heA[#delta-rs in the Delta Lake Slack workspace]
=== Development Meeting
We have a standing development sync meeting for those that are interested. The meeting is held every two weeks at **9am PST** on Tuesday mornings. The direct meeting URL is shared in the Slack channel above :point_up: before the meeting.
These meetings are also link:https://www.youtube.com/channel/UCSKhDO79MNcX4pIIRFD0UVg[streamed live via YouTube] if you just want to listen in.
=== Development
delta-rs requires the Rust compiler, which can be installed with the
link:https://rustup.rs/[rustup]
command.
Running tests can be done with `cargo test` in the root directory, or one of the directories below:
=== Rust
The `rust/` directory contains core Rust APIs for accessing Delta Lake from Rust, or for higher-level language bindings.
=== Python
The `python/` directory contains the `deltalake` Python package built on top of delta-rs