Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(python): Update example for writing to cloud storage #20265

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+38 −11
Open

docs(python): Update example for writing to cloud storage #20265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
14e7a1a
c
nameexhaustion Dec 12, 2024
d11ad1a
c
nameexhaustion Dec 23, 2024
f312b79
c
nameexhaustion Dec 23, 2024
b68d118
c
nameexhaustion Dec 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions docs/source/src/python/user-guide/io/cloud-storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,20 +77,39 @@ def get_credentials() -> pl.CredentialProviderFunctionReturn:
# --8<-- [end:scan_pyarrow_dataset]

# --8<-- [start:write_parquet]
import polars as pl

df = pl.DataFrame(
{
"foo": ["a", "b", "c", "d", "d"],
"bar": [1, 2, 3, 4, 5],
}
)

destination = "s3://bucket/my_file.parquet"

df.write_parquet(destination)

# --8<-- [end:write_parquet]

# --8<-- [start:write_file_object]
import polars as pl
import s3fs
import gzip

df = pl.DataFrame(
{
"foo": ["a", "b", "c", "d", "d"],
"bar": [1, 2, 3, 4, 5],
}
)

df = pl.DataFrame({
"foo": ["a", "b", "c", "d", "d"],
"bar": [1, 2, 3, 4, 5],
})
destination = "s3://bucket/my_file.csv.gz"

fs = s3fs.S3FileSystem()
destination = "s3://bucket/my_file.parquet"

# write parquet
with fs.open(destination, mode='wb') as f:
df.write_parquet(f)
# --8<-- [end:write_parquet]
with fs.open(destination, "wb") as cloud_f:
with gzip.open(cloud_f, "w") as f:
df.write_csv(f)
# --8<-- [end:write_file_object]
"""
3 changes: 3 additions & 0 deletions docs/source/src/rust/user-guide/io/cloud-storage.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,6 @@ async fn main() {

// --8<-- [start:write_parquet]
// --8<-- [end:write_parquet]

// --8<-- [start:write_file_object]
// --8<-- [end:write_file_object]
9 changes: 7 additions & 2 deletions docs/source/user-guide/io/cloud-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,12 @@ We first create a PyArrow dataset and then create a `LazyFrame` from the dataset

## Writing to cloud storage

We can write a `DataFrame` to cloud storage in Python using s3fs for S3, adlfs for Azure Blob
Storage and gcsfs for Google Cloud Storage. In this example, we write a Parquet file to S3.
`DataFrame`s can also be written to cloud storage by passing a cloud URL:

{{code_block('user-guide/io/cloud-storage','write_parquet',['write_parquet'])}}

Note that `DataFrame`s can also be written to any Python file object that supports writes. This can
be helpful for performing operations that are not yet natively supported, e.g. writing a compressed
CSV directly to cloud:

{{code_block('user-guide/io/cloud-storage','write_file_object',['write_csv'])}}
Loading