Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polars read_delta outputs too much information #14322

Closed
2 tasks done
0xdarkman opened this issue Feb 6, 2024 · 4 comments
Closed
2 tasks done

polars read_delta outputs too much information #14322

0xdarkman opened this issue Feb 6, 2024 · 4 comments
Labels
blocked Cannot be worked on due to external dependencies, or significant new internal features needed first bug Something isn't working P-low Priority: low python Related to Python Polars

Comments

@0xdarkman
Copy link

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

pl.read_delta(table_path, storage_options=storage_options)

Why am I seeing all the information below when reading delta table?

partition_values: {"name": String("foo")}
path: "name=foo/part-00000-aea5a35e-8164-4b72-b017-c85fe191c78c.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-91e85e96-1ec6-479f-ac6b-35a4d9edc058.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-b34142b8-ac55-4b08-be50-63cf803f2df6.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-cbc8c35d-b179-4449-b8e3-d812dc5c75c6.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-528c7682-3aec-4061-9a1b-b8e0b52568fb.c000.snappy.parquet"

Log output

partition_values: {"name": String("foo")}
path: "name=foo/part-00000-aea5a35e-8164-4b72-b017-c85fe191c78c.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-91e85e96-1ec6-479f-ac6b-35a4d9edc058.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-b34142b8-ac55-4b08-be50-63cf803f2df6.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-cbc8c35d-b179-4449-b8e3-d812dc5c75c6.c000.snappy.parquet"
partition_values: {"name": String("foo")}
path: "name=foo/part-00000-528c7682-3aec-4061-9a1b-b8e0b52568fb.c000.snappy.parquet"

Issue description

I am expecting NOT to see all the partition_values and path.

Expected behavior

To print output DF only.

Installed versions

0.19.9

@0xdarkman 0xdarkman added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Feb 6, 2024
@stinodego stinodego added P-low Priority: low and removed needs triage Awaiting prioritization by a maintainer labels Feb 6, 2024
@github-project-automation github-project-automation bot moved this to Ready in Backlog Feb 6, 2024
@stinodego stinodego added the blocked Cannot be worked on due to external dependencies, or significant new internal features needed first label Feb 6, 2024
@stinodego
Copy link
Member

I agree we shouldn't be showing this output. However, this is done by the deltalake package. Could you open an issue with them?
https://github.com/delta-io/delta-rs/issues

@0xdarkman
Copy link
Author

delta-io/delta-rs#2176

I have noted this problem in deltalake 0.15.2 and not in deltalake 0.15.1

@ion-elgreco
Copy link
Contributor

@stinodego this was fixed in the release 0.15.3 yesterday

@0xdarkman
Copy link
Author

yes, it works OK with 0.15.3

@github-project-automation github-project-automation bot moved this from Ready to Done in Backlog Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Cannot be worked on due to external dependencies, or significant new internal features needed first bug Something isn't working P-low Priority: low python Related to Python Polars
Projects
Archived in project
Development

No branches or pull requests

3 participants