Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nrows for partial parsing of csv/parquet #124

Merged
merged 2 commits into from
Jul 22, 2024
Merged

Add nrows for partial parsing of csv/parquet #124

merged 2 commits into from
Jul 22, 2024

Conversation

dberenbaum
Copy link
Contributor

No description provided.

@dberenbaum dberenbaum requested a review from volkfox July 22, 2024 13:07
Copy link
Contributor

@rlamy rlamy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

It would be nice to have a generic mechanism for stopping generators early though. Maybe applying limit(10) should tell a preceding generator to stop after 10 rows.

@dberenbaum
Copy link
Contributor Author

Closing in favor of #126

@dberenbaum dberenbaum closed this Jul 22, 2024
@dberenbaum
Copy link
Contributor Author

LGTM

It would be nice to have a generic mechanism for stopping generators early though. Maybe applying limit(10) should tell a preceding generator to stop after 10 rows.

Agreed. I don't really like this hack. I'm not sure how easy it is to apply limit() to the right part of the chain though.

@dberenbaum
Copy link
Contributor Author

Reopening since it looks like #126 will not be that simple.

@dberenbaum dberenbaum reopened this Jul 22, 2024
Copy link

cloudflare-workers-and-pages bot commented Jul 22, 2024

Deploying datachain-documentation with  Cloudflare Pages  Cloudflare Pages

Latest commit: b3cec7c
Status: ✅  Deploy successful!
Preview URL: https://ad36db00.datachain-documentation.pages.dev
Branch Preview URL: https://arrow-n-rows.datachain-documentation.pages.dev

View logs

Copy link

codecov bot commented Jul 22, 2024

The author of this PR, dberenbaum, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

@dberenbaum dberenbaum merged commit 53dbca3 into main Jul 22, 2024
19 checks passed
@dberenbaum dberenbaum deleted the arrow_n_rows branch July 22, 2024 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants