Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[LakeFSOutputCommitter] Measure time to write & speedup compared to FileOutputCommitter #4521

Closed
arielshaqed opened this issue Oct 31, 2022 · 1 comment
Closed

[LakeFSOutputCommitter] Measure time to write & speedup compared to FileOutputCommitter #4521

arielshaqed opened this issue Oct 31, 2022 · 1 comment
Assignees
@arielshaqed
Labels
team/ecosystem Team Ecosystem

Comments

@arielshaqed
Copy link
Contributor

Measure (e.g. on lakeFS Cloud) how long it takes to write a sizable file on
10, 200 and 4000 partitions. Compare with FileOutputCommitter and lakeFSFS,
and with S3A with as much magic as Spark can give us.
@arielshaqed arielshaqed changed the title [OC] Measure time to write & speedup compared to FileOutputCommitter [LakeFSOutputCommitter] Measure time to write & speedup compared to FileOutputCommitter Nov 2, 2022
@guy-har guy-har added the team/ecosystem Team Ecosystem label Nov 2, 2022
@arielshaqed arielshaqed self-assigned this Nov 28, 2022
@arielshaqed arielshaqed mentioned this issue Dec 5, 2022
Closed
@arielshaqed
Copy link
Contributor Author

Too slow, we closed this project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arielshaqed arielshaqed

Labels
team/ecosystem Team Ecosystem
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arielshaqed @guy-har