Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel download/upload of binaries from S3. #158

Closed
sauravexodus opened this issue Nov 23, 2018 · 12 comments
Closed

Parallel download/upload of binaries from S3. #158

sauravexodus opened this issue Nov 23, 2018 · 12 comments

Comments

@sauravexodus
Copy link

Enhancement Suggestion / Bug Report

Parallel uploads to S3 server using `--parallel` option

Why would the enhancement be useful to most users

I sometimes experience slow downloading/uploading on my mac. This would enable the uploads and downloads to use the full bandwidth of the internet.
@tmspzz
Copy link
Owner

tmspzz commented Nov 23, 2018

I think it makes sense even without the flag.

I am not sure why I didn't consider it at the time. Listing is already parallelized. https://github.com/blender/Rome/blob/582cc2d6139baefa0d6da5bb5adc9f34cef63d91/src/Caches/S3/Probing.hs#L26

@sauravexodus
Copy link
Author

Should I take this up?

@tmspzz
Copy link
Owner

tmspzz commented Nov 23, 2018

Sure. If you want to give it a try :)

@sauravexodus
Copy link
Author

sauravexodus commented Nov 25, 2018

I get this weird error when I try to run stack build

screenshot 2018-11-25 at 11 40 21 pm

Can you help me with this?

@tmspzz
Copy link
Owner

tmspzz commented Nov 25, 2018

do you get it reliably?

@sauravexodus
Copy link
Author

Yes. Even if I do a clean build.

@tmspzz
Copy link
Owner

tmspzz commented Nov 25, 2018

line 42 in stack.yml

try replacing the existing commit with this commit: e7077512c08f58a8e0a5e6448b6d73c16de46e8d

@sauravexodus
Copy link
Author

Yeah worked!! Thanks for the precise help :D

@tmspzz
Copy link
Owner

tmspzz commented Dec 6, 2018

Hey @sauravexodus were you able to make any progress?

@sauravexodus
Copy link
Author

No, @blender I haven't gotten time yet.

@tmspzz
Copy link
Owner

tmspzz commented Mar 10, 2019

This is implemented in #174

@sauravexodus
Copy link
Author

Awesome! 💃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants