Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to cp to stdout #20

Closed
fiendish opened this issue Sep 3, 2019 · 4 comments · Fixed by #139
Closed

Add option to cp to stdout #20

fiendish opened this issue Sep 3, 2019 · 4 comments · Fixed by #139

Comments

@fiendish
Copy link

fiendish commented Sep 3, 2019

aws s3 cp has a magic "-" option which, on linux and macos at least, works like writing to /dev/stdout. s5cmd does not look like it supports providing an in-order stream of bytes (writing to stdout or a fifo fails because they don't allow seeking).

Example use case: We would like to be able to pipe extremely large files (terabytes) from s3 into another program such as sha512sum without writing it to disk or storing the entire file in RAM.

@ilkinulas ilkinulas added this to the v1.0.0 milestone Mar 2, 2020
@seruman seruman self-assigned this Mar 23, 2020
igungor pushed a commit that referenced this issue Mar 31, 2020
@thecooltechguy
Copy link

Hi, will this be supported by s5cmd?

@igungor
Copy link
Member

igungor commented Dec 31, 2020

We decided to go with the cat subcommand instead of magic - character that indicates stdin/stdout. It's been there since v1.0.0.

@thecooltechguy
Copy link

Oh ok got it @igungor , do you by chance know why aws s3 cp s3://bucket/abc.tar - is a lot faster than s5cmd's cat? Is it due to some intentional design choice by s5cmd for this cat operation?

@igungor
Copy link
Member

igungor commented Dec 31, 2020

Writing to stdout requires data order guarantee (otherwise data will be corrupt), it needs to be done in a synchronous and sequential way.

I'm guessing awscli downloads the object using parallel future tasks, orders them, and writes to stdout sequentially. s5cmd doesn't do that for cat.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants