You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
aws s3 cp has a magic "-" option which, on linux and macos at least, works like writing to /dev/stdout. s5cmd does not look like it supports providing an in-order stream of bytes (writing to stdout or a fifo fails because they don't allow seeking).
Example use case: We would like to be able to pipe extremely large files (terabytes) from s3 into another program such as sha512sum without writing it to disk or storing the entire file in RAM.
The text was updated successfully, but these errors were encountered:
Oh ok got it @igungor , do you by chance know why aws s3 cp s3://bucket/abc.tar - is a lot faster than s5cmd's cat? Is it due to some intentional design choice by s5cmd for this cat operation?
aws s3 cp has a magic "-" option which, on linux and macos at least, works like writing to /dev/stdout. s5cmd does not look like it supports providing an in-order stream of bytes (writing to stdout or a fifo fails because they don't allow seeking).
Example use case: We would like to be able to pipe extremely large files (terabytes) from s3 into another program such as sha512sum without writing it to disk or storing the entire file in RAM.
The text was updated successfully, but these errors were encountered: