-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Behaviour of directories with S3Path
#224
Comments
Currently our implementation is "Option 1". I think "Option 2" is a faster and non-breaking option we should probably use. I should point out that with our current implementation: if a object is added into a bucket with external tooling then using |
Relatedly, should
edit: oops, clicked close by accident |
I think the typical
I may be misunderstanding this cases but if there is a non-empty file then I'd expect
Using Specifically, for your case since no object exists with that prefix |
In looking into how
S3Path
handles interactions with directories I realized the current behaviour wasn't documented very well and no comparison exists to alternatives that could exist. This issue will attempt to serve as a record of different options that can be used when treating non-hierarchical object storage akin to a file system.A little necessary context. S3 objects have a key name which uniquely identifies the object in the bucket. Objects can be grouped by using keys with similar prefixes and a delimiter (typically "/"). Finally, objects that use "/" as the last character in the key are treated as folders. If you upload an object with a key with a trailing "/" which is empty (0-bytes) then the S3 console will treat this as an empty directory (missing official reference)
Option 1
Make S3 be as file-system like as possible:
mkpath
: Make a zero-byte file to create each directory and sub-directory.isdir
: checks that an object exists with the specified prefixOption 2
Make S3 like a file-system but optimize using some object storage characteristics:
mkpath
: Make a zero-byte file to create only the lowest level directory.isdir
: checks that an object exists with the specified prefixOptions 3a
Embrace prefix storage:
mkpath
: no-opisdir
: checks that an object exists with the specified prefixisdir
aftermkpath
Options 3b
Same as 3a but:
isdir
: alwaysfalse
, embrace that directories don't really exist on S3. Should work fine withisdir(x) || mkpath(x)
Options 3c
Same as 3a but:
isdir
:true
, asisdir
is usually paired with directory creation just state that all directories exist. Probably not a good idea if someone expects that files/directories cannot exist with the same name (they can on S3)Related: I also think we should strive to have the
AbstractPath
interface work in a way that generic functions can be created which work on a variety of different paths without having to have any special cases.The text was updated successfully, but these errors were encountered: