Prefixed (HDFS, S3) paths should be treated as absolute/off-base

[slotrans]

When referencing an S3 path, e.g. s3://my-bucket/prefix/myfile, it is necessary to either blank-out BASE, or point it at S3, otherwise the default local-filesystem BASE will be prepended to the S3 path, giving a file reference like /Users/somejerk/someproject/s3://my-bucket/prefix/myfile, which won't work. For example...

This works:

BASE="s3://my-bucket"
drake_test.txt <-
    echo "foo" | aws s3 cp - $OUTPUT

This also works:

BASE=""
s3://my-bucket/drake_test.txt <-
    echo "foo" | aws s3 cp - $OUTPUT

This does not:

s3://my-bucket/drake_test.txt <-
    echo "foo" | aws s3 cp - $OUTPUT

[mikeivanov]

Hi @slotrans, I you should use an off-base path in this case:

!s3://my-bucket/drake_test.txt <-
    echo "foo" | aws s3 cp - $OUTPUT

Adding a "!" in front of your path turns the $BASE machinery off. Although it feels like a gotcha, I'm not sure if there's a better way of handling cases like this. If you think about how base directories work, it actually makes sense.

[slotrans]

HDFS paths are recognized as absolute paths without the off-base hint, are they not?. Why should S3 work differently?

Put a different way, any filename that begins with protocol colon slash slash cannot possibly be a relative path, just like any filename that begins with slash cannot possibly be a relative path. Drake already knows how to handle the latter, so handling the former seems both trivial and necessary.

[mikeivanov]

Uh, I actually came here to ask the same question about HDFS paths and found your question. So no, they are not off-base and behave exactly the same way as s3 paths.

Oh, I see what you mean about absolute paths. Yes, I agree, prefixed paths should be automatically treated as off-base, you're absolutely right.

[mikeivanov]

May I suggest renaming this issue to "Prefixed paths should be treated as absolute and off-base"?

[slotrans]

Done.