Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursive object get with . local filename does not behave like cp -r or mc -r #1826

Open
jhiemstrawisc opened this issue Dec 12, 2024 · 1 comment
Labels
client Issue affecting the OSDF client
Milestone

Comments

@jhiemstrawisc
Copy link
Member

This is a followup refinement from #1820, where I claimed that a directory structure a la:

.
└── /my-prefix3/
    ├── dir1/
    │   ├── foo.txt
    │   ├── subdir1/
    │   │   └── foo.txt
    │   └── subdir2/
    │       └── foo.txt
    └── dir2/
        └── foo.txt

wasn't being downloaded/recursed correctly with pelican object get -r pelican://<fed url>/my-prefix3/dir1 .. As it turns out, we do recurse, we just don't unpack things in the local filesystem like I expected. Instead of the previous command creating this:

.
└── dir1/
    ├── foo.txt
    ├── subdir1/
    │   └── foo.txt
    └── subdir2/
        └── foo.txt

we lose the dir1 and unpack like this:

foo.txt
subdir1/
└── foo.txt
subdir2/
└── foo.txt

I double checked both mc and cp to see how they handled recursion. Both behave the same where cp -r /some/dir/path . creates path/ and copies files into it and mc cp -r some/s3/path . creates a local path/ where /s in the object names are used for subdirectory nesting. I think we should modify Pelican to behave the same.

@jhiemstrawisc jhiemstrawisc added the client Issue affecting the OSDF client label Dec 12, 2024
@jhiemstrawisc jhiemstrawisc added this to the v7.13.0 milestone Dec 12, 2024
@brianaydemir
Copy link
Contributor

brianaydemir commented Dec 13, 2024

The discussion in #1638 and the ones we've had in the proverbial hallway are perhaps relevant here.

If I recall correctly, I was the one who argued, for example, that given pelican object get -r pelican://<fed url>/my-prefix3/dir1 ., it should be the case that dir1/<sequence_of_chars> should map to (be written into) ./<sequence_of_chars>.

But I suppose I hadn't considered the case where the local destination is ..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client Issue affecting the OSDF client
Projects
None yet
Development

No branches or pull requests

2 participants