Pelican's notion of "recursive" transfers -- Does it make any sense?

[jhiemstrawisc]

Recent conversation surrounding the Pelican plugin for HTCondor has me thinking about how we treat "recursive" transfers in our various clients. Consider the following:

pelican object put my-directory <remote pelican URL>

If you run this command as is, Pelican throws the error:
ERROR[2024-08-14T17:11:38Z] Failure putting my-directory: failed upload to : read my-directory: is a directory (0s since start)

If Pelican already knows the object I'm trying to PUT from the local filesystem is not an object but a directory, why do we require --recursive to make it work? Perhaps we're protecting users by forcing them to recognize there's some internal expansion going on?

On the other hand, a recursive GET might look like:

pelican object get --recursive pelican://osg-htc.org/some/prefix/path .

In this case, we're trying to convey to Pelican "Transfer everything under /some/prefix/path". If the origin hosting this prefix happens to be POSIX, there's an implicit assumption that /some/prefix/path is a directory with files, and potentially subdirectories. But we as Pelican developers claim we only understand objects, so this violates our principles. Even worse, if the origin exports a true object store, like S3 origins do, then "recursive" doesn't translate because there's nothing "beneath" the prefix -- to me it feels like the correct syntax for this is actually a glob pattern:

pelican object get pelican://osg-htc.org/some/prefix/path/*

How can we clean some of this up in the client to present a cohesive picture of what Pelican does and what it doesn't do?

[brianaydemir]

A lot of use cases are going to involve copying files on a traditional POSIX filesystem to and from objects in a Pelican federation. So, any Pelican UI that works with a filesystem should have some mapping between paths on a filesystem and names of objects.

What makes this is interesting is the fact that object names are more permissive that filesystem paths. I can have two distinct objects with names x/y/foo/ and x/y/foo. As filesystem paths, for both these strings to make any sort of sense, x/y/foo must be a directory, and it's up to applications to decide whether there's any significance to the trailing /.

Inspired by Amazon S3's treatment of "folders", what this suggests (to me) is that such a UI:

1. Should not allow the creation of an object whose name ends in /.
2. Should not allow the creation of an object with name x/y/foobar if x/y/foobar/ is the prefix of some other object's name.
3. Should not allow the creation of an object with name x/y/foobar if either x or x/y is an existing object's name.

Then, we can always treat a filesystem path x/y/dir to a directory as a prefix x/y/dir/ of some collection of objects, and a filesystem path x/y/file to a regular file as the name of an object x/y/file. In the case of pelican object {get,put}, because one side of the operation is a filesystem, the presence or lack of --recursive then signals whether the user intends to talk about a directory and a prefix, or a file and an object.

[Todd-L-Miller]

How can we clean some of this up in the client to present a cohesive picture of what Pelican does and what it doesn't do?

I think you're on the right track. --recursive means one thing on upload, and something else on download; I would be totally fine documenting that

pelican object get --recursive pelican://osg-htc.org/some/prefix/path .

means -- could literally be implemented as --

pelican object get 'pelican://osg-htc.org/some/prefix/path/*' ./path

(I'm assuming that --recursive presently creates path on the local filesystem; if it doesn't, adjust accordingly.) Just acknowledge that the former is a convenience shortcut for the latter. (And, as necessary, documenting what directories trees, if any, won't survive a round-trip through Pelican; see below, or, for instance, declare that dotfiles aren't copied by default, or whatever.)

You can define recursive puts in a very similar way (although one that's harder to abbreviate in command-line form): the client traverses the filesystem in the obvious way, generating object names as it goes in the obvious way. The question of what to do with (recursively) empty directories is presently unclear; see below.

---

I like the idea of having the UI maintain a (technically artificial) set of constraints such that there's always an unambiguous (and 1-1 and onto) mapping from a Pegasus origin's objects into a filesystem's directory tree; I think that would prevent the principle of least astonishment from being violated. (I'm also OK with allowing origins to maintain these restrictions, instead, either as an option for true object stores, or the default for origins backed by filesystems, or whatever. I guess I'd be miffed if uploads to some Pelican origins worked and others didn't for the same job, though.)

I don't think it's necessary for the user to specify that a directory on a filesystem is a directory; the Pelican code already knows. For the command-line client, however, I don't have a strong opinion about this UX issue.

When running as file-transfer plug-in, I will probably have a strong opinion. ;) For now, I feel like I don't know, precisely, what's causing the problems BrianA has described elsewhere, and that I should understand that before I start pontificating. There's a case that may not be important but could be and has the potential to very much change the solution, which is that of the empty directory, or the directory which contains only (recursively) empty directories. (I would be very surprised if I told HTCondor to move such a directory, it claimed to succeed, and yet no change occurs in the shared FS that's backing the origin in question.)

Relaxing rule 1 to allow the Pelican software -- but not the user -- to create (empty?) objects whose names end in / may be one way to represent/store/tunnel directories that don't (presently) contain anything.