Feature request: globus cat or get/put

[bd4]

Use case: I want to copy a small text file from an LCF to/from my laptop. I would normally use scp for this use case, but certain LCFs don't support control master and having to enter my PIN and OTP every. single. time. is maddening.

I wonder if there is a clever way to do this for globus endpoints running the HTTP server, without requiring gcp, for the cat/get case.

Another possibility, is to require running gcp on the laptop, and do some clever scripting around this to make it more seemless. Could involve staging through a directory that is shared with globus. Perhaps inside the cli is not the place for this, it could be done with wrapper script that handles my particular paths and gcp config. Just thought I would start a discussion because this seems like such a common use case that globus could help streamline. Of course doing this through the Web GUI is not so bad... I just miss my simple command line scp a lot.

[bd4]

Another non-globus way to handle this use case: reverse/remote port forwarding. Basically run an ssh server on laptop, optionally generate a key on the remote server and trust it on the local laptop ssh server, and add something like RemoteForward 8222 localhost:22 to .ssh/config, or -R 8222:localhost:22 on the command line. Then on the remote host to transfer files, can use something like scp -P 8222 parameters localhost:path/to/params/summit to copy a file from remote to laptop. For passwordless auth, another possibility is agent forwarding, if that is enabled on the remote server (in my experience if control master is not enabled, agent forwarding is often not also, following same school of thought to inconvenience^W lock down I guess).

[sirosen]

I wonder if there is a clever way to do this for globus endpoints running the HTTP server, without requiring gcp, for the cat/get case.

I think there is, but we won't get to start work on it until after we ship v3.0. If you look right now, main is the 3.0 work in progress.

Here's the basic situation and plan:

For version 3.0, globus-cli needs to add support direct operations against GCS v5. That means being able to get a token for urn:globus:auth:scopes:{GCS_ENDPOINT_ID}:manage_collections, to create/update/delete collections-née-endpoints.
The plumbing work for that will make it possible for the CLI to handle dynamic token/consent requirements.

There is a different consent required for HTTPS downloads. (I don't know what it is yet. I'll need to either ask the GCS guys or look at what the webapp does.) It should be possible to add some new command (globus download?) which uses that to do a GET.

For v3.0, globus login will take options to specify additional consents/tokens. But I have a plan slated for 3.1 to add a behavior which will prompt for immediate login after printing the explicit login command.

For download, it would look something like

$ globus download $epid:/path/file.txt
Globus CLI needs additional consents to perform this operation.
You can consent with
    globus login --scope "..."

Would you like to start this login flow now? [yes/no] yes
Beginning login flow...
[Either local browser flow or link text flow follows]
Login complete. Please run the command again in order to perform this operation.

$ globus download $epid:/path/file.txt
Download success!

All roads lead to globus-url-copy 😉

[bd4]

Another idea related to this use case, is have some to recognize when globus-cli is run on one of the endpoints and simplify the commands for that case, e.g. allow relative paths and omitting epid and it will automatically fill the epid with the current endpoint and make the path aboslute. I think I can do this pretty easily with a bash wrapper script, but might be interesting as a integrated feature if a significant number of users have similar use case.

[sirosen]

I don't know how many people want this feature. IIRC, one person expressed some interest in it years ago, but I couldn't devote any time to it then and haven't circled back.

I think I can do this pretty easily with a bash wrapper script

For the simple reason that we're (as always) trying to do a lot at the moment with limited resources, I give this approach a big 👍 . But I want to make sure we have the tools in place to make this as easy as we can.

To make sure you're aware, we already have a command for getting local endpoint IDs, but it's GCP only:

globus endpoint local-id

So it ought to be possible to emulate scp with something like

globus transfer $(globus endpoint local-id):$(realpath ./foo.txt) $dest_endpoint:$dest_path

Do you need local-id to work for GCS? I've only been meaning to work on that for like 4 years. 😬
It should be easy to support, especially in the GCSv5 case which provides a dedicated JSON file for us to read to get local endpoint metadata. Because the feature would be small, if it's blocking you from being able to do this, I could make sure to squeeze it in.

---

Regarding the topic of the original comment here, we now have a much better arrangement for storing tokens (We've been living with the same old ~/.globus.cfg setup up until now, and we've moved it to a sqlite DB), so it should be possible to implement upload/download commands. The trouble is that I still don't know how the HTTPS GET/PUT support in GCSv5 works, and I need to learn that before we can work on anything for the CLI.