Provide automatic consent for Globus

[forsyth2]

Provide automatic consent for Globus. Resolves #298 (alternative approach to #300).

[forsyth2]

@lukaszlacinski I'm trying to integrate @tylern4's globus_flow script (from #298 (comment)) into zstash. Then, users won't have to run it themselves before using zstash with Globus.

However, I'm getting quite confused as to where it belongs. I figured it should be part of the Globus activation, so I placed the calls in the globus_activate function of zstash/globus.py.

• If I comment out the calls (and I have already granted consent manually), then test_globus works fine.
• If I include the calls, the test hangs on the self.create(use_hpss, zstash_path, cache=self.cache) line of the test (https://github.com/E3SM-Project/zstash/blob/main/tests/test_globus.py#L170)

I'm going to keep looking into this, but I figured I should ask if you had any immediate insights on how to automate the consent granting. Thanks!

[forsyth2]

From what I can tell, the auth_code = get_input("Please enter the code you get after login here: ") line is causing the hang.

However, many lines that should be printed before that point don't print, which made me think incorrectly that the error was occurring much earlier. From what I can tell, the output, err = p.communicate() line (https://github.com/E3SM-Project/zstash/blob/main/tests/base.py#L55) that is used to run zstash create just doesn't print any output until the entire command completes. And that of course includes the line asking for user input.

I'm not quite sure how we would change that for testing purposes. In fact, we really don't want the unit tests to be asking for input anyway, as that would mean any test automation would also get stuck at that point. Then again, I suppose we could handle it the same way we do for endpoint activation -- have the test fail if the consents weren't granted ahead of time; once the user fixes the issue, the test can be run again.

A work-around would be to run a stand-alone globus_flow script before running the unit tests.

As for this pull request, I guess I can try testing the automated globus_flow functionality by manually transferring data to NERSC HPSS (from Perlmutter and from another machine).

[forsyth2]

Ah ok, so it looks like we just need the globus_flow(remote_endpoint) line below, not the globus_flow(local_endpoint) line.

Manual test on Chrysalis:

# Create something to archive
$ emacs setup.sh 
mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo 'file0 stuff' > zstash_demo/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt

$ chmod 755 setup.sh
$ ./setup.sh
# Activate environment with `globus_flow` included in `zstash`
$ conda activate zstash_dev_n298
$ zstash create --hpss=globus://nersc/~/n298_chrysalis_dev zstash_demo

# On Perlmutter:
$ hsi
$ cd n298_chrysalis_dev
$ ls
# 000000.tar   index.db  
# Success

# Now, uncomment the `globus_flow(remote_endpoint)` line, and run `pip install .`
$ zstash create --hpss=globus://nersc/~/n298_chrysalis_dev_v2 zstash_demo
# This now asks for the consent (even though it should still be granted from an earlier run...).
# Once I enter the code, it completes.

# On Perlmutter:
$ hsi
$ cd n298_chrysalis_dev_v2
$ ls
# 000000.tar   index.db
# Success

[forsyth2]

@tylern4 @lukaszlacinski Could you review this pull request, please?

I tested with the following two lines:

# This will prompt the user to enter consent information
zstash create --hpss=globus://nersc/~/n298_pm_dev_with_flag --globus-consent-prompt zstash_demo

# This will not. (It will fail unless consents have already been granted).
zstash create --hpss=globus://nersc/~/n298_pm_dev_no_flag zstash_demo 

My one remaining problem is I can't be entirely sure everything is working properly until I can disable currently granted consents. Nothing I do seems to actually accomplish this.

# Disable consents to test consent activation
# Go to NERSC HPSS on web interface
# https://app.globus.org/file-manager/collections/9cd89cfd-6d04-11e5-ba46-22000b92c6ec/overview
# > Manage Consent
# No consents found
# Go to settings page on web interface
# https://app.globus.org/settings/identities
# > Consents
# > Manage Your Consents
# All consents are from years ago

@golaz Would it be better to have the user include a --globus-consent-prompt (opt-in) flag (implementation in this pull request) or a --bypass-globus-consent (opt-out) flag? The former is better from a backwards-compatibility standpoint (i.e., users won't suddenly find they need to provide input before zstash can continue). The latter is better from a usability standpoint (i.e., users won't run into the 'ConsentRequired' error unless they explicitly disable the consent prompt).

@xylar @chengzhuzhang This is the pull request I would like to include in the immediate zstash / E3SM Unified patch release.

[golaz]

@forsyth2, @tylern4, @lukaszlacinski:

I'm not sure I fully understand the complication here. But I'm concerned that we are going to confuse users for no good reason if we have to add a command line argument to either opt-in or opt-out.

If we go with opt-in, users probably won't bother to figure out what they need to do and they'll complain that zstash is not working or will bother @forsyth2 unnecessarily.

If we go with opt-out, users need to constantly add an additional command line argument which just complicates the zstash syntax for not good reason.

There must be a way we can determine if consent is required, and walk users through the steps only when that is needed, no?

[tylern4]

Okay I looked through your code a bit more and I recommend against the flags, and removing my globus_flow code since it looks like you're using a module (fair_research_login) which handles the Globus OAuth flow for you. My code was just a quick hack to get things working 😂

What you can do is wrap your transfers in a try/execpt block to catch if there are problems and only then ask for the consent if needed. This is around 223 in your globus.py and should works to activate the source endpoint.

        try:
            task = transfer_client.submit_transfer(transfer_data)
        except TransferAPIError as err:
            if err.info.consent_required:
                native_client = NativeClient(
                    client_id="6c1629cf-446c-49e7-af95-323c6412397f", app_name="Zstash")
                native_client.login(
                    requested_scopes=f"urn:globus:auth:scope:transfer.api.globus.org:all[ *https://auth.globus.org/scopes/{remote_endpoint}/data_access ]")

The problems to solve though is multifold, since it may not always be the remorte_endpoint and if you request for an older v4 endpoint it will fail on the request. I also have some code that finds if an endpoint is v4 or v5 which could help decide which endpoints you need to get the data_access consent for

def check_endpoint_version_5(transfer_client, ep_id):
    logging.debug(f"Checking {ep_id}")
    output = json.loads(transfer_client.get_endpoint(ep_id))
    logging.debug(f"Version is {output['gcs_version']}")
    if int(output['gcs_version'].split('.')[0]) >= 5:
        return True
    else:
        return False

You can also get more than one data_access consent at the same time and it shouldn't hurt to get it again. To get multiple requests in one login you can put a list of endpoints in the request.

f"urn:globus:auth:scope:transfer.api.globus.org:all[ *https://auth.globus.org/scopes/{remote_endpoint}/data_access *https://auth.globus.org/scopes/{local_endpoint}/data_access ]"

[forsyth2]

@tylern4 Thanks for the updates! I've included them in the latest commit. How do the changes look now?

Re: #298 (comment) -- thank you, I was able to rescind consents, and I did get prompted for a new consent. However, I notice that I'm sometimes able to successfully transfer files even after rescinding a consent. Is rescinding a consent not instantaneous? Is there some sort of lag time?

For reference, I've tested using the following scripts.

Chrysalis:

DIR_NAME=test_298_pre_consent_chrysalis

rm -rf zstash_demo

mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo 'file0 stuff' > zstash_demo/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt

conda activate zstash_dev_n298
zstash create --hpss=globus://nersc/~/${DIR_NAME} zstash_demo # Successful even without any consents

Perlmutter:

DIR_NAME=test_298_pre_consent_pm

hsi rm -R /home/f/forsyth/${DIR_NAME}
rm -rf zstash_demo

mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo 'file0 stuff' > zstash_demo/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt

conda activate zstash_dev_n298
zstash create --hpss=globus://nersc/~/${DIR_NAME} zstash_demo # Successful even without any consents
hsi ls /home/f/forsyth/${DIR_NAME}

[tylern4]

I'm not sure how long the consents take to propagate but my experience with other Globus actions is that it may take a few seconds-minutes.

I added some changes in #306 to check both endpoints versions and get the consents. It's needed if both endpoints are v5 like the original NERSC HPSS to NERSC Perlmutter transfer.

The one note is that it will quit after an initial run and ask the user to submit another. I believe this is because it would need to re-new the transfer_client to get the scopes. There may be a more elegant way of doing it but restarting also works.

[forsyth2]

@tylern4 Thanks for your edits in #306. I ran zstash with them, but everything still works as if I have granted consents (even though I rescinded them yesterday). Is there some sort of hidden directory/cache on the machine? Am I missing something on the web page?

@mahf708 @golaz @chengzhuzhang @tomvothecoder I'm really trying to get this merged (by end of week), so it can be included in the upcoming patch release of E3SM Unified. I feel like the always-having-consents-granted problem may be specific to me. If any of you have time, can you please try running the following?

On Chrysalis:

cd zstash

# Set up branch
git remote add tylern4 git@github.com:tylern4/zstash.git
git fetch tylern4
git checkout -b issue-298v2-tylern4-chrysalis tylern4/issue-298v2

# Set up environment (if you don't have a zstash development environment set up)
mamba clean --all
mamba env create -f conda/dev.yml -n zstash_dev_issue_298

# Use latest code
pip install .

# Manual test
mkdir zstash_test
echo 'file0 stuff' > zstash_test/file0.txt
zstash create --hpss=globus://nersc/~/test_298_chrysalis zstash_test

# You should receive a prompt to enter a code to grant consent.
# Enter the code and then the command should complete successfully.

# I do not get this prompt, implying I already have the necessary consent, 
# even though I have no relevant consents listed on the web page.

# If you want to be extra sure it worked, switch to Perlmutter and run:
hsi
ls test_298_chrysalis
# You should see a tar file and an index file.

On Perlmutter:

cd zstash

# Set up branch
git remote add tylern4 git@github.com:tylern4/zstash.git
git fetch tylern4
git checkout -b issue-298v2-tylern4-perlmutter tylern4/issue-298v2

# Set up environment (if you don't have a zstash development environment set up)
mamba clean --all
mamba env create -f conda/dev.yml -n zstash_dev_issue_298

# Use latest code
pip install .

# Run test
python -m unittest tests/test_globus.py

# Ideally, you should receive a prompt to enter a code to grant consent.

# It's most likely you won't get a prompt, but rather the test will just hang after `create`.
# That's because the test uses `communicate` 
# (https://github.com/E3SM-Project/zstash/blob/main/tests/base.py#L55)
# which apparently doesn't pass on the prompt to the user.
# If the test just hangs, control + c to exit it. Then run `rm -rf zstash_test` to clean up.

# My test passes, with no prompt for consent and no hang, implying I already have the necessary consent, 
# even though I have no relevant consents listed on the web page.

It's not ideal that the test just hangs, but when GitHub Actions run the unit tests, it skips the tests requiring HPSS (including the Globus test), so it at least won't cause the build/release to fail. To get around the hang, you'd need to run the stand-alone globus_flow script (#298 (comment)), but I don't need you to worry about that.

[mahf708]

testing now, will update with results momentarily:

INFO: Got authorization URL: https://auth.globus.org/v2/oauth2/authorize?client_id=6c1629cf-446c-49e7-af95-323c6412397f&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&scope=urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall%5B+%2Ahttps%3A%2F%2Fauth.globus.org%2Fscopes%2F9cd89cfd-6d04-11e5-ba46-22000b92c6ec%2Fdata_access+%5D&state=_default&response_type=code&code_challenge=Zd1chqq71xXLUZmURt4yiMVOOuS7BnrQiIQ7X2mDmGo&code_challenge_method=S256&access_type=online&prefill_named_grant=Zstash+Login
Please paste the following URL in a browser:
https://auth.globus.org/v2/oauth2/authorize?client_id=6c1629cf-446c-49e7-af95-323c6412397f&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&scope=urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall%5B+%2Ahttps%3A%2F%2Fauth.globus.org%2Fscopes%2F9cd89cfd-6d04-11e5-ba46-22000b92c6ec%2Fdata_access+%5D&state=_default&response_type=code&code_challenge=Zd1chqq71xXLUZmURt4yiMVOOuS7BnrQiIQ7X2mDmGo&code_challenge_method=S256&access_type=online&prefill_named_grant=Zstash+Login
Please Paste your Auth Code Below: 

I noticed that in your steps you don't activate the environment you create. That is,

# Set up environment (if you don't have a zstash development environment set up)
mamba clean --all
mamba env create -f conda/dev.yml -n zstash_dev_issue_298
                                                            <----- need "mamba activate" here
# Use latest code
pip install .

[mahf708]

Do you know where the auth get stored btw? Is it ~/.local or ~/.config or is inside $CONDA_PREFIX somewhere?

[forsyth2]

you don't activate the environment you create

Sorry about that, thanks. I already had mine activated so I forgot that step when writing up the general directions.

Do you know where the auth get stored

No, but perhaps that may be causing the always-have-consent problem

[mahf708]

This is where my consent appears (Test)

When I rescind it and try to rerun the last command, I get:

globus_sdk.services.transfer.errors.TransferAPIError: ('POST', 'https://transfer.api.globus.org/v0.10/endpoint/61f9954c-a4fa-11ea-8f07-0a21f750d19b/autoactivate?if_expires_in=600', 'Bearer', 401, 'AuthenticationFailed', 'Token is not active', 'g2PFU4ZdE')

It appears it's working like intended, though I thought it would ask me for authenticate again rather than throw an error...

[forsyth2]

This is where my consent appears

Yes, that's where my consents appear as well. Did you delete just the most recent one ("Zstash Login Test") or all of them? (I've only been deleting the recent ones that I just granted).

It's interesting that you get 'AuthenticationFailed'. I thought you would get 'ConsentRequired' as in #298 (comment). I think that does mean there's an error somewhere. We want the following:

IF consents not granted:
    prompt for consent
    get consent
proceed with command

Are you logged into LCRC and NERSC on Globus? Perhaps you're missing some credentials.

[tylern4]

@forsyth2 @mahf708 Your Globus tokens are stored in ~/.globus-native-apps.cfg if you remove all the consents you've made the token stored in the file invalid (It's a security feature because if that token was compromised you should be able to log in and remove access making it invalid).

So for testing I've been removing all the consents, rm -rf ~/.globus-native-apps.cfg and going through the full Globus login process again.

I put a small fix in this morning to the and the two tests I've been running have been working from Perlmutter. The pytests don't seem to return the second prompt for auth which is why they hang.

# Test from Perlmutter (v5) to Globus test endpoint (v4) 
perlmutter $ zstash create --cache=zstash --hpss=globus://ddb59aef-6d04-11e5-ba46-22000b92c6ec/~/zstash_test/ zstash_test

# Test from Perlmutter (v5) to NERSC HPSS (v5)
perlmutter $ zstash create --cache=zstash --hpss=globus://nersc/~/zstash_test/ zstash_test

[mahf708]

Once I provided the consent, it worked (and I got two email notifications with SUCCEEDED in all caps lol).

I only deleted the last one (Zstash Login Test) because I wanted to only play with the newly added one.

This is from Chrysalis, but I on globus I am logged on both LCRC and NERSC.

I tried tracing where the auths are stored but I couldn't get anywhere. I suppose it is doing a lot of encryption. I checked ~/.globus.cfg and ~/.globus-native-apps.cfg --- doesn't appear anything related to globus is under ~/.config ...

[mahf708]

Ah, we wrote at the same time. The creds are encrypted in ~/.globus-native-apps.cfg. Makes sense.

[forsyth2]

I ran the following. Everything now appears to work correctly for me! Thank you @tylern4 and @mahf708!! I think I can actually proceed with merging this now.

A manual test:

# On Chrysalis:
cd zstash
git checkout issue-298v2-tylern4-chrysalis
git fetch tylern4
git rebase tylern4/issue-298v2 # Get second commit
conda activate zstash_dev_n298
pip install .
cd ../zstash_testing
mkdir zstash_test
echo 'file0 stuff' > zstash_test/file0.txt
# Confirmed no recent consents on https://auth.globus.org/v2/web/consents
rm -rf ~/.globus-native-apps.cfg
zstash create --hpss=globus://nersc/~/test_298_chrysalis_pre_consent zstash_test
# Asked for consent
# Completes successfully
mkdir zstash_test2
echo 'file0 stuff' > zstash_test2/file0.txt
zstash create --hpss=globus://nersc/~/test_298_chrysalis_post_consent zstash_test
# Completes successfully, without any prompt.

# On Perlmutter:
hsi ls /home/f/forsyth/test_298_chrysalis_pre_consent
hsi ls /home/f/forsyth/test_298_chrysalis_post_consent
# I see a tar file and an index file, for both.

The unit test:

# On Perlmutter
cd zstash
git checkout issue-298v2-tylern4
git fetch tylern4
git rebase tylern4/issue-298v2 # Get second commit
conda activate zstash_dev_issue298
pip install .
# Confirmed no recent Perlmutter consents on https://auth.globus.org/v2/web/consents
rm -rf ~/.globus-native-apps.cfg
python -m unittest tests/test_globus.py
# Asked for consent
# Test passes!
python -m unittest tests/test_globus.py
# Test passes! (No prompt this time)
python -m unittest tests/test_*.py
# All tests pass, no prompt for consent.

[forsyth2]

@tylern4 I'm wondering if we actually need this pull request. I noticed you had sys.exit(0) in your code, but zstash wasn't exiting after prompting for consents, so it seemed like that code block wasn't even being reached.

I then tried running with the latest official release of Unified, and it remarkably prompted for consent. I'm assuming either 1) the conda build picked up a very new version of a dependency that handles the consents, 2) removing ~/.globus-native-apps.cfg was all we ever had to do in the first place.

I feel like it has to be the latter case, since I haven't rebuilt the development environments today but I couldn't get the code working this morning. Then again, we were running into ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'ConsentRequired', 'Missing required data_access consent'... when using E3SM Unified, so that seems to imply something must have changed with the dependencies. Perhaps a combination of both things then?

Is there a case we do need the changes in this pull request then?

Unit test:

# On Perlmutter:
cd zstash
git checkout -b main upstream/main
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh # Use last official release
# Rescind relevant consents on https://auth.globus.org/v2/web/consents
rm -rf ~/.globus-native-apps.cfg
python -m unittest tests/test_globus.py
# Includes a "Please Paste your Auth Code Below:" prompt.
# If I enter the code, the test continues on and passes.

Manual test:

# On Chrysalis:
cd zstash
git checkout -b main upstream/main
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh # Use last official release
# Rescind relevant consents on https://auth.globus.org/v2/web/consents
rm -rf ~/.globus-native-apps.cfg
cd ../zstash_testing
mkdir zstash_test
echo 'file0 stuff' > zstash_test/file0.txt
zstash create --hpss=globus://nersc/~/test_298_chrysalis_unified zstash_test
# Includes a "Please Paste your Auth Code Below:" prompt.
# If I enter the code, the command continues on and completes successfully.
mkdir zstash_test2
echo 'file0 stuff' > zstash_test2/file0.txt
zstash create --hpss=globus://nersc/~/test_298_chrysalis_unified_post_consent zstash_test2
# Completes successfully, with no prompt.

On Perlmutter:
hsi ls /home/f/forsyth/test_298_chrysalis_unified 
hsi ls /home/f/forsyth/test_298_chrysalis_unified_post_consent
# I see a tar file and an index file, for both

[tylern4]

@forsyth2 I'm not sure why you can't remove the consents, maybe you're the owner of the client_id so the consents are different? When I do and try it does go to the sys.exit.

If I instead replace the sys.exit with re-trying the transfer.

diff --git a/zstash/globus.py b/zstash/globus.py
index 5d51f36..c3a057f 100644
--- a/zstash/globus.py
+++ b/zstash/globus.py
@@ -65,10 +65,7 @@ def submit_transfer_with_checks(transfer_data):
             native_client.login(
                 requested_scopes=scopes
             )
-            # Quit here and tell user to re-try
-            print("Consents added, please re-run the previous command to start transfer")
-            sys.exit(0)
-            # I think what's happening is that it needs to reload the token with the new consents
+            task = transfer_client.submit_transfer(transfer_data)
         else:
             raise err
     return task

I get (starting from the first submit_transfer that gets caught).

INFO: request done (success)
INFO: TransferClient.get_endpoint(ddb59aef-6d04-11e5-ba46-22000b92c6ec)
INFO: request done (success)
INFO: TransferClient.get_endpoint(6bdc7956-fc0f-4ad2-989c-7aa5ee643a79)
INFO: request done (success)
INFO: Creating client of type <class 'globus_sdk.services.auth.client.native_client.NativeAppAuthClient'> for service "auth"
INFO: Finished initializing client, client_id=6c1629cf-446c-49e7-af95-323c6412397f
INFO: ScopesMismatch: Requested scopes not found: {'urn:globus:auth:scope:transfer.api.globus.org:all[', '*https://auth.globus.org/scopes/6bdc7956-fc0f-4ad2-989c-7aa5ee643a79/data_access', ']'}. A login is required.
INFO: Starting Native App Grant Flow
INFO: Autogenerating verifier secret. On low-entropy systems this may be insecure
INFO: Got authorization URL: https://auth.globus.org/v2/oauth2/authorize?client_id=6c1629cf-446c-49e7-af95-323c6412397f.....
Please paste the following URL in a browser:
https://auth.globus.org/v2/oauth2/authorize?client_id=6c1629cf-446c-49e7-af95-323c6412397f.....
Please Paste your Auth Code Below:
zzzzzzzzzzzzzzzzzzzzzz
INFO: Final Step of 3-legged OAuth2 Flows: Exchanging authorization code for token(s)
INFO: Fetching new token from Globus Auth
INFO: request done (success)
INFO: Revoking token
INFO: request done (success)
INFO: Revoking token
INFO: request done (success)
INFO: TransferClient.submit_transfer(...)
INFO: request retry_sleep(0.7231529762715745) [max=10]
INFO: request retry_sleep(0.891779722599951) [max=10]
INFO: request retry_sleep(1.5567880858415035) [max=10]
INFO: request done (success)
ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 401, 'AuthenticationFailed', 'Token is not active', '8NGvP8pra')

The authorization fails because the transfer_client is still using the old tokens and needs to get the new ones with the right consents. When I just re-run the last zstash command again it gets a new tranfser_client with the right consents and succeeds. That's why I just put the print message and exited to tell users to try again.

[forsyth2]

I then tried running with the latest official release of Unified, and it remarkably prompted for consent.

Stepping through with pip install . && python -m unittest pdb tests/test_globus.py, I see that on the unit test (on Perlmutter, main branch), the prompt is triggered by native_client.login(no_local_server=True, refresh_tokens=True) (https://github.com/E3SM-Project/zstash/blob/main/tests/test_globus.py#L62).

On the manual test (on Chrysalis):

mkdir zstash_test
echo 'file0 stuff' > zstash_test/file0.txt
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
zstash create --hpss=globus://nersc/~/test_298_manual zstash_test
# Get this error, even though it just worked in my above comment...
# globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')

I'm not sure why you can't remove the consents

I thought I was successfully removing the consents. After all, everything worked fine in #304 (comment) and strangely, using the latest E3S Unified, in #304 (comment).

When I do and try it does go to the sys.exit.

Hmm yeah I don't seem to ever get there. I just don't understand why it's behaving differently.

Also re: the sys.exit should it be sys.exit(1) instead so it doesn't exit with the successful 0-code?

[forsyth2]

@lukaszlacinski Can you please review these comments starting at the 4th above this one (#304 (comment)). I think this is probably ready to merge, but it bothers me that I can't seem to get consistent behavior nor behavior matching what @tylern4 gets.

[lukaszlacinski]

@lukaszlacinski Can you please review these comments starting at the 4th above this one (#304 (comment)). I think this is probably ready to merge, but it bothers me that I can't seem to get consistent behavior nor behavior matching what @tylern4 gets.

Consents are per client and per scope. Tokens and consents (if have been rescinded or never given) for different scopes are requested in two different places in globus.py:

• in globus_activate() when calling native_client.login() for two scopes:
  -- openid
  -- urn:globus:auth:scope:transfer.api.globus.org:all
• in submit_transfer_with_checks() for three scopes:
  -- urn:globus:auth:scope:transfer.api.globus.org:all[
  -- *https://auth.globus.org/scopes/{ep_id}/data_access
  -- ]

Please check the log record

INFO: ScopesMismatch: Requested scopes not found: {'urn:globus:auth:scope:transfer.api.globus.org:all[', '*https://auth.globus.org/scopes/6bdc7956-fc0f-4ad2-989c-7aa5ee643a79/data_access', ']'}. A login is required.

in the last @tylern4 comment.

Scopes are specified in one string with a space as a separator. If data access scopes for GCSv5 mapped collections are added to scopes, that the NativeClient() in globus_activate() requests tokens for, then transfer_client.submit_transfer() should not throw TransferAPIError exception about a missing consent, docs.globus.org - activation is replaced by consent.

For example four scopes:
scopes = "openid urn:globus:auth:scope:transfer.api.globus.org:all urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/6bdc7956-fc0f-4ad2-989c-7aa5ee643a79/data_access] urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/61f9954c-a4fa-11ea-8f07-0a21f750d19b/data_access]"

First scope to talk to the Globus Auth API, second scope to talk to the Globus Transfer API, and two dependent scopes for Globus Transfer to talk to zstash source and destination GCSv5 mapped collections on behalf of a user.

[forsyth2]

@lukaszlacinski Sorry, I'm still confused about what these scopes mean for the difference in behavior between myself and @tylern4. Am I supposed to be rescinding more consents/authentications than I currently am between tests?

[forsyth2]

Latest tests:

Chrysalis

I notice that the Chrysalis endpoint (https://app.globus.org/file-manager/collections/61f9954c-a4fa-11ea-8f07-0a21f750d19b/overview) has a warning: "This endpoint must be migrated to Globus Connect Server version 5 by December 18, 2023 for continued support. ". So, it looks like it hasn't been updated to need consents yet. I wonder if that's why I was getting different behavior there.

Current Unified:

$ mkdir zstash_test
$ echo 'file0 stuff' > zstash_test/file0.txt
$ source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
$ zstash create --hpss=globus://nersc/~/test_20231024_chrysalis_1 zstash_test
globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')

Using this pull request:

$ cd zstash
$ git fetch upstream issue-298v2
$ git checkout -b n298-20231024 upstream/issue-298v2
$ mamba clean --all
$ mamba env create -f conda/dev.yml -n zstash_dev_n298_20231024
$ conda activate zstash_dev_n298_20231024
$ pip install .
$ cd <testing directory>
$ mkdir zstash_test
$ echo 'file0 stuff' > zstash_test/file0.txt
$ zstash create --hpss=globus://nersc/~/test_20231024_chrysalis_2 zstash_test
# Prompted for auth code
ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'PermissionDenied' ...

# Activated my credentials for NERSC HPSS
$ rm -rf zstash_test
$ mkdir zstash_test
$ echo 'file0 stuff' > zstash_test/file0.txt
$ zstash create --hpss=globus://nersc/~/test_20231024_chrysalis_3 zstash_test
# Same error

# Specifically went to NERSC HPSS in the file manager, where it said I need to log in. I logged in.
$ rm -rf zstash_test
$ mkdir zstash_test
$ echo 'file0 stuff' > zstash_test/file0.txt
$ zstash create --hpss=globus://nersc/~/test_20231024_chrysalis_4 zstash_test
# Success!
# Confirmed `test_20231024_chrysalis_4/ ` exists on NERSC `hsi`.

Perlmutter

Current Unified:

$ mkdir zstash_test
$ echo 'file0 stuff' > zstash_test/file0.txt
$ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
$ zstash create --hpss=globus://nersc/~/test_20231024_pm_1 zstash_test
globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')

Using this pull request:

$ cd zstash
$ git fetch upstream issue-298v2
$ git checkout -b n298-20231024 upstream/issue-298v2
$ mamba clean --all
$ mamba env create -f conda/dev.yml -n zstash_dev_n298_20231024
$ conda activate zstash_dev_n298_20231024
$ pip install .
$ cd <testing directory>
$ mkdir zstash_test
$ echo 'file0 stuff' > zstash_test/file0.txt
$ zstash create --hpss=globus://nersc/~/test_20231024_pm_2 zstash_test
# Success!
# Confirmed `test_20231024_pm_2/ ` exists on NERSC `hsi`.

Conclusions

The full logic of authentication, scopes, and consents is significantly more complicated than I would like. Luckily, it appears I now have my side working such that the current Unified fails but these changes succeed, thus matching results of others.

I will go ahead and merge these changes, and start a release candidate for a zstash patch.

Thank you for your help @tylern4 @lukaszlacinski.