Getting "Invalid JWT Signature" after upgrading to rsa==4.7

[alvaroabascar]

Environment details

• OS: Linux, Mac
• Python version: 3.7, 3.8
• pip version: 20.3.3
• google-auth version: 1.22.1

Steps to reproduce

We found this bug while using dvc and using Google Cloud Storage as a backend. Authentication with google is done via a service key file. When running dvc with rsa==4.6 everything works fine, but when upgrading to rsa==4.7, we encounter the following error:

  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/dvc/tree/gs.py", line 139, in isfile
    return blob.exists()
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 484, in exists
    _target_object=None,
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 431, in api_request
    timeout=timeout,
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 289, in _make_request
    method, url, headers, data, target_object, timeout=timeout
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 327, in _do_request
    url=url, method=method, headers=headers, data=data, timeout=timeout
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 460, in request
    self.credentials.before_request(auth_request, method, url, request_headers)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/service_account.py", line 361, in refresh
    access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{"error":"invalid_grant","error_description":"Invalid JWT Signature."}')

[busunkim96]

Hi,

Thanks for the report! I've marked this as external for now since I see you also opened sybrenstuvel/python-rsa#173. Please let me know if something needs to be fixed in this library.

[jamescooke]

Hi @busunkim96 , I've added a stacktrace for what I think is causing the error above to the ticket on python-rsa here: sybrenstuvel/python-rsa#173 (comment)

EDIT: the original issue on python-rsa contains new info that shows that this library's call to initialise the key is working fine. No need to read the below.

Looking at that stack trace, the call from /google/auth/crypt/_python_rsa.py to python-rsa looks like:

  File "/usr/local/lib/python3.6/site-packages/google/auth/jwt.py", line 112, in encode
    signature = signer.sign(signing_input)
  File "/usr/local/lib/python3.6/site-packages/google/auth/crypt/_python_rsa.py", line 136, in sign
    return rsa.pkcs1.sign(message, self._key, "SHA-256")

Is there any chance that the _key might not have been initialised correctly given the changes that introduced the blindfac attr here: sybrenstuvel/python-rsa@06ec1ea ?

[lechen26]

jumping in.
we started to get this error exactly recently,.
on our log we have the error

google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{"error":"invalid_grant","error_description":"Invalid JWT Signature."}')

we are using google service account to access google cloud storage on the python service.
nothing has changed on our serviec account or his key.
also, this error is intermittent , meaning we can fetch files from storage all the time and every couple of hours we get this error.

is this the same issue?

[busunkim96]

Hi @lechen26,

The root cause is still up in the air. See sybrenstuvel/python-rsa#173. Can you try explicitly pinning to rsa==4.6?

[fcollman]

I am experiencing the same problem.. i hotfixed pip installed rsa==4.6 and the problem went away.

[lechen26]

i'm trying to pin this rsa version.
i thought its not related to my case as i dont have this package on my requirements.txt file.
but checked with pipdeptree and indeed the rsa is dependency of the google packages and it was set to 4.7
pinned it now to 4.6 and will see if issue not reproduce again.

[sybrenstuvel]

I'm the author of the RSA package. As you can see in sybrenstuvel/python-rsa#173, there is a problem where attributes on the key objects are missing:

  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/local/lib/python3.6/site-packages/luigi/contrib/gcs.py", line 258, in _forward_args_to_put
    return self.put(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/luigi/contrib/gcs.py", line 255, in put
    self._do_put(media, dest_path)
  File "/usr/local/lib/python3.6/site-packages/luigi/contrib/gcs.py", line 173, in _do_put
    status, response = request.next_chunk()
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 993, in next_chunk
    headers=start_headers,
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 177, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google_auth_httplib2.py", line 190, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.6/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/service_account.py", line 360, in refresh
    assertion = self._make_authorization_grant_assertion()
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/service_account.py", line 354, in _make_authorization_grant_assertion
    token = jwt.encode(self._signer, payload)
  File "/usr/local/lib/python3.6/site-packages/google/auth/jwt.py", line 112, in encode
    signature = signer.sign(signing_input)
  File "/usr/local/lib/python3.6/site-packages/google/auth/crypt/_python_rsa.py", line 136, in sign
    return rsa.pkcs1.sign(message, self._key, "SHA-256")
  File "/usr/local/lib/python3.6/site-packages/rsa/pkcs1.py", line 331, in sign
    return sign_hash(msg_hash, priv_key, hash_method)
  File "/usr/local/lib/python3.6/site-packages/rsa/pkcs1.py", line 306, in sign_hash
    encrypted = priv_key.blinded_encrypt(payload)
  File "/usr/local/lib/python3.6/site-packages/rsa/key.py", line 463, in blinded_encrypt
    blinded = self.blind(message)  # blind before encrypting
  File "/usr/local/lib/python3.6/site-packages/rsa/key.py", line 165, in blind
    self._update_blinding_factor()
  File "/usr/local/lib/python3.6/site-packages/rsa/key.py", line 190, in _update_blinding_factor
    if self.blindfac < 0:
AttributeError: blindfac

Is Google Auth creating the keys in some non-standard way? The self.blindfac attribute is set in the key's __init__() function, so I don't see how it could be missing.

[jamescooke]

After looking at the RSA code and @sybrenstuvel 's patch, I would like to reinstate my original comment:

Is there any chance that the _key might not have been initialised correctly given the changes that introduced the blindfac attr?

Running the updated rsa==4.7.1 gives a new error: AttributeError: mutex < both this attr and blindfac are is initialised in AbstractKey.__init__() which you can see in the release diff: sybrenstuvel/python-rsa@version-4.7...version-4.7.1

[busunkim96]

@jamescooke @sybrenstuvel Thank you both for the analyses. 🙏 I'm not very familiar with this bit of the codebase so I will have to take a closer look. I will provide an update tomorrow.