oauth2client key error when instantiating a gspread client

[sr-murthy]

In my Google Developers Console account I have created OAuth 2.0 credentials for a Google spreadsheets client I am writing using the gspread library - I create the client using gspread.authorize(OAuth_credentials) where

OAuth2_credentials = SignedJwtAssertionCredentials(
  client_email,
  client_private_key,
  client_auth_scope
)

is the OAuth 2.0 credentials I created. But I get the following error which is related to the oauth2client library:

Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.6/dist-packages/gspread/client.py", line 335, in authorize
  client.login()
File "/usr/local/lib/python2.6/dist-packages/gspread/client.py", line 98, in login
  self.auth.refresh(http)
File "/usr/local/lib/python2.6/dist-packages/oauth2client/client.py", line 598, in refresh
  self._refresh(http.request)
File "/usr/local/lib/python2.6/dist-packages/oauth2client/client.py", line 769, in _refresh
  self._do_refresh_request(http_request)
File "/usr/local/lib/python2.6/dist-packages/oauth2client/client.py", line 795, in _do_refresh_request
  body = self._generate_refresh_request_body()
File "/usr/local/lib/python2.6/dist-packages/oauth2client/client.py", line 1425, in _generate_refresh_request_body
  assertion = self._generate_assertion()
File "/usr/local/lib/python2.6/dist-packages/oauth2client/client.py", line 1554, in _generate_assertion
  private_key, self.private_key_password), payload)
File "/usr/local/lib/python2.6/dist-packages/oauth2client/crypt.py", line 300, in from_string
  pkey = RSA.importKey(parsed_pem_key)
File "/usr/local/lib/python2.6/dist-packages/Crypto/PublicKey/RSA.py", line 638, in importKey
  if lines[1].startswith(b('Proc-Type:4,ENCRYPTED')):
IndexError: list index out of range

[dhermes]

Looks like it's related to the RSA library 😄 but let's see what we can do.

How are you loading client_private_key in your program?

Can you execute this for me:

import oauth2client
print oauth2client.__version__

[sr-murthy]

Hi, the strange thing was that it was working as written originally, and the problem only started occurring today (30/06). (Since you've replied I've replaced oauth2client.client.SignedJwtAssertionCredentials by oauth2client.GoogleCredentials and this does work.)

The oauth2client version is 1.4.11 (and the gspread version 0.2.5).

The old code still causes the same RSA key error. Here, the private key, along with the other authentication credentials and the spreadsheet key were obtained from one of the custom objects on our stack called brand - this information was in turn populated by the JSON file I downloaded from the API credential I created on Google Developers Console. The code used to look like this:

email = brand.get('oauth2_client_email')
private_key = brand.get('oauth2_client_private_key')
auth_scope = brand.get('oauth2_client_auth_scope')

spreadsheet_key = brand.get('google_spreadsheet_key')

OAuth2_credentials = SignedJwtAssertionCredentials(
  email, private_key, auth_scope
)

spreadsheet_client = gspread.authorize(OAuth2_credentials)
spreadsheet_client.session.headers['Connection'] = 'Keep-Alive'
spreadsheet = spreadsheet_client.open_by_key(spreadsheet_key)

[dhermes]

That's pretty strange. How do you load the JSON contents into brand? Do you open the file in 'rb' mode?

This may be related to #192, i.e. there may have been a malformed key.

[sr-murthy]

The JSON data are stored on a config web page, which is part of our stack, and this is accessible to the brand objects. It's one Django app talking to another. The data are stored in standard Django form fields.

[dhermes]

How did it get loaded into the Django form field?

From the RSA error (I assume you are on [release 2.6.1], which is latest on PyPI), it seems that the value you are using doesn't have the correct newlines / whitespace.

Check out the failing section (in the 2.6.1 release):
https://github.com/dlitz/pycrypto/blob/7fd528d03b5eae58eef6fd219af5d9ac9c83fa50/lib/Crypto/PublicKey/RSA.py#L632-L638

[sr-murthy]

Now I get a different error when I use the service_account._ServiceAccountCredentials() method to create the credentials object (the private key has been correctly copied into the text field from the JSON file downloaded from my Google API project.

Traceback (most recent call last):
File "<console>", line 6, in <module>
File "/usr/local/lib/python2.6/dist-packages/oauth2client/service_account.py", line 51, in __init__
  self._private_key = _get_private_key(private_key_pkcs8_text)
File "/usr/local/lib/python2.6/dist-packages/oauth2client/service_account.py", line 135, in _get_private_key
  der = rsa.pem.load_pem(private_key_pkcs8_text, 'PRIVATE KEY')
File "/usr/local/lib/python2.6/dist-packages/rsa/pem.py", line 85, in load_pem
  raise ValueError('No PEM start marker "%s" found' % pem_start)
ValueError: No PEM start marker "-----BEGIN PRIVATE KEY-----" found

[dhermes]

This is essentially the same as

File "/usr/local/lib/python2.6/dist-packages/oauth2client/crypt.py", line 300, in from_string
  pkey = RSA.importKey(parsed_pem_key)
File "/usr/local/lib/python2.6/dist-packages/Crypto/PublicKey/RSA.py", line 638, in importKey
  if lines[1].startswith(b('Proc-Type:4,ENCRYPTED')):
IndexError: list index out of range

which happens in PyCrypto (same link as above).

---

I'm closing out because this confirms that you aren't loading the key correctly.

---

Recommendations

1. Make sure you are loading the file correctly
2. Make sure you are using a PKCS12 key with

from oauth2client.client import SignedJwtAssertionCredentials
credentials = SignedJwtAssertionCredentials(
    service_account_name=client_email,
    private_key=open(pkcs12_key_file_path, 'rb').read(),
    scope=scope)

and that you are using a JSON key with

from oauth2client.client import _get_application_default_credential_from_file
credentials = _get_application_default_credential_from_file(
    json_key_file_path)

---

I'm happy to re-open if you're still having issues.

[sr-murthy]

Why does it complain that it cannot find the PEM marker when the private key does contain the marker? This is how the key starts:

-----BEGIN PRIVATE KEY-----\\n .....

[dhermes]

It looks like you're loading the key incorrectly into brand.

Notice that \\ is an escaped backslash. So instead of the newline \n, you have just a literal backslash \ followed by an n. So when the code tries to split on a newline (or any whitespace), it doesn't break up -----BEGIN PRIVATE KEY----- from the next line.

[sr-murthy]

I am not using SignedJwtAssertionCredentials to create the credentials, I am using the following code

credentials = service_account._ServiceAccountCredentials(
    service_account_id=client_id,
    service_account_email=client_email,
    private_key_id=client_private_key_id,
    private_key_pkcs8_text=client_private_key,
    scopes=client_auth_scopes
)

You can see from the source code http://google.github.io/oauth2client/_modules/oauth2client/client.html#GoogleCredentials that _get_application_default_credential_from_file just calls service_account._ServiceAccountCredentials() if the account type is service_account.

[sr-murthy]

I don't know how those double slashes got there.

[dhermes]

Yes https://github.com/google/oauth2client/blob/c47dcd7f66cb8463e5432ad1f2a2a54cc4653df4/oauth2client/client.py#L1353-L1358 is the relevant section of code and you are totally right.

This experience actually makes me think we need to

• merge together service_account._ServiceAccountCredentials and client.SignedJwtAssertionCredentials
• add a public factory constructor which does what _get_application_default_credential_from_file does
• better document the difference between a PKCS12/P12 key and a JSON key

[sr-murthy]

In my Sublime Text 3 I found and replaced every occurence of '\n' in the private key with an empty string, and it still complains:

>>> import rsa
>>> pk = '-----BEGIN PRIVATE KEY----- .... -----END PRIVATE KEY-----'
>>> rsa.pem.load_pem(pk, 'PRIVATE KEY')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.6/dist-packages/rsa/pem.py", line 85, in load_pem
  raise ValueError('No PEM start marker "%s" found' % pem_start)
ValueError: No PEM start marker "-----BEGIN PRIVATE KEY-----" found

[sr-murthy]

It would be helpful if the library itself was able to remove these newline characters. The JSON key I downloaded was "straight from the source" so to speak, from Google itself. I just copied and pasted the key into our config web page, which is then accessible to our Django objects.

[dhermes]

The copy-paste is the point of failure.

found and replaced every occurence of '\n' in the private key with an empty string,

You don't want an empty string, you want a newline character.

Google itself does remove the newline charactewrs, the issue is that the literal \\n is not a newline character, it's a literal \ followed by a literal n.

Do this to see the difference in Python

>>> a = '\n'
>>> a
'\n'
>>> print a


>>> b = '\\n'
>>> b
'\\n'
>>> print b
\n
>>> c = r'\n'
>>> c
'\\n'
>>> print c
\n

[sr-murthy]

I did this manually, i.e. replaced every occurence in the key of a '\n' character with a carriage return, and the call rsa.pem.load_pem(pk, 'PRIVATE KEY') now works.

I didn't think that OAuth 2.0 would be this complicated! :)

[dhermes]

Sorry for the long road here.

In the future, load the contents from the file you download instead of copying and pasting and you won't have these complications.