Private key decryption fails in py3

[andrewhollomon]

I am working to set up user-sync to use a private-encrypted.key. User sync is running on Windows server. Have created encrypted private key using openssl (openssl pkcs8 -in private.key -topk8 -v2 des3 -out private-encrypted.key)

Connector-umapi.yml is

enterprise:
org_id: "##############@AdobeOrg"
secure_api_key_key: ADOBE_API_KEY
secure_client_secret_key: ADOBE_API_CLIENT_SECRET
tech_acct: "###############@techacct.adobe.com"
secure_priv_pass_key: ADOBE_PRIV_KEY_PASSWORD
priv_key_path: private-encrypted.key

Entries have been created in Windows credential manager - and entries other than the one for secure_priv_pass_key, like client secret and API key, are successfully used.

Running user sync returns

2017-08-02 08:59:28 10672 DEBUG ldap - Connected
2017-08-02 08:59:28 10672 DEBUG umapi - UMAPI initialized with options: {'logger_name': 'umapi', 'test_mode': True, 'server': {'host': 'usermanagement.adobe.io', 'endpoint': '/v2/usermanagement', 'ims_host': 'ims-na1.adobelogin.com', 'ims_endpoint_jwt': '/ims/exchange/jwt', 'timeout': 120, 'retries': 3}, 'enterprise': {'org_id': '##########@AdobeOrg', 'tech_acct': '###########@techacct.adobe.com'}}
2017-08-02 08:59:28 10672 DEBUG umapi - umapi: reading private key data from file C:\PEX\private-encrypted.key
2017-08-02 08:59:28 10672 ERROR umapi - Found unused keys: ['secure_priv_pass_key'] in: umapi configuration.enterprise
2017-08-02 08:59:28 10672 INFO main - ========== End Run (User Sync version: 2.2) (Total time: 0:00:00)

It's possible I have misconfigured something, but I've checked everything over and it looks to be configured per the best practices docs.

[phil-levy]

Two thoughts:

1. Is the indentation in the connector-umapi.yml file correct? As pasted, the items under enterprise: are not indented.

2. Are you really running the 2.2 (or 2.1.1) version of user sync? You didn't include the part of the log where it lists the version.

[andrewhollomon]

Thanks, I can confirm we are running 2.2 and the exact connector-umapi.yml formatting shown below

[andrewhollomon]

In order to take Windows Credential Manager out of the equation - I tried instead using priv_key_pass and adding the passphrase directly to the umapi.yml file.

That results in the following:

2017-08-02 18:11:39 5268 DEBUG umapi - umapi: reading private key data from file C:\PEX\private-encrypted.key
2017-08-02 18:11:39 5268 DEBUG umapi - umapi: creating connection for org @AdobeOrg at endpoint https://usermanagement.adobe.io/v2/usermanagement
2017-08-02 18:11:39 5268 INFO main - ========== End Run (User Sync version: 2.2) (Total time: 0:00:00)
Traceback (most recent call last):
File "c:\pex\install\user_sync-2.2-py3-none-any.whl.c04b8967bba0a28ae4ad16fc13483e0aae93d864\user_sync-2.2-py3-none-any.whl\user_sync\connector\umapi.py", line 127, in init
retry_max_attempts=server_options['retries'] + 1,
File "c:\pex\install\umapi_client-2.5.2-py2.py3-none-any.whl.ea790a0014316bf9fd44be8a726cd6891a11e4c8\umapi_client-2.5.2-py2.py3-none-any.whl\umapi_client\connection.py", line 120, in init
self.auth = self._get_auth(ims_host=ims_host, ims_endpoint_jwt=ims_endpoint_jwt, **auth_dict)
File "c:\pex\install\umapi_client-2.5.2-py2.py3-none-any.whl.ea790a0014316bf9fd44be8a726cd6891a11e4c8\umapi_client-2.5.2-py2.py3-none-any.whl\umapi_client\connection.py", line 138, in _get_auth
jwt = JWT(self.org_id, tech_acct_id, ims_host, api_key, six.StringIO(private_key_data))
TypeError: initial_value must be str or None, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\pex\install\user_sync-2.2-py3-none-any.whl.c04b8967bba0a28ae4ad16fc13483e0aae93d864\user_sync-2.2-py3-none-any.whl\user_sync\app.py", line 362, in main
begin_work(config_loader)
File "c:\pex\install\user_sync-2.2-py3-none-any.whl.c04b8967bba0a28ae4ad16fc13483e0aae93d864\user_sync-2.2-py3-none-any.whl\user_sync\app.py", line 202, in begin_work
umapi_primary_connector = user_sync.connector.umapi.UmapiConnector(primary_name, primary_umapi_config)
File "c:\pex\install\user_sync-2.2-py3-none-any.whl.c04b8967bba0a28ae4ad16fc13483e0aae93d864\user_sync-2.2-py3-none-any.whl\user_sync\connector\umapi.py", line 130, in init
raise AssertionException("Connection to org %s at endpoint %s failed: %s" % (org_id, um_endpoint, e))
user_sync.error.AssertionException: Connection to org @AdobeOrg at endpoint https://usermanagement.adobe.io/v2/usermanagement failed: initial_value must be str or None, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\PEX\user-sync.pex.bootstrap_pex\pex.py", line 360, in execute
File "C:\PEX\user-sync.pex.bootstrap_pex\pex.py", line 288, in _wrap_coverage
File "C:\PEX\user-sync.pex.bootstrap_pex\pex.py", line 320, in _wrap_profiling
File "C:\PEX\user-sync.pex.bootstrap_pex\pex.py", line 403, in _execute
File "C:\PEX\user-sync.pex.bootstrap_pex\pex.py", line 461, in execute_entry
File "C:\PEX\user-sync.pex.bootstrap_pex\pex.py", line 466, in execute_module
File "C:\Program Files\Python36\lib\runpy.py", line 208, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "C:\Program Files\Python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\pex\install\user_sync-2.2-py3-none-any.whl.c04b8967bba0a28ae4ad16fc13483e0aae93d864\user_sync-2.2-py3-none-any.whl\user_sync\app.py", line 389, in
main()
File "c:\pex\install\user_sync-2.2-py3-none-any.whl.c04b8967bba0a28ae4ad16fc13483e0aae93d864\user_sync-2.2-py3-none-any.whl\user_sync\app.py", line 370, in main
logger.critical(e.message)
AttributeError: 'AssertionException' object has no attribute 'message'

[adobeDan]

So you had a misconfiguration in your initial setup: rather than secure_priv_pass_key you should have used secure_priv_key_pass_key. (You always take the base name, in this case priv_key_pass as in your second setup, and prepend secure_ and append _key. The result is unfortunately redundant in this case, which causes a lot of people - including myself - to make the same mistake you did by dropping the first key.) @phil-levy Can you check to see that we don't make this mistake in the docs somewhere?

The error encountered with your second setup looks like it might be a py3-specific bug: apparently the result of decrypting the private key file is a byte-string rather than a standard string, and this is not what six.StringIO is expecting. Can you confirm that you are running in py3?

[andrewhollomon]

Got it, thanks. I'm running Python 3.6.2 (2017-07-17). I can also confirm I do get further by using secure_priv_key_pass_key with the result being the same type error that I receive using priv_key_pass

[adobeDan]

OK, I've now got this as a bug. I'll try to push a fix this week, although we may not release immediately. @andrewhollomon Can you do a build yourself, or do you need me to post a build for you here? If so, it looks like you would need windows 10/py3, yes?

[andrewhollomon]

Thanks and Yep, I should be able to do a build.

[adobeDan]

@phil-levy noticed we have a bug in our sample umapi config file - it has the wrong value for the sample secure password key. I'll fix that too.

[adobeDan]

@andrewhollomon This is fixed in v2 head (app version 2.2.1). I have made a tag v2.2.1rc1; you can download the zip of that, expand it, and do make pex in the user-sync.py directory.

[adobeDan]

@andrewhollomon To get a clean build to work on Windows, I also had to fix #260. So now there is a tag v2.2.1rc2, and attached is a build of that for you to try.

[andrewhollomon]

Thanks @adobeDan I can confirm being able to read and use an encrypted private key w/ passphrase stored in Windows credential manager with the 2.2.1.rc2 build.

2017-08-04 16:55:07 5512 DEBUG umapi - umapi: reading private key data from file C:\PEX\private-encrypted.key
2017-08-04 16:55:07 5512 DEBUG umapi - umapi: creating connection for org @AdobeOrg at endpoint https://usermanagement.adobe.io/v2/usermanagement
2017-08-04 16:55:08 5512 DEBUG urllib3.connectionpool - Starting new HTTPS connection (1): ims-na1.adobelogin.com
2017-08-04 16:55:09 5512 DEBUG urllib3.connectionpool - https://ims-na1.adobelogin.com:443 "POST /ims/exchange/jwt HTTP/1.1" 200 976

[adobeDan]

Yay! Thanks so much @andrewhollomon for confirming. Enjoy syncing! :)