ansible-runner playbook fails with ssh-key created on rhel 8.

[tinaafitz]

We're experiencing an issue when using ansible-runner to run a playbook with an ssh_key generated on rhel8.

I work with CloudForms, and our 5.11 version runs on rhel8 where we're seeing the issue.

My environment:

Red Hat Enterprise Linux release 8.2 (Ootpa)

Ansible-runner version 1.4.6

ansible 2.9.13
config file = /root/.ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Dec 5 2019, 15:45:45) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

My playbook command line:
ansible-runner -vvv --playbook hello_world_all.yml run /a

When using an ssh_key generated on rhel 8, results in:
Error loading key "/a/2303c3f4-70d5-40b5-9461-4ff0c90d419e/ssh_key_data": invalid format

Invalid key - The ssh_key generated on rhel 8 contains an OPENSSH private key header/footer:
-----BEGIN OPENSSH PRIVATE KEY----------END OPENSSH PRIVATE KEY-----
Running the playbook with this key fails. (complete invalid private key below)

Valid key - The ssh_key generated on rhel 8 using the -m PEM option creates an RSA private_key header/footer.
-----BEGIN RSA PRIVATE KEY----------END RSA PRIVATE KEY-----
Running the playbook with this key is successful. (complete valid private key below)

Invalid Key
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAvUUsURerQKVvztS1jp1fspvs6ceQx085aEab7Ku3Qmddrz5viAZm
1YwXqcj/XD7xoJ+pXIKvxeTgRCkOfF2X1gHBeQzezCgmRqpMIfrCW+rSnZ0fbIgE9XZQwX
QCvCKg+IQ/copKxgiF268s2VfWJSfR2Nfy8p+6FMqIxB4IMvUOEhvV7yZ+Ko002vHSrAkH
5rp5V71nyOnAc8/1yz01thh+0UYomBJsG0D6Y8g2fdJU6438ujAvMzrSjmG/SNEIwhlHYZ
+YFYTn0m8xGCgP5oPrSMLe62YC3qjg0Y7tkxY5BqTg1qHcvkzpe2IkKJHSv38lP59OzCuw
Aukhc19bkadF4rfXiWbShLiOItFZD3kKTrCa+bI9auOjGvElNmlB/LaGPyJ74l4UopTxd2
sYrnl2Ozasrv0bgiDg19GUdcNTE4cH717+ASQNjGAYhJy+7K0jyMRCJ25zWutxk6MsW7FA
OFzi0lQaDKlG23cK14KJkirV4BQmaPknlQlKybJhAAAFsMG6ucPBurnDAAAAB3NzaC1yc2
EAAAGBAL1FLFEXq0Clb87UtY6dX7Kb7OnHkMdPOWhGm+yrt0JnXa8+b4gGZtWMF6nI/1w+
8aCfqVyCr8Xk4EQpDnxdl9YBwXkM3swoJkaqTCH6wlvq0p2dH2yIBPV2UMF0ArwioPiEP3
KKSsYIhduvLNlX1iUn0djX8vKfuhTKiMQeCDL1DhIb1e8mfiqNNNrx0qwJB+a6eVe9Z8jp
wHPP9cs9NbYYftFGKJgSbBtA+mPINn3SVOuN/LowLzM60o5hv0jRCMIZR2GfmBWE59JvMR
goD+aD60jC3utmAt6o4NGO7ZMWOQak4Nah3L5M6XtiJCiR0r9/JT+fTswrsALpIXNfW5Gn
ReK314lm0oS4jiLRWQ95Ck6wmvmyPWrjoxrxJTZpQfy2hj8ie+JeFKKU8XdrGK55djs2rK
79G4Ig4NfRlHXDUxOHB+9e/gEkDYxgGIScvuytI8jEQiduc1rrcZOjLFuxQDhc4tJUGgyp
Rtt3CteCiZIq1eAUJmj5J5UJSsmyYQAAAAMBAAEAAAGBAINdso+qPOuGp97HlighJJ3ESa
BNKjibzT2J3Ilc+bo2IjkSvbiMsO1A6d+nKU4uRej1SEh0UGmrHNlSIcLB9zPLTiyleTH8
H+zmICdpc2L2zi4lMv1SPxbbgttymL9ZzZ4PtDXctF9U5QuAP0GvDGwHgkfeQrqRkEcD9/
8e1RQBMD3zYf42GlWQYoIgW9eaAQjsnnH+fT1euHvpeDETD+eOgt7fsicyiiQ0alW9Tczw
oo5Ho4TbErec+S1nvKteoWFkdFeYljtG9299mOgi0BH7p4bHZJRjqrWn6ohepKUqlQ9APV
R0uALwQgRimcDkLIKZlUItpxJanNMO92I38vXkET959fXiGJ//VWqEeOfAOmPIjR3+AAZV
xmS4vRl0qhcwe43lWJg7VnyFkXVs68E2GI4yKruQTTXBZs+Etx0PV2XNq0bzLEITfD4OG9
+0CQ9lzGHcEbkZJ3KQX5ym+OJYbFK55i328nh6Lozoflvdf2c5MvgXxhJ9hvlv6Rcg0QAA
AMBJqTWBH6+kaVsO9w6VMvXr/7iGBC4iVbeCKYLCeDRnRL9AGFlVocZYGeavNCZOHvCaEa
KSZjYoZQQ0h2rnIkoZULECGhKHGbatfX/9SPqU8uttzGvDebDXbdBPe11z9Lm7DsgdnolD
rBVB+y1p5PiIQfJqQK+0EmeTNWQqSCMXrPa8QHhrUcJETMM93Rg5yIS2tpfXOP4p9/TJig
LSjNbbxJxtbcuQJP0n7FtF0HFxSmdVM3a9KyO1en68KzoFVEIAAADBAPPcHf1JIOQ4t0i1
+qdxX10fg4AfXetz1H9Dz40FNZG6E+RYI2wxeQCVxd6msR+y0bi50EwCgSpSi1SFhp+3t/
xF34kmvXZPbv1Zelra3C4P3UkiqOMznFCZSQYXkzDH/YwH5ftqkmW106s2Wma6fOdMjV4M
st0IQROLUkianfAKoO0WlcsX4N2fxLGK6TyHoj8/h3VWzGcHAk4Jx3CkU0hhfyaG/tKyhT
JDndHgDiECksl8PIRdz7Mn18N1A2C/XQAAAMEAxrFWAqLqxQIteUoi9p275SWQ7W9Um4TY
hYbHKwld76id5nBZuqMyR1ctOKEktQ9C/kiGyepQ56SA9w8KuKTvwleeGmzgsUsp0aZUsI
AbQZvzYq5XQ8EUK6dlK+D7w9n3V+BbN+OSOMovu/RfkVUHawYRDTvvZg0MmXdKwnRa5g8G
zMBYisto3KDMvleI5xFdeE/j+NZIkbcL3F0aJQeQC4Jp241oiz5p+g0DiO8TJ781se+bVn
C2yiXtBni8/sLVAAAANHJvb3RAZGhjcC04LTk5LTkyLmNsb3VkZm9ybXMubGFiLmVuZy5y
ZHUyLnJlZGhhdC5jb20BAgMEBQY=
-----END OPENSSH PRIVATE KEY-----

Valid Key
-----BEGIN RSA PRIVATE KEY-----
MIIG4gIBAAKCAYEAp9gDqo/hHo15vco6v8vjckeHu/OT0Bijv5pxYTF2YfruVH1B
F7vhyzcFJyMSOPAhH3XCRwfqEZcLWaop/AegAwlLZuPwFGjLKM5mZ4ZJLqmzyEMI
JdiZq0w5VtuPw9RATwJups5Rcfx9p9p+Vlol+60BDqV7t1o2q5N47WyevAAfcwMI
8Gco9V0ddphEfZOwTwmZjf+SyZnEWW6wTDCgJe4x4oqqDMim/akbzs0xmmhH84BC
7qjksIhxsr9Uen6x8Gt2Ktjx+l0jHvXapHDzlFvO5EgUMhgJkTnYc7wMtA1KSzVW
thPIWKRUTgdtkItvjqSMbgp/v3w20awc974Q97S85iNBBwlv0hoIVNlwcYabS4xv
VSoDPpikMiR+bSkXipPjgOpUCAY8aa77tIuMcj3juHd2P1WAhv6/CQ/TCIoi4NsW
gHR9/VxlRu74jhyYh7vMZhbHyIZoRo+IWN+lePQJ/5UsNDoJLVjiYlOy7PmOZs8C
ZC7T1MglWkmrLBO1AgMBAAECggGATR2ruY73sNDxsWicC4lDDDufOpTUQhVADwWH
uE9AhA4FuJ3/7UaunG7AGFY9xhlWLp0tUPNq8QAEAAN13M0oEJ0Rnsl1gfsP9MS4
OjJxp1XjO0zJDpJxMEoKQkUIzcHUdji43D0JIKUQmrGBwG0U7Ih1ZuRTbk+ttLFC
K1vvppMr3+zyC3pqUCmjIt0dAtF7fST8/0gAik3y7+Aych8ZRbQmCV8KeGsshnlP
RHn0o2vsOw2gMwU4ZX+bNbRAvTM+EalRL6RegeP6F1YLF9hnRNH1cYNidY4Tlczs
oXDsWdMxWMnH7KRqJ7XUGxhqEFVqJSsJFG7cBwQOiF0gzuJUWyE0aXpxmDV4IonW
w6wd4q0y7fKNn9uTlgvpTH+XaXF7ZJ95PWoteyicNurjavzJ8QQNF1HElY2JcERE
G4wReh2tjl7kBT9XsWIHRmgCGgrpyz9+jHtxE6EcYdOcPeq6YkF8rqDCQkCgXH0M
VOE4kut1Y5FPSj8fHaGv2dcwo8hBAoHBANG6Ieef4HO6zmIn48hfqTwtRNm3jhFb
IZcITIKi4unCBiJ+j7r1FXtJKUqGn1CjuLBWhc5Ry2AKuy5B2VHmdELNzGkrdqvy
IwcyQXj4J1Ed1Ir2jaRX84/kSnlzDVEW6b4J40gD3kIgozGlRIdaM+NfZXFocX9Z
WYrxAPLogG/JBmD9f2p8k6P5Mdqn8H4osAfjoAEteC2+YcQ/TOEnbnC6xEsYwYIc
f8YXXxNUrtORZ+PXIdJEoN6AjFjHSxAQBQKBwQDM4DfiOGVCOVDkWqh/sMApXCqj
QE8YgxVt7N1lKUR/KrPdh+IkDbayrUWvSaK438/gPd7FidJCCwfbMLatmEM6n5Qc
BbLrSnHADHfjniQ/SZHAFphbyHelhthyauTe87FL7qw5hG8uZIUU+9z1veQcCvc2
r2ydWOhnaM2Nk/WcMBJ1ViKMpsqA47wyjB0JOBpnbt7zZyR5V4UemE+E46YECIxu
YDM0NBzMpVOUZGwrZYMhfQJivjC2DLfUR2ksM/ECgcAzaqGTSpPZa4vbbpIedq1j
zmlZwvF2edQTVfqD8HEXA/VDEaolGJ7KmdYgqj52o5zMxnmJam4kvpzDK2ue85O/
KPsbMYengfZqdjys7pGdBmYsJ2P0TYet2DcV7UnvfmGRsJTi++fKaHv52N2jHU8p
SCLzmbpxkTMtqAYmhy+tuCfzMVLChYYgpiA9zfxcwZbeaES65S3YcRtA5I/xhxxB
UNh83LXcMyZ0JxitAY7W2YgRyJVHGoIh6nv96+YQKw0CgcAVm5BJS6liUqk1bFpZ
HUsnEZFtplqxOyIHdfPjt3aB8v8FXk97ffbG4X93/UwsM265N81HCEIktQUfGQQr
SuPMSwhT7EXyiud0WdxLJWKtShUVLUzkYUA9NPufFacN3KXWXFXNuyn+uptOm0Gy
qVRloMJyAg5yie34+hSz2ecNkgRgW7VJIE47ScmPizeEwWYHgqoBAd/W7GQd1fKN
EXVw93XYLk04NEbVTpqCDBVcflhWL9kQkP6SNMUunll5UAECgcBkJE9oQRzkt3w8
XY3ZVxfRpDfs83NMkgNHQB7ZD9S5DQAgXVu1zS9/Ve0KvtqotaV418vE+oTYtrKN
Nu5X8n8u4QNks0IRX293BjXlpsxKxwT5sWRnBK8gUnM+NOWkzDAY9RT/megMeLqi
w7U8GvKYYUYor1NY+L4mnOst8QFUnbzCuVfMViTzrtH3zA6XOXbC9uJ83SEr094f
f5HP+BF5ehhHoE4E2hnk/eHnlp86p3Pch3SfifvhoIATjDs5vYY=
-----END RSA PRIVATE KEY-----

[wenottingham]

Hopefully those are throwaway keys.

That error message is from ssh-add, not runner, and just trying to load that key on the commandline does not work.

[notting@nostromo: ~]$ ssh-add openssh-key 
Error loading key "openssh-key": invalid format
[notting@nostromo: ~]$ ssh-keygen -l
Enter file in which the key is (/home/notting/.ssh/id_rsa): openssh-key
openssh-key is not a key file.

This doesn't look like a runner bug. If you have a key that can be loaded by ssh-agent/read by ssh-keygen that still fails, please reopen.

[wenottingham]

More information:

[cloud-user@runner-testing ~]$ ssh-add openssh-key-bad
Error loading key "openssh-key-bad": invalid format
[cloud-user@runner-testing ~]$ ssh-add openssh-key-good
Identity added: openssh-key-good (root@somewhere.over.the.rainbow)
[cloud-user@runner-testing ~]$ diff -u openssh-key-bad openssh-key-good
--- openssh-key-bad	2020-10-29 16:13:09.573227346 -0400
+++ openssh-key-good	2020-10-29 16:13:11.459224012 -0400
@@ -36,4 +36,4 @@
 zMBYisto3KDMvleI5xFdeE/j+NZIkbcL3F0aJQeQC4Jp241oiz5p+g0DiO8TJ781se+bVn
 C2yiXtBni8/sLVAAAANHJvb3RAZGhjcC04LTk5LTkyLmNsb3VkZm9ybXMubGFiLmVuZy5y
 ZHUyLnJlZGhhdC5jb20BAgMEBQY=
------END OPENSSH PRIVATE KEY-----
\ No newline at end of file
+-----END OPENSSH PRIVATE KEY-----

In runner:

[cloud-user@runner-testing ~]$ cp openssh-key-bad a/env/ssh_key 
[cloud-user@runner-testing ~]$ ansible-runner --playbook playbook.yml run ./a
Error loading key "/home/cloud-user/a/artifacts/5f16dab0-27e3-44da-b172-0a82f790798d/ssh_key_data": invalid format
[cloud-user@runner-testing ~]$ cp openssh-key-good a/env/ssh_key 
[cloud-user@runner-testing ~]$ ansible-runner --playbook playbook.yml run ./a
Identity added: /home/cloud-user/a/artifacts/7c816f97-1ed6-4efc-b5c5-d81d635a8edc/ssh_key_data (root@somewhere.over.the.rainbow)
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
...

For whatever reason, ssh requires OpenSSH format keys to have a full ending newline. It does not require this for old-style PEM keys.

If you are templating out keys for runner, you'll need to ensure they have a full ending newline to be safe.

[tinaafitz]

Thanks so much, @wenottingham. That's great that a newline resolves the issue.

[dsfranks]

I updated the case that I have with Red Hat on Thursday to say that I added a new line to the credential that is being fed to Ansible, and we are still seeing the same error about being unable to parse the SSH key.

[tinaafitz]

Hi @dsfranks, We found that the newline is being stripped off when the credential is saved. That's why it didn't change your results.

[dsfranks]

@tinaafitz ahhh.... That makes sense. Thank you for the update.

[tinaafitz]

Hi @dsfranks, I'm glad that it all makes sense now. We have the code change to resolve the issue under review.