Playbook fails on "Get etcd certificate serials"

[mazdader]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Environment:

• Cloud provider or hardware configuration:
  Virtual Machines with CentOS 7 on XenServer

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

$ printf "$(uname -srm)\n$(cat /etc/os-release)\n"
Linux 3.10.0-862.14.4.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Version of Ansible (ansible --version):

$ ansible --version
ansible 2.7.0
  config file = /home/aaudonin/TEMP/playgrounds/kubespray/ansible.cfg
  configured module search path = [u'/home/aaudonin/TEMP/playgrounds/kubespray/library']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

Kubespray version (commit) (git rev-parse --short HEAD):
$ git rev-parse --short HEAD
ccc3f89

Network plugin used:
Calico

Copy of your inventory file:

[all]
node01.demo etcd_member_name=etcd01
node02.demo etcd_member_name=etcd02
node03.demo etcd_member_name=etcd03
node11.demo
node12.demo
node13.demo

[kube-master]
node01.demo
node02.demo
node03.demo

[etcd]
node01.demo
node02.demo
node03.demo

[kube-node]
node11.demo
node12.demo
node13.demo

[k8s-cluster:children]
kube-master
kube-node

Command used to invoke ansible:
ansible-playbook -i inventory/demo/hosts.ini -v -D --become --become-user=root cluster.yml

Anything else do we need to know:
Ansible playbook fails with the following error:

TASK [etcd : Gen_certs | Get etcd certificate serials] *****************************************************************************************************************************************************
Monday 22 October 2018  14:12:11 -0700 (0:00:00.122)       0:05:32.289 ******** 
ok: [node01.demo] => {"changed": false, "cmd": ["openssl", "x509", "-in", "/etc/ssl/etcd/ssl/node-node01.demo.pem", "-noout", "-serial"], "delta": "0:00:00.027898", "end": "2018-10-22 14:12:12.197832", "rc": 0, "start": "2018-10-22 14:12:12.169934", "stderr": "", "stderr_lines": [], "stdout": "serial=D054C43DE1853707", "stdout_lines": ["serial=D054C43DE1853707"]}
ok: [node02.demo] => {"changed": false, "cmd": ["openssl", "x509", "-in", "/etc/ssl/etcd/ssl/node-node02.demo.pem", "-noout", "-serial"], "delta": "0:00:00.026823", "end": "2018-10-22 14:12:12.200513", "rc": 0, "start": "2018-10-22 14:12:12.173690", "stderr": "", "stderr_lines": [], "stdout": "serial=D054C43DE1853708", "stdout_lines": ["serial=D054C43DE1853708"]}
ok: [node03.demo] => {"changed": false, "cmd": ["openssl", "x509", "-in", "/etc/ssl/etcd/ssl/node-node03.demo.pem", "-noout", "-serial"], "delta": "0:00:00.015154", "end": "2018-10-22 14:12:12.218872", "rc": 0, "start": "2018-10-22 14:12:12.203718", "stderr": "", "stderr_lines": [], "stdout": "serial=D054C43DE1853709", "stdout_lines": ["serial=D054C43DE1853709"]}
fatal: [node11.demo]: FAILED! => {"changed": false, "cmd": ["openssl", "x509", "-in", "/etc/ssl/etcd/ssl/node-node11.demo.pem", "-noout", "-serial"], "delta": "0:00:00.015413", "end": "2018-10-22 14:12:12.243781", "msg": "non-zero return code", "rc": 1, "start": "2018-10-22 14:12:12.228368", "stderr": "Error opening Certificate /etc/ssl/etcd/ssl/node-node11.demo.pem\n140520446543760:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/ssl/etcd/ssl/node-node11.demo.pem','r')\n140520446543760:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:\nunable to load certificate", "stderr_lines": ["Error opening Certificate /etc/ssl/etcd/ssl/node-node11.demo.pem", "140520446543760:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/ssl/etcd/ssl/node-node11.demo.pem','r')", "140520446543760:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:", "unable to load certificate"], "stdout": "", "stdout_lines": []}
fatal: [node12.demo]: FAILED! => {"changed": false, "cmd": ["openssl", "x509", "-in", "/etc/ssl/etcd/ssl/node-node12.demo.pem", "-noout", "-serial"], "delta": "0:00:00.026319", "end": "2018-10-22 14:12:12.318069", "msg": "non-zero return code", "rc": 1, "start": "2018-10-22 14:12:12.291750", "stderr": "Error opening Certificate /etc/ssl/etcd/ssl/node-node12.demo.pem\n140534781015952:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/ssl/etcd/ssl/node-node12.demo.pem','r')\n140534781015952:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:\nunable to load certificate", "stderr_lines": ["Error opening Certificate /etc/ssl/etcd/ssl/node-node12.demo.pem", "140534781015952:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/ssl/etcd/ssl/node-node12.demo.pem','r')", "140534781015952:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:", "unable to load certificate"], "stdout": "", "stdout_lines": []}
fatal: [node13.demo]: FAILED! => {"changed": false, "cmd": ["openssl", "x509", "-in", "/etc/ssl/etcd/ssl/node-node13.demo.pem", "-noout", "-serial"], "delta": "0:00:00.026202", "end": "2018-10-22 14:12:12.580187", "msg": "non-zero return code", "rc": 1, "start": "2018-10-22 14:12:12.553985", "stderr": "Error opening Certificate /etc/ssl/etcd/ssl/node-node13.demo.pem\n140332697167760:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/ssl/etcd/ssl/node-node13.demo.pem','r')\n140332697167760:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:\nunable to load certificate", "stderr_lines": ["Error opening Certificate /etc/ssl/etcd/ssl/node-node13.demo.pem", "140332697167760:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/ssl/etcd/ssl/node-node13.demo.pem','r')", "140332697167760:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:", "unable to load certificate"], "stdout": "", "stdout_lines": []}

[mirwan]

Duplicate of #3464. Long story short downgrade to ansible 2.6.X until PR #3486 is merged.
Closing

[hatt]

@mirwan is this actually a duplicate? I applied the PR locally and saw no change.

[mirwan]

@hatt My bad, I was hasty by closing it...
After the PR that made the missing certs being detected earlier in the play was merged, I thought it was related with the other issue.
Btw we're investigating the issue

[hatt]

Thanks, I tested with CentOS 7 and Ansible 2.6.6 on Python 2.7.5 with current master HEAD. If there's anything I can do to assist in debugging, please let me know. I tried to resolve it myself in code but can't quite figure out what the logic should be.

[Atoms]

this PR should fix this #3605

[hatt]

Just confirming, with the merge of this PR my instance of the issue is now resolved. Thanks @Atoms!

[mirwan]

@mazdader Are you still facing the issue?

[woopstar]

fixed