"baselayout : run ccache installer" fails running ansible against LinuxONE/smartos

[richardlau]

Running the create worker ansible playbook against our LinuxONE hosts errors on the baselayout : run ccache installer task:

TASK [baselayout : run ccache installer] *******************************************************************************************************************
fatal: [test-ibm-rhel7-s390x-1]: FAILED! => {"msg": "The conditional check 'os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists' failed. The error was: error while evaluating conditional (os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists): 'dict object' has no attribute 'stat'\n\nThe error appears to be in '/home/rlau/sandbox/github/build/ansible/roles/baselayout/tasks/main.yml': line 202, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: run ccache installer\n  ^ here\n"}
fatal: [test-ibm-rhel7-s390x-2]: FAILED! => {"msg": "The conditional check 'os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists' failed. The error was: error while evaluating conditional (os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists): 'dict object' has no attribute 'stat'\n\nThe error appears to be in '/home/rlau/sandbox/github/build/ansible/roles/baselayout/tasks/main.yml': line 202, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: run ccache installer\n  ^ here\n"}
fatal: [test-ibm-rhel7-s390x-3]: FAILED! => {"msg": "The conditional check 'os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists' failed. The error was: error while evaluating conditional (os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists): 'dict object' has no attribute 'stat'\n\nThe error appears to be in '/home/rlau/sandbox/github/build/ansible/roles/baselayout/tasks/main.yml': line 202, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: run ccache installer\n  ^ here\n"}
fatal: [test-ibm-rhel7-s390x-4]: FAILED! => {"msg": "The conditional check 'os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists' failed. The error was: error while evaluating conditional (os == \"rhel7\" and arch == \"s390x\" and not has_ccache.stat.exists): 'dict object' has no attribute 'stat'\n\nThe error appears to be in '/home/rlau/sandbox/github/build/ansible/roles/baselayout/tasks/main.yml': line 202, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: run ccache installer\n  ^ here\n"}

cc @AshCripps

[richardlau]

I stuck some debug into the ansible script to log has_ccache and it looks like this:

"has_ccache": {
        "changed": false,
        "skip_reason": "Conditional result was False",
        "skipped": true
    }

With -vv we see:

TASK [baselayout : check for a recent enough ccache] *******************************************************************************************************
task path: /home/rlau/sandbox/github/build/ansible/roles/baselayout/tasks/main.yml:59
ok: [test-ibm-rhel7-s390x-3] => {"changed": false, "stat": {"atime": 1615538416.8339357, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 1512, "charset": "binary", "checksum": "c68c426ab8b17d962af12cc35e05518b3a211a6b", "ctime": 1576234214.1039357, "dev": 24065, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 154377, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "application/x-executable", "mode": "0755", "mtime": 1576234212.213935, "nlink": 1, "path": "/usr/local/bin/ccache", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 770072, "uid": 0, "version": "787343009", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}

TASK [baselayout : check for ccache aix] *******************************************************************************************************************
task path: /home/rlau/sandbox/github/build/ansible/roles/baselayout/tasks/main.yml:64
skipping: [test-ibm-rhel7-s390x-3] => {"changed": false, "skip_reason": "Conditional result was False"}

so we're overwriting the has_ccache variable by registering it twice:

build/ansible/roles/baselayout/tasks/main.yml

Lines 59 to 68 in 7034b89

	- name: check for a recent enough ccache
	when: os in ccache_no_binpkg or 'rhel7-s390x' in inventory_hostname
	stat: path="{{ binary_dest }}"
	register: has_ccache
	- name: check for ccache aix
	when: os|startswith("aix")
	stat:
	path: /opt/ccache-3.7.4/bin/ccache
	register: has_ccache

[richardlau]

Updated the ansible inventory (#2568) so I could look at the smartos "Failed on 'baselayout : run ccache installer'" failures in #2531 and have verified that they're failing for the same reason.

[richardlau]

The only platforms we're explicitly installing ccache (as opposed to from a package manager) are AIX, LinuxONE and smartos. Since the AIX task that registers has_ccache comes last the scripts work on AIX but break on LinuxONE and smartos as the value they set has_ccache to gets overwritten by the skipped AIX task. I'm going to rework the ccache install into partials as we do elsewhere.