Cannot create Execution Environment with Ansible 2.10

[dsmaugy]

ISSUE TYPE

• Bug Report

SUMMARY

I have a set of playbooks which requires Ansible 2.10 to run (not compatible with Ansible 2.11). I've been using a custom execution environment on Ansible 2.10 successfully for the past month that is based off the quay.io/ansible/ansible-runner:stable-2.10-devel image. When I went to add some more package dependencies and rebuild the environment however, I ran into this error upon trying to execute a job:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 1397, in run
    res = receptor_job.run()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 2957, in run
    return self._run_internal(receptor_ctl)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 3008, in _run_internal
    raise RuntimeError(detail)
RuntimeError: Pod Running

If I change my execution-environment.yml to use quay.io/ansible/awx-ee:0.1.1 however, the job is able to run (although it later fails as it is on Ansible 2.11 as mentioned above).

This seems to be similar to #9917
I have tried a lot of the workarounds mentioned in that thread such as reinstalling the ansible-runner package with no success.

ENVIRONMENT

• AWX version: 19.0.0
• AWX install method: kubernetes
• Ansible version: X.Y.Z
• Operating System: MacOS
• Web Browser: Chrome

STEPS TO REPRODUCE

Build a custom execution environment with quay.io/ansible/ansible-runner:stable-2.10-devel as the base runner image.
Run a job using this custom EE.

EDIT: The problem seems to be limited not only to just the quay.io/ansible/ansible-runner:stable-2.10-devel image but every image that isn't quay.io/ansible/awx-ee:0.1.1. See comment below

EXPECTED RESULTS

The job to successfully start on AWX.

ACTUAL RESULTS

Job fails with a non-descriptive RunTime error message of "Pod Running"

ADDITIONAL INFORMATION

Ansible 2.10 execution-environment.yml

---
version: 1

build_arg_defaults:
  ANSIBLE_RUNNER_IMAGE: quay.io/ansible/ansible-runner:stable-2.10-devel

dependencies:
  galaxy: dependencies/requirements.yml
  python: dependencies/requirements.txt

additional_build_steps:
  append:
    - RUN alternatives --set python /usr/bin/python3
    - COPY --from=quay.io/project-receptor/receptor:0.9.7 /usr/bin/receptor /usr/bin/receptor
    - RUN mkdir -p /var/run/receptor
    - CMD ansible-runner worker --private-data-dir=/runner
    - USER 1000

Ansible 2.11 execution-environment.yml

---
version: 1

build_arg_defaults:
  ANSIBLE_RUNNER_IMAGE: quay.io/ansible/awx-ee:0.1.1

dependencies:
  galaxy: dependencies/requirements.yml
  python: dependencies/requirements.txt

[dsmaugy]

EDIT: It seems like the issue has less to do with the Ansible version and more with the specific awx-ee:0.1.1 image that was built in April. I tried with the awx-ee:0.2.0 image (without the additional build steps), ansible-runner:stable-2.10-devel, ansible-runner:stable-2.11-devel, and ansible-runner:devel (with the additional build steps) and none of these worked.

[dsmaugy]

EDIT: Upgrading to AWX 19.1.0 seemed to fix the issue for me. My guess is that when I recompiled my execution environment, it pulled the latest image from ansible-runner:stable-2.10-devel which broke compatibility with AWX 19.0.0? (#10013)

If that were the case, it would be nice to tag the leave compatible versions of the ansible runner image up as to not force users to update AWX if they want to recompile their execution environment.

[shanemcd]

Sorry you ran into this. We will definitely start to stabilize these new components over time. Since upgrading resolves your issue, I'm going to close this.