job failed with binascii.Error: Incorrect padding error from time to time

[mick1627]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

From time to time some jobs failed with the following error :

AWX version

21.5.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

yes

Ansible version

2.12

Operating system

No response

Web browser

No response

Steps to reproduce

A job template has a schedule in AWX, from time to time it failed with binascii.Error: Incorrect padding error

Expected results

Job launch correctly

Actual results

job failed with binascii.Error: Incorrect padding error

Additional information

EE custom

[fosterseth]

likely the pod running the job terminated unexpectedly, or receptor itself restarted while getting the stdout back from the pod.

was hoping to see a better error message on job_explanation

If you feel like spelunking a bit,

kubectl logs deploy/awx -c awx-ee | grep ERROR

any errors around the time the job ran?

for advanced debugging,

kubectl edit awx
under "spec" add this extra setting

  extra_settings:
   - setting: RECEPTOR_RELEASE_WORK
     value: "False"

kubectl delete deploy/awx to recreate the awx pod

Then,

1. kubectl exec -it deploy/awx -c awx-task -- bash
2. source /var/lib/awx/venv/awx/bin/activate
3. receptorctl --socket /var/run/receptor/receptor.sock work list --unit_id work_unit_id
   a. work_unit_id can be found on api/v2/jobs/<job id>

what does Detail in the output say?

remove that extra_setting after debugging

[mick1627]

@fosterseth Thanks I updated extra setting to investigate for the next error.

[stanislav-zaprudskiy]

This could be ansible/ansible-runner#998

[shanemcd]

This might have been fixed with ansible/ansible-runner#1161