project update hook fail with many jobs no playbook found

[chris93111]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Summary

Hi

With many job launch (33), the first ones fail with error no playbook found

In the tasks we see hooks error

2022-04-01 11:18:34,947 INFO [f5566a273d2e4198b607ab96113598d7] awx.main.commands.run_callback_receiver Event processing is finished for Job 444885, sending notifications
2022-04-01 11:18:34,947 INFO [f5566a273d2e4198b607ab96113598d7] awx.main.commands.run_callback_receiver Event processing is finished for Job 444885, sending notifications
2022-04-01 11:18:35,048 DEBUG [f5566a273d2e4198b607ab96113598d7] awx.main.tasks.jobs project_update 444885 (running) finished running, producing 25 events.
2022-04-01 11:18:35,054 DEBUG [f5566a273d2e4198b607ab96113598d7] awx.analytics.job_lifecycle projectupdate-444885 post run
2022-04-01 11:18:35,055 DEBUG [f5566a273d2e4198b607ab96113598d7] awx.main.tasks.jobs project_update 444885 (running) wrote to cache at /var/lib/awx/projects/.__awx_cache/_76__iac/444885
2022-04-01 11:18:35,058 ERROR [f5566a273d2e4198b607ab96113598d7] awx.main.tasks.jobs project_update 444885 (running) Post run hook errored.
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 589, in run
self.post_run_hook(self.instance, status)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 1436, in post_run_hook
self.make_local_copy(instance, self.job_private_data_dir)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 1370, in make_local_copy
raise RuntimeError('Unexpectedly could not determine a revision to run from project.')
RuntimeError: Unexpectedly could not determine a revision to run from project.

the project refresh stdout is incomplete

the job fail with no found playbook

AWX version

20.1.0

Select the relevant components

• UI
• API
• Docs

Installation method

kubernetes

Modifications

no

Ansible version

2.9.9

Operating system

redhat

Web browser

Chrome

Steps to reproduce

launch many jobs with ovveride scm branch

Expected results

no error on jobs

Actual results

first jobs fail with no playbook found

Additional information

No response

[AlanCoding]

the project refresh stdout is incomplete

That's the most important detail to me here. This task in the project update playbook has to run for us to obtain a revision from the project update.

awx/awx/playbooks/project_update.yml

Lines 45 to 48 in b885fc2

	- name: Set the git repository version
	set_fact:
	scm_version: "{{ git_result['after'] }}"
	when: "'after' in git_result"

Since it does not run, all these other errors happen as a fairly direct consequence. This is a highly abnormal situation, where the project update was successful, but we failed to gather all its output.

More debugging information could help us here. Do you have any relevant logs from callback_receiver.log? Could you navigate to the events /api/v2/project_updates/N/events/ and paste those details? The final text in the screenshot is garbled stuff from the internals of ansible-runner, and we've seen this same thing in other issues.

In particular, I think this comes back to ansible/receptor#597, and is not truly a distinct issue on its own.

[chris93111]

Hi @AlanCoding sorry for the time, I no longer have this problem since increase the limit resources of awx