The project update is triggered every time, when the job template changes the source control branch

[elibogomolnyi]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Context

We were struggling with this bug for a couple of weeks in our RND environment:
#14395
The bug troubleshooting took much more time than expected because of the following bugs in UI (I've opened them accordingly):
#14431
#14430

The root cause of the issue

We found a root cause of this issue and see that it is related to the project update mechanism that does not work as we expected.

We have many projects and a lot of workflows that are running simultaneously. We defined the Source Control Branch/Tag/Commit and Cache Timeout for each project and expect the project will not be updated too often because the whole git repo is cached.

But it doesn't work like that. Each time some job triggers the project with another branch (the branch different from the branch it was triggered before), the project update is triggered, ignoring the cache. Because of the UI bugs from above, it took us time to find these "invisible" project update jobs.

I'll illustrate this issue with an example. Let's say we have a project named awx-project; its Source Control Branch/Tag/Commit is set to master. We have three workflows: awx-workflow-A (branch A), awx-workflow-B (branch B), and awx-workflow-C (branch C), that are based on this project. The project cache is configured to 20 minutes, so we expect the project update won't be triggered more often than once in 20 minutes. We can even see in UI that it works as expected (and the project update is "kind of" triggered only every 20 minutes), but it does not since there are some hidden project updates we don't see in UI, and we can find them only in logs. Looks like it works for real like that:

1. awx-workflow-A is triggered. The project update has started. The A branch was cached.
2. awx-workflow-A is triggered. The project update is not triggered since branch A is already cached
3. awx-workflow-B is triggered. The project update has started. The B branch was cached.
4. awx-workflow-B is triggered. The project update is not triggered since branch B is already cached
5. awx-workflow-C is triggered. The project update has started. The C branch was cached.
6. awx-workflow-C is triggered. The project update is not triggered since branch C is already cached, and so on...

That means we can trigger only the workflows from the project's default branch; otherwise, it will trigger additional project updates.

So, cache works on the branch level, not the project level. We have many running workflows from the same project but with different branches. It makes the cache configuration almost useless for us. It overloads our system and RDS, so our AWX stops functioning or does it too slowly because all our jobs are waiting for project updates (see the source issue above for all the details).

AWX version

22.5

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Run workflows from the same project with different branches.

Expected results

The whole project is cached according to the Cache configuration.

Actual results

Only a specific branch is cached, and the cache is overridden for each branch. It makes the project updates triggered too often.

Additional information

No response

[iamperson347]

Has there been any updates on this? We are facing this issue as well - significantly slowly down jobs.

[nikiforova13]

Hello! Has there been any updates on this?