Make da Playground Project name unique with timestamp

[plocket]

Our test implementation is on a dev branch/release, but is mixed in with some other experimental changes.

X of the problem: We're finding we're having some caching issues with @CaroRob's tests - tests on the same branch will use previous yml code, not updating to the newly pulled code. When a branch's code is updated in relevant ways, its tests will continue to behave as they previously had. A new branch checked out of that branch with no code changes will behave properly. Also, after a few days, the problem will disappear and the tests will behave appropriately again. [This last implies that if a Project is being cached, da does eventually clean it up.]

Possibly, through the API the Project's name gets deleted, but the Project itself doesn't get fully deleted, and pulling new code doesn't actually pull new code. That seems strange, but we don't have other guesses right now.

Y of the problem (and the purpose of this issue): We think making sure the Project name is unique will solve this. We implemented it on a dev branch/release and the problem hasn't showed up again yet. We hope that if this is indeed the problem, lingering Projects will clean themselves up - after all, the problem does disappear after a few days.

[plocket]

GitHub ticket

In case GitHub stuff is involved with this as well.

Product

GitHub

Description

Actions don't seem to be updating based on code updates - test output for a branch doesn't change even when the code changes until the user makes a new branch with the same code. Not blocking, but our users are getting frustrated working around this. We don't mind that the tests are failing - the new branch's tests are failing in expected ways - we just mind that the action output doesn't change when the code changes.

What we saw a few weeks ago on a different repo too:

Branch 1 starts failing specific end-to-end tests (e.g. #20, #21, #22). Rerunning results in the same consistently failing tests 5 times in a row.
Checkout a new branch from Branch 1 (Branch 2a). Make no changes.
Branch 2a (correctly) fails only one test, e.g. #22, consistently.
Checkout another new branch from Branch 1 (Branch 2b). Make no changes.
Branch 2b (correctly) fails only one test, e.g. #22, consistently, same as Branch 2a.
Branch 1 continues to fail all its same own tests consistently (e.g. #20, #21, #22)

We're now having this issue again:

Branch 1: https://github.com/A2JatAKCourts/docassemble-ChangingChildCustody/actions/runs/4318924770
Branch 2: https://github.com/A2JatAKCourts/docassemble-ChangingChildCustody/actions/runs/4318784095
Branch 3: https://github.com/A2JatAKCourts/docassemble-ChangingChildCustody/actions/runs/4319050758

I'm not sure GitHub is the only culprit. I think the site being tested could potentially have something to do with it. Maybe it thinks the action is a returning user. Does the GitHub branch keep the same IP address for all its actions however many times they're run? Does it keep some other information around? Is there a way to clear that information?

When we had this issue for a few days a few weeks ago, there was a post in GitHub about caching issue with something else. We hoped it was related, but maybe it's not.

Reproduction Steps

We can't reproduce this consistently, but this is what we have seen sometimes:

Branch 1 starts failing specific end-to-end tests (e.g. #20, #21, #22). Rerunning results in the same consistently failing tests 5 times in a row.
Checkout a new branch from Branch 1 (Branch 2a). Make no changes.
Branch 2a (correctly) fails only one test, e.g. #22, consistently.
Checkout another new branch from Branch 1 (Branch 2b). Make no changes.
Branch 2b (correctly) fails only one test, e.g. #22, consistently, same as Branch 2a.
Branch 1 continues to fail all its same own tests consistently (e.g. #20, #21, #22)

Fault Type

Intermittent - Occurring irregularly

Blocking?

No

Last working date

16 February

Logs

These are the actions that are providing the same output, even when rerun. Note that each branch contains the same code during these actions.

"Branch 1" is failing the wrong way: https://github.com/A2JatAKCourts/docassemble-ChangingChildCustody/actions/runs/4318924770

20 scenarios (5 failed, 15 passed)
150 steps (5 failed, 25 skipped, 120 passed)

"Branch 2" is failing the right way: https://github.com/A2JatAKCourts/docassemble-ChangingChildCustody/actions/runs/4318784095

20 scenarios (1 failed, 19 passed)
150 steps (1 failed, 5 skipped, 144 passed)

"Branch 3" is failing the right way: https://github.com/A2JatAKCourts/docassemble-ChangingChildCustody/actions/runs/4319050758

20 scenarios (1 failed, 19 passed)
150 steps (1 failed, 5 skipped, 144 passed)

I'm not sure what info would be useful for you to see.

Clarifying note
The other repo eventually fixed itself after a couple days or so - suddenly re-running the tests on the branch updated to the new code and everything was fine. This repo hasn't done that yet.

Response

Hello plocket,

Thank you for reaching out to GitHub Support! I apologize for the delayed response here.

Actions don't seem to be updating based on code updates - test output for a branch doesn't change even when the code changes until the user makes a new branch with the same code.

With checking out code in Actions, actions/checkout will by default check out the repository on the commit SHA of the triggering event. Re-running the job will have it execute in the same context of when it first triggered with the same commit SHA, whereas triggering another workflow run will bring a fresh triggering event with perhaps an updated commit SHA if changes were made.

In the logs for a workflow run, the last log line of the actions/checkout step will be this commit SHA (example). If the same SHA or code is being checked out, I'm not certain what the difference would be here with making a new branch - as that shouldn't have an impact on what is getting pulled and executed in the workflow run.

Does the GitHub branch keep the same IP address for all its actions however many times they're run? Does it keep some other information around? Is there a way to clear that information?

GitHub-hosted runners execute on hosts provisioned from a shared pool of VMs that each have their own IPs. It's possible that two separate jobs may execute from the same VM by chance since they come from the same pool, but that would be pretty rare given the large size of the pool. Altogether, no information like that is maintained exclusively for a branch or repository that would cause it to persist and affect workflow runs.

As long as you can verify that the same code is being used to execute the tests, I believe the cause of the differences in results would come from the test execution itself, as you mentioned. I see from your examples that a third-party action SuffolkLITLab/ALKiln is being used to do the testing - is this the same for the instances you're still seeing of this issue? Are you aware of any cases where differences in results could stem from non-code related differences?