is version 3.1.0 broken?

[jamengual]

Yesterday the world was fine, today not so much.
To fix my issue today I just downgrade to V2, this literally worked yesterday and did not today.

Bad run: https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098

Good run: https://github.com/runatlantis/atlantis/actions/runs/3185943896/jobs/5196059932

error:

##[debug]Evaluating condition for step: 'Post Run actions/checkout@v3'
##[debug]Evaluating: always()
##[debug]Evaluating always:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Post Run actions/checkout@v3
##[debug]Loading inputs
##[debug]Evaluating: github.repository
##[debug]Evaluating Index:
##[debug]..Evaluating github:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'repository'
##[debug]=> 'runatlantis/atlantis'
##[debug]Result: 'runatlantis/atlantis'
##[debug]Evaluating: github.token
##[debug]Evaluating Index:
##[debug]..Evaluating github:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'token'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Post job cleanup.
/usr/bin/docker exec  9ff[2](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:2)8c2640f[3](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:3)6d36de6e5ae0ffe0378b7672e70[4](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:4)[5](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:5)472288278dc5a2f[6](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:6)6516c22 sh -c "cat /etc/*release | grep ^ID"
##[debug]ID=debian
##[debug]Running JavaScript Action with default external tool: node16
node:internal/fs/utils:344
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_2[7](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:7)[8](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:8)3dec5-b4f7-4[9](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:9)c6-b083-53a8c72b0af3'
    at Object.openSync (node:fs:585:3)
    at Object.writeFileSync (node:fs:2153:35)
    at Object.appendFileSync (node:fs:2215:6)
    at Object.issueFileCommand (/__w/_actions/actions/checkout/v3/dist/index.js:2293:8)
    at Object.saveState (/__w/_actions/actions/checkout/v3/dist/index.js:11873:31)
    at Object.153 (/__w/_actions/actions/checkout/v3/dist/index.js:4044:[10](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:10))
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:30)
    at Object.287 (/__w/_actions/actions/checkout/v3/dist/index.js:70[13](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:13):34)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:30)
    at Object.853 (/__w/_actions/actions/checkout/v3/dist/index.js:3[18](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:18)01:36) {
  errno: -13,
  syscall: 'open',
  code: 'EACCES',
  path: '/__w/_temp/_runner_file_commands/save_state_[27](https://github.com/runatlantis/atlantis/actions/runs/3185910307/jobs/5196021098#step:9:27)83dec5-b4f7-49c6-b083-53a8c72b0af3'
}
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Post Run actions/checkout@v3

Thanks.

[jamengual]

new run on 3.0.2

https://github.com/runatlantis/atlantis/actions/runs/3185997771/jobs/5196164236

[ollie-etl]

Yes. We have the same bug. Reverting to v3.0.2 fixes for us. This occurs only on our self hosted runners

[rentziass]

Thank you for reporting this, looking into this now!

[rentziass]

We are repointing v3 to v3.0.2 while we look into this.

[simoneb]

Still broken here https://github.com/nearform/sql/actions/runs/3196327180/jobs/5218105824

[bel0v]

We're also facing the same unzipping error ^
Can't use 'tar -xzf' extract archive file
using actions/checkout@v3

[Loic-Hakisa]

Same error here :

Download action repository 'actions/checkout@v3' (SHA:2541b1294d2704b0964813337f33b291d3f8596b)
Error: Can't use 'tar -xzf' extract archive file:

[xSAVIKx]

It looks like the whole GH universe is going to add their copy of the error here :-D We have hundreds of failed builds, but I personally am not sure the action is the problem.

[sh-TU]

same here

Error: Can't use 'tar -xzf' extract archive file: . return code: 2.

[MarijnS95]

Still broken here https://github.com/nearform/sql/actions/runs/3196327180/jobs/5218105824

This is a failure to download the tar archive at 2541b12 from the v3 / v3.0.2 tag, which doesn't seem to have any relation to the original issue reported here. Could it perhaps be related to the tag being repointed, though?

Alas, it simply seems that the tarball returned by https://github.com/actions/checkout/tarball/2541b1294d2704b0964813337f33b291d3f8596b is empty:

$ file ~/Downloads/actions-checkout-v3-0-g2541b12.tar.gz
~Downloads/actions-checkout-v3-0-g2541b12.tar.gz: empty

EDIT: The tarball from 93ea575 for v3.1.0 is fine: https://github.com/actions/checkout/tarball/93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8

[bel0v]

actually we tried - uses: actions/checkout@v3.0.2 and it didn't help

[trevorah]

For those that are getting the Can't use 'tar -xzf' extract archive file when using actions/checkout@v3, we've found that switching to actions/checkout@v3.1.0 has fixed it for us.

[ssbarnea]

I don't really know if v3.1.0 is broken but we can see that v3 tag was not moved to point to it, it still points to the version from April, see https://github.com/actions/checkout/tags

Also this morning I started to see failures from this action, and the outcome was the same even after doing a rerun,

[30](https://github.com/ansible/vscode-ansible/actions/runs/3196395729/jobs/5218346155#step:1:34)
Getting action download info
[31](https://github.com/ansible/vscode-ansible/actions/runs/3196395729/jobs/5218346155#step:1:35)
Download action repository 'actions/checkout@v3' (SHA:2541b1294d2704b0964813337f33b291d3f8596b)
[32](https://github.com/ansible/vscode-ansible/actions/runs/3196395729/jobs/5218346155#step:1:36)
Error: Can't use 'tar -xzf' extract archive file: /home/runner/work/_actions/_temp_9684c06e-c225-4f6f-a208-25bca24c56a2/b65d6f4f-ef1f-4b97-a59e-f4e293641bcf.tar.gz. return code: 2.

Funny bit is that https://www.githubstatus.com/ reports all green, when in fact checkout no longer works

[christian-steinmeyer]

we ran into the tar issue and are pointing to actions/checkout@v3

[bel0v]

switching to actions/checkout@v3.1.0 has fixed it for us.

Unfortunately this didn't work in our case

[casperbakker]

Seems like everyone with problems is running ubuntu-20.04. I think Github has launched a new version of that container that is broken. We fixed it with moving to ubuntu-22.04, but it was an intermitted problem on 20.04 so who knows how long it will work. 22.04 also fails sometimes now.

I think we have to wait for Github to fix it.

[ssbarnea]

@casperbakker I wonder how long does it takes for them to report the outage, if they will ever do it.

That reminded me of the non-sense naming issue with runners, where -latest ones are almost never latest. Many complained but they still refuse to fix the bad naming. They should have only used -stable for what now is named -latest and have a real -latest one that points to the future-stable, like ubuntu-22.04 as of today.

[bel0v]

No luck with ubuntu-22.04 either :/

[ssbarnea]

I confirm that bump to v3.1.0 did sort the issue for me. And that could have being avoided if they would have moved the v3 tag when they made the release.

[brotich]

deployed a walkaround on my end using:

• checkout action v3.1.0
• using ubuntu-22-04 as the OS
  i am not sure of both changes are needed, or just the updated of the actions/checkout@v3.1.0

[mcalthrop]

@casperbakker I wonder how long does it takes for them to report the outage, if they will ever do it.

@ssbarnea It's been reported, a few minutes ago:
https://www.githubstatus.com/incidents/gq1x0j8bv67v

We identified an issue with the checkout action download which was causing elevated actions failures. We have performed some mitigations and are continuing investigation.

[fhammerl]

UPDATE: the comment below is regarding the 'permission denied' exception happening on jobContainers introduced in v3.1.0, and is unrelated to the Can't use 'tar -xzf' extract archive file: /home/runner/work/[...] problem experienced on Thu 6 Oct.

@jamengual @ollie-etl

Thanks for reporting this so quickly!

Summary

In short: the jobContainers in these workflows seem to follow an unsupported path. Namely,

Do not use the USER instruction in your Dockerfile, because you won't be able to access the GITHUB_WORKSPACE

I believe the jobContainer operates as the user 'circleci' in the workflow you linked.

We will roll out the changes again as we haven't seen issues with the officially supported paths.

It is recommended 'not use the USER instruction in your Dockerfile'.
In most cases, this means 'run the container as root'.

I understand this introduces some friction, so see below for workarounds and a technical summary.

Detailed Summary

1. In actions/checkout@3.1.0, we began phasing out the use of some workflow commands that write to stdout with functions that write to temp files instead.
2. If you're running in a jobContainer, these temp files are created by the runner app on your host OS and are owned by the user who started the runner (in case of ubuntu-latest, this user is 'runner' with UID-1001). These files are then mounted to your jobContainer.
3. actions/checkout is a .js file executed using node on your jobContainer. As per the changes in 1., actions/checkout@v3.1.0 attempts to write to these temp files on your jobContainer.

This can have a couple of outcomes:

1. If the jobContainer user is 'root': writes successfully.
2. If the jobContainer user is NOT 'root':
   a) If the UID of the jobContainer user == UID of host runner user (UID-1001 on ubuntu-latest): write is successful
   b) If the UID of the jobContainer user != UID of host runner user (UID-1001 on ubuntu-latest): permission denied

The jobContainers in the above workflow seem to run into 2/b.

Fix

As per the docs, don't modify the default user in the Dockerfile. If that's not feasible for you: 👇

Workarounds (unsupported)

• Use actions/checkout@v3.0.2 in your workflow (will be deprecated)
• Match the UID of the host runner user (e.g. UID-1001 on ubuntu-latest today) to the UID of the user on the jobContainer
• Override the default container user and use 'root':

container: 
    image: alpine:latest
    options: --user root

We will roll out the changes again as we haven't seen issues with the officially supported paths.

[fhammerl]

Follow-up in #956