Failed to upload Artifact: ECONNRESET and Socket Disconnection

[killermoehre]

Describe the bug
While try to upload the upload fails with either ECONNRESET or socket disconnected.

Version

• V1
• V2

Environment

• self-hosted
• Linux
• Windows
• Mac

Run actions/upload-artifact@v2.1.4
  with:
    name: undionly.kpxe
    path: ipxe/src/bin/undionly.kpxe
    if-no-files-found: error
  env:
    GITHUB_TOKEN: ***
    filename: undionly.kpxe
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]Search path '/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin/undionly.kpxe'
##[debug]File:/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin/undionly.kpxe was found using the provided searchPath
With the provided path, there will be 1 file(s) uploaded
##[debug]Root artifact directory is /home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin
##[debug]Artifact Url: https://pipelines.actions.githubusercontent.com/bINcheClQHqTZNbVjfeRDuwAx9xssj4r3uOpnOJv4gG2xmi8qY/_apis/pipelines/workflows/212542507/artifacts?api-version=6.0-preview
##[error]Client network socket disconnected before secure TLS connection was established
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Upload bin/undionly.kpxe artifact as undionly.kpxe

Run actions/upload-artifact@v2.1.4
  with:
    name: ipxe.efi
    path: ipxe/src/bin-x86_64-efi/ipxe.efi
    if-no-files-found: error
  env:
    GITHUB_TOKEN: ***
    filename: ipxe.efi
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]followSymbolicLinks 'true'
##[debug]implicitDescendants 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]Search path '/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin-x86_64-efi/ipxe.efi'
##[debug]File:/home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin-x86_64-efi/ipxe.efi was found using the provided searchPath
With the provided path, there will be 1 file(s) uploaded
##[debug]Root artifact directory is /home/runner/work/custom-ipxe/custom-ipxe/ipxe/src/bin-x86_64-efi
##[debug]Artifact Url: https://pipelines.actions.githubusercontent.com/bINcheClQHqTZNbVjfeRDuwAx9xssj4r3uOpnOJv4gG2xmi8qY/_apis/pipelines/workflows/212542507/artifacts?api-version=6.0-preview
##[error]read ECONNRESET
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Upload bin-x86_64-efi/ipxe.efi artifact as ipxe.efi

Run/Repo Url
https://github.com/killermoehre/custom-ipxe/runs/995234331?check_suite_focus=true
https://github.com/killermoehre/custom-ipxe/runs/995234353?check_suite_focus=true

How to reproduce
I don't know. I just run the run.

Additional context
It started suddenly with the linked run. I didn't change anything, my files are quite small, too.

[killermoehre]

Btw, it works now again. This was probably a problem in the actual GitHub API. But even than shouldn't the action handle this better?

[eddieantonio]

I have also had this failure, but re-running the jobs fixes it. Flakey builds are annoying!

https://github.com/UAlbertaALTLab/gunaha/actions/runs/214200167
https://github.com/UAlbertaALTLab/gunaha/actions/runs/212474456

[konradpabjan]

I think I know what is going on here. During artifact upload, there are effectively 3 internal steps

• 1 POST call to create the artifact
• Multiple concurrent PUT calls to upload everything (depends on the size of the artifact and the number of files)
• 1 PATCH call to indicate we are done uploading the artifact

Historically, the second step has been the flakiest since this can be hundreds or even thousands of http calls. There is a robust retry mechanism for this part of the upload (multiple retries with exponential backoff if something goes wrong). Now the first POST call (same with the last PATCH call) does not have any retry mechanism and I suspect something is failing along the way and there is no retry so it aborts and the action fails.

The specific section of code is here for those interested: https://github.com/actions/toolkit/blob/main/packages/artifact/src/internal/upload-http-client.ts#L54:L100

[dmbarry86]

I am experiencing this ECONNRESET issue as well. I'm trying to upload coverage artifacts. In total there are 92 files but they are all very small and the overall size of the bundle is only a few hundred kb.

[kiancross]

I'm also having this problem. I've seen it a couple of times in the past, but suddenly it's become quite frequent today. I'm trying to upload a JSON coverage artifact - the file isn't very big.

EDIT:
This is now happening frequently.

[kakuzei]

I'm having the same issue: a lot of workflow executions failed in the last 3 days with the following error:

Run actions/upload-artifact@v2
With the provided path, there will be 31 file(s) uploaded
Error: read ECONNRESET

[peteryule]

This is now happening for us frequently in the last couple of days.

[ogrisel]

It seems to me that it would be possible to reduce the likelihood of those failures by implementing a retry mechanism with exponential backoff:

• try upload;
• if it fails, print warning message and wait 1s and retry;
• if it still fails, print warning message and wait 2s and retry;
• if it still fails, print warning message and wait 4s and retry;
• if it still fails, print warning message and wait 8s and retry;
• if it still fails, print warning message and wait 16s and retry;
• if it still fails, print error message and abort.

However I have never coded in TypeScript so I am not confident enough to try to contribute this myself.

[kakuzei]

Hello,
We are still encountering the upload artifact issue during the execution of our workflows:

2020-11-30T16:14:03.8299997Z ##[group]Run actions/upload-artifact@v2
2020-11-30T16:14:03.8300677Z with:
2020-11-30T16:14:03.8301354Z   name: ...
2020-11-30T16:14:03.8302112Z   path: ...
2020-11-30T16:14:03.8302824Z   if-no-files-found: warn
2020-11-30T16:14:03.8303422Z env:
...
2020-11-30T16:14:03.8429868Z   ACTIONS_STEP_DEBUG: true
2020-11-30T16:14:03.8430458Z ##[endgroup]
2020-11-30T16:14:03.9223335Z With the provided path, there will be xxx file(s) uploaded
2020-11-30T16:14:05.6239533Z Total size of all the files uploaded is xxx bytes
2020-11-30T16:14:05.6390587Z ##[error]read ECONNRESET

Do you understand why I don't have any debug messages in the logs even if I set the environment variable ACTIONS_STEP_DEBUG to true ?

[jnewland]

Just adding a note here to say that I too ran into this several times this morning, paired with intermittent occurrences of actions/download-artifact#55. I'm curious if the telemetry mentioned over in #137 (comment) might be showing a trend that matches the reports in this issue.

[newhoggy]

Also ran into this issue:

[kroggen]

Me too, the same problem.

I just clicked on Re-run jobs

[konradpabjan]

Should be good now. Retries with exponential back-off have been added to all HTTP calls so any socket disconnects, timeouts, 503s or ECONNRESET errors should gracefully now retry and the artifact upload should go through (previously only a subset of HTTP calls during upload were retried). Recent updates should significantly improve reliability.

If this issue appears again, please open a new issue