Intermittent errors in deploy error handling #10

ilkka · 2016-08-30T10:49:10Z

We are deploying a Netlify site as part of a Jenkins build, and sometimes are seeing what look like network connection issues between our CI provider and Netlify. The troublesome aspect here is that Jenkins fails to flag these builds as failed, thinking that they've succeeded regardless of this error output at the bottom of the console log:

+ netlify deploy -t deadbeef .
Deploying folder: .
{ client: 
   { access_token: 'acdc1234',
     client_id: 'abba4321',
     client_secret: undefined,
     redirect_uri: undefined,
     ENDPOINT: 'https://api.netlify.com',
     VERSION: 'v1',
     hostname: 'api.netlify.com',
     ssl: [ 'https://', index: 0, input: 'https://api.netlify.com' ],
     port: 443,
     http: 
      { Server: [Object],
        createServer: [Function],
        globalAgent: [Object],
        Agent: [Object],
        request: [Function],
        get: [Function] } },
  data: 
   { Error: socket hang up
       at createHangUpError (_http_client.js:250:15)
       at TLSSocket.socketOnEnd (_http_client.js:342:23)
       at emitNone (events.js:91:20)
       at TLSSocket.emit (events.js:185:7)
       at endReadableNT (_stream_readable.js:926:12)
       at _combinedTickCallback (internal/process/next_tick.js:74:11)
       at process._tickCallback (internal/process/next_tick.js:98:9) code: 'ECONNRESET' },
  meta: null }
Potentially unhandled rejection [3] SyntaxError: Unexpected token E in JSON at position 0
    at Object.parse (native)
    at Object.exports.log (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/lib/helpers/error_logger.js:11:21)
    at /data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/lib/commands/deploy.js:144:17
    at tryCatchReject (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:845:30)
    at runContinuation1 (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:804:4)
    at Rejected.when (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:625:4)
    at Pending.run (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/makePromise.js:483:13)
    at Scheduler._drain (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/Scheduler.js:62:19)
    at Scheduler.drain (/data/jenkins-vrci/home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node.js_6.2.0/lib/node_modules/netlify-cli/node_modules/when/lib/Scheduler.js:27:9)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)

The "error handling error" seems to be trying to parse error.data, containing the string "Error: ..." in error_logger.js. Don't know why that would cause an exit with return code 0 though, or if there are more Jenkins shenanigans actually causing that.

The text was updated successfully, but these errors were encountered:

schickling · 2016-09-28T12:04:35Z

Same here when running via CircleCI. Unfortunately this blocks us from using Netlify.

schickling · 2016-09-28T12:07:34Z

Any idea @biilmann @cassiozen?

biilmann · 2016-09-28T16:54:25Z

I pushed an update to netlify-cli yesterday that mainly pulls in an updated version of the node client library. The fix in the node client was related to deploys with many concurrent uploads failing with that kind of error. Can you make sure you're on netlify-cli v1.0.3 and let me know if that version still has the problem?

schickling · 2016-09-28T18:37:32Z

Hey @biilmann. I've installed the netlify-cli today, so it's using version 1.0.3. Could there be a different issue?

fool · 2016-09-28T23:00:57Z

ah, I filed #13 today - to give us some better debugging tools around this same issue.

schickling · 2016-10-05T12:21:20Z

Any update on this? 80% of our builds are seeing this error...

fool · 2016-10-19T20:35:57Z

Hi @schickling and @ilkka - I think we just fixed the longer term problem with larger deploys. Can you give v1.1.0 a spin and let us know how it goes?

ilkka · 2016-10-20T06:48:33Z

Thanks, we'll bump our dependency and see how it goes!

fool · 2016-10-26T00:51:48Z

OK, this is still an issue with version 1.1.0 and I've now seen it firsthand and think we (@netlify) will be able to reproduce well enough to debug. Is there anyone attached to this issue (mainly not sure about @ilkka ) who isn't using CircleCI when they get the error?

It felt to me like the "working" deploys completed at least an order of magnitude faster than the failing ones, so while I'm not suggesting this as a long-term workaround or a fix, you may get some mileage out of a stopgap with something like the first answer here to restart a failed deploy rather than having to redo the build:

http://stackoverflow.com/questions/5161193/bash-script-that-kills-a-child-process-after-a-given-timeout

fool · 2016-11-11T23:28:42Z

We've hit some walls in debugging this and want to take it to CircleCI - but are not paying customers.

Does one of you who is a paying customer mind writing in with a real ticket (rather than a forum post which is all we can do for free) and cc:ing myself (chrism@netlify.com) and our buildmaster (ryan@netlify.com) so we can present our evidence and ask them for help? After that point we probably don't actually need you to spend any time at it, just trying to get in the door :)

If someone does it please comment here so we don't hit them multiple times in parallel. (cc @schickling and @ilkka)

fool · 2016-11-16T16:58:24Z

hmm, haven't heard back from anyone yet but would like to make some progress on this - @schickling any chance you can email Circle and cc: us?

schickling · 2016-11-16T17:21:37Z

Sorry we're also not on the needed pricing plan and they haven't responded to our previous inquiries yet.

ilkka · 2016-11-18T07:23:13Z

Yeah, sorry, we're still only freeloaders on CircleCI too.

fool · 2016-11-21T23:19:32Z

seems to be happening to TravisCI customers as well sometimes, grr! (https://app.intercom.io/a/apps/q245f50x/inbox/conversation/6853522502)

schickling · 2016-11-22T09:31:01Z

Yeah, it's gotten pretty bad for us. Now happening for 9/10 builds for some repos...

schickling · 2016-12-09T18:12:21Z

Any progress on this?

rybit · 2016-12-09T19:31:43Z

I was looking for this ticket!
Ok ~ so we went and built up a simple cli tool in go based on our open-api client. It doesn't have feature parity with the current cli, but we are slowly working on that. I built up releases for linux and osx too. I can build more, but just haven't had the motivation :)

check it out! https://github.com/netlify/netlifyctl

If you want to see an example of using it in circle-ci check out this https://github.com/rybit/circle-ci-test-repo

schickling · 2016-12-10T12:38:29Z

Thanks for this update @rybit! Are there any plans to phase out the JS version of netlify-cli in favor of the Go implementation or is this rather considered as a workaround for now?

rybit · 2016-12-12T18:09:39Z

That is definitely part of an ongoing discussion we're having, but it will definitely be supported for the foreseeable future. Idealy we would bundle it inside a node module, but that is still a ways out.

aj0strow · 2016-12-16T22:49:05Z

I actually fixed it twice, while I was waiting for the first one to go through Circle Ci.

Hope this helps.

schickling · 2016-12-17T09:38:38Z

@aj0strow can you create a PR for the original netlify-cli tool? 🙂

jppurcell9 · 2016-12-30T17:44:11Z

I'm also seeing this error every time using bitbucket pipelines. The call that hangs for me is the createDeploy call in node-client: https://github.com/netlify/node-client/blob/22117378564500d082b281f4e8c0f62e4ca5fe56/lib/site.js#L144

It's failing with a socket hangup, which looks like it's actually making the connection but not receiving any data. Do you see the connection being made on your side? I'm able to make the GET request for the site before this call, so at least something is making a connection.

Here's the request data if it helps (without the 161 files from the post body):

making request /sites/<my site id>/deploys {"files":{...},"draft":false}
{ method: 'post',
  hostname: 'api.netlify.com',
  path: '/api/v1/sites/<my site id>/deploys',
  port: 443,
  headers: 
   { 'Content-Type': 'application/json',
     'Content-Length': 12835,
     Authorization: 'Bearer $NETLIFY_TOKEN' },
  auth: null }

jppurcell9 · 2016-12-31T21:15:27Z

I was able to hack something together to get this to work, but I'm still not sure of the root cause. I found that simply retrying the request in node wasn't doing the trick. Instead I was able to fall back to curl to make the request instead. It's a total hack, but the code is here: https://github.com/jppurcell9/node-client/tree/feature/retry-if-no-data

rybit · 2017-01-03T17:59:26Z

@jppurcell9 have you tried the go client? It might be easier than subshelling out to curl

jppurcell9 · 2017-01-04T02:52:36Z

@rybit I got the go client to work. Thanks!

rozhok · 2017-02-16T21:56:20Z

All of sudden I've starting to get the same error in my jenkins machine. My build is very small and consists only of few dozens of files less than 2 MiBs in total.
Also, when tried to upload stuff from my laptop and faced same issue. Will give a try for go client instead.

schickling · 2017-02-17T17:32:31Z

Any ETA on when the go binary will be shipped via NPM? It's a pretty big deal for us.

rozhok · 2017-02-20T10:53:51Z

@schickling from what I understand, CircleCI allows to install many runtimes/libs into build image and also allow to download custom binaries so you could just use netlifyctl binaries taken from https://github.com/netlify/netlifyctl/releases

schickling · 2017-02-20T11:08:00Z

@rozhok it's rather a matter of convenience/maintainability since we're mainting dozens of repos using Netlify for deployment.

BorisVaskin · 2017-05-02T09:58:27Z

Having the same issue on Visual Studio Team Services build system. The netlify deploy is almost always failing with mentioned error.

2017-05-01T16:26:46.5666661Z ##[command]netlify deploy -s ******** -p C:\a\1\a -t ********
2017-05-01T16:26:47.7018809Z Deploying folder: C:\a\1\a
2017-05-01T16:28:47.8972080Z { client: 
2017-05-01T16:28:47.8972080Z    { access_token: '********',
2017-05-01T16:28:47.8972080Z      client_id: '********',
2017-05-01T16:28:47.8982076Z      client_secret: undefined,
2017-05-01T16:28:47.8982076Z      redirect_uri: undefined,
2017-05-01T16:28:47.8982076Z      ENDPOINT: 'https://api.netlify.com',
2017-05-01T16:28:47.8982076Z      VERSION: 'v1',
2017-05-01T16:28:47.8982076Z      hostname: 'api.netlify.com',
2017-05-01T16:28:47.8982076Z      ssl: [ 'https://', index: 0, input: 'https://api.netlify.com' ],
2017-05-01T16:28:47.8982076Z      port: 443,
2017-05-01T16:28:47.8982076Z      http: 
2017-05-01T16:28:47.8982076Z       { Server: [Object],
2017-05-01T16:28:47.8982076Z         createServer: [Function],
2017-05-01T16:28:47.9052085Z Potentially unhandled rejection [3] SyntaxError: Unexpected token E in JSON at position 0
2017-05-01T16:28:47.9052085Z         globalAgent: [Object],
2017-05-01T16:28:47.9062085Z     at Object.parse (native)
2017-05-01T16:28:47.9062085Z         Agent: [Object],
2017-05-01T16:28:47.9072087Z     at Object.exports.log (C:\NPM\Modules\node_modules\netlify-cli\lib\helpers\error_logger.js:11:21)
2017-05-01T16:28:47.9072087Z         request: [Function],
2017-05-01T16:28:47.9072087Z     at C:\NPM\Modules\node_modules\netlify-cli\lib\commands\deploy.js:144:17
2017-05-01T16:28:47.9072087Z         get: [Function] } },
2017-05-01T16:28:47.9072087Z     at tryCatchReject (C:\NPM\Modules\node_modules\netlify-cli\node_modules\when\lib\makePromise.js:845:30)
2017-05-01T16:28:47.9072087Z   data: 
2017-05-01T16:28:47.9072087Z     at runContinuation1 (C:\NPM\Modules\node_modules\netlify-cli\node_modules\when\lib\makePromise.js:804:4)
2017-05-01T16:28:47.9072087Z    { Error: socket hang up
2017-05-01T16:28:47.9072087Z     at Rejected.when (C:\NPM\Modules\node_modules\netlify-cli\node_modules\when\lib\makePromise.js:625:4)
2017-05-01T16:28:47.9082084Z        at createHangUpError (_http_client.js:254:15)
2017-05-01T16:28:47.9082084Z     at Pending.run (C:\NPM\Modules\node_modules\netlify-cli\node_modules\when\lib\makePromise.js:483:13)
2017-05-01T16:28:47.9082084Z        at TLSSocket.socketOnEnd (_http_client.js:346:23)
2017-05-01T16:28:47.9082084Z     at Scheduler._drain (C:\NPM\Modules\node_modules\netlify-cli\node_modules\when\lib\Scheduler.js:62:19)
2017-05-01T16:28:47.9082084Z        at emitNone (events.js:91:20)
2017-05-01T16:28:47.9082084Z     at Scheduler.drain (C:\NPM\Modules\node_modules\netlify-cli\node_modules\when\lib\Scheduler.js:27:9)
2017-05-01T16:28:47.9082084Z        at TLSSocket.emit (events.js:185:7)
2017-05-01T16:28:47.9082084Z     at _combinedTickCallback (internal/process/next_tick.js:67:7)
2017-05-01T16:28:47.9082084Z        at endReadableNT (_stream_readable.js:974:12)
2017-05-01T16:28:47.9082084Z        at _combinedTickCallback (internal/process/next_tick.js:74:11)
2017-05-01T16:28:47.9082084Z        at process._tickCallback (internal/process/next_tick.js:98:9) code: 'ECONNRESET' },
2017-05-01T16:28:47.9092083Z   meta: null }

schickling · 2017-05-02T11:10:26Z

Any update on this @netlify? This is still a huge pain for us.

fool · 2017-05-02T14:58:08Z

While pull requests are welcomed, we will probably not put a lot more development effort into netlify-cli

https://github.com/netlify/netlifyctl is the tool that can reliably deploy today, and will definitely be developed further in the future.

bcomnes · 2018-08-27T22:19:18Z

Closing since this issue appears stale. Feel free to reopen if we run into similar issues on the 2.0.0 node cli release.

Cjay123 · 2019-02-06T06:36:43Z

After deploying my project to netlify. I saw something wrong to my website. My imported assets cant read by netlify but i can run it well on localhost. Anything can help me about my problem?

bcomnes · 2019-02-06T23:23:09Z

Sorry @Cjay123, doesn't ring a bell. If you have a reproducible problem you can demonstrate in an issue, gist or repo, please open an issue.

worrawutp · 2019-02-21T17:02:03Z

I have this problem when I do $ netlify deploy --prod on the first time. It show something like...

Deploying to live site URL...
√ Finished hashing 7 files
√ CDN requesting 7 files
√ Finished uploading 7 assets
| Waiting for deploy to go live... »   Warning:
 »   {
 »      "message": "request to
 »   https://api.netlify.com/api/v1/sites/04c14fe2-e27a-4ed4-b10c-f81a2bfbd27c/deplo
 »   ys/5c6ed79fef12041c97a94db0 failed, reason: read ECONNRESET",
 »      "type": "system",
 »      "errno": "ECONNRESET",
 »      "code": "ECONNRESET",
 »      "name": "FetchError",

but, after few seconds I run the same command again it then work fine.
and I got

Deploying to live site URL...
√ Finished hashing 7 files
√ CDN requesting 0 files
√ Finished uploading 0 assets
√ Deploy is live!

Not sure if it help to point out to the cause of the problem...

bcomnes · 2019-02-21T17:13:59Z

Nice catch @worrawutp, we should make the polling step while waiting for the deploy to finish up more resilient to random errors or transport errors.

* chore: change package details * chore: update package * chore: bump * chore: add @netlify/edge-bundler * chore: bump * chore: remove npmrc line * chore: bump version * chore: update @netlify/edge-bundler * feat: deploy Edge Handlers to internal path * feat: add local development experience * chore: bump version * chore: update @netlify/edge-bundler * refactor: move `watcher` util * feat: add `configWatcher` * feat: updates to Edge Handlers dev server * chore: update @netlify/edge-bundler * refactor: remove unused error binding * chore: update package name * chore: update deps * fix: use correct URL path when matching Edge Handlers * chore: bump version * chore: bump version * feat: add support for internal handlers in local dev (#1) * feat: add support for internal handlers in local dev * refactor: correctly merge declarations * chore: bump version * fix: run config watcher only on certain commands (#2) * chore: bump version * chore: bump version * feat: add support for user import maps (#3) * feat: add support for user import maps * feat: warn when failing to read import map file * chore: bump version * chore: bump version * fix: improve error handling for Edge Handlers (#4) * chore: bump version * feat: listen for `NETLIFY_DENO_DEBUG` environment variable (#5) * feat: listen for `NETLIFY_DENO_DEBUG` environment variable * chore: upgrade @netlify-labs/edge-bundler * chore: bump version * fix: use `0.0.0.0` for Deno server host (#6) * fix: wait for Edge Handlers server to become available (#7) * fix: wait for Edge Handlers server to become available * refactor: wait for server inside request * refactor: increase timeout * refactor: use watcher from correct location * chore: update package.json * chore: disable release-please * feat: replace @netlify/build with @netlify-labs/build-internal * chore: use token in CI * refactor: use existing config watcher * chore: remove unused variables * chore: remove actions * chore: remove action * chore: update test * chore: add tests * chore: move env to job * chore: remove duplicate env * chore: fix tests * chore: remove unused imports * fix: check if EH directory exists before deploy * feat: rename Edge Handlers to Edge Functions (#9) * chore: fix linting issue * chore: fix tests * chore: fix test * chore: add `withEdgeFunction` test helper * chore: remove `console.log` * feat: add `generate` command * chore: add tests * refactor: rename variable * refactor: remove old Edge Handlers code * chore: update packages * fix: remove reference to traffic-mesh * chore: fix linting issue * chore: fix linting issue * refactor: remove unused code * refactor: remove `dev:trace` command * chore: setup Deno in the CI * refactor: use different localhost IP * chore: fix linting issues * refactor: remove nullish coalesce * refactor: remove nullish coalesce * chore: undo test change * fix: unixify Windows paths * feat: set `deno.path` in VS Code settings * chore: update test snapshot * refactor: update message * refactor: rename command * chore: update command name in test * chore: update test snapshot * chore: add test snapshot

fool mentioned this issue Sep 28, 2016

Add a verbose mode #13

Closed

fool added the bug label Oct 19, 2016

EvanLovely mentioned this issue Jan 23, 2018

Adding Netlify deploy script that only deploys to main site on main branch boltdesignsystem/bolt#374

Merged

bcomnes closed this as completed Aug 27, 2018

bcomnes mentioned this issue Feb 21, 2019

Deploy step fails on random steps due to network errors netlify/js-client#38

Open

Intermittent errors in deploy error handling #10

Intermittent errors in deploy error handling #10

Comments

ilkka commented Aug 30, 2016

schickling commented Sep 28, 2016 • edited Loading

schickling commented Sep 28, 2016

biilmann commented Sep 28, 2016

schickling commented Sep 28, 2016

fool commented Sep 28, 2016 • edited Loading

schickling commented Oct 5, 2016

fool commented Oct 19, 2016

ilkka commented Oct 20, 2016

fool commented Oct 26, 2016

fool commented Nov 11, 2016 • edited Loading

fool commented Nov 16, 2016

schickling commented Nov 16, 2016

ilkka commented Nov 18, 2016

fool commented Nov 21, 2016

schickling commented Nov 22, 2016

schickling commented Dec 9, 2016

rybit commented Dec 9, 2016

schickling commented Dec 10, 2016

rybit commented Dec 12, 2016

aj0strow commented Dec 16, 2016

schickling commented Dec 17, 2016

jppurcell9 commented Dec 30, 2016

jppurcell9 commented Dec 31, 2016

rybit commented Jan 3, 2017

jppurcell9 commented Jan 4, 2017

rozhok commented Feb 16, 2017 • edited Loading

schickling commented Feb 17, 2017

rozhok commented Feb 20, 2017

schickling commented Feb 20, 2017

BorisVaskin commented May 2, 2017

schickling commented May 2, 2017

fool commented May 2, 2017

bcomnes commented Aug 27, 2018

Cjay123 commented Feb 6, 2019

bcomnes commented Feb 6, 2019

worrawutp commented Feb 21, 2019

bcomnes commented Feb 21, 2019

schickling commented Sep 28, 2016 •

edited

Loading

fool commented Sep 28, 2016 •

edited

Loading

fool commented Nov 11, 2016 •

edited

Loading

rozhok commented Feb 16, 2017 •

edited

Loading