-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stacktrace in heartbeat #9556
Comments
Pinging @elastic/uptime |
The issue here is that we don't handle the case where a certificate doesn't have either Thanks for the report! Working on a fix. |
OK, we've got a PR in #9566 Out of curiosity @smartrics where did this cert come from? It's quite rare to find a cert with that field missing? |
Hi @andrewvc this is a snippet from the script that builds the certificate(s)
|
Thanks for the info @tonybaines . Weirdly checking that cert I see both dates set:
Could you check the cert you have with |
Yup @andrewvc, the x509 dump of the certificate on a 'real' server does include both dates (creation date and then 9999 days later) |
@tonybaines that's really weird then. The way the failure happened made it seem pretty clear to me that it was missing that date, but if it's there then... weird? Any chance that is a public endpoint or that you can share the cert? |
Or, is there any chance you could build heartbeat off the branch in #9566 and see if the problem still occurs? |
The cert is a one-off for dev only, generated by the command above |
Hi @andrewvc - what official version of heartbeat is this patch likely to be available in? |
Some certs in the wild don't set these standard fields and can cause an NPE Fixes elastic#9556
* [Heartbeat] Handle TLS certs missing notBefore/notAfter Some certs in the wild don't set these standard fields and can cause an NPE Fixes #9556 * Add changelog entry
* [Heartbeat] Handle TLS certs missing notBefore/notAfter Some certs in the wild don't set these standard fields and can cause an NPE Fixes elastic#9556 * Add changelog entry
@smartrics I'm aiming for 6.6 with this backport in #9758 , but I can't promise it'll make that. If not, we'll try for 6.6.1 or 6.7 |
…ore/notAfter (#9759) * Add heartbeat test for TLS client cert auth (#8984) (#9676) * Add heartbeat test for TLS client cert auth We were missing a test for this specific case. I wrote this hoping to confirm #8979, but actually wound up disproving it. That said, this is still a good test to have, so we should merge it. * [Heartbeat] Handle TLS certs missing notBefore/notAfter (#9566) Some certs in the wild don't set these standard fields and can cause an NPE Fixes #9556
* [Heartbeat] Handle TLS certs missing notBefore/notAfter Some certs in the wild don't set these standard fields and can cause an NPE Fixes elastic#9556 * Add changelog entry (cherry picked from commit 337113e)
Hello I have rebuilt the master branch but we still get a NPE
|
Hey thanks for checking out master. I'll look into this soon, NPEs are
always serious business.
…On Wed, Jan 9, 2019, 4:25 PM Fabrizio Cannizzo ***@***.*** wrote:
Hello I have rebuilt the master branch but we still get a NPE
2019-01-09T14:07:21.818Z ERROR scheduler/scheduler.go:355 Panic in job ***@***.***://localhost/gecko/health'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2019-01-09T14:07:21.818Z ERROR scheduler/scheduler.go:355 Panic in job ***@***.***://localhost/agamid/health'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2019-01-09T14:07:21.819Z ERROR scheduler/scheduler.go:357 Stacktrace: goroutine 9971 [running]:
runtime/debug.Stack(0x1651dc9, 0x3a, 0xc00049e8e8)
/home/fabrizio/Apps/go-1.11.2/src/runtime/debug/stack.go:24 +0xa7github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1.1(0xc0001421e0, 0xc000204080)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:357 +0xf7
panic(0x14ab860, 0x21e0640)
/home/fabrizio/Apps/go-1.11.2/src/runtime/panic.go:513 +0x1b9github.com/elastic/beats/heartbeat/monitors/active/dialchain.TLSLayer.func1.1(0x1752020, 0xc00042f880, 0x3, 0xc00042a7c0, 0xd, 0x1752020)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/tls.go:83 +0x455github.com/elastic/beats/heartbeat/monitors/active/dialchain.afterDial.func1(0x1615705, 0x3, 0xc00042a7c0, 0xd, 0x0, 0x0, 0x0, 0x0)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/util.go:89 +0xbagithub.com/elastic/beats/libbeat/outputs/transport.DialerFunc.Dial(0xc000503880, 0x1615705, 0x3, 0xc00042a7c0, 0xd, 0xc0004a8de0, 0x95247f, 0xc00038a870, 0x10)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/libbeat/outputs/transport/transport.go:40 +0x4egithub.com/elastic/beats/heartbeat/monitors/active/http.(*SimpleTransport).RoundTrip(0xc00041d500, 0xc0005dea00, 0x0, 0x0, 0x0)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/simple_transp.go:84 +0xfe
net/http.send(0xc0005de900, 0x17363a0, 0xc00041d500, 0xbf059dea6851c8b8, 0x5bd8dbb557, 0x223e5e0, 0xc00000e408, 0xbf059dea6851c8b8, 0xc0004a9010, 0x1)
/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc00041d530, 0xc0005de900, 0xbf059dea6851c8b8, 0x5bd8dbb557, 0x223e5e0, 0xc00000e408, 0x0, 0x1, 0x7f9b2aaa3758)
/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc00041d530, 0xc0005de900, 0x0, 0x0, 0x0)
/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc00041d530, 0xc0005de900, 0x223e5e0, 0x15b6fe0, 0xc000153201)
/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:509 +0x35github.com/elastic/beats/heartbeat/monitors/active/http.execRequest(0xc00041d530, 0xc0005de900, 0x1668520, 0xbf059de66851c77a, 0x581f2f1416, 0x223e5e0, 0x0, 0x0, 0x0, 0x0, ...)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:271 +0xb7github.com/elastic/beats/heartbeat/monitors/active/http.execPing(0xc00041d530, 0xc000168900, 0x0, 0x0, 0x0, 0x3b9aca000, 0x1668520, 0x0, 0x0, 0x0, ...)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:245 +0x13cgithub.com/elastic/beats/heartbeat/monitors/active/http.createPingFactory.func1(0xc00041cde0, 0x0, 0x0, 0x0)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:188 +0x477github.com/elastic/beats/heartbeat/monitors.MakePingIPFactory.func1.1(0x203000, 0x203000, 0x95247f)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:194 +0x2egithub.com/elastic/beats/heartbeat/monitors.MakeSimpleCont.func1(0x10, 0x14e67c0, 0xc0004a9770, 0x95247f, 0xc000503740, 0x20)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:184 +0x26github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc00038a780, 0x18, 0x20, 0x20, 0xc000503740, 0x0, 0xc000120000)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x27github.com/elastic/beats/heartbeat/monitors.WithFields.func1(0xc0004a9701, 0xc000503740, 0xc0004a97c0, 0xd21361, 0x15189e0, 0xc000503740)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:398 +0x4agithub.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc000503740, 0x1738a60, 0xc00038a780, 0x1738a60, 0xc000503740, 0x223e5e0, 0x17a17)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x27github.com/elastic/beats/heartbeat/monitors.makeByHostAnyIPJob.func1(0xde6839, 0xc0002e00f0, 0x1508d60, 0xc000502520, 0x99, 0x0)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:323 +0x8edgithub.com/elastic/beats/heartbeat/monitors.annotated.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0001fcac0, 0xa, 0xffffffffffff, ...)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:140 +0xc7github.com/elastic/beats/heartbeat/monitors.MakeJob.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xacc910, 0xc00005c530, 0xbe7a63, ...)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:127 +0xe8github.com/elastic/beats/heartbeat/monitors.(*funcJob).Run(0xc0001b0760, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00005c608, 0x951ff3, ...)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:438 +0x82github.com/elastic/beats/heartbeat/monitors.Job.Run-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbe693c, 0xc000284600, 0x1735fa0, ...)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/task.go:120 +0x86github.com/elastic/beats/heartbeat/monitors.(*task).prepareSchedulerJob.func1(0x10, 0x1668570, 0xc0001421e0)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/task.go:85 +0x7agithub.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1(0xc0001421e0, 0xc000204080, 0xc0001421e0, 0xc0001b0940)
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:362 +0x5b
created by github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask
/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:352 +0x5c
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#9556 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAIBY9DV2Z0TsvNJt45-_mv428zByb3kks5vBmxCgaJpZM4ZTvWV>
.
|
@smartrics are you certain you're running the current version master? Which commit hash did you build? Looking at the trace I see the line: That line is currently a comment. I suspect you're running an older commit |
@andrewvc I have rebuilt from the latest master branch (which I believe is a 7.0.0 version) and I am not seeing any NPE. I am happy to verify it on any other branch if you suggest which one should I checkout |
@smartrics glad to hear it :) #9758 is the backport FYI. It will probably make it into 6.7 |
… notBefore/notAfter (elastic#9759) * Add heartbeat test for TLS client cert auth (elastic#8984) (elastic#9676) * Add heartbeat test for TLS client cert auth We were missing a test for this specific case. I wrote this hoping to confirm elastic#8979, but actually wound up disproving it. That said, this is still a good test to have, so we should merge it. * [Heartbeat] Handle TLS certs missing notBefore/notAfter (elastic#9566) Some certs in the wild don't set these standard fields and can cause an NPE Fixes elastic#9556
For our http module configuration
We see this stack trace
The text was updated successfully, but these errors were encountered: