Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stacktrace in heartbeat #9556

Closed
smartrics opened this issue Dec 14, 2018 · 16 comments
Closed

stacktrace in heartbeat #9556

smartrics opened this issue Dec 14, 2018 · 16 comments
Assignees
Labels
Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Comments

@smartrics
Copy link

- Version:  6.5.1
- Operating System: linux centos 7 

For our http module configuration

- type: http
  name: foo/bar
  # List or urls to query
  urls: [
    "http://localhost/analytics",
    "https://localhost/analytics"
  ]
  # Configure task schedule
  schedule: "@every 30s"
  username: yyy
  password: zzz
  ssl:
    verification_mode: none

We see this stack trace

2018-12-14T14:54:24.421Z	ERROR	scheduler/scheduler.go:355	Panic in job foo/bar@https://localhost/analytics'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2018-12-14T14:54:24.422Z	ERROR	scheduler/scheduler.go:357	Stacktrace: goroutine 17450 [running]:
runtime/debug.Stack(0x16ad273, 0x3a, 0xc42058c8b8)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1.1(0xc4201aa6e0, 0xc4203a6080)
	/go/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:357 +0xf7
panic(0x14f98a0, 0x1f689f0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/elastic/beats/heartbeat/monitors/active/dialchain.TLSLayer.func1.1(0x1757340, 0xc4201de700, 0x3, 0xc4201733c0, 0xd, 0x1757340)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/tls.go:83 +0x45b
github.com/elastic/beats/heartbeat/monitors/active/dialchain.afterDial.func1(0x1670dc2, 0x3, 0xc4201733c0, 0xd, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/util.go:89 +0xba
github.com/elastic/beats/libbeat/outputs/transport.DialerFunc.Dial(0xc42000d320, 0x1670dc2, 0x3, 0xc4201733c0, 0xd, 0x9bfe99, 0xc4206a5630, 0x50, 0x48)
	/go/src/github.com/elastic/beats/libbeat/outputs/transport/transport.go:40 +0x4e
github.com/elastic/beats/heartbeat/monitors/active/http.(*SimpleTransport).RoundTrip(0xc420693650, 0xc42019ee00, 0x0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/simple_transp.go:84 +0xfe
net/http.send(0xc42019ed00, 0x173b820, 0xc420693650, 0xbefd0eec14077578, 0xba8640061a, 0x1fc6900, 0xc4200bc2b0, 0xbefd0eec14077578, 0xc4205a9068, 0x1)
	/usr/local/go/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0xc420693680, 0xc42019ed00, 0xbefd0eec14077578, 0xba8640061a, 0x1fc6900, 0xc4200bc2b0, 0x0, 0x1, 0x3)
	/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0xc420693680, 0xc42019ed00, 0x1fc6900, 0xc42019ed00, 0xb1597b)
	/usr/local/go/src/net/http/client.go:615 +0x28d
github.com/elastic/beats/heartbeat/monitors/active/http.execRequest(0xc420693680, 0xc42019ed00, 0x16bffd0, 0xbefd0ee814077485, 0xb6cc93652a, 0x1fc6900, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:252 +0xa6
github.com/elastic/beats/heartbeat/monitors/active/http.execPing(0xc420693680, 0xc4201bca00, 0x0, 0x0, 0x0, 0x3b9aca000, 0x16bffd0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:226 +0x13c
github.com/elastic/beats/heartbeat/monitors/active/http.createPingFactory.func1(0xc420693500, 0x9bf995, 0x7f8cec1221c8, 0x81)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:171 +0x442
github.com/elastic/beats/heartbeat/monitors.MakePingIPFactory.func1.1(0x6, 0xc4205a9778, 0x9bfe99)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:194 +0x31
github.com/elastic/beats/heartbeat/monitors.MakeSimpleCont.func1(0x10, 0x1534380, 0xc4205a97a8, 0x9bfe99, 0xc42000d060, 0x20)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:184 +0x26
github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc4200705e0, 0x18, 0x20, 0x20, 0xc42000d060, 0x0, 0xc42008cc00)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x27
github.com/elastic/beats/heartbeat/monitors.WithFields.func1(0xc4205a9701, 0xc42000d060, 0xc4205a97f8, 0xd62c21, 0x1566780, 0xc42000d060)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:398 +0x4a
github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc42000d060, 0x173dee0, 0xc4200705e0, 0x173dee0, 0xc42000d060, 0x1fc6900, 0x14a912)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x27
github.com/elastic/beats/heartbeat/monitors.makeByHostAnyIPJob.func1(0x0, 0x0, 0xc4203389b0, 0x9e43ab, 0x0, 0xc4203389c0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:323 +0x939
github.com/elastic/beats/heartbeat/monitors.annotated.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42040e7c0, 0xa, 0xffffffffffffffff, ...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:140 +0xc7
github.com/elastic/beats/heartbeat/monitors.MakeJob.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420346000, 0xc420296f98, 0x0, ...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:127 +0xe8
github.com/elastic/beats/heartbeat/monitors.(*funcJob).Run(0xc4201cf3c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42015e388, 0xc42015e438, ...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/util.go:438 +0x82
github.com/elastic/beats/heartbeat/monitors.(Job).Run-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420381701, 0xc4201b6020, 0xa27918, ...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/task.go:120 +0x86
github.com/elastic/beats/heartbeat/monitors.(*task).prepareSchedulerJob.func1(0x10, 0x16c0020, 0xc4201aa6e0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/task.go:85 +0x7d
github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1(0xc4201aa6e0, 0xc4203a6080, 0xc4201aa6e0, 0xc4201cfc40)
	/go/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:362 +0x5b
created by github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask
	/go/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:352 +0x5c



@ruflin ruflin added Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Dec 14, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime

@andrewvc
Copy link
Contributor

The issue here is that we don't handle the case where a certificate doesn't have either notBefore or notAfter field. It looks like, according the RFC you only need to specify one of the two.

Thanks for the report! Working on a fix.

@andrewvc
Copy link
Contributor

OK, we've got a PR in #9566

Out of curiosity @smartrics where did this cert come from? It's quite rare to find a cert with that field missing? openssl won't let you make a cert like that.

@tonybaines
Copy link

Hi @andrewvc this is a snippet from the script that builds the certificate(s)

    # Generate
    openssl req -x509 -newkey rsa:4096 -keyout "$CERT_NAME-privateKey.pem" -out "$CERT_NAME-certificate.pem" -days 9999 -nodes -subj "/C=GB/ST=Suffolk/L=Ipswich/O=BT/OU=UTA/OU=IPT/CN=www.bt.com"

    # haproxy needs a combined private-key and public cert
    cat "$CERT_NAME-privateKey.pem" "$CERT_NAME-certificate.pem" > "$CERT_NAME-combined.pem"

@andrewvc
Copy link
Contributor

Thanks for the info @tonybaines . Weirdly checking that cert I see both dates set:

➜  ~ openssl x509 -in -certificate.pem -text
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 12732667726718559499 (0xb0b385d63e23e50b)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=GB, ST=Suffolk, L=Ipswich, O=BT, OU=UTA, OU=IPT, CN=www.bt.com
        Validity
            Not Before: Dec 17 16:23:32 2018 GMT
            Not After : May  3 16:23:32 2046 GMT
...

Could you check the cert you have with openssl x509 -in -certificate.pem -text and let me know if you see both NotBefore and NotAfter?

@tonybaines
Copy link

Yup @andrewvc, the x509 dump of the certificate on a 'real' server does include both dates (creation date and then 9999 days later)

@andrewvc
Copy link
Contributor

@tonybaines that's really weird then. The way the failure happened made it seem pretty clear to me that it was missing that date, but if it's there then... weird?

Any chance that is a public endpoint or that you can share the cert?

@andrewvc
Copy link
Contributor

Or, is there any chance you could build heartbeat off the branch in #9566 and see if the problem still occurs?

@tonybaines
Copy link

The cert is a one-off for dev only, generated by the command above
issue-9556-example-cert.pem.zip

@smartrics
Copy link
Author

Hi @andrewvc - what official version of heartbeat is this patch likely to be available in?

andrewvc added a commit to andrewvc/beats that referenced this issue Dec 20, 2018
Some certs in the wild don't set these standard fields and can cause an NPE

Fixes elastic#9556
andrewvc added a commit that referenced this issue Dec 21, 2018
* [Heartbeat] Handle TLS certs missing notBefore/notAfter

Some certs in the wild don't set these standard fields and can cause an NPE

Fixes #9556

* Add changelog entry
andrewvc added a commit to andrewvc/beats that referenced this issue Dec 21, 2018
* [Heartbeat] Handle TLS certs missing notBefore/notAfter

Some certs in the wild don't set these standard fields and can cause an NPE

Fixes elastic#9556

* Add changelog entry
@andrewvc
Copy link
Contributor

andrewvc commented Dec 21, 2018

@smartrics I'm aiming for 6.6 with this backport in #9758 , but I can't promise it'll make that. If not, we'll try for 6.6.1 or 6.7

andrewvc added a commit that referenced this issue Dec 30, 2018
…ore/notAfter (#9759)

* Add heartbeat test for TLS client cert auth (#8984) (#9676)

* Add heartbeat test for TLS client cert auth

We were missing a test for this specific case. I wrote this hoping to confirm #8979, but actually wound up disproving it.

That said, this is still a good test to have, so we should merge it.

* [Heartbeat] Handle TLS certs missing notBefore/notAfter (#9566)

Some certs in the wild don't set these standard fields and can cause an NPE

Fixes #9556
andrewvc added a commit to andrewvc/beats that referenced this issue Dec 31, 2018
* [Heartbeat] Handle TLS certs missing notBefore/notAfter

Some certs in the wild don't set these standard fields and can cause an NPE

Fixes elastic#9556

* Add changelog entry

(cherry picked from commit 337113e)
andrewvc added a commit that referenced this issue Jan 2, 2019
* [Heartbeat] Handle TLS certs missing notBefore/notAfter

Some certs in the wild don't set these standard fields and can cause an NPE

Fixes #9556

* Add changelog entry

(cherry picked from commit 337113e)
@smartrics
Copy link
Author

Hello I have rebuilt the master branch but we still get a NPE

2019-01-09T14:07:21.818Z	ERROR	scheduler/scheduler.go:355	Panic in job 'dev-cluster/alpha@https://localhost/gecko/health'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2019-01-09T14:07:21.818Z	ERROR	scheduler/scheduler.go:355	Panic in job 'dev-cluster/alpha@https://localhost/agamid/health'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2019-01-09T14:07:21.819Z	ERROR	scheduler/scheduler.go:357	Stacktrace: goroutine 9971 [running]:
runtime/debug.Stack(0x1651dc9, 0x3a, 0xc00049e8e8)
	/home/fabrizio/Apps/go-1.11.2/src/runtime/debug/stack.go:24 +0xa7
github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1.1(0xc0001421e0, 0xc000204080)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:357 +0xf7
panic(0x14ab860, 0x21e0640)
	/home/fabrizio/Apps/go-1.11.2/src/runtime/panic.go:513 +0x1b9
github.com/elastic/beats/heartbeat/monitors/active/dialchain.TLSLayer.func1.1(0x1752020, 0xc00042f880, 0x3, 0xc00042a7c0, 0xd, 0x1752020)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/tls.go:83 +0x455
github.com/elastic/beats/heartbeat/monitors/active/dialchain.afterDial.func1(0x1615705, 0x3, 0xc00042a7c0, 0xd, 0x0, 0x0, 0x0, 0x0)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/util.go:89 +0xba
github.com/elastic/beats/libbeat/outputs/transport.DialerFunc.Dial(0xc000503880, 0x1615705, 0x3, 0xc00042a7c0, 0xd, 0xc0004a8de0, 0x95247f, 0xc00038a870, 0x10)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/libbeat/outputs/transport/transport.go:40 +0x4e
github.com/elastic/beats/heartbeat/monitors/active/http.(*SimpleTransport).RoundTrip(0xc00041d500, 0xc0005dea00, 0x0, 0x0, 0x0)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/simple_transp.go:84 +0xfe
net/http.send(0xc0005de900, 0x17363a0, 0xc00041d500, 0xbf059dea6851c8b8, 0x5bd8dbb557, 0x223e5e0, 0xc00000e408, 0xbf059dea6851c8b8, 0xc0004a9010, 0x1)
	/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc00041d530, 0xc0005de900, 0xbf059dea6851c8b8, 0x5bd8dbb557, 0x223e5e0, 0xc00000e408, 0x0, 0x1, 0x7f9b2aaa3758)
	/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc00041d530, 0xc0005de900, 0x0, 0x0, 0x0)
	/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc00041d530, 0xc0005de900, 0x223e5e0, 0x15b6fe0, 0xc000153201)
	/home/fabrizio/Apps/go-1.11.2/src/net/http/client.go:509 +0x35
github.com/elastic/beats/heartbeat/monitors/active/http.execRequest(0xc00041d530, 0xc0005de900, 0x1668520, 0xbf059de66851c77a, 0x581f2f1416, 0x223e5e0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:271 +0xb7
github.com/elastic/beats/heartbeat/monitors/active/http.execPing(0xc00041d530, 0xc000168900, 0x0, 0x0, 0x0, 0x3b9aca000, 0x1668520, 0x0, 0x0, 0x0, ...)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:245 +0x13c
github.com/elastic/beats/heartbeat/monitors/active/http.createPingFactory.func1(0xc00041cde0, 0x0, 0x0, 0x0)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:188 +0x477
github.com/elastic/beats/heartbeat/monitors.MakePingIPFactory.func1.1(0x203000, 0x203000, 0x95247f)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:194 +0x2e
github.com/elastic/beats/heartbeat/monitors.MakeSimpleCont.func1(0x10, 0x14e67c0, 0xc0004a9770, 0x95247f, 0xc000503740, 0x20)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:184 +0x26
github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc00038a780, 0x18, 0x20, 0x20, 0xc000503740, 0x0, 0xc000120000)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x27
github.com/elastic/beats/heartbeat/monitors.WithFields.func1(0xc0004a9701, 0xc000503740, 0xc0004a97c0, 0xd21361, 0x15189e0, 0xc000503740)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:398 +0x4a
github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc000503740, 0x1738a60, 0xc00038a780, 0x1738a60, 0xc000503740, 0x223e5e0, 0x17a17)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x27
github.com/elastic/beats/heartbeat/monitors.makeByHostAnyIPJob.func1(0xde6839, 0xc0002e00f0, 0x1508d60, 0xc000502520, 0x99, 0x0)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:323 +0x8ed
github.com/elastic/beats/heartbeat/monitors.annotated.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0001fcac0, 0xa, 0xffffffffffff, ...)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:140 +0xc7
github.com/elastic/beats/heartbeat/monitors.MakeJob.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xacc910, 0xc00005c530, 0xbe7a63, ...)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:127 +0xe8
github.com/elastic/beats/heartbeat/monitors.(*funcJob).Run(0xc0001b0760, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00005c608, 0x951ff3, ...)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/util.go:438 +0x82
github.com/elastic/beats/heartbeat/monitors.Job.Run-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbe693c, 0xc000284600, 0x1735fa0, ...)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/task.go:120 +0x86
github.com/elastic/beats/heartbeat/monitors.(*task).prepareSchedulerJob.func1(0x10, 0x1668570, 0xc0001421e0)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/task.go:85 +0x7a
github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1(0xc0001421e0, 0xc000204080, 0xc0001421e0, 0xc0001b0940)
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:362 +0x5b
created by github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask
	/home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:352 +0x5c

@andrewvc
Copy link
Contributor

andrewvc commented Jan 9, 2019 via email

@andrewvc
Copy link
Contributor

@smartrics are you certain you're running the current version master? Which commit hash did you build?

Looking at the trace I see the line: /home/fabrizio/Apps/go-1.11.2/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/tls.go:83 +0x455

That line is currently a comment. I suspect you're running an older commit

@smartrics
Copy link
Author

@andrewvc I have rebuilt from the latest master branch (which I believe is a 7.0.0 version) and I am not seeing any NPE.

I am happy to verify it on any other branch if you suggest which one should I checkout

@andrewvc
Copy link
Contributor

@smartrics glad to hear it :)

#9758 is the backport FYI. It will probably make it into 6.7

leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
… notBefore/notAfter (elastic#9759)

* Add heartbeat test for TLS client cert auth (elastic#8984) (elastic#9676)

* Add heartbeat test for TLS client cert auth

We were missing a test for this specific case. I wrote this hoping to confirm elastic#8979, but actually wound up disproving it.

That said, this is still a good test to have, so we should merge it.

* [Heartbeat] Handle TLS certs missing notBefore/notAfter (elastic#9566)

Some certs in the wild don't set these standard fields and can cause an NPE

Fixes elastic#9556
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

No branches or pull requests

5 participants