Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xud sometimes not stopping gracefully via SIGTERM or shutdown #1278

Closed
kilrau opened this issue Oct 8, 2019 · 7 comments
Closed

xud sometimes not stopping gracefully via SIGTERM or shutdown #1278

kilrau opened this issue Oct 8, 2019 · 7 comments
Assignees
@rsercano
Labels
bug Something isn't working P1 top priority

Comments

@kilrau
Copy link
Contributor

kilrau commented Oct 8, 2019

Discovered on master pre 1.0.0-mainnet tag by @offerm

admin@lightning:~ $ sudo systemctl start xud
admin@lightning:~ $ sudo systemctl stop xud
admin@lightning:~ $ sudo systemctl status xud
● xud.service - xud Daemon
   Loaded: loaded (/etc/systemd/system/xud.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-10-03 16:57:26 -03; 3s ago
  Process: 30032 ExecStart=/bin/bash /home/bitcoin/scripts/run_xud (code=killed, signal=TERM)
 Main PID: 30032 (code=killed, signal=TERM)
    Tasks: 12 (limit: 2319)
   Memory: 53.1M
   CGroup: /system.slice/xud.service
           └─30033 node /home/bitcoin/github/xud/bin/xud --mainnet --lnd.LTC.disable

Oct 03 16:57:22 lightning systemd[1]: Started xud Daemon.
Oct 03 16:57:26 lightning systemd[1]: Stopping xud Daemon...
Oct 03 16:57:26 lightning systemd[1]: xud.service: Main process exited, code=killed, status=15/TERM
Oct 03 16:57:26 lightning systemd[1]: xud.service: Succeeded.
Oct 03 16:57:26 lightning systemd[1]: Stopped xud Daemon.
admin@lightning:~ $ ps -ef |grep xud
bitcoin  30033     1 61 16:57 ?        00:00:07 node /home/bitcoin/github/xud/bin/xud --mainnet --lnd.LTC.disable
admin    30074 28920  0 16:57 pts/1    00:00:00 grep --color=auto xud

Please confirm that this is still the case on latest master @offerm , then @sangaman can take over.
@kilrau kilrau added the bug Something isn't working label Oct 8, 2019
@kilrau kilrau added the P2 mid priority label Oct 8, 2019
@kilrau kilrau assigned michael1011 and unassigned sangaman and offerm Oct 9, 2019
@michael1011
Copy link
Contributor

@offerm what does your xud.service file look like? Did you use the one from the docs?

@kilrau kilrau assigned moshababo and unassigned michael1011 Oct 31, 2019
@kilrau
Copy link
Contributor Author

kilrau commented Nov 12, 2019

To be done: test manual xud setup with sample service fail and test if it starts/stops properly.

@kilrau kilrau assigned LePremierHomme and unassigned moshababo Nov 18, 2019
@engwarrior
Copy link
Contributor

Pls provide estimate for this task.

@kilrau kilrau assigned sangaman and unassigned LePremierHomme Feb 28, 2020
@kilrau kilrau removed the 51-100 XUC label Feb 28, 2020
@sangaman
Copy link
Collaborator

There are still a handful of scenarios where xud doesn't shutdown gracefully given a sigterm or shutdown command, I've tried a couple of times to diagnose what the issue is but with no success so far. I don't think it's related to systemd necessarily, so I will rename this issue and use it to track general shutdown issues.

@sangaman sangaman changed the title xud not stopping when using systemd xud sometimes not stopping gracefully via SIGTERM or shutdown Mar 31, 2020
@kilrau
Copy link
Contributor Author

kilrau commented Apr 28, 2020 •
edited
Loading

I found one more: when xud is Waiting for lndbtc admin.macaroon or lnd-BTC is WaitingUnlock it doesn't react to shutdown/sigterm:

29/04/2020 11:10:20.002 [LND-BTC] error: getinfo error: 4.2 - lnd-BTC is WaitingUnlock
29/04/2020 11:10:20.003 [LND-LTC] error: getinfo error: 4.2 - lnd-LTC is WaitingUnlock
29/04/2020 11:10:30.035 [RPC] debug: received call /xudrpc.Xud/GetBalance
29/04/2020 11:10:31.240 [GLOBAL] info: XUD is shutting down
29/04/2020 11:10:31.242 [LND-BTC] info: LndClient status: Disconnected
29/04/2020 11:10:31.243 [LND-LTC] info: LndClient status: Disconnected
29/04/2020 11:10:31.243 [CONNEXT] info: ConnextClient status: Disconnected
29/04/2020 11:10:31.245 [P2P] debug: Peer 03193508d52dc8235c1bf6e902dae30c6dc1823f1e3cdc3c1a97652af9142cd4fe (MeltWedding): closing socket. reason: Shutdown
29/04/2020 11:10:31.246 [P2P] debug: Peer 02d532758864c49dd3ba9715d5e9e4d10c37f0fdc52b37c26e34c226ce8d548001 (PhraseHidden): closing socket. reason: Shutdown
29/04/2020 11:10:31.247 [P2P] debug: Peer 02514021570812ff0ad137698de74bbe4665869765f9c4758f8470d5f57a4e9044 (ExampleShift): closing socket. reason: Shutdown
29/04/2020 11:10:31.248 [P2P] debug: Peer 02516c6b0e9d37ed130b0b5a13251b828759393ebb79e19e4479e3a4fbe7ad9599 (TapeUniverse): closing socket. reason: Shutdown
29/04/2020 11:10:31.249 [P2P] debug: Peer 030a33024f634bccd5ca781928c9bcb03d9b0b2ca2a9014f9d7ef717ffda812479 (AcousticRemind): closing socket. reason: Shutdown
29/04/2020 11:10:31.249 [P2P] debug: Peer 02e8408be3392249f5caf7542598cc0d070f9c976b8d2a127549d87e6816830e34 (HockeyPunch): closing socket. reason: Shutdown
29/04/2020 11:10:31.251 [HTTP] info: http server has closed
29/04/2020 11:10:31.252 [RPC] info: gRPC server completed shutdown
29/04/2020 11:10:31.253 [ORDERBOOK] debug: removed all orders for peer 03193508d52dc8235c1bf6e902dae30c6dc1823f1e3cdc3c1a97652af9142cd4fe (MeltWedding)
29/04/2020 11:10:31.253 [ORDERBOOK] debug: removed all orders for peer 02d532758864c49dd3ba9715d5e9e4d10c37f0fdc52b37c26e34c226ce8d548001 (PhraseHidden)
29/04/2020 11:10:31.254 [ORDERBOOK] debug: removed all orders for peer 02514021570812ff0ad137698de74bbe4665869765f9c4758f8470d5f57a4e9044 (ExampleShift)
29/04/2020 11:10:31.255 [ORDERBOOK] debug: removed all orders for peer 02516c6b0e9d37ed130b0b5a13251b828759393ebb79e19e4479e3a4fbe7ad9599 (TapeUniverse)
29/04/2020 11:10:31.255 [ORDERBOOK] debug: removed all orders for peer 030a33024f634bccd5ca781928c9bcb03d9b0b2ca2a9014f9d7ef717ffda812479 (AcousticRemind)
29/04/2020 11:10:31.256 [ORDERBOOK] debug: removed all orders for peer 02e8408be3392249f5caf7542598cc0d070f9c976b8d2a127549d87e6816830e34 (HockeyPunch)
29/04/2020 11:10:31.257 [GLOBAL] info: XUD shutdown gracefully
29/04/2020 11:10:31.258 [P2P] info: Peer 02e8408be3392249f5caf7542598cc0d070f9c976b8d2a127549d87e6816830e34 (HockeyPunch) socket closed
29/04/2020 11:10:31.258 [P2P] info: Peer 030a33024f634bccd5ca781928c9bcb03d9b0b2ca2a9014f9d7ef717ffda812479 (AcousticRemind) socket closed
29/04/2020 11:10:31.258 [P2P] info: Peer 02516c6b0e9d37ed130b0b5a13251b828759393ebb79e19e4479e3a4fbe7ad9599 (TapeUniverse) socket closed
29/04/2020 11:10:31.259 [P2P] info: Peer 02514021570812ff0ad137698de74bbe4665869765f9c4758f8470d5f57a4e9044 (ExampleShift) socket closed
29/04/2020 11:10:31.259 [P2P] info: Peer 02d532758864c49dd3ba9715d5e9e4d10c37f0fdc52b37c26e34c226ce8d548001 (PhraseHidden) socket closed
29/04/2020 11:10:31.259 [P2P] info: Peer 03193508d52dc8235c1bf6e902dae30c6dc1823f1e3cdc3c1a97652af9142cd4fe (MeltWedding) socket closed

@kilrau
Copy link
Contributor Author

kilrau commented Jul 8, 2020

Can you help testing if in above two scenarios (easy reproducable with a new xud-docker environment, just don't "create") xud is still not shutting down gracefully? If it works now, we can close here. @raladev

@kilrau kilrau added P1 top priority and removed P2 mid priority labels Aug 3, 2020
@kilrau kilrau assigned rsercano and unassigned sangaman Aug 17, 2020
@kilrau
Copy link
Contributor Author

kilrau commented Oct 23, 2020

Closing here in favor of #1668

@kilrau kilrau closed this as completed Oct 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rsercano rsercano

Labels
bug Something isn't working P1 top priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@sangaman @rsercano @michael1011 @offerm @moshababo @kilrau @engwarrior @LePremierHomme