Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

abstract_tcp_server2: fix busy calling of idle IO service #3970

Merged
merged 1 commit into from
Nov 6, 2018

Conversation

moneromooo-monero
Copy link
Collaborator

This would make monerod use 100% CPU when running with torsocks
without Tor running

This would make monerod use 100% CPU when running with torsocks
without Tor running
@garlicgambit
Copy link

Thanks for looking into this issue.

The issue still persists. Tested on Tails with the local Tor proxy with the following command:
DNS_PUBLIC=tcp torsocks ./monerod --p2p-bind-ip 127.0.0.1 --no-igd --rpc-bind-ip 127.0.0.1

We have some new/improved info. If you configure iptables to drop monerod traffic to Tor and allow it again monerod will establish new connections with Tor and will have normal cpu use. However if you do a simple restart the of the Tor process monerod won't create new connections to Tor and will keep using 100% of the cpu.

@anonimal
Copy link
Contributor

Tested on Tails with the local Tor proxy with the following command:
DNS_PUBLIC=tcp torsocks ./monerod --p2p-bind-ip 127.0.0.1 --no-igd --rpc-bind-ip 127.0.0.1

There's no need to torsocks behind a local transproxy.

If you configure iptables to drop monerod traffic to Tor, and allow it again, monerod will establish new connections with Tor (and monero will have normal cpu use). However, if you do a simple restart of the Tor process, monerod won't create new connections to Tor and will continue to use 100% of the cpu.

Don't play with iptables rules unless you know what you're doing. Flushing and then restarting rules on a local transproxy leaves a potential window for application leaks. If you must, use a sandboxed internal network machine with gateway transproxy.

@moneromooo-monero
Copy link
Collaborator Author

Does #3997 help ?

@garlicgambit
Copy link

The issue still persists in #3997.

@anonimal, thanks for the comment. We're using an offline Tails system to troubleshoot this issue. So no worries.

@moneromooo-monero
Copy link
Collaborator Author

moneromooo-monero commented Jun 17, 2018

OK, then run "sudo perf top -p `pidof monerod`" while it's doing this, let it run for 20 seconds, then paste the state.

@garlicgambit
Copy link

Tested on Debian Stretch x64 with PR #3997.

Action: stop Tor process
Command: sudo perf top -p `pidof monerod`
Output:

Overhead  Shared Object         Symbol
  11.99%  [kernel]              [k] do_select
   9.67%  [kernel]              [k] memset_erms
   4.42%  [kernel]              [k] tcp_poll
   4.38%  [kernel]              [k] core_sys_select
   4.10%  [kernel]              [k] __fget
   3.74%  [kernel]              [k] _raw_spin_lock_irqsave
   3.68%  [kernel]              [k] entry_SYSCALL_64
   3.41%  [kernel]              [k] fput
   3.17%  [kernel]              [k] entry_SYSCALL_64_after_swapgs
   2.96%  libpthread-2.24.so    [.] __libc_recv
   2.85%  [kernel]              [k] _raw_spin_lock_bh
   2.74%  [kernel]              [k] tcp_recvmsg
   2.44%  libc-2.24.so          [.] __select
   2.23%  [kernel]              [k] __local_bh_enable_ip
   1.97%  [kernel]              [k] _raw_spin_unlock_irqrestore
   1.88%  [kernel]              [k] __check_object_size
   1.62%  [kernel]              [k] __pollwait
   1.60%  [kernel]              [k] SYSC_recvfrom

Action: stop Tor process
Command: sudo perf top --sort comm,dso
Output:

Overhead  Command          Shared Object
  83.86%  monerod          [kernel]
   5.30%  monerod          libpthread-2.24.so
   4.81%  monerod          libc-2.24.so
   3.82%  monerod          libtorsocks.so.0.0.0

Action: drop traffic to Tor with iptables
Command: sudo perf top -p `pidof monerod`
Output:

Overhead  Shared Object         Symbol
   9.30%  [kernel]              [k] _raw_spin_lock_bh
   8.88%  libc-2.24.so          [.] connect
   7.90%  [kernel]              [k] __fget
   7.66%  [kernel]              [k] __local_bh_enable_ip
   6.51%  [kernel]              [k] entry_SYSCALL_64
   6.08%  [kernel]              [k] SYSC_connect
   5.48%  [kernel]              [k] entry_SYSCALL_64_after_swapgs
   4.45%  [kernel]              [k] __inet_stream_connect
   3.74%  [kernel]              [k] fput

Action: drop traffic to Tor with iptables
Command: sudo perf top --sort comm,dso
Output:

Overhead  Command          Shared Object
  78.25%  monerod          [kernel]
  18.20%  monerod          libc-2.24.so
   1.26%  monerod          libtorsocks.so.0.0.0
   0.48%  monerod          libpthread-2.24.so

Action: run without 'DNS_PUBLIC=tcp', drop traffic to Tor with iptables
Command: sudo perf top -p `pidof monerod`
Output:

Overhead  Shared Object              Symbol
  11.37%  [kernel]                   [k] __local_bh_enable_ip
  10.53%  libc-2.24.so               [.] connect
  10.53%  [kernel]                   [k] _raw_spin_lock_bh
   8.35%  [kernel]                   [k] entry_SYSCALL_64_after_swapgs
   7.64%  [kernel]                   [k] entry_SYSCALL_64
   7.53%  [kernel]                   [k] __fget
   4.84%  [kernel]                   [k] fput

Action: run without 'DNS_PUBLIC=tcp', drop traffic to Tor with iptables
Command: sudo perf top --sort comm,dso
Output:

Overhead  Command          Shared Object
  73.56%  monerod          [kernel]
  20.57%  monerod          libc-2.24.so
   1.34%  monerod          libtorsocks.so.0.0.0
   0.23%  monerod          libpthread-2.24.so

@moneromooo-monero
Copy link
Collaborator Author

What command(s) are you using for "drop traffic to Tor with iptables" ?

1 similar comment
@moneromooo-monero
Copy link
Collaborator Author

What command(s) are you using for "drop traffic to Tor with iptables" ?

@moneromooo-monero
Copy link
Collaborator Author

Well, that patch fixes at least one case so let's add it for now.

Copy link
Contributor

@fluffypony fluffypony left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed

@fluffypony fluffypony merged commit 3381b65 into monero-project:master Nov 6, 2018
fluffypony added a commit that referenced this pull request Nov 6, 2018
3381b65 abstract_tcp_server2: fix busy calling of idle IO service (moneromooo-monero)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants