Skip to content

Limit the number of parallel mongodump workers based on CPU count #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 17, 2017
Merged

Limit the number of parallel mongodump workers based on CPU count #77

merged 7 commits into from
Jan 17, 2017

Conversation

timvaillancourt
Copy link
Contributor

@timvaillancourt timvaillancourt commented Jan 14, 2017

On hosts with many shards there can be some significant CPU contention and high load average when using mongodump 3.2.0+, which defaults to using 4 x dump workers per instance of mongodump.

For example on a system with 4 x CPUs and 4 x shards requiring backup this will create 16 x mongodump threads (4 shards x 4 collection dump threads), and the threads start to starve and fight for resources, especially considering gzip is involved in 3.2+. This contention seems to be limiting performance of the backup and causing the system to be very slow.

This PR moves to:

  1. If you're using mongodump <= 3.2, these changes do nothing aside from logging this new warning:
[2017-01-14 18:25:44,467] [WARNING] [MainProcess] [Dumper:run:78] Threading unsupported by mongodump version 2.6.12. Use mongodump 3.2.0 or greater to enable per-dump threading.
  1. If you're using mongodump >= 3.2, this change considers the CPU count of a host and divides the amount of work by cpu count to stabilise the system CPU workload. If you have more shards than CPUs, 1 thread is used and we hope for the best.

There is a new log message to specify how many dump threads are used here ('0' = no threading used):

[2017-01-14 18:25:44,467] [INFO] [MainProcess] [Dumper:run:103] Starting backups using mongodump 3.2.11 (inline gzip: True, threads per dump: 2)

Tested on 2.6.x and 3.2.x.

@timvaillancourt timvaillancourt changed the title Limit the number of parallel mongodump workers based on CPU Limit the number of parallel mongodump workers based on CPU count Jan 14, 2017
@timvaillancourt
Copy link
Contributor Author

Done making tweaks, finally. Ready for review.

@dbmurphy dbmurphy merged commit bb3b2aa into Percona-Lab:master Jan 17, 2017
@timvaillancourt timvaillancourt deleted the thread_limit branch January 19, 2017 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants