Skip to content

Latest commit

 

History

History
111 lines (76 loc) · 5.48 KB

README.md

File metadata and controls

111 lines (76 loc) · 5.48 KB
Raw

Wordnik MongoDB Admin Tools

Overview

This project contains a number of tools for administering a MongoDB Deployment

To run

cd ../..
./sbt dist:stage

cd modules/mongo-admin-utils
./bin/run.sh

To get tool options

Run any tool with a -? parameter to see the options:

./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -?
usage: SnapshotUtil
 -d : database name
 -o : output directory
 [-h : hostname]
 [-c : CSV collection string (prefix with ! to exclude)]
 [-t : threads]
 [-s : max file size in MB]
 [-Z : compress files]
 [-J : output in JSON (default is BSON)]
 [-u : username]
 [-p : password]

Tools included

####com.wordnik.system.mongodb.SnapshotUtil This is pretty straight forward, it's meant for taking backups of your data. Some differences between it and mongodump are:

  • - It splits files based on a configurable size
  • - It will let you select what you want to backup with inclusion and exclusion operators
  • - It will automatically gzip the files as it rotates them
  • - It supports a JSON export
  • - It runs a configurable thread pool so you can backup multiple collections simultaneously

    • examples: backup localhost/test to folder backup

    ./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -o backup

    backup only collection "users" in localhost/test to folder "backups"

    ./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -c users -o backups

    backup collection "user_links" in localhost/test to folder backups in JSON format in 32mb files, then gzip

    ./bin/run.sh com.wordnik.system.mongodb.SnapshotUtil -h localhost -d test -c user_links -o backups -J -s 32 -Z

    com.wordnik.system.mongodb.RestoreUtil

    Operates against either mongodump files or files made with the SnapshotUtil with either uncompressed or compressed bson files. Also supports inclusion/exclusion of files

    examples: Restore all files in folder "backups" to database "restored_data"

    ./bin/run.sh com.wordnik.system.mongodb.RestoreUtil -i backup -h localhost -d restored_data

    com.wordnik.system.mongodb.IncrementalBackupUtil

    This queries a master server's oplog and maintains a set of files which can be replayed against a snapshot of the database. The procedure we use is to snapshot the db (either at the filesystem or with the tool) and apply the incremental changes created by this tool.

    The tool looks for a file called "last_timestamp.txt" in the output directory. This file sets a starting point for querying the oplog--it should contain a single line in the format:

    [time-in-seconds]|[counter]

    The [time-in-seconds] is the seconds since epoch, which you can grab from the OS or from a tool like this:

    http://www.esqsoft.com/javascript_examples/date-to-epoch.htm

    The counter should typically be set to 0. As the tool runs, every operation flushed to disk will cause this file to be updated. If you want to stop a running process, create a file called "stop.txt" in the CWD of the application. It will cause the app to stop within one second.

    examples: Save incremental backup on everything in the database "ugc" (note the -c args are scoped to the database)

    ./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -c ugc -o backups

    Save incremental backup on the database "ugc" with collection "login_info" to the folder "backups"

    ./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -c ugc.login_info -o backups

    com.wordnik.system.mongodb.ReplayUtil

    Takes a series of files created by the IncrementalBackupUtil and replays them. The tool allows applying the operations against alternate databases and collections. It also supports skipping records which fall outside a specified timepoint, if for instance you want to roll back to a particular point in time.

    Note that not all operations can be remapped, especially adding indexes and applying database-level commands.

    examples: Replay incremental backup files in folder "backups" to host "localhost:27018"

    ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups -h localhost:27018

    Replay files in folder "backups" to host "localhost:27018", collection "test1", and map from db "test" to db "foo". Note! This doesn't replay commands applied to remapped databases

    ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups  -h localhost:27018 -R test=foo -c test1

    Replay files in folder backups to host localhost:27018, collection test1, and map from test1 to test5

    ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i backups  -h localhost:27018 -r test1=test5 -c test1

    com.wordnik.system.mongodb.ReplicationUtil

    Tool to replicate from server to server. Use caution with the config to avoid getting in a replication loop! You can add mappings to replicate from one database to another.

    examples: Replicate everything from localhost:27017 to localhost:27018

    ./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018

    Replicate everything from localhost:27017 to localhost:27018, and map from source database "foo" to target database "bar"

    ./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018 -m foo:bar

    Replicate collection test.foo from localhost:27017 to localhost:27018, including commands, index operations

    ./bin/run.sh com.wordnik.system.mongodb.ReplicationUtil -h localhost -H localhost:27018 -c test.foo,test.$cmd,test.system.indexes