Skip to content

v3.0 Usage

Jump to bottom
Andrey Kurilov edited this page Feb 22, 2017 · 1 revision

Contents

  1. Environment Requirements
  2. Deployment
  3. Hello World
  4. Running
  5. Results Analysis
    1. Standard Output
    2. Log Files
    3. Items List Files
  6. Docker
    1. Get an image
    2. Run standalone
    3. Run the storage driver service
    4. Run the controller
    5. Run the storage mock
  7. Web GUI

Environment Requirements

  • JRE 8+
  • OS open files limit is at least a bit higher than specified concurrency level
  • Few gigabytes of free memory. High-load tests may allocate up to 2-4 GB of the memory depending on the scenario.
  • (Remote Storage) Connectivity with the endpoint nodes via the ports used
  • (Distributed Mode) Connectivity with the load servers via port #1099

Deployment

  1. Download the latest Mongoose tar file
  2. Copy the tar file to a proper directory and unpack the file:
tar xvf mongoose-<VERSION>.tgz

This command extracts the contents of the distribution into the directory ./mongoose- which will contain several directories and files. One of the files must be mongoose.jar.

Note:

In case of the distributed mode Mongoose should be also deployed to the driver hosts. Each of Mongoose driver serivces should be started using the following command:





java -jar mongoose-/mongoose-storage-driver-service.jar



# Hello World

An user can try Mongoose without a real storage. Mongoose distribution contains a storage mock implementation for the demo/testing purposes.

Start the web storage mock first:
```bash
java -jar <MONGOOSE_DIR>/mongoose-storage-mock.jar



Then open another console and start Mongoose itself:


java -jar <MONGOOSE_DIR>/mongoose.jar


The line starts Mongoose with default scenario and the default configuration. By default Mongoose writes using Amazon S3 API to storage with IP address 127.0.0.1 (local host). Storage mock is already there to store the new objects.


Now you can switch between the two consoles to see how Mongoose regularly reports about objects created and the storage mock regularly reports about objects stored.


Running


The following line contains a typical command that starts Mongoose run to create 100,000 data items of 8KB size:


 java -jar <MONGOOSE_DIR>/mongoose.jar --storage-node-addrs=10.64.84.<X>,10.64.84.<Y>,10.64.84.<Z> --load-limit-count=100000 --item-data-size=8KB --storage-auth-id=wuser1@SANITY.LOCAL --storage-auth-secret=A5JKVKuSHp5Kme2qcMFlvMqEKbN+QBNF0tRuFleT


Where:


    

  • --storage-node-addrs - the list of storage node IP addresses
    • 

  • --load-limit-count - the number of data items to create
    • 

  • --item-data-size - the size data items to create
    • 

  • --storage-auth-id - "user" authentication identifier
    • 

  • --storage-auth-secret - "user" authentication secret key
    • 



Note:




    

  • If no configuration properties are set in the command line the default configuration is used.
    • 

  • The default configuration is bundled in the <MONGOOSE_DIR>/mongoose-ui.jar file's under the "/defaults.json" path.
    • 

  • If you're running against ECS, use its provision environment to obtain credentials and other information like bucket, subtenant, container, etc.
    • 

  • Please keep in mind that high load require a lot of system resources such as open file descriptors and Java heap memory. It may be necessary to specify the resource limits manually:
    • 







ulimit -n 1048576


Results Analysis


Standard Output


The expected console output looks like below:


TODO


After several initialization statements Mongoose starts reporting its progress every 10 seconds. Progress information is printed with the following layout:


<TIMESTAMP> <LEVEL> <CLASS> <LOAD_JOB_NAME>↩
n=(<SUCCESS>/<FAIL>);↩
t[s]=(<JOB_TIME>/<DURATION_SUM>)↩
size=(<SIZE>);↩
TP[op/s]=(<AVG>/<AVG_LAST>);↩
BW[MB/s]=(<AVG>/<AVG_LAST>)↩
dur[us]=(<AVG>/<MIN>/<MAX>);↩
lat[us]=(<AVG>/<MIN>/<MAX>)



The summary metrics statement has a bit different metrics layout:


n=(<SUCCESS>/<FAIL>);↩
t[s]=(<JOB_TIME>/<DURATION_SUM>)↩
size=(<SIZE>);↩
TP[op/s]=(<AVG>/<AVG_LAST>);↩
BW[MB/s]=(<AVG>/<AVG_LAST>)↩
dur[us]=(<AVG>/<MIN>/<LOQ>/<MED>/<HIQ>/<MAX>);↩
lat[us]=(<AVG>/<MIN>/<LOQ>/<MED>/<HIQ>/<MAX>)



Where "LOQ" is a low quartile of the distribution and "HIQ" is a high quartile.


Log Files


The tool saves rich logging information about the events and performance metrics. The particular path for the logs for every run may be determined as log/RUN_ID. The directory contains a subset of the following files:


    

  • 

    3rdparty.log
    

    Third party libraries logging goes here
    

    • 

  • 

    config.log
    

    Effective run configuration. In case a run was restarted or resumed several times, the log file contains several configurations
    

    • 

  • 

    errors.log
    

    The file contains error and warning messages
    

    • 

  • 

    items.csv
    

    The file contains information about processed data items with the entries pattern:
    

    <ITEM_PATH>,<ITEM_RING_OFFSET>,<ITEM_SIZE>,<RING_LAYER_#>/<UPDATED_RANGES_MASK>

    

    This file is without a header line because some users shuffle it.
    

    • 

  • 

    messages.log
    

    The file contains Mongoose internal log messages.
    

    • 

  • 

    metrics.periodic.csv
    

    The file contains periodic performance metrics records. The header line describes the content:
    

      

    • DateTimeISO8601,
      • 

    • LoadId,
      • 

    • TypeAPI,
      • 

    • TypeLoad,
      • 

    • CountConn,
      • 

    • CountNode,
      • 

    • CountLoadServer,
      • 

    • CountSucc,
      • 

    • CountFail,
      • 

    • Size,
      • 

    • JobDuration[s],
      • 

    • DurationSum[s],
      • 

    • TPLast[op/s],
      • 

    • TPAvg[op/s],
      • 

    • BWAvg[MB/s],
      • 

    • BWLast[MB/s],
      • 

    • DurationAvg[us],
      • 

    • DurationMin[us],
      • 

    • DurationMax[us],
      • 

    • LatencyAvg[us],
      • 

    • LatencyMin[us],
      • 

    • LatencyMax[us]
      • 

    

    • 

  • 

    metrics.total.csv
    

    The file contains summary performance information calculated at the end of the run. The header line describes the content:
    

      

    • DateTimeISO8601,
      • 

    • LoadId,
      • 

    • TypeAPI,
      • 

    • TypeLoad,
      • 

    • CountConn,
      • 

    • CountNode,
      • 

    • CountLoadServer,
      • 

    • CountSucc,
      • 

    • CountFail,
      • 

    • Size,
      • 

    • JobDuration[s],
      • 

    • DurationSum[s],
      • 

    • TPAvg[op/s],
      • 

    • TPLast[op/s],
      • 

    • BWAvg[MB/s],
      • 

    • BWLast[MB/s],
      • 

    • DurationAvg[us],
      • 

    • DurationMin[us],
      • 

    • DurationLoQ[us],
      • 

    • DurationMed[us],
      • 

    • DurationHiQ[us],
      • 

    • DurationMax[us],
      • 

    • LatencyAvg[us],
      • 

    • LatencyMin[us],
      • 

    • LatencyLoQ[us],
      • 

    • LatencyMed[us],
      • 

    • LatencyHiQ[us],
      • 

    • LatencyMax[us]
      • 

    

    • 

  • 

    io.trace.csv
    

    The file contains detailed performance metrics per I/O task finished. The entry pattern is described by header:
    

      

    • StorageDriver,
      • 

    • ThreadName,
      • 

    • StorageNode,
      • 

    • ItemPath,
      • 

    • OpCode,
      • 

    • StatusCode,
      • 

    • ReqTimeStart[us],
      • 

    • Duration[us],
      • 

    • RespLatency[us],
      • 

    • DataLatency[us],
      • 

    • TransferSize
      • 

    

    The operation code ("OpCode" column) in the io.trace.csv file may have one of the values below
    

    • 







Code
Description






O
CREATE




1
READ




2
UPDATE




3
DELETE






The status code in the io.trace.csv file may have one of the values below







Code
Description
HTTP response codes






O
Pending (internal)





1
Active (internal)





2
Cancelled





3
Unknown failure
all other codes




4
Success
2xx




5
I/O Failure





6
Timeout
504




7
Unrecognized storage response





8
Client failure or invalid request
100, 400, 405, 409, 411, 414, 416




9
Internal storage failure
413, 415, 429, 500, 501, 502, 503, 505




10
Item not found on the storage
404




11
Authentication/access failure
401, 403




12
Data item corrupted





13
Not enough space on the storage
507






Items List Files


The items processed during the test may be stored in the output file for a further reuse.
For example, this output file may be used as input file to read the items written before.
To specify the output file the CLI option "--item-output-file=" should be used:


java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-file=items.csv [<OTHER_CLI_OPTIONS>]


To use the item list file the CLI option "--item-input-file=" should be used:


java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-file=items.csv --load-type=read [<OTHER_CLI_OPTIONS>]


Docker


Get an image


To get the latest stable version:


docker pull emcmongoose/mongoose


Otherwise, get the specific version:


docker pull emcmongoose/mongoose:3.0.3


Run standalone


docker run emcmongoose/mongoose java -jar /opt/mongoose/mongoose.jar <ARGS>


Run the storage driver service


docker run -d --expose 1099 emcmongoose/mongoose java -jar /opt/mongoose/mongoose-storage-driver-service.jar


Run the controller


docker run emcmongoose/mongoose --storage-driver-remote --storage-driver-addrs=<ADDR1,ADDR2,...> <ARGS>


Run the storage mock


docker run -d --expose 9020-9024 emcmongoose/mongoose java -jar /opt/mongoose/mongoose-storage-mock.jar <ARGS>


Web GUI


TODO


              


          



  
          

            

              

                

                  

                    
About



Documentation



Development


Notes



Archive



Links



                  

                

              


            
Clone this wiki locally