Skip to content

Commit

Permalink
Merge pull request #113 from informalsystems/thane/55-naming
Browse files Browse the repository at this point in the history
Drop master/slave terminology
  • Loading branch information
thanethomson authored Nov 1, 2021
2 parents d8fb3d5 + f59832e commit ab4685d
Show file tree
Hide file tree
Showing 18 changed files with 1,357 additions and 1,340 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Changelog

## v1.0.0
* [\#113](https://github.com/informalsystems/tm-load-test/pull/113) - Dropped
the master/slave terminology. "Master" nodes have now been renamed to
"Coordinator", and "Slave" nodes have been renamed to "Worker".

## v0.9.0
* [\#47](https://github.com/informalsystems/tm-load-test/pull/47) - Makes sure
that the KVStore client's `GenerateTx` method honours the preconfigured
Expand Down
116 changes: 63 additions & 53 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,32 @@ be the successor to [`tm-bench`](https://github.com/tendermint/tendermint/tree/m

Naturally, any transactions sent to a Tendermint network are specific to the
ABCI application running on that network. As such, the `tm-load-test` tool comes
with built-in support for the `kvstore` ABCI application, but you can
[build your own clients](./pkg/loadtest/README.md) for your own apps.
with built-in support for the `kvstore` ABCI application, but you can [build
your own clients](./pkg/loadtest/README.md) for your own apps.

**NB: `tm-load-test` is currently alpha-quality software. Semantic versioning is
not strictly adhered to prior to a v1.0 release, so breaking API changes can
emerge with minor version releases.**

## Requirements

`tm-load-test` is currently tested using Go v1.17.

## Building

To build the `tm-load-test` binary in the `build` directory:

```bash
make
```

## Usage

`tm-load-test` can be executed in one of two modes: **standalone**, or
**master/slave**.
**coordinator/worker**.

### Standalone Mode

In standalone mode, `tm-load-test` operates in a similar way to `tm-bench`:

```bash
Expand All @@ -42,64 +46,67 @@ To see a description of what all of the parameters mean, simply run:
tm-load-test --help
```

### Master/Slave Mode
In master/slave mode, which is best used for large-scale, distributed load
testing, `tm-load-test` allows you to have multiple slave machines connect to
a single master to obtain their configuration and coordinate their operation.
### Coordinator/Worker Mode

In coordinator/worker mode, which is best used for large-scale, distributed load
testing, `tm-load-test` allows you to have multiple worker machines connect to a
single coordinator to obtain their configuration and coordinate their operation.

The master acts as a simple WebSockets host, and the slaves are WebSockets
The coordinator acts as a simple WebSockets host, and the workers are WebSockets
clients.

On the master machine:
On the coordinator machine:

```bash
# Run tm-load-test with similar parameters to the standalone mode, but now
# specifying the number of slaves to expect (--expect-slaves) and the host:port
# to which to bind (--bind) and listen for incoming slave requests.
# Run tm-load-test with similar parameters to the standalone mode, but now
# specifying the number of workers to expect (--expect-workers) and the host:port
# to which to bind (--bind) and listen for incoming worker requests.
tm-load-test \
master \
--expect-slaves 2 \
coordinator \
--expect-workers 2 \
--bind localhost:26670 \
-c 1 -T 10 -r 1000 -s 250 \
--broadcast-tx-method async \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket
```

On each slave machine:
On each worker machine:

```bash
# Just tell the slave where to find the master - it will figure out the rest.
tm-load-test slave --master localhost:26680
# Just tell the worker where to find the coordinator - it will figure out the rest.
tm-load-test worker --coordinator localhost:26680
```

For more help, see the command line parameters' descriptions:

```bash
tm-load-test master --help
tm-load-test slave --help
tm-load-test coordinator --help
tm-load-test worker --help
```

### Endpoint Selection Strategies

As of v0.5.1, an endpoint selection strategy can now be given to `tm-load-test`
as a parameter (`--endpoint-select-method`) to control the way in which
as a parameter (`--endpoint-select-method`) to control the way in which
endpoints are selected for load testing. There are several options:

1. `supplied` (the default) - only use the supplied endpoints (via the
1. `supplied` (the default) - only use the supplied endpoints (via the
`--endpoints` parameter) to submit transactions.
2. `discovered` - only use endpoints discovered through the supplied endpoints
(by way of crawling the Tendermint peers' network info), but do not use any
of the supplied endpoints.
3. `any` - use both the supplied and discovered endpoints to perform load
3. `any` - use both the supplied and discovered endpoints to perform load
testing.

**NOTE**: These selection strategies only apply if, and only if, the
`--expect-peers` parameter is supplied and is non-zero. The default behaviour
if `--expect-peers` is not supplied is effectively the `supplied` endpoint
**NOTE**: These selection strategies only apply if, and only if, the
`--expect-peers` parameter is supplied and is non-zero. The default behaviour if
`--expect-peers` is not supplied is effectively the `supplied` endpoint
selection strategy.

### Minimum Peer Connectivity
As of v0.6.0, `tm-load-test` can now wait for a minimum level of P2P
connectivity before starting the load testing. By using the

As of v0.6.0, `tm-load-test` can now wait for a minimum level of P2P
connectivity before starting the load testing. By using the
`--min-peer-connectivity` command line switch, along with `--expect-peers`, one
can restrict this.

Expand All @@ -109,13 +116,15 @@ minimum address book size is. Once the minimum address book size reaches the
configured value, the load testing can begin.

### Customizing

To implement your own client type to load test your own Tendermint ABCI
application, see the [`loadtest` package docs here](./pkg/loadtest/README.md).

## Monitoring
As of v0.4.1, `tm-load-test` exposes a number of metrics when in master/slave
mode, but only from the master's web server at the `/metrics` endpoint. So if
you bind your master node to `localhost:26670`, you should be able to get these

As of v0.4.1, `tm-load-test` exposes a number of metrics when in coordinator/worker
mode, but only from the coordinator's web server at the `/metrics` endpoint. So if
you bind your coordinator node to `localhost:26670`, you should be able to get these
metrics from:

```bash
Expand All @@ -124,33 +133,34 @@ curl http://localhost:26670/metrics

The following kinds of metrics are made available here:

* Total number of transactions recorded from the master's perspective (across
all slaves)
* Total number of transactions sent by each slave
* The status of the master node, which is a gauge that indicates one of the
* Total number of transactions recorded from the coordinator's perspective
(across all workers)
* Total number of transactions sent by each worker
* The status of the coordinator node, which is a gauge that indicates one of the
following codes:
* 0 = Master starting
* 1 = Master waiting for all peers to connect
* 2 = Master waiting for all slaves to connect
* 0 = Coordinator starting
* 1 = Coordinator waiting for all peers to connect
* 2 = Coordinator waiting for all workers to connect
* 3 = Load test underway
* 4 = Master and/or one or more slave(s) failed
* 5 = All slaves completed load testing successfully
* The status of each slave node, which is also a gauge that indicates one of the
following codes:
* 0 = Slave connected
* 1 = Slave accepted
* 2 = Slave rejected
* 4 = Coordinator and/or one or more worker(s) failed
* 5 = All workers completed load testing successfully
* The status of each worker node, which is also a gauge that indicates one of
the following codes:
* 0 = Worker connected
* 1 = Worker accepted
* 2 = Worker rejected
* 3 = Load testing underway
* 4 = Slave failed
* 5 = Slave completed load testing successfully
* Standard Prometheus-provided metrics about the garbage collector in
* 4 = Worker failed
* 5 = Worker completed load testing successfully
* Standard Prometheus-provided metrics about the garbage collector in
`tm-load-test`
* The ID of the load test currently underway (defaults to 0), set by way of the
`--load-test-id` flag on the master
`--load-test-id` flag on the coordinator

## Aggregate Statistics
As of `tm-load-test` v0.7.0, one can now write simple aggregate statistics to
a CSV file once testing completes by specifying the `--stats-output` flag:

As of `tm-load-test` v0.7.0, one can now write simple aggregate statistics to a
CSV file once testing completes by specifying the `--stats-output` flag:

```bash
# In standalone mode
Expand All @@ -159,10 +169,10 @@ tm-load-test -c 1 -T 10 -r 1000 -s 250 \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket \
--stats-output /path/to/save/stats.csv

# From the master in master/slave mode
# From the coordinator in coordinator/worker mode
tm-load-test \
master \
--expect-slaves 2 \
coordinator \
--expect-workers 2 \
--bind localhost:26670 \
-c 1 -T 10 -r 1000 -s 250 \
--broadcast-tx-method async \
Expand Down
27 changes: 14 additions & 13 deletions cmd/tm-load-test/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ import (
"github.com/informalsystems/tm-load-test/pkg/loadtest"
)

const appLongDesc = `Load testing application for Tendermint with optional master/slave mode.
Generates large quantities of arbitrary transactions and submits those
const appLongDesc = `Load testing application for Tendermint with optional coordinator/worker mode.
Generates large quantities of arbitrary transactions and submits those
transactions to one or more Tendermint endpoints. By default, it assumes that
you are running the kvstore ABCI application on your Tendermint network.
Expand All @@ -14,28 +14,29 @@ To run the application in a similar fashion to tm-bench (STANDALONE mode):
--broadcast-tx-method async \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket
To run the application in MASTER mode:
To run the application in COORDINATOR mode:
tm-load-test \
master \
--expect-slaves 2 \
coordinator \
--expect-workers 2 \
--bind localhost:26670 \
--shutdown-wait 60 \
-c 1 -T 10 -r 1000 -s 250 \
--broadcast-tx-method async \
--endpoints ws://tm-endpoint1.somewhere.com:26657/websocket,ws://tm-endpoint2.somewhere.com:26657/websocket
To run the application in SLAVE mode:
tm-load-test slave --master localhost:26680
To run the application in WORKER mode:
tm-load-test worker --coordinator localhost:26680
NOTES:
* MASTER mode exposes a "/metrics" endpoint in Prometheus plain text format
which shows total number of transactions and the status for the master and
all connected slaves.
* The "--shutdown-wait" flag in MASTER mode is specifically to allow your
* COORDINATOR mode exposes a "/metrics" endpoint in Prometheus plain text
* format
which shows total number of transactions and the status for the coordinator
and all connected workers.
* The "--shutdown-wait" flag in COORDINATOR mode is specifically to allow your
monitoring system some time to obtain the final Prometheus metrics from the
metrics endpoint.
* In SLAVE mode, all load testing-related flags are ignored. The slave always
takes instructions from the master node it's connected to.
* In WORKER mode, all load testing-related flags are ignored. The worker always
takes instructions from the coordinator node it's connected to.
`

func main() {
Expand Down
2 changes: 1 addition & 1 deletion pkg/loadtest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ func main() {
panic(err)
}
// The loadtest.Run method will handle CLI argument parsing, errors,
// configuration, instantiating the load test and/or master/slave
// configuration, instantiating the load test and/or coordinator/worker
// operations, etc. All it needs is to know which client factory to use for
// its load testing.
loadtest.Run(&loadtest.CLIConfig{
Expand Down
56 changes: 28 additions & 28 deletions pkg/loadtest/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,63 +59,63 @@ func buildCLI(cli *CLIConfig, logger logging.Logger) *cobra.Command {
rootCmd.PersistentFlags().StringVar(&cfg.BroadcastTxMethod, "broadcast-tx-method", "async", "The broadcast_tx method to use when submitting transactions - can be async, sync or commit")
rootCmd.PersistentFlags().StringSliceVar(&cfg.Endpoints, "endpoints", []string{}, "A comma-separated list of URLs indicating Tendermint WebSockets RPC endpoints to which to connect")
rootCmd.PersistentFlags().StringVar(&cfg.EndpointSelectMethod, "endpoint-select-method", SelectSuppliedEndpoints, "The method by which to select endpoints")
rootCmd.PersistentFlags().IntVar(&cfg.ExpectPeers, "expect-peers", 0, "The minimum number of peers to expect when crawling the P2P network from the specified endpoint(s) prior to waiting for slaves to connect")
rootCmd.PersistentFlags().IntVar(&cfg.ExpectPeers, "expect-peers", 0, "The minimum number of peers to expect when crawling the P2P network from the specified endpoint(s) prior to waiting for workers to connect")
rootCmd.PersistentFlags().IntVar(&cfg.MaxEndpoints, "max-endpoints", 0, "The maximum number of endpoints to use for testing, where 0 means unlimited")
rootCmd.PersistentFlags().IntVar(&cfg.PeerConnectTimeout, "peer-connect-timeout", 600, "The number of seconds to wait for all required peers to connect if expect-peers > 0")
rootCmd.PersistentFlags().IntVar(&cfg.MinConnectivity, "min-peer-connectivity", 0, "The minimum number of peers to which each peer must be connected before starting the load test")
rootCmd.PersistentFlags().StringVar(&cfg.StatsOutputFile, "stats-output", "", "Where to store aggregate statistics (in CSV format) for the load test")
rootCmd.PersistentFlags().BoolVarP(&flagVerbose, "verbose", "v", false, "Increase output logging verbosity to DEBUG level")

var masterCfg MasterConfig
masterCmd := &cobra.Command{
Use: "master",
Short: "Start load test application in MASTER mode",
var coordCfg CoordinatorConfig
coordCmd := &cobra.Command{
Use: "coordinator",
Short: "Start load test application in COORDINATOR mode",
Run: func(cmd *cobra.Command, args []string) {
logger.Debug(fmt.Sprintf("Configuration: %s", cfg.ToJSON()))
logger.Debug(fmt.Sprintf("Master configuration: %s", masterCfg.ToJSON()))
logger.Debug(fmt.Sprintf("Coordinator configuration: %s", coordCfg.ToJSON()))
if err := cfg.Validate(); err != nil {
logger.Error(err.Error())
os.Exit(1)
}
if err := masterCfg.Validate(); err != nil {
if err := coordCfg.Validate(); err != nil {
logger.Error(err.Error())
os.Exit(1)
}
master := NewMaster(&cfg, &masterCfg)
if err := master.Run(); err != nil {
coord := NewCoordinator(&cfg, &coordCfg)
if err := coord.Run(); err != nil {
os.Exit(1)
}
},
}
masterCmd.PersistentFlags().StringVar(&masterCfg.BindAddr, "bind", "localhost:26670", "A host:port combination to which to bind the master on which to listen for slave connections")
masterCmd.PersistentFlags().IntVar(&masterCfg.ExpectSlaves, "expect-slaves", 2, "The number of slaves to expect to connect to the master before starting load testing")
masterCmd.PersistentFlags().IntVar(&masterCfg.SlaveConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to wait for all slaves to connect")
masterCmd.PersistentFlags().IntVar(&masterCfg.ShutdownWait, "shutdown-wait", 0, "The number of seconds to wait after testing completes prior to shutting down the web server")
masterCmd.PersistentFlags().IntVar(&masterCfg.LoadTestID, "load-test-id", 0, "The ID of the load test currently underway")
coordCmd.PersistentFlags().StringVar(&coordCfg.BindAddr, "bind", "localhost:26670", "A host:port combination to which to bind the coordinator on which to listen for worker connections")
coordCmd.PersistentFlags().IntVar(&coordCfg.ExpectWorkers, "expect-workers", 2, "The number of workers to expect to connect to the coordinator before starting load testing")
coordCmd.PersistentFlags().IntVar(&coordCfg.WorkerConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to wait for all workers to connect")
coordCmd.PersistentFlags().IntVar(&coordCfg.ShutdownWait, "shutdown-wait", 0, "The number of seconds to wait after testing completes prior to shutting down the web server")
coordCmd.PersistentFlags().IntVar(&coordCfg.LoadTestID, "load-test-id", 0, "The ID of the load test currently underway")

var slaveCfg SlaveConfig
slaveCmd := &cobra.Command{
Use: "slave",
Short: "Start load test application in SLAVE mode",
var workerCfg WorkerConfig
workerCmd := &cobra.Command{
Use: "worker",
Short: "Start load test application in WORKER mode",
Run: func(cmd *cobra.Command, args []string) {
logger.Debug(fmt.Sprintf("Slave configuration: %s", slaveCfg.ToJSON()))
if err := slaveCfg.Validate(); err != nil {
logger.Debug(fmt.Sprintf("Worker configuration: %s", workerCfg.ToJSON()))
if err := workerCfg.Validate(); err != nil {
logger.Error(err.Error())
os.Exit(1)
}
slave, err := NewSlave(&slaveCfg)
worker, err := NewWorker(&workerCfg)
if err != nil {
logger.Error("Failed to create new slave", "err", err)
logger.Error("Failed to create new worker", "err", err)
os.Exit(1)
}
if err := slave.Run(); err != nil {
if err := worker.Run(); err != nil {
os.Exit(1)
}
},
}
slaveCmd.PersistentFlags().StringVar(&slaveCfg.ID, "id", "", "An optional unique ID for this slave. Will show up in metrics and logs. If not specified, a UUID will be generated.")
slaveCmd.PersistentFlags().StringVar(&slaveCfg.MasterAddr, "master", "ws://localhost:26670", "The WebSockets URL on which to find the master node")
slaveCmd.PersistentFlags().IntVar(&slaveCfg.MasterConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to keep trying to connect to the master")
workerCmd.PersistentFlags().StringVar(&workerCfg.ID, "id", "", "An optional unique ID for this worker. Will show up in metrics and logs. If not specified, a UUID will be generated.")
workerCmd.PersistentFlags().StringVar(&workerCfg.CoordAddr, "coordinator", "ws://localhost:26670", "The WebSockets URL on which to find the coordinator node")
workerCmd.PersistentFlags().IntVar(&workerCfg.CoordConnectTimeout, "connect-timeout", 180, "The maximum number of seconds to keep trying to connect to the coordinator")

versionCmd := &cobra.Command{
Use: "version",
Expand All @@ -129,8 +129,8 @@ func buildCLI(cli *CLIConfig, logger logging.Logger) *cobra.Command {
},
}

rootCmd.AddCommand(masterCmd)
rootCmd.AddCommand(slaveCmd)
rootCmd.AddCommand(coordCmd)
rootCmd.AddCommand(workerCmd)
rootCmd.AddCommand(versionCmd)
return rootCmd
}
Expand Down
4 changes: 2 additions & 2 deletions pkg/loadtest/client_kvstore_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,8 @@ func BenchmarkKVStoreClient_GenerateTx_100kB(b *testing.B) {
}

func TestKVStoreClient(t *testing.T) {
testCases := []struct{
config loadtest.Config
testCases := []struct {
config loadtest.Config
clientCount int
}{
{loadtest.Config{Size: 32, Count: 1000}, 5},
Expand Down
Loading

0 comments on commit ab4685d

Please sign in to comment.