Skip to content

Commit

Permalink
Orderer Traffic Engine (OTE) FAB-1805
Browse files Browse the repository at this point in the history 
This solves FAB-1805, and partially solves FAB-972 by providing
some of the tests for our Orderer Functional Test Plan.

Updated 2/16 to add linter corrections and "% complete" progress logs.
Updated 2/17 to allow running on multiple architectures and to
clean up some tests under development.

This OTE tool is useful for Functional Testing the Orderer Service.
The user can specify network test configuration parameters, such as
the number of orderers, ordererType, total number of transactions
to send, etc., and also may set environment variables to override
several of the configuration parameters defined in orderer/orderer.yaml
such as batchSize and batchTimeout.

OTE leverages another tool driver.sh to launch a network, using
docker containers for the orderers and kafka-brokers. The driver.sh
tool is under development at https://github.com/dongmingh/v1framework
and some days is unstable, so we include a working version of driver.sh
here, along with its helper file scripts network.json and json2yml.js.
It creates a docker-compose file and uses that to launch a network.

OTE also creates a CA and PEER, but those are not currently used by OTE.

OTE borrows the orderer/sample_clients when it creates Producers to
send/broadcast transactions and Consumers to receive/deliver batches.
These are created as go-threads for concurrency.

OTE counts and verifies totals of transactions and batches, and
provides report logs.

Users can run OTE from the command line, "go build; ./ote",
as well as from "go test" functional tests in file ote_test.go,
which will be useful for continuous integration test suites.

OTE also supports multi-channels. Below are all the environment
variables which are currently supported (and defaults):

  OTE_TXS                                      55
  OTE_CHANNELS                                 1
  OTE_ORDERERS                                 1
  OTE_KAFKABROKERS                             0
  OTE_MASTERSPY                                false

  CONFIGTX_ORDERER_ORDERERTYPE                 solo
  CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT   10
  CONFIGTX_ORDERER_BATCHTIMEOUT                10

Examples:

  git clone https://github.com/hyperledger-fabric.git
  cd $GOPATH/src/github.com/hyperledger/fabric
  make native
  make docker
  cd $GOPATH/src/github.com/hyperledger/fabric/bddtests/regression/ote
  go build
  ./ote
  CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT=20 ./ote
  OTE_TXS=100  OTE_CHANNELS=4  ./ote
  CONFIGTX_ORDERER_ORDERERTYPE=kafka OTE_KAFKABROKERS=3  ./ote

  go test -run ORD7[79]
  go test -timeout 90m

  go get github.com/jstemmer/go-junit-report
  go test -run ORD7 -v | go-junit-report > report.xml

Change-Id: I327c926bf51ac348d61fbe29b4888dbeaf8f55b2
Signed-off-by: Scott Zwierzynski <scottz@us.ibm.com>
Signed-off-by: vagrant <vagrant@ubuntu.vagrantup.com>
Signed-off-by: Scott Zwierzynski <scottz@us.ibm.com>
  • Loading branch information
@scottz64
scottz64 committed Feb 18, 2017
1 parent 60ecaf1 commit 29954d6
Show file tree
Hide file tree
Showing 9 changed files with 2,596 additions and 3 deletions.
19 changes: 16 additions & 3 deletions bddtests/regression/go/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
## GO tests, scripts, and supporting files
(For all scripts added to this directory, add information below to
describe pre-requisites, setup steps, and usage examples.)
## GO tests, scripts, tools

OTE is the Orderer Traffic Engine application that
launches a network as specified with environment variable parameters,
creates Producer clients to broadcast transactions to all channels on
all orderers,
creates Consumer clients to deliver/receive batches of transactions,
and counts and reports all results.

TDK is the Test Development Kit application test tool and suite of tests for
high level system testing with an eye towards consensus and reliability.
This tool is still under development; it requires enhancements to be converted
from v0.6 to v1.0 interfaces.

For all scripts added to this directory, refer to information below
describing pre-requisites, setup steps, and usage examples.

## SETUP

Expand Down
164 changes: 164 additions & 0 deletions bddtests/regression/go/ote/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# Orderer Traffic Engine (OTE)

## What does OTE do?

+ This Orderer Traffic Engine (OTE) tool creates and tests the operation of a
hyperledger fabric ordering service.
+ The focus is strictly on the orderers themselves.
No peers are involved: no endorsements or validations or committing to ledgers.
No SDK is used.

+ OTE sends transactions to
every channel on every orderer, and verifies that the correct number
of transactions and blocks are delivered on every channel from every orderer.
+ OTE generates report logs and returns
a pass/fail boolean and a resultSummaryString.

## How does OTE do it?

+ OTE invokes a local copy of the tool driver.sh (including helper files
network.json and json2yml.js) -
which is a close copy of the original version at
https://github.com/dongmingh/v1FabricGenOption.
+ The driver.sh launches an orderer service network per the user-provided
parameters including number of orderers, orderer type,
number of channels, and more.
+ Producer clients are created to connect via
grpc ports to the orderers to concurrently send traffic until the
requested number of transactions are sent.
Each client generates unique transactions - a fraction of the total
requested number of transactions.
+ Consumer clients are created to connect via
grpc ports to the orderers to concurrently receive delivered traffic
until all batches of transactions are tallied.
OTE checks if the correct number of blocks and TXs are delivered
by all the orderers on all the channels

## Prerequisites
- <a href="https://git-scm.com/downloads" target="_blank">Git client</a>
- <a href="https://www.docker.com/products/overview" target="_blank">Docker v1.12 or higher</a>
- [Docker-Compose v1.8 or higher](https://docs.docker.com/compose/overview/)
- GO

Check your Docker and Docker-Compose versions with the following commands:
```bash
docker version
docker-compose version
```

### Prepare binaries and images:

- Alternative 1: Prepare all binaries and images using a script
```bash
cd $GOPATH/src/github.com/hyperledger/fabric/bddtests/regression/ote
./docker_images.sh
```

- Alternative 2: Prepare binaries and images manually
- - Clone the fabric repository, build the binaries and images
```bash
cd $GOPATH/src/github.com/hyperledger/fabric
make native docker
```
- - Clone the fabric-ca repository, build the images
```bash
cd $GOPATH/src/github.com/hyperledger/

# Use ONE of these methods to clone the repository:
go get github.com/hyperledger/fabric-ca
git clone https://github.com/hyperledger/fabric-ca.git
git clone ssh://YOUR-ID@gerrit.hyperledger.org:29418/fabric-ca

cd $GOPATH/src/github.com/hyperledger/fabric-ca
make docker
```

### Environment Variables for test setup, with defaults:
```
OTE_TXS 55
OTE_CHANNELS 1
OTE_ORDERERS 1
OTE_KAFKABROKERS 0
OTE_MASTERSPY false
OTE_PRODUCERS_PER_CHANNEL 1
```

### Environment Variables for configuration
Find default values of all variables in hyperledger/fabric/orderer/orderer.yaml
and hyperledger/fabric/peer/core.yaml.
```
CONFIGTX_ORDERER_ORDERERTYPE solo
CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT 10
CONFIGTX_ORDERER_BATCHTIMEOUT 10
Others:
CORE_LOGGING_LEVEL <unset>
CORE_LEDGER_STATE_STATEDATABASE leveldb
CORE_SECURITY_LEVEL 256
CORE_SECURITY_HASHALGORITHM SHA2
```

## Execute OTE on shell command line
There are several environment variables to control the test parameters,
such as number of transactions, number of orderers, ordererType, and more.
To see an example test using default settings, simply execute the following.
```bash
cd $GOPATH/src/github.com/hyperledger/fabric/bddtests/regression/ote
go build
./ote
CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT=20 ./ote
OTE_TXS=100 OTE_CHANNELS=4 ./ote
CONFIGTX_ORDERER_ORDERERTYPE=kafka OTE_KAFKABROKERS=3 ./ote
```

Choose which variables to modify from default values. For example:

+ This test will create eight Producer clients.
+ Each Producer will send 125 transactions to a different orderer and channel.
+ 250 total TXs will be broadcast on each channel.
+ 500 total TXs will be broadcast to each orderer.
+ Four Consumer clients will be created to receive the delivered
batches on each channel on each orderer.
+ 50 batches (with 10 TX each) will be delivered on channel 0, and
a different 50 batches will be delivered on channel 1. On both Orderers.
+ 100 batches will be received on every orderer; this is the sum of the
totals received on each channel on the orderer.
```bash
OTE_TXS=1000 OTE_CHANNELS=4 OTE_ORDERERS=2 CONFIGTX_ORDERER_ORDERERTYPE=kafka ./ote
```

## Execute OTE GO Tests
The tester may optionally define environment variables to
set the test parameters and to
override certain orderer configuration parameters.
Then use "go test" to execute Test functions
to execute either one test, or all go tests, or
a subset of existing functional go tests using a regular expression
to choose tests in local test files.
```bash
cd $GOPATH/src/github.com/hyperledger/fabric/bddtests/regression/ote
go test -run ORD77
go test -run ORD7[79]
go test -run batchSz -timeout 20m
go test -timeout 90m

go get github.com/jstemmer/go-junit-report
go test -run ORD7 -v | go-junit-report > report.xml

```

## Execute OTE GO Tests for Continuous Improvement
Optionally, one can translate the "go test" output to xml for reports.
This is useful for automated test suites that are automatically
executed from Jenkins by Continuous Improvement processes.

#### Pre-requisite to convert "go test" output to xml
```bash
cd $GOPATH/src/github.com/hyperledger/fabric/bddtests/regression/ote
go get github.com/jstemmer/go-junit-report
```
#### Example command to execute all "go tests" and convert to xml:
```
cd $GOPATH/src/github.com/hyperledger/fabric/bddtests/regression/ote
go test -v -timeout 120m | go-junit-report > ote_report.xml
```
21 changes: 21 additions & 0 deletions bddtests/regression/go/ote/docker_images.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash
# Clone fabric git repository
#############################
echo "Fabric Images"
cd $GOPATH/src/github.com/hyperledger/fabric
#FABRIC_COMMIT=$(git log -1 --pretty=format:"%h")
make docker && make native

# Clone fabric-ca git repository
################################
echo "Ca Images"
CA_REPO_NAME=fabric-ca
cd $GOPATH/src/github.com/hyperledger/
if [ ! -d "$CA_REPO_NAME" ]; then
git clone --depth=1 https://github.com/hyperledger/$CA_REPO_NAME.git
#CA_COMMIT=$(git log -1 --pretty=format:"%h")
fi
cd $CA_REPO_NAME
make docker
echo "List of all Images"
docker images | grep hyperledger
170 changes: 170 additions & 0 deletions bddtests/regression/go/ote/driver.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
#!/bin/bash

InvalidArgs=0

#init var
nBroker=0
nPeer=1

while getopts ":l:d:w:x:b:c:t:a:o:k:p:" opt; do
case $opt in
# peer environment options
l)
CORE_LOGGING_LEVEL=$OPTARG
export CORE_LOGGING_LEVEL=$CORE_LOGGING_LEVEL
echo "CORE_LOGGING_LEVEL: $CORE_LOGGING_LEVEL"
;;
d)
db=$OPTARG
echo "ledger state database type: $db"
;;
w)
CORE_SECURITY_LEVEL=$OPTARG
export CORE_SECURITY_LEVEL=$CORE_SECURITY_LEVEL
echo "CORE_SECURITY_LEVEL: $CORE_SECURITY_LEVEL"
;;
x)
CORE_SECURITY_HASHALGORITHM=$OPTARG
export CORE_SECURITY_HASHALGORITHM=$CORE_SECURITY_HASHALGORITHM
echo "CORE_SECURITY_HASHALGORITHM: $CORE_SECURITY_HASHALGORITHM"
;;

# orderer environment options
b)
CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT=$OPTARG
export CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT=$CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT
echo "CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT: $CONFIGTX_ORDERER_BATCHSIZE_MAXMESSAGECOUNT"
;;
c)
CONFIGTX_ORDERER_BATCHTIMEOUT=$OPTARG
export CONFIGTX_ORDERER_BATCHTIMEOUT=$CONFIGTX_ORDERER_BATCHTIMEOUT
echo "CONFIGTX_ORDERER_BATCHTIMEOUT: $CONFIGTX_ORDERER_BATCHTIMEOUT"
;;
t)
CONFIGTX_ORDERER_ORDERERTYPE=$OPTARG
export CONFIGTX_ORDERER_ORDERERTYPE=$CONFIGTX_ORDERER_ORDERERTYPE
echo "CONFIGTX_ORDERER_ORDERERTYPE: $CONFIGTX_ORDERER_ORDERERTYPE"
if [$nBroker == 0 ] && [ $CONFIGTX_ORDERER_ORDERERTYPE == 'kafka' ]; then
nBroker=1 # must have at least 1
fi
;;

# network options
a)
Req=$OPTARG
echo "action: $Req"
;;
k)
nBroker=$OPTARG
echo "# of Broker: $nBroker"
;;
p)
nPeer=$OPTARG
echo "# of peer: $nPeer"
;;
o)
nOrderer=$OPTARG
echo "# of orderer: $nOrderer"
;;

# else
\?)
echo "Invalid option: -$OPTARG" >&2
InvalidArgs=1
;;
:)
echo "Option -$OPTARG requires an argument." >&2
InvalidArgs=1
;;
esac
done


if [ $InvalidArgs == 1 ]; then
echo "Usage: "
echo " ./driver.sh [opt] [value] "
echo " network variables"
echo " -a: action [create|add] "
echo " -p: number of peers "
echo " -o: number of orderers "
echo " -k: number of brokers "
echo " "
echo " peer environment variables"
echo " -l: core logging level [(deafult = not set)|CRITICAL|ERROR|WARNING|NOTICE|INFO|DEBUG]"
echo " -w: core security level [256|384]"
echo " -x: core security hash algorithm [SHA2|SHA3]"
echo " -d: core ledger state DB [goleveldb|couchdb] "
echo " "
echo " orderer environment variables"
echo " -b: batch size [10|msgs in batch/block]"
echo " -t: orderer type [solo|kafka] "
echo " -c: batch timeout [10s|max secs before send an unfilled batch] "
echo " "
exit
fi

if [ $nBroker -gt 0 ] && [ $CONFIGTX_ORDERER_ORDERERTYPE == 'solo' ]; then
echo "reset Kafka Broker number to 0 due to the CONFIGTX_ORDERER_ORDERERTYPE=$CONFIGTX_ORDERER_ORDERERTYPE"
nBroker=0
fi

#OS
##OSName=`uname`
##echo "Operating System: $OSName"


dbType=`echo "$db" | awk '{print tolower($0)}'`
echo "action=$Req nPeer=$nPeer nBroker=$nBroker nOrderer=$nOrderer dbType=$dbType"
VP=`docker ps -a | grep 'peer node start' | wc -l`
echo "existing peers: $VP"


echo "remove old docker-composer.yml"
rm -f docker-compose.yml

# form json input file
if [ $nBroker == 0 ]; then
#jsonFILE="network_solo.json"
jsonFILE="network.json"
else
# jsonFILE="network_kafka.json"
jsonFILE="network.json"
fi
echo "jsonFILE $jsonFILE"

# create docker compose yml
if [ $Req == "add" ]; then
N1=$[nPeer+VP]
N=$[N1]
VPN="peer"$[N-1]
else
N1=$nPeer
N=$[N1 - 1]
VPN="peer"$N
fi

## echo "N1=$N1 VP=$VP nPeer=$nPeer VPN=$VPN"

node json2yml.js $jsonFILE $N1 $nOrderer $nBroker $dbType

## sed 's/-x86_64/TEST/g' docker-compose.yml > ss.yml
## cp ss.yml docker-compose.yml
# create network
if [ $Req == "create" ]; then

#CHANGED FROM ORIG SCRIPT
#docker-compose -f docker-compose.yml up -d --force-recreate $VPN cli
docker-compose -f docker-compose.yml up -d --force-recreate $VPN
for ((i=1; i<$nOrderer; i++))
do
tmpOrd="orderer"$i
docker-compose -f docker-compose.yml up -d $tmpOrd
done
fi

if [ $Req == "add" ]; then
docker-compose -f docker-compose.yml up -d $VPN

fi

exit
Loading

0 comments on commit 29954d6

Please sign in to comment.