Merge "create regression test dirs"

bddtests/regression/README.md

@@ -0,0 +1,87 @@
+## regression
+Regression Test Suites scripts, and folders for: execution results for
+significant releases, and
+configuration scripts and supporting files in a variety of languages:
+
+- [GO](https://github.com/hyperledger/fabric/bddtests/regression/go)
+  language test scripts
+- [NODE](https://github.com/hyperledger/fabric/bddtests/regression/node)
+  language test scripts
+  Note: a flexible-use performance engine, which can be configured and used
+  for a variety of tests, is located in the
+  [node/performance](https://github.com/hyperledger/fabric/bddtests/regression/node/performance)
+  folder.  For examples how to use it, check there, or look for examples in the
+  daily test suite and long-run test suite scripts.
+- [results](https://github.com/hyperledger/fabric/bddtests/regression/results)
+  folder containing logfiles and results of running the Test Suites on
+  significant releases, for reference
+
+
+### Continous Integration Setup
+Continuous Integration team will execute the *daily test suite* each day when a
+merge commit has been pushed to fabric repository.
+We have configured Jenkins in vLaunch to execute the Daily Test Suite:
+the jobs listed below are downstream jobs which are run after successfully
+executing the upstream job.
+After the build is successfully executed, Jenkins will post build results
+back to **rel-criteria-build** slack channel,
+and generate a test summary report, to be available for viewing for 30 days.
+For significant releases, the results will be stored
+in **bddtests/regression/results** folder.
+
+
+### Daily Test Suite - daily_test_suite.sh
+Expected total time duration is between 13 - 16 hours.
+
+* Consensus Acceptance Tests (CAT) - using chaincode example02, GO, gRPC
+  - Objective of CAT tests is to ensure the stability and resiliency of the
+    BFT Batch design.
+* Large Networks Basic API And Consensus Tests - using chaincode example02,
+  GO, gRPC
+  - Objective of Basic gRPC API test is to ensure basic gRPC API functions
+    are working as expected
+* Ledger Stress Tests (LST) for API and 20K runs with concurrency and
+  1K payload - using chaincode addrecs, GO, gRPC
+* Speed Tests for measuring turnaround performance of the communication path
+  and blockchain network processing - using chaincode addrecs, GO, gRPC
+* Concurrency Tests - using chaincode addrecs, GO, gRPC
+  - Objective of Concurrency test is to ensure system is able to accept
+    multiple threads concurrently for specified number of mins.
+* Complex Transactions Tests - using chaincode auction, node, gRPC
+  - Objective of this test is to send complex transactions using
+    auction chaincode
+* Performance Tests, random sized payloads - using chaincodes example02
+  and auction, node, gRPC
+  - CI performs below tests in Performance testing
+     * Invoke on chaincode_example02 using 4 peers and
+       4 Threads for 180 secs
+     * Query on chaincode_example02 using 4 peers and
+       4 Threads for 180 secs
+     * Invoke on auction chaincode using 4 peers and
+       1000 Tx for each of the 4 Threads
+     * Query on auction chaincode using 4 peers and
+       1000 Tx for each of the 4 Threads
+     * Invoke on auction chaincode using 4 peers and 4 Threads.
+       Each Invoke is followed by a Query on every Thread
+     * Invoke on example02 chaincode using 4 peers and 4 Threads.
+Each Invoke is followed by a Query on every Thread
+
+
+### LongRun Test Suite - longrun_test_suite.sh
+Expected total time duration is 72 hours when executed from
+Jenkins automation scripts, which run several parallel jobs.
+
+* Consensus peers restarts tests, 2 tests, each approximately 10 hours -
+  using chaincode example02, GO, gRPC
+* Ledger Stress Test 1 Million transactions, approximately 36 hours -
+  using chaincode addrecs, GO, gRPC
+* Ledger Stress Test, 72 hours - using chaincode addrecs, GO, gRPC
+  - Send parallel Invoke requests on all 4 peers
+* Performance Tests, 72 hours, random sized payloads - using chaincode auction,
+  node, gRPC (includes variable traffic, concurrency, complex transactions)
+  - Invoke 1 Tx per sec on 1 peer for 72 hrs (1 Thread)
+
+
+### Full Regression Test Suite
+Execute both the Daily Test Suite AND the Long Run Test Suite.
+

bddtests/regression/daily_test_suite.sh

@@ -0,0 +1,130 @@
+#!/bin/bash
+#
+# Copyright IBM Corp. 2016 All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# DESCRIPTION:
+# These are the tests in the Daily Test Suite.
+# To run these tests, user may clone the fabric, make code changes if
+# desired, build images, and run this script.
+
+
+# Setup
+
+WORKDIR=`pwd`
+
+
+
+
+
+# Note to test developers: Many tests in the Daily Test Suite are
+# specific tests that already contain the TEST OBJECTIVE and STEPS.
+# However, in some cases, such as running a flexible test engine to
+# perform different tests by using configuration parameters or
+# environment variables, it must be provided here.
+# For EACH of those test cases, before invoking the test, please
+# provide a comment block like this:
+
+#-----------------------------------------------------------------------
+# TEST OBJECTIVE : basic network setup, deploy, invoke, query all peers
+#                  using gRPC interface, and exercise chaincode APIs
+# SETUP STEPS:
+#   1. Set environment variable: export TEST_NETWORK=LOCAL
+# TEST DETAILS (Optional):
+#   1. Setup local docker network with security with 4 peer nodes
+#   2. Deploy chaincode example02
+#   3. Send Invoke Requests on all peers using go routines
+#   4. Verify query results and chainheights match on all peers
+#-----------------------------------------------------------------------
+
+
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# GO TDK          gRPC                Consensus Acceptance Tests
+########################################################################
+
+cd $WORKDIR/go/tdk/CAT
+#UNDER CONSTRUCTION...
+#go run CAT_100_Startup_DIQ_API.go
+#go run CAT_101_BasicConsensus_S1_R1_S2_S1_R1_R2.go
+#go run CAT_102_S1_IQDQIQ.go
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# GO TDK          gRPC                Transaction Speed Measurement
+########################################################################
+
+cd $WORKDIR/go/tdk/ledgerstresstest
+#./run_speed_tests.sh
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# GO TDK          gRPC                Ledger Stress Tests
+########################################################################
+
+cd $WORKDIR/go/tdk/ledgerstresstest
+#go run BasicFuncExistingNetworkLST.go
+#go run LST_1client1peer20K.go
+#go run LST_2client1peer20K.go
+#go run LST_2client2peer20K.go
+#go run LST_4client1peer20K.go
+#go run LST_4client4peer20K.go
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# GO TDK          gRPC                Concurrency/Ledger
+########################################################################
+
+cd $WORKDIR/go/tdk/ledgerstresstest
+#go run conc4p1min1000Thrd1TxPerLoop_LOCAL.go
+#go run conc4p1min400Thrd_LOCAL.go
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# Node.js         gRPC                Concurrency with Auction
+########################################################################
+
+cd $WORKDIR/node/
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# Node.js         gRPC                Complex Transactions with Auction
+########################################################################
+
+cd $WORKDIR/node/
+
+
+
+########################################################################
+# Language        API                 Functional Area
+# Node.js         gRPC                Performance engine Baseline tests
+########################################################################
+
+cd $WORKDIR/node/performance/
+
+

bddtests/regression/go/README.md

@@ -0,0 +1,8 @@
+## GO tests, scripts, and supporting files
+(For all scripts added to this directory, add information below to
+describe pre-requisites, setup steps, and usage examples.)
+
+## SETUP
+
+## USAGE
+

bddtests/regression/go/tdk/CAT/README.md

@@ -0,0 +1,74 @@
+## Consensus Acceptance Testcases (CAT)
+Hyperledger Fabric Go language Test Development Kit (TDK)
+tests can be executed on local fabric in vagrant environment with docker
+containers.
+Additional details (the date the tests were run, commit image,
+test config parameters, the test steps, output summary and details, etc.)
+are available in the GO_TEST files in this folder.
+
+### CAT test naming convention
+
+The testnames themselves indicate the steps performed. For example:
+
+```
+    CAT_210_S2S1_IQ_R1_IQ.go    Consensus Acceptance Test (CAT) suite, testcase number 210
+           _S2S1                Stop Validating PEERs VP2 and VP1 at virtually the same time
+                _IQ             Send some Invoke requests to all running peers, and Query all to validate A/B/ChainHeight results
+                   _R1          Restart VP1
+                      _IQ       Send some Invoke requests to all running peers, and Query all to validate A/B/ChainHeight results
+```
+
+### Test Coverage
+
+The objective of Consensus Acceptance Tests (CAT) is to ensure the
+stability and resiliency of the
+PBFT Batch design when Byzantine faults occur in a 4 peer network.
+Test areas coverage:
+
+Stop 1 peer: the network continues to process deploys, invokes, and query
+transactions.
+Perform this operation on each peer in the fabric.
+While exactly 3 peers are running, their ledgers will remain in synch.
+(In a 4 peer network, 3 represents 2F+1, the minimum number required for
+consensus.)
+Restarting a 4th peer will cause it to join in the network operations again.
+Note: when queried after having been restarted, the ledger on extra
+peers ( additional nodes beyond  2(F+1) )
+may appear to lag for some time. It will catch up if another peer is
+stopped (leaving it as one of exactly 2F+1 participating peers), OR, with
+no further disruptions in the network,
+it could catch up after huge numbers of transaction batches are processed.
+
+Stop 2 peers: the network should halt advancement, due to a lack of consensus.
+Restarting 1 or 2 peers will cause the network to resume processing
+transactions because enough nodes are available to reach consensus. This may
+include processing transactions that were received and queued by any running
+peers while consensus was halted.
+
+Stop 3 peers: the network should halt advancement due to a lack of consensus.
+Restarting just one of the peers should not resume consensus.
+Restarting 2 or 3 peers should cause the network to resume consensus.
+
+Deploys should be processed, or queued if appropriate, with any number of
+running peers.
+
+
+### RESULTS SUMMARY
+
+```
+    Consensus - Failed Testcases            Opened Bugs        Description
+
+    CAT_111_SnIQRnIQ_cycleDownLoop.go       FAB-337            dup Tx
+    CAT_303_S0S1S2_IQ_R0R1R2_IQ.go          FAB-333            lost Tx after restart 3 peers together
+    CAT_305_S1S2S3_IQ_R1R2R3_IQ.go          FAB-333            lost Tx after restart 3 peers together
+    CAT_407_S0S1S2_D_I_R0R1_IQ.go           FAB-911            lost Tx from vp0 after stop vp0/vp1/vp2, deploy, restart vp0/vp1, invokes
+    CAT_408_S0S1S2_D_I_R0R1R2_IQ.go         FAB-335            deploy failed when restart 3 peers together
+    CAT_409_S1S2S3_D_I_R1R2_IQ.go           FAB-334/FAB-912    lost Tx after all peers after stop vp1/vp2/vp3, deploy, restart vp1/vp2, invokes
+    CAT_410_S1S2S3_D_I_R1R2R3_IQ.go         FAB-335            deploy failed when restart 3 peers together
+    CAT_412_S0S1S2_D_I_R1R2_IQ.go           FAB-334/FAB-912    lost Tx after all peers after stop vp1/vp2/vp3, deploy, restart vp1/vp2, invokes
+
+    CRT_502_StopAndRestart1or2_10Hrs.go     FAB-331            acked Tx lost after stop/restart peers many times
+
+    CAT_201_S2S1_IQDIQ.go (using Pauses)    FAB-336            no http response from vp0, while peers vp1,vp2 docker paused
+```
+