FAB-3932 standardize test names

+ Renamed some files and testnames to meet the standardized naming conventions, for a better user experience when viewing test results on CI reports page for all the tests in the daily and weekly test suite. + Added some scripts to facilitate running the weekly test suite in groups in parallel. + Use Readme doc syntax strategy to use italics font format for filenames and paths, and use bold font to highlight other key phrases. Change-Id: Icdecd4c5db74e10103daad2f2e4424cd5028a26a Signed-off-by: Scott Zwierzynski <scottz@us.ibm.com>
hyperledger · May 15, 2017 · a4dbc17 · a4dbc17
1 parent 0509bed
commit a4dbc17
Show file tree

Hide file tree

Showing 9 changed files with 89 additions and 35 deletions.
diff --git a/test/regression/daily/test_example.py → test/regression/daily/Example.py b/test/regression/daily/test_example.py → test/regression/daily/Example.py
@@ -2,7 +2,7 @@
 # To run this:
 # Install: sudo apt-get install python python-pytest
 # Install: sudo pip install xmlrunner
-# At command line: py.test -v --junitxml results.xml ./test_example.py
+# At command line: py.test -v --junitxml results_sample.xml Example.py
 
 import unittest
 import xmlrunner
@@ -21,7 +21,7 @@ def test_skipped(self):
     # This runs on ubuntu x86 laptop, but it fails when run by CI, because
     # "bc" is not installed on the servers used for CI jobs.
     @unittest.skip("skipping")
-    def test_SampleAdditionTestWillPass(self):
+    def test_SampleAdditionTestSkippedButWillPassIfInstallBC(self):
         '''
         This test will pass.
         '''

diff --git a/test/regression/daily/README.md b/test/regression/daily/README.md
@@ -1,61 +1,61 @@
 # Daily Test Suite
 
-This readme explains everything there is to know about our daily regression test suite.  *Note 1*: This applies similarly for both the **test/regression/daily/** and **test/regression/weekly/** test suites.  *Note 2*: The Release Criteria (**test/regression/release/**) test suite is a subset of all the Daily and Weekly tests.
+This readme explains everything there is to know about our daily regression test suite.  **Note 1**: This applies similarly for both the *test/regression/daily/* and *test/regression/weekly/* test suites.  **Note 2**: The Release Criteria (*test/regression/release/*) test suite is a subset of all the Daily and Weekly tests.
 
 - How to Run the Tests
 - Where to View the Results produced by the daily automation tests
 - Where to Find Existing Tests
 - How to Add New Tests to the Automated Test Suite
-  * Why Test Output Format Must Be *xml* and How to Make It So
+  * Why Test Output Format Must Be **xml** and How to Make It So
   * Alternative 1: Add a test using an existing tool and test driver script
   * Alternative 2: Add a new test with a new tool and new test driver script
   * How to Add a New Chaincode Test
 
 ## How to Run the Tests, and Where to View the Results
 
-Everything starts with [runDailyTestSuite.sh](./runDailyTestSuite.sh), which invokes all test driver scripts, such as **test_pte.py** and **test_chaincodes.py**. Together, these driver scripts initiate all tests in the daily test suite. You can manually execute **runDailyTestSuite.sh** in its entirety, or, run one any one of the test driver scripts on the command line. Or, you may simply view the results generated daily by an automated Continuous Improvement (CI) tool which executes **runDailyTestSuite.sh**. Reports are displayed on the [Daily Test Suite Results Page](https://jenkins.hyperledger.org/view/Daily/job/fabric-daily-chaincode-tests-x86_64/test_results_analyzer). When you look at the reports; click the buttons in the **'See children'** column to see the results breakdown by component and by individual tests.
+Everything starts with [runDailyTestSuite.sh](./runDailyTestSuite.sh), which invokes all test driver scripts, such as *systest_pte.py* and *chaincodes.py*. Together, these driver scripts initiate all tests in the daily test suite. You can manually execute *runDailyTestSuite.sh* in its entirety, or, run one any one of the test driver scripts on the command line. Or, you may simply view the results generated daily by an automated Continuous Improvement (CI) tool which executes *runDailyTestSuite.sh*. Reports are displayed on the [Daily Test Suite Results Page](https://jenkins.hyperledger.org/view/Daily/job/fabric-daily-chaincode-tests-x86_64/test_results_analyzer). When you look at the reports; click the buttons in the **'See children'** column to see the results breakdown by component and by individual tests.
 
 #### Where to Find Existing Tests
 
-Examine the driver scripts to find the individual tests, which are actually stored in several locations under **/path/to/fabric/test/**. Some tests are located in test suite subdirectories such as
+Examine the driver scripts to find the individual tests, which are actually stored in several locations under */path/to/fabric/test/*. Some tests are located in test suite subdirectories such as
 
-- **test/regression/daily/chaincodeTests/**
+- *test/regression/daily/chaincodeTests/*
 
 whereas other tests are located in the tools directories themselves, such as
 
-- **test/feature/ft/** - User-friendly *Behave* functional tests feature files
-- **test/tools/PTE/** - Performance Traffic Engine *(PTE)* tool and tests
-- **test/tools/OTE/** - Orderer Traffic Engine *(OTE)* tool and tests
+- *test/feature/ft/* - User-friendly **Behave** functional tests feature files
+- *test/tools/PTE/* - Performance Traffic Engine **(PTE)** tool and tests
+- *test/tools/OTE/* - Orderer Traffic Engine **(OTE)** tool and tests
 
 Each testcase title should provide the test objective and a Jira FAB issue which can be referenced for more information. Test steps and specific details can be found in the summary comments of the test scripts themselves. Additional information can be found in the README files associated with the various test directories.
 
 ## How to Add New Tests to the Automated Test Suite
 
-We love contributors! Anyone may add a new test to an existing test driver script, or even create a new tool and new test driver script. The steps for both scenarios are provided further below as *Alternative 1* and *Alternative 2*. First, a few things to note:
+We love contributors! Anyone may add a new test to an existing test driver script, or even create a new tool and new test driver script. The steps for both scenarios are provided further below as **Alternative 1** and **Alternative 2**. First, a few things to note:
 
 - Before linking a test case into the CI automation tests, please merge your (tool and) testcase into gerrit, and create a Jira task, as follows:
 
-  1. First merge your tool and tests to gerrit in appropriate folders under **/path/to/fabric/test/**.
+  1. First merge your tool and tests to gerrit in appropriate folders under */path/to/fabric/test/*.
   1. Of course, all tests must pass before being submitted. We do not want to see any false positives for test case failures.
   1. To integrate your new tests into the CI automation test suite, create a new Jira task FAB-nnnn for each testcase, and use 'relates-to' to link it to epic FAB-3770.
-  1. You will this new Jira task to submit a changeset to gerrit, to invoke your testcase from a driver script similar to **/path/to/fabric/test/regression/daily/test_example.py**. In the comments of the gerrit merge request submission, include the
+  1. You will this new Jira task to submit a changeset to gerrit, to invoke your testcase from a driver script similar to */path/to/fabric/test/regression/daily/Example.py*. In the comments of the gerrit merge request submission, include the
       - Jira task FAB-nnnn
       - the testcase title and objective
       - copy and fill in the template from Jira epic FAB-3770
-  1. Follow all the steps below in either *Alternative*, and then the test will be executed automatically as part of the next running of the CI daily test suite. The results will show up on the daily test suite display board - which can be viewed by following the link at the top of this page.
+  1. Follow all the steps below in either **Alternative**, and then the test will be executed automatically as part of the next running of the CI daily test suite. The results will show up on the daily test suite display board - which can be viewed by following the link at the top of this page.
 
-#### Why Test Output Format Must Be *xml* and How to Make It So
+#### Why Test Output Format Must Be **xml** and How to Make It So
 
-The Continuous Improvement (CI) team utilizes a Jenkins job to execute the full test suite, **runDailyTestSuite.sh**. The CI job consumes xml output files, creates reports, and displays them. *Note: When adding new scripts that generate new xml files, if you do not see the results displayed correctly, please contact us on [Rocket.Chat channel #fabric-ci](https://chat.hyperledger.org).* For this reason, we execute tests in one of the following ways:
+The Continuous Improvement (CI) team utilizes a Jenkins job to execute the full test suite, *runDailyTestSuite.sh*. The CI job consumes xml output files, creates reports, and displays them. **Note:** When adding new scripts that generate new xml files, if you do not see the results displayed correctly, please contact us on [Rocket.Chat channel #fabric-ci](https://chat.hyperledger.org). For this reason, we execute tests in one of the following ways:
 
-  1. Invoke the individual testcase from within a test driver script in **regression/daily/**. There are many examples here, such as **test_example.py** and **test_pte.py**. These test driver scripts are basically wrappers written in python, which makes it easy to produce the desired junitxml output format required for displaying reports. This method is useful for almost any test language, including bash, tool binaries, and more. More details are provided below explaining how to call testcases from within a test driver script. Here we show how simple it is to execute the test driver and all the testcases within it. *Note: File 'example_results.xml' will be created, containing the test output.*
+  1. Invoke the individual testcase from within a test driver script in *regression/daily/*. There are many examples here, such as *Example.py* and *systest_pte.py*. These test driver scripts are basically wrappers written in python, which makes it easy to produce the desired junitxml output format required for displaying reports. This method is useful for almost any test language, including bash, tool binaries, and more. More details are provided below explaining how to call testcases from within a test driver script. Here we show how simple it is to execute the test driver and all the testcases within it. **Note:** File *results_sample.xml* will be created, containing the sample testcases output.
 
   ```
      cd /path/to/fabric/test/regression/daily
-     py.test -v --junitxml example_results.xml ./test_example.py
+     py.test -v --junitxml example_results.xml ./Example.py
   ```
 
-  1. Execute 'go test', and pipe the output through tool github.com/jstemmer/go-junit-report to convert to xml. *Note: In the example shown, file 'results.xml' will be created with the test output.*
+  1. Execute 'go test', and pipe the output through tool github.com/jstemmer/go-junit-report to convert to xml. **Note:** In the example shown, file 'results.xml' will be created with the test output.
 
   ```
      cd /path/to/fabric/test/tools/OTE
@@ -67,14 +67,14 @@ The Continuous Improvement (CI) team utilizes a Jenkins job to execute the full
 
 ### Alternative 1:  Add a test using an existing tool and test driver script
 
-To add another test using an existing tool (such as **PTE**), simply add a test inside the existing test driver (such as **test_pte.py**). It is as simple as copying a block of ten lines and modify these things:
+To add another test using an existing tool (such as **PTE**), simply add a test inside the existing test driver (such as *systest_pte.py*). It is as simple as copying a block of ten lines and modify these things:
 
   1. Insert the testcase in the correct test component class and edit the test name
   1. Edit the testcase description
   1. Edit the specified command and arguments to be executed
   1. Edit the asserted test result to be matched
 
-Refer to **test_example.py** for a model to clone and get started quickly. The testcases should use the format shown in this example:
+Refer to *Example.py* for a model to clone and get started quickly. The testcases should use the format shown in this example:
 
   ```
       def test_FAB9876_1K_Payload(self):
@@ -94,9 +94,9 @@ Refer to **test_example.py** for a model to clone and get started quickly. The t
 
 Adding a new test with a new tool involves a few more steps.
 
-  1. Create and merge a new tool, for example, **/path/to/fabric/test/tools/NewTool/newTool.sh**
-  1. Create a new test driver script such as **/path/to/fabric/test/regression/daily/test_newTool.py**.  Model it after others like **test_example.py**, found under driver scripts under **/path/to/test/regression/daily/** and **test/regression/weekly/**.  Note: the filename must start with 'test_'.
-  1. Add your new testcases to **test_newTool.py**. The testcases should use the following format. Refer also to the steps described in Alternative 1, above.
+  1. Create and merge a new tool, for example, */path/to/fabric/test/tools/NewTool/newTool.sh*
+  1. Create a new test driver script such as */path/to/fabric/test/regression/daily/newTool.py*.  Model it after others like *Example.py*, found under driver scripts under */path/to/test/regression/daily/* and *test/regression/weekly/*.
+  1. Add your new testcases to *newTool.py*. The testcases should use the following format. Refer also to the steps described in Alternative 1, above.
 
   ```
     class <component_feature>(unittest.TestCase):
@@ -109,10 +109,10 @@ Adding a new test with a new tool involves a few more steps.
         self.assertIn("<string from stdout of newTool that indicates PASS>", result)
   ```
 
-  1. Edit **/path/to/test/regression/daily/runDailyTestSuite.sh** to run the new testcases. Add a new line, or append your new test driver scriptname **test_newTool.py** to an existing line:
+  1. Edit */path/to/test/regression/daily/runDailyTestSuite.sh* to run the new testcases. Add a new line, or append your new test driver scriptname *newTool.py* to an existing line:
 
   ```
-     py.test -v --junitxml results.xml test_example.py  test_newTool.py
+     py.test -v --junitxml results_newTool.xml newTool.py
   ```
 
 ### How to Add a New Chaincode Test

diff --git a/test/regression/daily/runDailyTestSuite.sh b/test/regression/daily/runDailyTestSuite.sh
@@ -1,8 +1,24 @@
 #!/bin/bash
 
-echo "========== Example tests and PTE system tests..."
-py.test -v --junitxml results.xml test_example.py test_pte.py
+DAILYDIR="$GOPATH/src/github.com/hyperledger/fabric/test/regression/daily"
 
-echo "========== Chaincode tests..."
-chaincodeTests/runChaincodes.sh
+#echo "========== Sample Tests..."
+#py.test -v --junitxml results_sample.xml Example.py
 
+echo "========== System Test Performance Stress tests driven by PTE tool..."
+py.test -v --junitxml results_systest_pte.xml systest_pte.py
+
+echo "========== Test Your Chaincode ..."
+# TBD - after changeset https://gerrit.hyperledger.org/r/#/c/9163/ is merged,
+# replace the previous 2 lines with this new syntax to run all the chaincode tests;
+# and when making this change we should also remove file chaincodeTests/runChaincodes.sh)
+#
+#cd $DAILYDIR/chaincodeTests/envsetup
+#py.test -v --junitxml ../../results_testYourChaincode.xml testYourChaincode.py
+
+# TBD - after changeset https://gerrit.hyperledger.org/r/#/c/9251/ is merged,
+# and integrated with this, lines like these should be executed too:
+#echo "========== Ledger component performance tests..."
+#cd $DAILYDIR/ledgerperftests
+#py.test -v --junitxml results_perf_goleveldb.xml test_perf_goleveldb.py
+#py.test -v --junitxml results_perf_couchdb.xml test_perf_couchdb.py
diff --git a/test/regression/daily/test_pte.py → test/regression/daily/systest_pte.py b/test/regression/daily/test_pte.py → test/regression/daily/systest_pte.py
@@ -2,7 +2,7 @@
 ######################################################################
 # To execute:
 # Install: sudo apt-get install python python-pytest
-# Run on command line: py.test -v --junitxml results.xml ./test_pte.py
+# Run on command line: py.test -v --junitxml results.xml ./systest_pte.py
 
 import unittest
 import subprocess
@@ -14,7 +14,7 @@
 ### LEVELDB
 ######################################################################
 
-class LevelDB_Perf_Stress(unittest.TestCase):
+class Perf_Stress_LevelDB(unittest.TestCase):
     @unittest.skip("skipping")
     def test_FAB3808_TPS_Queries_1_Thread_TinyNtwk(self):
         '''
@@ -70,7 +70,7 @@ def test_FAB3835_TPS_Invokes_8_Thread_TinyNtwk(self):
 ### COUCHDB
 ######################################################################
 
-class CouchDB_Perf_Stress(unittest.TestCase):
+class Perf_Stress_CouchDB(unittest.TestCase):
     @unittest.skip("skipping")
     def test_FAB3807_TPS_Queries_1_Thread_TinyNtwk(self):
         '''

diff --git a/test/regression/weekly/README.md b/test/regression/weekly/README.md
@@ -3,3 +3,17 @@
 
 ## Test Details
 Refer to [../daily/README](../daily/README.md) Everything there is relevant.
+
+## Execute all tests in this directory
+The tests are split into groups; each could be executed by different CI jobs in parallel.
+
+```
+    cd /path/to/fabric/test/regression/weekly
+```
+
+```
+    ./runGroup1.sh
+    ./runGroup2.sh
+    ./runGroup3.sh
+```
+
diff --git a/test/regression/weekly/runGroup1.sh b/test/regression/weekly/runGroup1.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+######################################################################
+### Run one group of the tests in weekly test suite.
+
+echo "========== Performance Stress PTE Scaleup tests"
+py.test -v --junitxml results_systest_pte_Scaleup.xml systest_pte.py -k Scaleup
+
diff --git a/test/regression/weekly/runGroup2.sh b/test/regression/weekly/runGroup2.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+######################################################################
+### Run one group of the tests in weekly test suite.
+
+echo "========== Performance Stress PTE 12Hr test"
+py.test -v --junitxml results_systest_pte_12Hr.xml systest_pte.py -k TimedRun_12Hr
+
diff --git a/test/regression/weekly/runGroup3.sh b/test/regression/weekly/runGroup3.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+######################################################################
+### Run one group of the tests in weekly test suite.
+
+echo "========== Performance Stress PTE 72Hr test"
+py.test -v --junitxml results_systest_pte_72Hr.xml systest_pte.py -k TimedRun_72Hr
+
diff --git a/test/regression/weekly/test_pte.py → test/regression/weekly/systest_pte.py b/test/regression/weekly/test_pte.py → test/regression/weekly/systest_pte.py
@@ -2,7 +2,7 @@
 ######################################################################
 # To execute:
 # Install: sudo apt-get install python python-pytest
-# Run on command line: py.test -v --junitxml results.xml ./test_pte.py
+# Run on command line: py.test -v --junitxml results.xml ./systest_pte.py
 
 import unittest
 import subprocess
@@ -14,7 +14,7 @@
 ### COUCHDB
 ######################################################################
 
-class CouchDB_Perf_Stress(unittest.TestCase):
+class Perf_Stress_CouchDB(unittest.TestCase):
 
     @unittest.skip("skipping")
     def test_FAB3820_TimedRun_12Hr(self):