[FABN-1169] auto-gen test certs in build

- Add gulp scripts to automatically download fabric binaries and generate certs
- Add generate certs step to Jenkins build scripts
- Remove git-cached certs from repository
- Add to ReadMe files the required initial steps to get certs locally

Change-Id: Id13634c6bcc1bb8daab531bd8185da58cb5093f8
Signed-off-by: nkl199@yahoo.co.uk <nkl199@yahoo.co.uk>

.gitignore

@@ -19,11 +19,13 @@ tmp
 *.heapsnapshot
 
 #test files
+test/fixtures/crypto-material/crypto-config/
+test/fixtures/crypto-material/channel-config/
+test/fixtures/crypto-material/config-base/twoorgs.genesis.block
+test/fixtures/fabricca/enroll*.*
+test/fixtures/fabricca/test*.*
 test/typescript/**/*.js
 test/typescript/**/*.js.map
-test/fixtures/src/node_cc/example_cc/.npmrc
-test/fixtures/src/node_cc/example_cc1/.npmrc
-test/fixtures/src/github.com/example_cc/junk.go
 
 #fabric client
 fabric-client/.DS_Store

README.md

@@ -23,6 +23,7 @@ Clone the project and launch the following commands to install the dependencies
 In the project root folder:
 * `npm install` to install dependencies
 * optionally, `gulp docs` to generate API docs if you want to review the doc content
+* `install-and-generate-certs` to generate the required crypto material used by the tests
 * `npm test` or `gulp test-headless` to run the headless tests that do not require any additional set up
 
 The following tests require setting up a local blockchain network as the target. You need to build the necessary Docker images required to run the network. Follow the steps below to set it up.

build/tasks/certs.js

@@ -0,0 +1,66 @@
+/*
+# Copyright IBM Corp. All Rights Reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+*/
+
+const gulp = require('gulp');
+const shell = require('gulp-shell');
+const runSequence = require('run-sequence');
+
+const version = '1.4.0';
+const binariesPath = '/tmp/fabric-binaries';
+const darwinTarFile = 'hyperledger-fabric-darwin-amd64-' + version + '.tar.gz';
+const amd64TarFile = 'hyperledger-fabric-linux-amd64-' + version + '.tar.gz';
+const s390TarFile = 'hyperledger-fabric-linux-s390x-' + version + '.tar.gz';
+const darwin = 'darwin-amd64-' + version + '/' + darwinTarFile;
+const amd64 = 'linux-amd64-' + version + '/' + amd64TarFile;
+const s390 = 'linux-s390x-' + version + '/' + s390TarFile;
+const binariesRoot = 'https://nexus.hyperledger.org/content/repositories/releases/org/hyperledger/fabric/hyperledger-fabric/';
+const darwinBinaries =  binariesRoot + darwin;
+const amd64Binaries = binariesRoot + amd64;
+const s390Binaries = binariesRoot + s390;
+
+// Retrieve the cryptogen material binaries, pinned at 1.4
+// Download and xxtract binaries from tar file
+// Set to path via export
+gulp.task('get-crypto-binaries-amd64', shell.task(
+	'mkdir -p ' + binariesPath + ';' +
+	'wget ' + amd64Binaries + ' -P ' + binariesPath + ';' +
+	'tar xvzf ' + binariesPath + '/' + amd64TarFile + ' -C ' + binariesPath + ';')
+);
+
+gulp.task('get-crypto-binaries-mac', shell.task(
+	'mkdir -p ' + binariesPath + ';' +
+	'wget ' + darwinBinaries + ' -P ' + binariesPath + ';' +
+	'tar xvzf ' + binariesPath + '/' + darwinTarFile + ' -C ' + binariesPath + ';')
+);
+
+gulp.task('get-crypto-binaries-s390', shell.task(
+	'mkdir -p ' + binariesPath + ';' +
+	'wget ' + s390Binaries + ' -P ' + binariesPath + ';' +
+	'tar xvzf ' + binariesPath + '/' + s390TarFile + ' -C ' + binariesPath + ';')
+);
+
+// Generate required crypto material, channel tx blocks, and fabric ca certs
+// - shell command to run the required test file scripts
+gulp.task('generate-test-certs', shell.task(
+	'./test/fixtures/crypto-material/generateAll.sh ' + binariesPath + '/bin;' +
+	'./test/fixtures/fabricca/generateCSR.sh;')
+);
+
+// Perform both of the above sequentially
+gulp.task('install-and-generate-certs', (done) => {
+	const tasks = ['get-crypto-binaries-amd64', 'generate-test-certs'];
+	runSequence(...tasks, done);
+});
+
+gulp.task('install-and-generate-certs-mac', (done) => {
+	const tasks = ['get-crypto-binaries-mac', 'generate-test-certs'];
+	runSequence(...tasks, done);
+});
+
+gulp.task('install-and-generate-certs-s390', (done) => {
+	const tasks = ['get-crypto-binaries-s390', 'generate-test-certs'];
+	runSequence(...tasks, done);
+});

scripts/Jenkins_Scripts/CI_Script.sh

@@ -92,7 +92,7 @@ function removeUnwantedImages() {
 rm -rf $HOME/.node-gyp/ $HOME/.npm/ $HOME/.npmrc  || true
 
 # remove tmp/hfc and hfc-key-store data
-rm -rf /home/jenkins/npm /tmp/fabric-shim /tmp/hfc* /tmp/npm* /home/jenkins/kvsTemp /home/jenkins/.hfc-key-store
+rm -rf /home/jenkins/npm /tmp/fabric-shim /tmp/hfc* /tmp/npm* /home/jenkins/kvsTemp /home/jenkins/.hfc-key-store /tmp/fabric-binaries
 
 rm -rf /var/hyperledger/*
 
@@ -191,6 +191,15 @@ sdk_E2e_Tests() {
         # Install NPM before start the tests
         install_Npm
 
+		# Generate crypto material before running the tests
+		if [ $ARCH == "s390x" ]; then
+			# Run the s390x gulp task
+			gulp install-and-generate-certs-s390 || err_Check "ERROR!!! gulp install and generation of test certificates failed"
+		else
+			# Run the amd64 gulp task
+			gulp install-and-generate-certs || err_Check "ERROR!!! gulp install and generation of test certificates failed"
+		fi
+
         echo -e "\033[32m Execute Headless and Integration Tests" "\033[0m"
         gulp test || err_Check "ERROR!!! gulp test failed"
 

test/README.md

@@ -13,6 +13,12 @@ The functional tests are currently written in [Tape](https://github.com/substack
 
 The scenario tests are written in [Cucumber](https://github.com/cucumber/cucumber-js), with the intention of providing high level test coverage from a scenario perspective. For more information, please refer to the [README](./scenario/README.md) within the scenario directory.
 
+Test certificates are set to expire a year after generation. Due to this the test suite generates new certificates as part of the build process, and is a manual requirement prior to running the tests locally. This process is orchestrated using gulp files that:
+ - Download, install and export the path to the 1.4 Hyperledger Fabric binaries used for generating crypto material
+ - Generate the crypto-material, matching channel blocks and fabric-ca certificates required by the docker-compose files and test suites
+
+Use the gulp task `gulp install-and-generate-certs` to perform the above on a linux x64 machine, or `gulp install-and-generate-certs-mac` for a mac. This is only required to be performed upon initial project clone, and then yearly afterwards.
+
 ## Structure
 
 The folder structure is the following:
@@ -28,7 +34,7 @@ test
 └───unit
 ```
 
-- `fixtures` holds all the configuration files used by the integration tests
+- `fixtures` holds all the configuration files used by the integration and scenario tests
 - `integration` contains the interation test suite
 - `scenario` contains the sceanrio test suite
 - `typescript` contains the typescript test suite

test/fixtures/crypto-material/README.md

@@ -1,12 +1,3 @@
-Cryptogen created using fabric 1.4 level cryptogen using `generate.sh`
-
-Mulitple configs are generated:
-- config-base, does not include anchor peers
-- config-update, used to update anchor peers
-
-Both folders contain a ./generate.sh to run, that creates new material to be used. A smart person would use a script to run them both, that's what `generateAll.sh` does
-
-Don't forget:
-- export the cryptogen path before you run the command, toherwise it will fail, and you will cry
-
-The above is to be automated in the build so that we create new material prior to evey test, which will prevent certifaicate expiration issues.
+All crypto material here is generated via the contained shell scripts files and orchestrated by the parent gulp files `/build/tasks/certs.js`, please use the gulp files to create any required certificates by:
+- Obtain crypto-gen binaries (`gulp get-crypto-binaries`)
+- Generate the crypto material required for tests (`gulp generate-test-certs`)