Merge pull request hyperledger#1689 from pSchlarb/newNetworkDoc

Added Documentation for creating a new Network

docs/source/NewNetwork/CLIInstall.md

@@ -0,0 +1,116 @@
+# Installing the `indy-cli`
+
+The `indy-cli` is developed under the [indy-sdk](https://github.com/hyperledger/indy-sdk). This documentation may be not up to date.
+
+You will need to perform the following once for each `indy-cli` machine you would like to set up (only 1 is required). 
+It is recommended that you install the `indy-cli` in your native work environment if possible, so you always have it available even when traveling. 
+
+## Containerized `indy-cli` Environment
+
+The following sections describe how to install and configure the `indy-cli` directly on a machine or VM. However, possibly the most convenient option is to use a containerized `indy-cli` environment like the one included with [von-network](https://github.com/bcgov/von-network).  For information on how to use the containerized `indy-cli` in `von-network`, refer to [Using the containerized indy-cli](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md)
+
+## Windows:
+To install the `indy-cli` on Windows 10 perform the following steps:
+1. Download https://repo.sovrin.org/windows/indy-cli/stable/1.16.0/indy-cli_1.16.0.zip and unzip it.
+   If there is a newer version under https://repo.sovrin.org/windows/indy-cli/stable/ it instead.
+2. Open a command prompt.  (This will work differently if you use Windows Terminal).
+3. `cd` to the directory where you unzipped the `indy-cli` package.  For example, if you unzipped directly in your ‘downloads’ directory like I did you would type:  `cd \Users\<Username>\Downloads\indy-cli_1.14.2`
+4. Create a JSON Config file containing your taaAcceptanceMechanism in the directory where indy-cli.exe resides (I created \Users\<Username>\Downloads\indy-cli_1.14.2\cliconfig.json on my machine)
+   ```json
+   {
+   "taaAcceptanceMechanism": "for_session" 
+   } 
+   ```
+5. Run  `indy-cli.exe --config cliconfig.json`  to verify proper installation.  You should see a new window appear with an `indy>` prompt,  (If you are double clicking to start `indy-cli`, you need to right click on the .exe in your window and add the --config parameter first.)  If you get an error stating that it is missing vcruntime140.dll then do the following:
+6. Download and install vc_redist.x64.exe from the Visual Studio 2017 section on the https://support.microsoft.com/en-ae/help/2977003/the-latest-supported-visual-c-downloads page, and then rerun indy-cli.exe to see if it works as described in previous step.
+7. Type ‘exit’ in the `indy-cli`
+
+## Ubuntu:
+To install the `indy-cli` on Ubuntu, perform the following steps from the ubuntu command line:
+
+1. `sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE7709D068DB5E88`
+2. `sudo add-apt-repository "deb https://repo.sovrin.org/sdk/deb xenial stable"`
+3. `sudo add-apt-repository "deb https://repo.sovrin.org/deb xenial stable"`
+4. `sudo apt-get update -y`
+5. `sudo apt-get upgrade -y `
+6. `sudo apt-get install -y indy-cli`
+7. `cd ~`
+8. Create a JSON Config file containing your taaAcceptanceMechanism in your home directory:
+`vim ~/cliconfig.json`
+
+   Press the “i” key and paste the following into the file:
+   ```json
+   {
+   "taaAcceptanceMechanism": "for_session" 
+   } 
+   ```
+   Press the “esc” key then the following characters to write the file and quit
+`:wq` 
+9. Run `indy-cli --config ~/cliconfig.json` to start the `indy-cli`
+
+## Mac:
+
+Since there is not a prepackaged version of the `indy-cli` prepared for the Mac, the following steps will help you to create an environment, build, and run the `indy-cli` in a Mac terminal.
+
+Open a Terminal
+Run the following commands in the terminal:
+
+1. `cd ~`
+2. `mkdir github`
+3. `cd github`
+4. `git clone https://github.com/hyperledger/indy-sdk.git`(might need xcode-select --install if error occurs)
+5. `/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"`
+6. `curl https://sh.rustup.rs -sSf | sh`
+7. Follow onscreen instructions to install rust
+8. `brew install pkg-config libsodium automake autoconf cmake openssl zeromq zmq`
+   NOTE: the openssl path needs to match what you currently have on your system
+
+9. Run  > `ls /usr/local/Cellar/openssl/`
+   Note the name of the directory shown (the example below shows 1.0.2p but the latest version is 1.1.1l) 
+
+   Use this directory in place of the one listed below in your .profile file 
+
+10. Add the following lines to your` ~/.profile file `(making the correction shown in the previous step if needed)
+      ```
+      export PATH="$HOME/.cargo/bin:$PATH:~/github/indy-sdk/libindy/target/debug:~/github/indy-sdk/cli/target/debug"
+      export PKG_CONFIG_ALLOW_CROSS=1
+      export CARGO_INCREMENTAL=1
+      export RUST_LOG=indy=trace
+      export RUST_TEST_THREADS=1
+      export OPENSSL_DIR=/usr/local/Cellar/openssl/1.0.2p  #use your path
+      export LIBRARY_PATH=~/github/indy-sdk/libindy/target/debug/
+      export LIBINDY_DIR=~/github/indy-sdk/libindy/target/debug/
+      ```
+11. Run the following commands from your terminal to build the `indy-cli`:
+      ```
+      source ~/.profile
+      cd ~/github/indy-sdk/libindy
+      cargo build
+      cd ../cli
+      cargo build
+      ```
+12. Create a JSON Config file containing your taaAcceptanceMechanism in your home directory:
+    `vim ~/cliconfig.json`
+   
+      Press the “i” key and paste the following into the file:
+      ```json
+      {
+      "taaAcceptanceMechanism": "for_session" 
+      } 
+      ```
+      Press the “esc” key then the following characters to write the file and quit
+   `:wq` 
+13. You can now run `indy-cli` from within a terminal by typing
+
+      `indy-cli --config ~/cliconfig.json`
+   
+      `indy> exit`     (To exit from the `indy-cli` prompt when you ar done) 
+
+      If the above gives error regarding library not loaded libssl.1.0.0, you will probably need to run the following command (all in one line should work) to revert your version: 
+  
+      ```
+      brew uninstall --ignore-dependencies openssl; brew uninstall openssl;
+      brew install https://github.com/tebelorg/Tump/releases/download/v1.0.0/openssl.rb
+      ```
+
+

docs/source/NewNetwork/CreateDID.md

@@ -0,0 +1,33 @@
+# Creating a DID using the `indy-cli`
+
+You will need to perform the following commands once for each `indy-cli` machine that you want to run on. The following commands contain suggestions to save certain values in a secure place.  Please do not share those values or that place with anyone.
+
+_If you just need to quickly generate a set of secrets (Seed and wallet key), or a Seed, DID, and Verkey and do not have an `indy-cli` environment already setup, you can use the `indy-cli` features integrated into `von-network`.  Refer to [Generate a set of Secrets](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md#generate-a-set-of-secrets), and [Generate your DID](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md#generate-your-did) for details._
+
+
+1. Start your `indy-cli` using the instructions from [Installing the `indy-cli`](./CLIInstall.md) for your platform.
+
+   All following commands are executing inside the `indy-cli`.
+
+2. Create a wallet with:
+
+   `wallet create <wallet_name> key`
+
+   You will be prompted for a wallet key.  What you type will not be displayed on the console.  Your wallet key is a secure key that only you should know, and it should be randomly generated.  Save it in a secure place for later use.  You will use it every time you need to send transactions to the ledger from the `indy-cli`.
+
+3. `wallet open wallet_name key`
+
+   You will be prompted for your wallet key.  What you type will not be displayed on the console.
+
+4. `did new seed`
+
+   You will be prompted for a seed.  What you type will not be displayed on the console.
+
+   If you have lost your original seed or have never created one, then create a new one. This seed is used to regenerate your DID and to add your DID to your wallet(s).  
+
+   The seed is a 32 character string that only you can know. 
+
+   > WARNING: Whoever knows your Seed can recreate your exact DID in their own wallet and use it to manage the ledger.
+   
+   Save your Seed in a secure place so that only you can recreate your DID whenever needed.  
+   Also save the public DID and verkey generated from this step so that you will know and can verify your public DID.

docs/source/NewNetwork/NewNetwork.md

@@ -0,0 +1,144 @@
+# Setting up a New Network
+
+## Introduction
+
+   The purpose of this document is to describe in some detail the process of building a brand-new Indy Node Network (Network) using 4 Stewards on their own separate nodes.
+   It goes into more details than [Starting a Network](../start-nodes.md).
+   These instructions are intended to be used for a distributed or “production” level environment but can be adapted to be used to build a private network.
+
+   This document is heavily based on [Create New Indy Network](https://docs.google.com/document/d/1XE2QOiGWuRzWdlxiI9LrG9Am9dCfPXBXnv52wGHorNE) and the [Steward Validator Preparation Guide v3](https://docs.google.com/document/d/18MNB7nEKerlcyZKof5AvGMy0GP9T82c4SWaxZkPzya4).
+
+## I. Create Network Governance documents (Optional)
+
+   Network Governance describes the policies and procedures by which your new network will run and be maintained. Here’s an example: [Sovrin Governance Framework](https://docs.google.com/document/d/1K8l5MfXQWQtpT49-FHuYn_ZnRC5m0Nwk)
+
+
+## II. Assign Network Trustees
+
+   Trustees are the people who manage the network and protect the integrity of the Network Governance.  This includes managing the `auth_rules`.
+
+   For a production Network, at least 3 Trustees representing three different persons are required and more are preferred.  For a test Network one Trustee is required and 3 or more are preferred (all Trustee DID’s may belong to the same user on a test network if needed). 
+
+   Initial Trustees (3 preferred) must create and submit a Trustee DID and Verkey so that the domain genesis file can be built.
+
+   Each trustee has to [instal the `indy-cli`](./CLIInstall.md) and [create a Trustee DID](./CreateDID.md).
+
+   Once the Trustees have created their DID and Verkey give the Trustees access to a spreadsheet like [this one](https://docs.google.com/spreadsheets/d/1LDduIeZp7pansd9deXeVSqGgdf0VdAHNMc7xYli3QAY/edit#gid=0) and have them fill out their own row of the Trustees sheet.  The completed sheet will be used to generate the genesis transaction files for the network.   
+
+
+## III. Genesis Stewards
+
+   A Steward is an organization responsible for running a Node on the Network
+
+   Exactly 4 “Genesis” Stewards are needed to establish the network, more Stewards can be added later.
+
+   Each Genesis Steward’s node information will be included in the Genesis Pool file, so they should be willing to install and maintain a Node on the new Network for an extended period of time.
+
+   The Stewards must:
+   1. Generate Steward DIDs as described in [Creating DID](./CreateDID.md).
+   1. Install their node as described in [Installation and configuration of Indy-Node](../installation-and-configuration.md) (with some small adjustments):
+      1. Determine a name for the new network and have the stewards substitute it in the appropriate places in the guide, such as when setting the network name and creating the directory when creating the keys for the node.
+      1. They all need to stop at the normal place ([3.5. Add Node to a Pool](../installation-and-configuration.md#3.5.-Add-Node-to-a-Pool)) as instructed in the guide as the steps that follow differ when creating a new network.  The following sections of this guide describe the steps required to start the new network.
+
+   Once the Stewards have created their DID and Verkey, and performed the initial setup of they node, give the Stewards access to a spreadsheet like [this one](https://docs.google.com/spreadsheets/d/1LDduIeZp7pansd9deXeVSqGgdf0VdAHNMc7xYli3QAY/edit#gid=0) and have them fill out their own row of the Stewards sheet.  The completed sheet will be used to generate the genesis transaction files for the network.   
+
+## IV. Create and Distribute genesis transaction files
+
+   Save the sheets filled out by the Trustees and Stewards as separate files in csv format, and use the [genesis_from_files.py](https://github.com/sovrin-foundation/steward-tools/tree/master/create_genesis) script to generate the `pool_transactions_genesis` and `domain_transactions_genesis` files for the network.
+
+   >Tip: The `generategenesisfiles` in `von-network` provides a convenient wrapper around the `genesis_from_files.py` and runs it in a container including all of the dependencies.  For more information refer to [Generate Genesis Files](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md#generate-genesis-files).
+
+   Double check the files contain the correct information:
+   - The `domain_transactions_genesis` file should contain all of the DIDs and Verkeys for the Trustees (`"role":"0"`) and the Stewards (`"role":"2"`).
+   - The `pool_transactions_genesis` file should contain each of the nodes with all their unique information.
+
+   Publish the genesis files to a public location, such as a GitHub repository associated with your network.  The Stewards and end users will need this information.
+
+   Inform the Stewards and Trustees where they can download the genesis files.
+
+   - The Trustees and Stewards will need to register the `pool_transactions_genesis` with their `indy-cli` to complete the setup and to be able to connect to the network once it's running.  How and where they need to register the `pool_transactions_genesis` depends on how they setup their `indy-cli` environment; [Installing the `indy-cli`](./CLIInstall.md)
+
+   - The Stewards will also need to download the genesis files onto their nodes while completing the setup.  All of the following steps are to be completed on the node.
+      1.  Set the network name in `/etc/indy/indy_config.py`, replacing `<the_network_name>` in the following command with the actual network name;
+
+            `sudo sed -i -re "s/(NETWORK_NAME = ')\w+/\1<the_network_name>/" /etc/indy/indy_config.py`
+
+      1. Create a network directory and download the genesis files into it.  _The directory name must be the same on all of the nodes and it must match the name of the network._
+         1. `sudo -i -u indy mkdir /var/lib/indy/<the_network_name>`
+         1. `cd  /var/lib/indy/<the_network_name>`
+         1. `sudo curl -o domain_transactions_genesis <url_to_the_raw_domain_transactions_genesis_file>`
+         1. `sudo curl -o pool_transactions_genesis  <url_to_the_raw_pool_transactions_genesis_file>`
+         1. `sudo chown indy:indy *`
+            - It is important the files are owned by `indy:indy`.
+
+## V. Schedule a meeting to instantiate the new network
+
+   Invite all Genesis Stewards to a meeting where they can execute commands and share their screens for both an `indy-cli` and for their Validator Nodes being added to the Network.
+
+   > NOTE: It is very useful to go through some checks for each node to verify their setup before continuing. Some large amounts of debug and recovery work can be avoided by 5-10 minutes of checking configs of each node at the beginning of the meeting.
+   > - `/etc/indy/indy_config.py`
+   >   - all nodes need to have the same network name.
+   >   - the name of the network should correspond to the `/var/lib/indy/<the_network_name>` directory on each node which contains the genesis files for the network, and the files in the directory should be owned by `indy:indy`.
+   > - `/etc/indy/indy.env`
+   >   - all nodes should have local ip addresses in this file and be pointing at the correct ports.
+   > - Genesis files
+   >   - Ensure both `pool_transactions_genesis` and `domain_transactions_genesis` files contain the expected content.
+   > - Verify the software version on all the nodes match
+   >   ```
+   >   dpkg -l | grep indy
+   >   dpkg -l | grep sovrin
+   >   ```
+   > - Network Connectivity
+   >   - Use `nc -l <local_ip_address> <port>` (on the host), and `nc -vz <public_ip_address> <port>` (on the remote) to test the following.
+   >     - Check the network connectivity between nodes using the `node_ip:port` combinations.  Ensure that each node can communicate with all of the other nodes.
+   >     - Check the network connectivity between the nodes and a client using the `client_ip:port` combinations.  Ensure each node is accessible to client machines.
+
+   Once all of the checks are complete have the Stewards simultaneously start their nodes as described in section [3.5.2. Enable the Service](../installation-and-configuration.md#3.5.2.-Enable-the-Service) of the Installation and configuration of Indy-Node guide, and walk though the remainder of that guide.
+
+## VI. Configure the Indy Network
+
+### `auth_rules`
+   Update the network's `auth_rules` to help enforce the governance rules for the network.
+
+   For more information on `auth_rules` refer to:
+   - [Default AUTH_MAP Rules](../auth_rules.md)
+   - [auth_rules Walkthough](https://docs.google.com/document/d/1xk0A5FljKOZ2Fazri6J5mAfnYWXdOMl2LwrFK16MJIY)
+
+### `TAA` (Transaction Author Agreements)
+   Add a `TAA` to the network.
+   
+   For more information on `TAA`s refer to:
+   - [Transaction Author Agreement - `indy-sdk`](https://github.com/hyperledger/indy-sdk/blob/master/docs/how-tos/transaction-author-agreement.md)
+   - [Transaction Author Agreement - `indy-plenum`](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/transaction_author_agreement.md)
+   - [Transaction Author Agreement Design](../../../design/txn_author_agreement.md)
+   - [TAA for CLI Walkthrough](https://docs.google.com/document/d/1Ma-EJkYpRfPOZApyEvcWrkb4EKn71XrIFd9KvZL0Whg)
+
+## Where to go from here?
+
+   ### Add more Nodes
+   
+   For the network to remain in write consensus in the event of node failures the network needs to be comprised of `3f+1` nodes, where `f` is the number of failed nodes.
+
+   For a network of 4 nodes the network can remain in write consensus if a single node at a time fails, however if more than a single node fails at a time the network will loose write consensus and go into a read-only state.  Similarly, a network comprised of 7 nodes can withstand up to 2 nodes failing at any given time.  Therefore, it's recommended to have at least 7 nodes running in your network.
+
+   Examples:
+
+   | Failures to Withstand | 3f+1 | Required Nodes |
+   |--|--|--|
+   | 1 | 3(1)+1 | 4 |
+   | 2 | 3(2)+1 | 7 |
+   | 3 | 3(3)+1 | 10 |
+
+   ### Network Monitoring
+
+   [hyperledger/indy-node-monitor](https://github.com/hyperledger/indy-node-monitor) is the community supported and maintained tool for network monitoring.
+
+   #### Manual
+   - Run `indy-node-monitor` at least three times a day to detect any issues with the network.
+
+   #### Automated
+   - Run `indy-node-monitor` on a schedule (every 15-30 minutes) and add a notification plugin to alert you to any issues.  _Please consider contributing your work back to the project._
+
+## Hands On Walkthrough
+
+   An example walkthrough of the above mentioned steps can be found in the `sample/Network` [folder](../../../sample/Network/README.md).