Skip to content

Commit affbdad

Browse files
yeasyyeasy
committed
[CE-85] Fix setup scripts and docs
As new lib imported, there's problem by rebuilding images. In this patchset, we add new image-clean scripts and re-org the scripts and docs, to make the setup much easier to follow. Change-Id: I4a9ba38546b0965902019f899974fd5f2772d09b Signed-off-by: Baohua Yang <yangbaohua@gmail.com>
1 parent 0817214 commit affbdad

31 files changed

+452
-343
lines changed

‎Makefile

Lines changed: 36 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
# Copyright IBM Corp, All Rights Reserved.
32
#
43
# SPDX-License-Identifier: Apache-2.0
@@ -36,17 +35,19 @@ else
3635
endif
3736

3837
.PHONY: \
39-
all \
40-
check \
41-
clean \
42-
doc \
43-
log \
44-
logs \
45-
redeploy \
46-
restart \
47-
setup \
48-
start \
49-
stop \
38+
all \ # default to run check
39+
check \ # ci checking
40+
clean \ # clean up the env, remove temp files
41+
doc \ # start a doc server locally
42+
image-clean \ # clean up all cello related images
43+
log \ # show logs of specify service
44+
logs \ # show logs of all services
45+
setup-master \ # setup the master node
46+
setup-worker \ # setup the worker node
47+
redeploy \ # redeploy service(s)
48+
start \ # start all services
49+
restart \ # restart all services
50+
stop \ # stop all services
5051

5152
all: check
5253

@@ -55,22 +56,28 @@ check: ##@Code Check code format
5556
make start && sleep 10 && make stop
5657

5758
clean: ##@Code Clean tox result
58-
rm -rf .tox
59+
rm -rf .tox .cache *.egg-info
60+
find . -name "*.pyc" -o -name "__pycache__" -exec rm -rf "{}" \;
5961

6062
doc: ##@Create local online documentation
6163
pip install mkdocs
6264
mkdocs serve
6365

6466
# Use like "make log service=dashboard"
6567
log: ##@Log tail special service log, Use like "make log service=dashboard"
66-
docker-compose logs -f ${service} --tail=100
68+
docker-compose logs -f ${service} --tail=200
6769

6870
logs: ##@Log tail for all service log
6971
docker-compose logs -f --tail=200
7072

7173
# Use like "make redeploy service=dashboard"
7274
redeploy: ##@Service Redeploy single service, Use like "make redeploy service=dashboard"
73-
bash scripts/redeploy.sh ${service}
75+
bash scripts/master_node/redeploy.sh ${service}
76+
77+
image-clean: ##@Clean all existing images to rebuild
78+
echo "Clean all cello related images, may need to remove all containers before"
79+
docker images | grep "cello-" | awk '{print $1}' | xargs docker rmi -f
80+
docker rmi $(docker images -f dangling=true -q)
7481

7582
initial-env: ##@Configuration Initial Configuration for dashboard
7683
$(SED) 's/\(STATIC_FOLDER=\).*/\1${STATIC_FOLDER}/' .env
@@ -79,24 +86,31 @@ initial-env: ##@Configuration Initial Configuration for dashboard
7986

8087
start: ##@Service Start service
8188
@$(MAKE) $(START_OPTIONS)
82-
bash scripts/start.sh
89+
echo "Start all services..."
90+
docker-compose up -d --no-recreate
8391

8492
stop: ##@Service Stop service
85-
bash scripts/stop.sh
93+
echo "Stop all services..."
94+
docker-compose stop
95+
echo "Remove all services..."
96+
docker-compose rm -f -a
8697

8798
restart: stop start ##@Service Restart service
8899

89-
setup: ##@Environment Setup dependency for service environment
90-
bash scripts/setup.sh
100+
setup-master: ##@Environment Setup dependency for master node
101+
bash scripts/master_node/setup.sh
102+
103+
setup-worker: ##@Environment Setup dependency for worker node
104+
bash scripts/worker_node/setup.sh
91105

92106
build-js: ##@Nodejs Build js files for react
93-
bash scripts/build_reactjs.sh
107+
bash scripts/master_node/build_reactjs.sh
94108

95109
watch-mode: ##@Nodejs Run watch mode with js files for react
96-
bash scripts/watch_mode.sh
110+
bash scripts/master_node/watch_mode.sh
97111

98112
npm-install: ##@Nodejs Install modules with npm package management
99-
bash scripts/npm_install.sh
113+
bash scripts/master_node/npm_install.sh
100114

101115
HELP_FUN = \
102116
%help; \

‎docs/installation.md

Lines changed: 5 additions & 188 deletions
Original file line numberDiff line numberDiff line change
@@ -12,201 +12,18 @@ Cello follows a typical Master-Worker architecture. Hence there will be two type
1212
For each Node, it is suggested as a Linux-based (e.g., Ubuntu 14.04+) server/vm:
1313

1414
## Worker Node
15-
Currently we support Docker Host or Swarm Cluster as Worker Node. More types will be added soon.
16-
17-
For the Worker Node with meeting the [system requirements](#system-requirements), three steps are required:
18-
19-
* [Docker daemon setup](#docker-daemon-setup)
20-
* [Docker images pulling](#docker-images-pulling)
21-
* [Firewall Setup](#firewall-setup)
22-
23-
### System Requirements
24-
* Hardware: 8c16g100g
25-
* Docker engine:
26-
- 1.10.0~1.13.0
27-
* aufs-tools (optional): Only required on ubuntu 14.04.
28-
29-
### Docker Daemon Setup
30-
31-
Let Docker daemon listen on port 2375, and make sure Master can reach Worker Node through this port.
32-
33-
#### Ubuntu 14.04
34-
Simple add this line into your Docker config file `/etc/default/docker`.
35-
36-
```sh
37-
DOCKER_OPTS="$DOCKER_OPTS -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --api-cors-header='*' --default-ulimit=nofile=8192:16384 --default-ulimit=nproc=8192:16384"
38-
```
39-
40-
Then restart the docker daemon with:
41-
42-
```sh
43-
$ sudo service docker restart
44-
```
45-
46-
#### Ubuntu 16.04
47-
Update `/lib/systemd/system/docker.service` like
48-
49-
```
50-
[Service]
51-
DOCKER_OPTS="$DOCKER_OPTS -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --api-cors-header='*' --default-ulimit=nofile=8192:16384 --default-ulimit=nproc=8192:16384"
52-
EnvironmentFile=-/etc/default/docker
53-
ExecStart=
54-
ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS
55-
```
56-
57-
Regenerate the docker service script and restart the docker engine:
58-
59-
```sh
60-
$ sudo systemctl daemon-reload
61-
$ sudo systemctl restart docker.service
62-
```
63-
64-
### Alternatively (for all Linux distro):
65-
This will run the docker-daemon on port 2375 as long as the system is restarted or docker-daemon is killed.
66-
67-
```sh
68-
$ sudo systemctl stop docker.service
69-
$ sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --api-cors-header='*' --default-ulimit=nofile=8192:16384 --default-ulimit=nproc=8192:16384 -D &
70-
```
71-
72-
At last, run the follow test at Master node and get OK response, to make sure it can access Worker node successfully.
73-
74-
```sh
75-
[Master] $ docker -H Worker_Node_IP:2375 version
76-
```
77-
78-
### Docker Images Pulling
79-
Pulling the necessary images.
80-
81-
#### Fabric v1.0
82-
83-
```bash
84-
$ cd scripts/worker_node_setup && bash download_images.sh
85-
```
86-
87-
#### Fabric v0.6 (TODO: deprecated soon)
88-
89-
For fabric v0.6, need to run the following command.
90-
91-
```bash
92-
$ docker pull hyperledger/fabric-peer:x86_64-0.6.1-preview \
93-
&& docker pull hyperledger/fabric-membersrvc:x86_64-0.6.1-preview \
94-
&& docker pull yeasy/blockchain-explorer:latest \
95-
&& docker tag hyperledger/fabric-peer:x86_64-0.6.1-preview hyperledger/fabric-peer \
96-
&& docker tag hyperledger/fabric-peer:x86_64-0.6.1-preview hyperledger/fabric-baseimage \
97-
&& docker tag hyperledger/fabric-membersrvc:x86_64-0.6.1-preview hyperledger/fabric-membersrvc
98-
```
99-
100-
101-
### Artifacts Preparation
102-
103-
Copy required artifacts to the `/opt/cello`, e.g.,
104-
105-
```bash
106-
$ cp -r src/agent/docker/_compose_files/fabric-1.0 /opt/cello
107-
```
10815

109-
### Firewall Setup
110-
Make sure ip forward is enabled, you can simply run the follow command.
16+
Currently we support Docker Host or Swarm Cluster as Worker Node. More types will be added soon.
11117

112-
```sh
113-
$ sysctl -w net.ipv4.ip_forward=1
114-
```
115-
And check the os iptables config, to make sure host ports are open (e.g., 2375, 7050~10000)
18+
`Docker Host`: See [Installation on Worker Docker Node](installation_worker_docker.md).
19+
`Docker Swarm `: TODO.
20+
`Kubernetes`: TODO.
11621

11722
## Master Node
118-
The Master Node includes several services:
119-
120-
* dashboard: Provide Web UI for operators.
121-
* restserver: Provide RESTful APIs for chain consumers.
122-
* watchdog: Watch for health checking.
123-
124-
*More details can be found at the [architecture doc](docs/arch.md).*
125-
126-
It can be deployed by in 3 steps.
127-
128-
* Clone code
129-
* Pull Docker images
130-
* Run setup script
131-
132-
### System Requirement
133-
* Hardware: 8c16g100g
134-
* Docker engine: 1.10.0~1.13.0
135-
* docker-compose: 1.8.0+
136-
137-
### Clone Code
138-
139-
You may check `git` and `make` are installed to clone the code.
140-
141-
```sh
142-
$ sudo aptitude install git make -y
143-
$ git clone http://gerrit.hyperledger.org/r/cello && cd cello
144-
```
145-
146-
### Docker Images Pulling
147-
148-
Pull the following images
149-
150-
```bash
151-
$ docker pull python:3.5 \
152-
&& docker pull mongo:3.2 \
153-
&& docker pull yeasy/nginx:latest \
154-
&& docker pull mongo-express:0.30
155-
```
156-
157-
*Note: mongo-express:0.30 is for debugging the db, which is optional for basic setup.*
158-
159-
### Run Setup
160-
161-
For the first time running, please setup the master node with
162-
163-
```sh
164-
$ make setup
165-
```
166-
167-
Make sure there is no error during the setup. Otherwise, please check the log msgs.
168-
169-
### Usage
170-
171-
#### Start/Stop/Restart
172-
To start the whole services, please run
173-
174-
```sh
175-
$ make start
176-
```
177-
178-
To stop or restart the whole services, run `make stop` or `make restart`.
179-
180-
#### Redploy a service
181-
To redeploy one specific service, e.g., dashboard, please run
182-
183-
```sh
184-
$ make redeploy service=dashboard
185-
```
186-
187-
#### Check Logs
188-
To check the logs for all the services, please run
189-
190-
```sh
191-
$ make logs
192-
```
193-
194-
To check the logs for one specific service, please run
195-
```sh
196-
$ make log service=watchdog
197-
```
198-
199-
Now you can access the `MASTER_NODE_IP:8080` to open the Web-based [operational dashboard](docs/dashboard.md).
200-
201-
### Configuration
202-
The application configuration can be imported from file named `CELLO_CONFIG_FILE`.
20323

204-
By default, it also loads the `config.py` file as the configurations.
24+
See [Installation on Master Node](installation_master.md).
20525

206-
### Data Storage
207-
The mongo container will use local `/opt/cello/mongo` directory for persistent storage.
20826

209-
Please keep it safe by backups or using more high-available solutions.
21027

21128
*Licensed under Creative Commons Attribution 4.0 International License
21229
https://creativecommons.org/licenses/by/4.0/*

‎docs/installation_master.md

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
## Master Node Setup
2+
The Master Node includes several services:
3+
4+
* dashboard: Provide Web UI for operators.
5+
* restserver: Provide RESTful APIs for chain consumers.
6+
* watchdog: Watch for health checking.
7+
8+
*More details can be found at the [architecture doc](docs/arch.md).*
9+
10+
It can be deployed by in 2 steps.
11+
12+
* Clone code
13+
* Run setup script
14+
15+
### System Requirement
16+
* Hardware: 8c16g100g
17+
* Docker engine: 1.10.0~1.13.0 (Docker 17.0+ support is experimental)
18+
* docker-compose: 1.8.0~1.12.0
19+
20+
### Clone Code
21+
22+
You may check `git` and `make` are installed to clone the code.
23+
24+
```sh
25+
$ sudo aptitude install git make -y
26+
$ git clone http://gerrit.hyperledger.org/r/cello && cd cello
27+
```
28+
29+
### Run Setup
30+
31+
For the first time running, please setup the master node with the [setup.sh](https://github.com/hyperledger/cello/blob/master/scripts/setup.sh).
32+
33+
```sh
34+
$ make setup-master
35+
```
36+
37+
Make sure there is no error during the setup. Otherwise, please check the log msgs.
38+
39+
### Usage
40+
41+
#### Start/Stop/Restart
42+
To start the whole services, please run
43+
44+
```sh
45+
$ make start
46+
```
47+
48+
To stop or restart the whole services, run `make stop` or `make restart`.
49+
50+
#### Redploy a service
51+
To redeploy one specific service, e.g., dashboard, please run
52+
53+
```sh
54+
$ make redeploy service=dashboard
55+
```
56+
57+
#### Check Logs
58+
To check the logs for all the services, please run
59+
60+
```sh
61+
$ make logs
62+
```
63+
64+
To check the logs for one specific service, please run
65+
```sh
66+
$ make log service=watchdog
67+
```
68+
69+
Now you can access the `MASTER_NODE_IP:8080` to open the Web-based [operational dashboard](docs/dashboard.md).
70+
71+
72+
### Configuration
73+
The application configuration can be imported from file named `CELLO_CONFIG_FILE`.
74+
75+
By default, it also loads the `config.py` file as the configurations.
76+
77+
### Data Storage
78+
The mongo container will use local `/opt/cello/mongo` directory for persistent storage.
79+
80+
Please keep it safe by backups or using more high-available solutions.

0 commit comments

Comments
 (0)