Prerequisites:
- Ubuntu 18.04.
- golang.
apt-get update
sudo locale-gen "en_US.UTF-8"
- Add to
/etc/environment
:
LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
- Check if language/locale settings are ok:
perl -e exit
. apt install golang
- this installs Go 1.10.1.apt install git psmisc jsonlint yamllint gcc
mkdir /data; mkdir /data/dev
- Configure Go:
- For example add to
~/.bash_profile
:
GOPATH=/data/dev; export GOPATH PATH=$PATH:$GOPATH/bin; export PATH
- Logout and login again.
- golint:
go get -u golang.org/x/lint/golint
- goimports:
go get -u golang.org/x/tools/cmd/goimports
- goconst:
go get -u github.com/jgautheron/goconst/cmd/goconst
- usedexports:
go get -u github.com/jgautheron/usedexports
- errcheck:
go get -u github.com/kisielk/errcheck
- cors:
go get -u github.com/rs/cors
- jsoniter:
go get -u github.com/json-iterator/go
- If you want to use ElasticSearch output: elastic:
go get -u github.com/olivere/elastic
.
- For example add to
- Go to
$GOPATH/src/github.com/cncf
and clone devstats and devstatscode there:git clone https://github.com/cncf/devstats.git
.git clone https://github.com/cncf/devstatscode.git
.- cd
devstats
. - Set reuse TCP connections:
./cron/sysctl_config.sh
- If you want to make changes and PRs, please clone
devstats
anddevstatscode
from GitHub UI, and clone your forked version instead, like this:git clone https://github.com/your_github_username/devstats.git
git clone https://github.com/your_github_username/devstatscode.git
- Go to
devstatscode
directory, so you are in~/dev/go/src/github.com/cncf/devstatscode
directory and compile binaries:make
- If compiled sucessfully then execute test coverage that doesn't need databases:
make test
- Tests should pass.
- Install binaries:
sudo mkdir /etc/gha2db
sudo chmod 777 /etc/gha2db
sudo make install
- Go to
devstats
directory, so you are in~/dev/go/src/github.com/cncf/devstats
and install scripts, metrics and other config/data files:make install
- Install Postgres database (link):
- Make sure you database uses
en_US.UTF-8
locale and collation. apt install postgresql
(you can use specific version, for examplepostgresql-9.6
)devstats
repo directory must be available for postgres user.chmod -R ugo+r /data/
.sudo -i -u postgres
,psql
and as rootsudo -u postgres psql
to test installation.- Postgres only allows local connections by default so it is secure, we don't need to disable external connections:
- Config file is:
/etc/postgresql/10/main/pg_hba.conf
, instructions to enable external connections (not recommended):http://www.thegeekstuff.com/2014/02/enable-remote-postgresql-connection/?utm_source=tuicool
- You need to set something like
host all all 1.2.3.4/32 md5
in this file and alsolisten_addresses = '*'
inpostgresql.conf
. - Set bigger maximum number of connections, at least 4x number of your CPU cores or more:
/etc/postgresql/10/main/postgresql.conf
. Default is 100.max_connections = 300
. - You can also set
shared_buffers = ...
to something like 25% of your RAM. This is optional. service postgresql restart
orsystemctl restart postgresql@10-main
../devel/set_psql_password.sh
to set postgres user password.chown -R postgres /var/log/postgresql
.
- Make sure you database uses
- Setup GitHub OAuth
- You need to have GitHub OAuth token, either put this token in
/etc/github/oauth
file or specify token value viaGHA2DB_GITHUB_OAUTH=deadbeef654...10a0
(here you token value) - If you want to use multiple tokens, create
/etc/github/oauths
file that contain list of comma separated OAuth keys or specify token values viaGHA2DB_GITHUB_OAUTH=key1,key2,...,keyN
- If you really don't want to use GitHub OAuth2 token, specify
GHA2DB_GITHUB_OAUTH=-
- this will force tokenless operation (via public API), it is a lot more rate limited than OAuth2 which gives 5000 API points/h - If you set
GHA2DB_GHAPISKIP=1
all GitHub API calls will be skipped. You can setGHA2DB_GHAPISKIP=1
then, because artificial events cleanup is not needed when GitHub API is not needed. If both those variables are set,ghapi2db
won't be called at all. - If your project(s) use icons (some of them has value other than
ICON="-"
, then you need to clone CNCF artwork repo into~/dev/cncf/artwork
:cd ~/dev/cncf/
,git clone https://github.com/cncf/artwork.git
. - You can have artwork elsewhere, then you must use
ARTWORK=/path/to/artwork/repo
. If all projects useICON="-"
artwork is not needed. - You also need ImageMagick's convert utility:
apt install imagemagick
. - You need to have
/var/www/html/img
directory available for deploy user.
- You need to have GitHub OAuth token, either put this token in
- Install Grafana.
- Go to:
https://grafana.com/grafana/download
. wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.x.x_amd64.deb
.sudo dpkg -i grafana_5.x.x_amd64.deb
- Go to:
- Install Apache & SSL (You need to have a working DNS name for this). This is optional.
apt-get install apache2
.- Create "web" directory:
mkdir /var/www/html/
(it will hold gha databases dumps and other static info on the main domain.) chmod -R ugo+rwx /var/www
.- Enable mod proxy and mod rewrite:
ln /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled/
ln /etc/apache2/mods-available/proxy.conf /etc/apache2/mods-enabled/
ln /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled/
ln /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/
ln /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled/
- You can enable SSL, to do so You need to follow SSL instruction in SSL (that requires domain name).
service apache2 restart
- Test via:
apt install links
,links 127.0.0.1
- If you don't use Apache/SSL proxy, you should add EXTERNAL=1 when deploying - this will expose grafana to outside world by binding to
0.0.0.0
instead of local only127.0.0.1
. Seedevel/create_grafana.sh
.
- Automatic deploy of first database (few examples)
- You should probably use last two command examples or tweak any of commands listed below.
- Top deploy one of CNCF projects see
projects.yaml
chose oneprojectname
and specify to deploy only this project:ONLY=projectname
. - Use
INIT=1
to specify that you want to initialize logs databasedevstats
, you need to provide passowrd for adminPG_PASS
,ro_user
(PG_PASS_RO
used by Grafana for read-only access) anddevstats_team
(PG_PASS_TEAM
- this is just another role to allow readonly access from the bastion ssh server). - You must use
INIT=1
when running for the first time, this creates shared logs database and postgres users. - This will generate all data, without fetching anything from backups:
INIT=1 ONLY=projectname PG_PASS=... PG_PASS_RO=... PG_PASS_TEAM=... ./devel/deploy_all.sh
- You shoud add
SETPASS=1
on the first run to set main postgres database user password. Thsi requires user interaction, so it shouldn't be used from automatic scripts. - If you do not specify
ONLY="project1 project2 ... projectN"
it will deploy all projects defined inprojects.yaml
. - Use
SKIPWWW=1
to skip Apache/SSL config (which requires DNS name for a server), the final result will be Grafana via HTTP on port 3xxx on accessible by server IP or name. - Example with skipping GitHub API and custom ARTWORK:
ARTWORK=/data/artwork SKIPWWW=1 GHA2DB_GITHUB_OAUTH=- GHA2DB_GHAPISKIP=1 IGET=1 GET=1 ONLY=someproj PG_PASS=... ./devel/deploy_all.sh
- Other possible env variables used for automatic deploy are in the comments section in a first lines of deploy files:
devel/deploy_all.sh devel/deploy_proj.sh devel/create_databases.sh devel/init_database.sh devel/create_grafana.sh devel/create_www.sh all/add_project.sh
. - Example command that worked on a clean Ubuntu 18 without domain name:
EXTERNAL=1 SKIPTEMP=1 ARTWORK=/data/artwork INIT=1 PG_PASS=p PG_PASS_RO=p PG_PASS_TEAM=p ONLY=cncf GET=1 SKIPWWW=1 GHA2DB_GITHUB_OAUTH=- GHA2DB_GHAPISKIP=1 GHA2DB_PROJECTS_OVERRIDE="+cncf" ./devel/deploy_all.sh
. - Assuming you have GitHub OAuth file and artwork is in the default localtion and there is no domain name:
EXTERNAL=1 SKIPTEMP=1 INIT=1 PG_PASS=p PG_PASS_RO=p PG_PASS_TEAM=p ONLY=cncf GET=1 SKIPWWW=1 GHA2DB_PROJECTS_OVERRIDE="+cncf" ./devel/deploy_all.sh
. - You can also take a look at
ADDING_NEW_PROJECT.md
file for more info about setting up new projects. - Assuming that you deployed cncf project (like in the last example), you should see response using
links 127.0.0.1:3255
. You should albo be able to access your Grafana from outside world via:http://ip:3255
- Now when postgres users are created, you test all stuff that require databases (on
cncf/devstats
repo):PG_PASS=p make
. - Tests should pass.
- To enable Continuous deployment using Travis, please follow instructions here.
- We need to setup cron job that will call sync every hour (10 minutes after 1:00, 2:00, ...)
- Setup
devstats
,webhook
(to handle CI/CD deployments) andcron_db_backup_all.sh
cron jobs, similar to this:
7 * * * * PATH=$PATH:/path/to/GOPATH/bin PG_PASS="..." devstats 2>> /tmp/gha2db_sync.err 1>> /tmp/gha2db_sync.log 30 3 * * * PATH=$PATH:/path/to/GOPATH/bin cron_db_backup_all.sh 2>> /tmp/gha2db_backup.err 1>> /tmp/gha2db_backup.log */5 * * * * PATH=$PATH:/path/to/GOPATH/bin GOPATH=/go/path GHA2DB_CMDDEBUG=1 GHA2DB_PROJECT_ROOT=/path/to/repo GHA2DB_DEPLOY_BRANCHES="production,master" PG_PASS=... GHA2DB_SKIP_FULL_DEPLOY=1 webhook 2>> /tmp/gha2db_webhook.err 1>> /tmp/gha2db_webhook.log
- First crontab entry is for automatic GHA sync.
- Second crontab entry is for automatic daily backup of postgres databases.
- Third crontab entry is for Continuous Deployment - this a Travis Web Hook listener server, it deploys project when specific conditions are met, details here.
- You need to change "..."
PG_PASS
to the real postgres password value and copy this line. - You need to change "/path/to/your/GOPATH/bin" to the value of "$GOPATH/bin", you cannot use $GOPATH in crontab directly.
- Run
crontab -e
and put this line at the end of file and save. - Cron job will update Postgres database at 0:08, 1:08, ... 23:08 every day.
- It outputs logs to
/tmp/gha2db_sync.log
and/tmp/gha2db_sync.err
and also to gha Postgresdevstats
database: into tablegha_logs
. - Check database values and logs about 25 minutes after full hours, like 14:25:
- Check max event created date:
select max(created_at) from gha_events
and logsselect * from gha_logs order by dt desc limit 20
.
- Setup