Skip to content

OBP API Install Guide Bare Metal

Jump to bottom Edit New page
Mulugeta Ayalew edited this page Aug 4, 2017 · 42 revisions

Assumption

  • Instructions assume Debian Jessie server (but should work on other GNU/Linux distributions or UNIX as well)

1. Run OBP API in mapped mode using built in H2 database

  • Install git, maven, java 8, Jetty 9

Option 1 (Build packaged OBP war file from source)

  • Clone the OBP-API source code from the repositoty

    git clone https://github.com/OpenBankProject/OBP-API.git

  • Change the working directory to OBP-API and copy the configuration file

    cd OBP-API
cp src/main/resources/props/sample.props.template src/main/resources/props/default.props

  • Edit the configuration file src/main/resources/props/default.props and uncomment the following lines

    connector=mapped
db.driver=org.h2.Driver
db.url=jdbc:h2:./lift_proto.db;DB_CLOSE_ON_EXIT=FALSE
hostname=http://127.0.0.1:8080

  • Comment out the following lines in the configuration file

    #db.driver=org.postgresql.Driver
#db.url=jdbc:postgresql://localhost:5432/dbname?user=dbusername&password=thepassword

  • Build the WAR package for the OBP-API

    mvn package

Note that the packaging take a few minutes to finish.

  • When the packaging is completed, you will find the OBP-API-1.0.war file under the directory OBP-API/target/

  • To run the application on Jetty, copy the OBP-API-1.0.war file to the /var/lib/jetty9/webapps directory and rename it as ROOT.war

  • You can now point your browser to http://localhost:8080

Option 2 (Update configuration file to already packaged OBP war file)

  • If you have to change settings on the configuration file after the application has been packaged, you can update the configuration in the packaged WAR file like so

    • First, make a directory structure as follows (this exact directory structure is required by the jar command, as only files already in the archive having the same pathname as a file being added will be overwritten)

      mkdir -p WEB-INF/classes/props/

    • Copy the edited props file to the props directory

      cp default.props WEB-INF/classes/props/

    • Update the default.props file inside the package using the jar command

      jar uf /path/to/package/OBP-API-1.0.war WEB-INF/classes/props/default.props

  • To run the application on Jetty, copy the OBP-API-1.0.war file to the /var/lib/jetty9/webapps directory and rename it as ROOT.war

  • You can now point your browser to http://localhost:8080

Run OBP API in mapped mode using built in H2 database (using Docker containers)

  • Please note that this is an alternative setup using Docker containers

Assumption

  • This assumes that Docker Community Edition or Docker Enterprise Edition is installed on the server

Run OBP API in Docker Container

  • Using the steps from above, make the packaged OBP API software ready

  • Run the OBP API package inside the Jetty 9 Docker container (whose official image can be found on Docker Hub https://hub.docker.com/_/jetty/). You may want to change the container name or port forwarding as per your requirements.

    docker run -d \
--name obp-api \
-v /path/to/package/OBP-API-1.0.war:/var/lib/jetty/webapps/ROOT.war \ 
-p 8080:8080 \
jetty

  • You can now point your browser to http://localhost:8080

2. Run OBP API in mapped mode using local database

Assumptions

  • Instructions in this section assume that you have a PostgreSQL (or Oracle) database setup on the local server
  • Following instructions about database creation are for PostgreSQL only. Please use the equivalent commands if you have an Oracle DB

Run OBP API with local database

  • Create a database on PostgreSQL

    root@sandbox ~ # sudo -i -u postgres
postgres@sandbox:~$ psql
postgres=# CREATE DATABASE <dbname>;
postgres=# CREATE USER <username> WITH PASSWORD 'xxx';
postgres=# GRANT ALL PRIVILEGES ON DATABASE <dbname> to <username>;

  • Edit the API configuration file default.props so that the API uses the local database. Uncomment the following lines in the file

    db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://localhost:5432/dbname?user=dbusername&password=thepassword

  • Make sure to replace dbname, dbusername, and thepassword as per your database's configuration

  • Comment out the following lines

    #db.driver=org.h2.Driver
#db.url=jdbc:h2:./lift_proto.db;DB_CLOSE_ON_EXIT=FALSE

  • Copy the edited props file to the props directory

    cp default.props WEB-INF/classes/props/

  • Update the default.props file inside the package using the jar command

    jar uf /path/to/package/OBP-API-1.0.war WEB-INF/classes/props/default.props

  • To run the application on Jetty, copy the OBP-API-1.0.war file to the /var/lib/jetty9/webapps directory and rename it as ROOT.war

  • You can now point your browser to http://localhost:8080

Run OBP API in mapped mode using local database (using Docker Containers)

  • Please note that this is an alternative setup using Docker containers

Assumption

  • This assumes that Docker Community Edition or Docker Enterprise Edition is installed on the server

Run PostgresSQL and OBP API in Docker Containers

  • Edit the API configuration file default.props so that the API uses the PostgreSQL container. Update the following line in the file

    db.url=jdbc:postgresql://obp-postgres:5432/obp-api-db?user=obp-api-db-user&password=thepassword

  • Replace obp-postgres, obp-api-db, obp-api-db-user, and thepassword as required (but make sure that they are the same as the respective values of the environment variables for the PostgreSQL container below)

  • Update the default.props file inside the package using the jar command

    jar uf /path/to/package/OBP-API-1.0.war WEB-INF/classes/props/default.props

  • Run PostgreSQL using the postgres:9.5 official Docker image from Docker Hub. Use a local directory and map it to the /var/lib/postgresql/data of the container to persist the data.

    docker run -d \
--name obp-postgres \
-v /path/to/data/directory/dbdata:/var/lib/postgresql/data \
-e POSTGRES_USER=obp-api-db-user \
-e POSTGRES_PASSWORD=thepassword \
-e POSTGRES_DB=obp-api-db \
postgres:9.5

  • Run the OBP API package inside the Jetty 9 Docker container (whose official image can be found on Docker Hub https://hub.docker.com/_/jetty/). You may want to change the container name or port forwarding as per your requirements.

    docker run -d \
--name obp-api \
--link obp-postgres:obp-postgres \
-v /path/to/obp/OBP-API/target/OBP-API-1.0.war:/var/lib/jetty/webapps/ROOT.war \
-p 8080:8080 \
jetty9

  • You can now point your browser to http://localhost:8080

3. Run OBP API in mapped mode and OBP storage connected to PostgreSQL (Oracle) database

Intro

  • Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM. See http://akka.io

  • To use Akka, you want to have two different machines:

    • One considered remote which stores the data
    • One considered local which is the public facing side of the API, where your users will connect

  • Both run current versions of the Open Bank Project API, but configured and run differently.

Remote Models

  • Not all models have been ported yet to be retrieved via Akka. See modelsRemotedata in src/main/scala/bootstrap/liftweb/Boot.scala to get a current list.

Remote Side

  • Configure src/main/resources/default.props:
# Remote end gets data 'locally'
remotedata.enable=false
# Your remote's external IP address
remotedata.hostname=xx.xx.xx.xx # Replace with external IP address of the host  
# Arbitrary port of your choosing
remotedata.port=5448

# Optionally configure postgres, otherwise file-based H2 will be used 
remotedata.db.driver=org.postgresql.Driver
remotedata.db.url=jdbc:postgresql://localhost:5432/dbname?user=user&password=password

Run

Manual

#!/bin/sh

cd ${HOME}/OBP-API/ && /usr/bin/nohup /usr/bin/mvn compile exec:java -Dexec.mainClass="code.remotedata.RemotedataActors" -Dexec.args="standalone" > ${HOME}/akka_remote_api.log &

systemd

  • You can run with above using this unit: obp-remotedata.service 
[Unit]
Description=Open Bank Project Remotedata Standalone

[Service]
Type=simple
User=deploy
Group=deploy
SyslogIdentifier=obp-remotedata
Restart=on-failure
WorkingDirectory=/home/deploy/OBP-API
ExecStart=/usr/bin/mvn compile exec:java -Dexec.mainClass=code.remotedata.RemotedataActors -Dexec.args=standalone

[Install]
WantedBy=multi-user.target

Local OBP API Side

  • Configure src/main/resources/default.props:
# Local end gets data remotely
remotedata.enable=true
# Your remote's public IP address
remotedata.hostname=xx.xx.xx.xx # Replace with external IP address of the remote host
# Arbitrary port of your choosing, has to match remote above
remotedata.port=5448
# Timeout for connection attempts
remotedata.timeout = 1

Run

#!/bin/sh

cd ${HOME}/OBP-API/ && /usr/bin/nohup /usr/bin/mvn jetty:run -Djetty.port=8080 -DskipTests  > ${HOME}/akka_local_api.log &

  • Optionally, like the steps in the previous sections, copy the edited props file to the props directory

    cp default.props WEB-INF/classes/props/

  • Update the default.props file inside the package using the jar command

    jar uf /path/to/package/OBP-API-1.0.war WEB-INF/classes/props/default.props

  • To run the application on Jetty, copy the OBP-API-1.0.war file to the /var/lib/jetty9/webapps directory and rename it as ROOT.war

4. Run OBP API in Production mode connected to Kafka and Adapter

Assumptions

  • In this section we assume Debian Jessie server for hosting the OBP API, PostgreSQL host, Zookeeper cluster and Kafka cluster in place

Zookeeper/Kafka Cluster Setup

  • The Zookeeper/Kafka cluster contains at least 3 nodes each (the number of nodes needs to be odd).

  • Kafka binary can be downloaded from https://kafka.apache.org/downloads. Version kafka_2.12-0.10.2.1.tgz can be used. Extract the compressed file and copy the extracted directory to all nodes.

Zookeeper cluster configuration

  • In production, you should run ZooKeeper in replicated mode. A replicated group of servers in the same application is called a quorum, and in replicated mode, all servers in the quorum have copies of the same configuration file.

  • The following configurations need to be done on each of the Zookeeper nodes.

  • Create directory /var/zookeeper

  • Inside the Kafka directory, edit the file conf/zookeeper.properties and include these lines

dataDir=/var/zookeeper
server.1=host1:2888:3888 # The name of the servers shall be changed as appropriate
server.2=host2:2888:3888
server.3=host3:2888:3888
tickTime=2000
clientPort=2181
initLimit=5
syncLimit=2
  • Create a myid file under dataDir which is /var/zookeeper in this example
echo “1” > /var/zookeeper/myid       #Insert unique id’s on each of the machines
  • Start the zookeeper daemons on each of the 3 machines
bin/zookeeper-server-start.sh config/zookeeper.properties &

  • You may consider to automate starting/stopping the Zookeeper service using systemd

  • Sample systemd configuration for zookeeper /etc/systemd/system/zookeeper.service

[Unit]
Description=Zookeeper
After=network.target

[Service]
User=kafka
Group=kafka
Type=simple
SyslogIdentifier=zookeeper
Restart=on-failure
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties

[Install]
WantedBy=multi-user.target

  • Reload systemd systemctl daemon-reload

  • To start zookeeper with systemd systemctl start kafka-zookeeper.service

Kafka cluster configuration

The following configurations need to be done on each of the three nodes.

Inside the Kafka directory, edit the file conf/server.properties and include these lines

broker.id=1 # The broker.id should be unique for each host
listeners=PLAINTEXT://host_ip:9092
advertised.listeners=PLAINTEXT://host_ip:9092
num.partitions=10
zookeeper.connect=host1_ip:2181,host2_ip:2181,host3_ip:2181 # Replace the hostnames as applicable
  • Start the kafka broker daemons on all the machines
bin/kafka-server-start.sh config/server.properties &

  • You may consider to automate starting/stopping the Kafka service using systemd

  • Example systemd configuration for kafka /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
PIDFile=/var/run/kafka.pid
User=kafka
Group=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
SyslogIdentifier=kafka

[Install]
WantedBy=multi-user.target

  • Reload systemd systemctl daemon-reload

  • To start kakfka with systemd systemctl start kafka.service

Create Kafka Topics

  • Create two topics named Request and Response on the Kafka cluster (this assumes a 3-host cluster, please adjust as necessary)

    bin/kafka-topics.sh --create --zookeeper host1:2181,host2:2181,host3:2181 \ 
--replication-factor 3 --partitions 10 --topic Request

bin/kafka-topics.sh --create --zookeeper host1:2181,host2:2181,host3:2181 \
--replication-factor 3 --partitions 10 --topic Response

Update configurations

  • Rename the default.props file as production.default.props

  • Change the connector and include settings for Kafka and Zookeeper clusters

    connector=kafka_vJun2017

kafka.request_topic=Request
kafka.response_topic=Response
kafka.bootstrap_hosts=kafka_host_1_IP:9092,kafka_host_2_IP:9092,kafka_host_3_IP:9092 # Replace the IP addresses of the Kafka nodes

Copy configuration file to OBP API package and run the API

  • Copy the edited props file to the props directory

    cp production.default.props WEB-INF/classes/props/

  • Update the production.default.props file inside the package using the jar command

    jar uf /path/to/package/OBP-API-1.0.war WEB-INF/classes/props/production.default.props

  • To run the application on Jetty, copy the OBP-API-1.0.war file to the /var/lib/jetty9/webapps directory and rename it as ROOT.war

  • You can now point your browser to http://localhost:8080

Running the Adapter

  • Get the OBP-Adapter_xxx.jar from OBP-API team.

  • Get the application.conf and update_jar.sh from OBP-API team.

  • Modify the Kafka bootstrap.servers in application.conf file (please replace the IP addresses with those of the nodes in the Kafka cluster)

bootstrap {
  servers = "10.1.2.171:9092,10.1.2.95:9092,10.1.2.180:9092"
}

  • Save the file as application.conf.new

  • Run the update_jar.sh script to update this cofniguration file in the jar

./update_jar.sh
  • Run the project's jar file with java 
java -jar OBP-Adapter_xxx.jar
Add a custom footer
Add a custom sidebar
Clone this wiki locally