cenote (Katavothra) is a Big Data Management System (BDMS) for event processing and analytics. Our goal was to build an open source equivalent of keen.io analytics and learn about scalable systems in the process. The technology stack is based on:

• Kafka
• Storm
• CockroachDB
• MERN stack

and inludes code from programming languages JavaScript, Java and Python.

This is the logistics repository to hold issues, documentation, installation instructions and any code that is across cenote systems.

Repositories

Since cenote is a distributed system it spans across 5 repositories:

• cenote: This one, used for gathering all the issues related to cenote, hosting IaC files and containing installation instructions.
• cenote-api: API server & web management client, used also for data reading.
• cenote-cockroach: Read/Write related code for CockroachDB.
• cenote-write: Apache Storm topology used for data writing.
• cenote-read: Apache Storm Topology used for querying data. (Obsolete, due to latencies in DRPC server, switched the reading to cenote-api).

Installation

Our demo server can be found here along with the online API docs.

Prerequisites

Before you can install cenote, you need to have already set up:

• A zookeeper instance/cluster which will be used to share configuration files between Apache's Kafka & Storm.
• A kafka instance/cluster with:
  • A topic for incoming messages (e.g. cenoteIncoming).
• A storm instance/cluster.
• A cockroachDB instance/cluster with:
  • A DB for cenote to store its data (e.g. cenote)
  • A new user (e.g. cockroach) with all rights on the DB above.
• Node.js & MongoDB for the API server to run on.

How to install on kubernetes

Every service should be deployed under the namespace cenote. redis, cockroach, mongodb can be installed using the tiller-helm. After that, the following services should be installed: zookeeper, storm, kafka, node. Those services can be installed using its yaml files.

Login at your kubernetes cluster and execute the following commands:

kubectl create -f https://github.com/AuthEceSoftEng/cenote/tree/master/iac/zookeeperAlone.yaml
kubectl create -f https://github.com/AuthEceSoftEng/cenote/tree/master/iac/kafkaAll.yaml
kubectl create -f https://github.com/AuthEceSoftEng/cenote/tree/master/iac/nodejs.yaml
kubectl create -f https://github.com/AuthEceSoftEng/cenote/tree/master/iac/storm.yaml
kubectl create -f https://github.com/AuthEceSoftEng/cenote/tree/master/iac/storm2.yaml
kubectl create -f https://github.com/AuthEceSoftEng/cenote/tree/master/iac/storm3.yaml

Attention! Storm instances require the following IPs:

• Storm 1: 10.43.192.184
• Storm 2: 10.43.215.24
• Storm 3: 10.43.122.136

If you want to use different IPs, you have to rebuild the docker image.

Cenote components

• The Apache Storm Topology used by cenote to write events to the database can be found here. You just need to clone the source code, configure a .env file and compile it to a jar that you will then submit to the Storm cluster. Instructions on how to do this can be found in the repo's README file.

  Note: cenote-write uses cenote-cockroach internally so check cenote-cockroach@README for its required environment variables.

• The API server & UI used by cenote can be found here. You just need to clone the source code, configure a .env file and start it. Instructions on how to do this can be found in the repo's README file.

Tests

To run the tests:

1. Configure a .env file.
2. Run npm i
3. Run npm test