Luix Data Analytic Engine with Apache Flink

Overview

Data analytic engine is to consume a stream of inputs and evaluate them against a set of rules. Any input that matches the rules is flagged as a output. The engine is implemented using Apache Flink and is deployed as a standalone application.

Snapshots

Modules

This project consists of three main components:

• luix-dae-commons: Common library for the DAE Flink Job

• luix-dae-flink-job: Data analytic engine application with Apache Flink

• luix-fraud-detection-simulator: Fraud detection web simulator demo, which is used to generate transactions and configure rules

Requirements

Before you can build this project, you must install JDK on your machine. You can install JDK by SDK Man. e.g.

sdk install java 11.0.20-tem

Note	Java compatability: JDK 11 only

Dependencies

• Apache Flink 1.15.4

• Spring Boot 2.7.1

Build

Run the following command to build the project:

./mvnw clean verify

Run

This project contains multiple sub-modules.

Run luix-dae-flink-job module with main() method of: cn.luixtech.dae.flinkjob.DaeFlinkJobApplication

Access Flink Web UI (Dashboard) with URL: http://localhost:8081

Run luix-fraud-detection-simulator module with main() method of: cn.luixtech.frauddetection.simulator.SimulatorApplication

Access swagger with URL: http://localhost:5656

Install Docker

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

Using Docker to simplify development (optional)

To start zookeeper and Kafka in a docker container, run:

docker-compose -f docker/all.yml up -d

To stop it and remove the container, run:

docker-compose -f docker/all.yml down

VM Options

Add below Key/Value pair to env variable of rancher.

key: JAVA_OPTS
value: -Dspring.profiles.active=XXX

Troubleshooting

Error:

./mvnw: Permission denied

Solution:

chmod +x mvnw
git update-index --chmod=+x mvnw

Contacts

• Email: louis@luixtech.com

References

Fraud detection demo code repository

基于 Flink 的动态欺诈检测系统（上）

Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System

Flink Docs