Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Threat detection #1783

Draft
wants to merge 67 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
b07151a
pushing malicious requests to kafka
ag060 Nov 6, 2024
1151e88
added flush kafka messages task
ag060 Nov 9, 2024
4550ea3
added window based aggregation support for malicious requests
ag060 Nov 12, 2024
731d25e
using separate mongo for storing suspect data
ag060 Nov 12, 2024
6de974d
fixed kafka host
ag060 Nov 12, 2024
6660653
fixed source ip generation logic
ag060 Nov 12, 2024
5d21d82
not syncing writes and read for RedisWriteBackCache at the same time
ag060 Nov 13, 2024
534ad26
using redis backed increment counter cache
ag060 Nov 13, 2024
44362c9
added group id to aggregate notifier
ag060 Nov 14, 2024
df11b6a
added notification cooldown support
ag060 Nov 14, 2024
8a344d3
added start and end bucket ids with each detected alerts
ag060 Nov 14, 2024
3862ca1
added ttl for redis entries
ag060 Nov 14, 2024
d1f0f9a
added new collections for sample malicious requests and detected alerts
ag060 Nov 15, 2024
f99611d
addded cleanup sample malicious records task
ag060 Nov 16, 2024
a203173
refactor code
ag060 Nov 18, 2024
fb9bc69
refactor code
ag060 Nov 18, 2024
d4219c2
refactor code
ag060 Nov 18, 2024
491cddb
add aggregation parse layer
ayushaga14 Nov 21, 2024
28bcd9a
grpc service for consuming malicious and smart events (#1748)
ag060 Nov 22, 2024
cc8c31f
apply aggregations
ayushaga14 Nov 22, 2024
bf81ce2
removed db service and using mongo client everywhere
ag060 Nov 22, 2024
740413e
moved clean up task to threat protection service
ag060 Nov 22, 2024
e19b55e
added authorization interceptor for grpc server
ag060 Nov 22, 2024
8562788
removed account_id field from consume event service requests (threat-…
ag060 Nov 22, 2024
3abaa08
added client side authentication support while saving malicious and s…
ag060 Nov 23, 2024
99a0675
added new api threat detection
ag060 Nov 25, 2024
68a4e24
saving malicious sample data to postgres db
ag060 Nov 25, 2024
f7cd995
abstracted out common kafka polling consumer task
ag060 Nov 26, 2024
ad5b13b
added tasks for sending alerts to protection backend
ag060 Nov 28, 2024
bc83b53
exposing single rpc method for recording alerts
ag060 Nov 28, 2024
e265de2
updated proto definition
ag060 Nov 29, 2024
8ae9eed
removed cleanup task from threat protection backend module
ag060 Nov 29, 2024
15f04fd
updated kafka topic names
ag060 Nov 29, 2024
2382b2d
running flyway migrations programatically
ag060 Nov 29, 2024
2438ea6
refactored code
ag060 Nov 29, 2024
d3704b7
added clean up for deleting all the malicious entries older than 7 days
ag060 Nov 29, 2024
89fec06
added hibernate orm (#1786)
ag060 Dec 10, 2024
3a90345
threat protection producer consumer db writes
ayushaga14 Dec 10, 2024
4d3e463
added proto generation to ci
ag060 Dec 10, 2024
8616f46
fix create mongo client params
ayushaga14 Dec 10, 2024
95f09f0
deleted generated proto files
ag060 Dec 10, 2024
861835c
added dashboard rpc service (#1808)
ag060 Dec 10, 2024
74bafd6
updated proto definition to include malicious event type (SINGLE or A…
ag060 Dec 10, 2024
80b04e1
storing api_collection_id in postgres
ag060 Dec 10, 2024
2fe1e23
fixed paths in ci for threat-detection module
ag060 Dec 11, 2024
a1e8b76
moved kafka config from threat detection to utils
ag060 Dec 11, 2024
d1595ca
moved consumer and producer on same machine for threat protection bac…
ag060 Dec 11, 2024
7bd52a6
refactored dashboard proto messages
ag060 Dec 11, 2024
9f3cb08
added check if buf is installed in proto-gen script
ag060 Dec 12, 2024
c5f627b
fixed MaliciousEventModel not populating data from mongo
ag060 Dec 16, 2024
5daab8d
refactored code
ag060 Dec 17, 2024
f4ced15
disabled debug logging for hibernate
ag060 Dec 17, 2024
8d45294
removed unused imports
ag060 Dec 17, 2024
36b9995
disabled debug logging for hibernate
ag060 Dec 17, 2024
9c1c757
committing kafka offsets manually now
ag060 Dec 17, 2024
8806577
renamed threat detection backend service
ag060 Dec 17, 2024
7d3b407
added health check server
ag060 Dec 17, 2024
b506fe4
building threat detection backend image in staging workflow
ag060 Dec 17, 2024
cfe9b1b
refactored generated proto files
ag060 Dec 18, 2024
3ae54be
added health check for rpc and enabled reflection
ag060 Dec 18, 2024
e30f66c
added health service for grpc
ag060 Dec 18, 2024
43c3912
using vertx http server as threat detection backend (#1842)
ag060 Dec 19, 2024
9264881
reading akto threat detection backend url from env
ag060 Dec 20, 2024
b46e067
added try catch
ag060 Dec 21, 2024
b5e9675
fixed sample not being to backend
ag060 Dec 21, 2024
e273f0b
clearing cache as soon the alert is raised
ag060 Dec 23, 2024
f640fdd
sending sample malicious events to backend only once
ag060 Dec 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .github/workflows/prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
workflow_dispatch:
inputs:
release_version:
required: true
required: true

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
Expand All @@ -17,7 +17,7 @@ jobs:

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
Expand All @@ -27,6 +27,11 @@ jobs:
- uses: actions/setup-node@v2
with:
node-version: '17'
- uses: bufbuild/buf-action@v1
with:
setup_only: true
- name: Generate Proto files
run: make proto-gen
- name: Download Akto templates zip and PII files
working-directory: ./apps/dashboard/src/main/resources
run: |
Expand Down Expand Up @@ -82,8 +87,8 @@ jobs:
cd ../internal
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-internal:$IMAGE_TAG_1 -t $ECR_REGISTRY/akto-internal:$IMAGE_TAG_2 . --push
echo "::set-output name=image::$ECR_REGISTRY/akto-internal:$IMAGE_TAG"
cd ../api-threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-api-protection:$IMAGE_TAG_1 -t $ECR_REGISTRY/akto-api-protection:$IMAGE_TAG_2 -t $ECR_REGISTRY/akto-api-protection:$IMAGE_TAG_3 . --push
cd ../threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-protection:$IMAGE_TAG_1 -t $ECR_REGISTRY/akto-threat-detection:$IMAGE_TAG_2 -t $ECR_REGISTRY/akto-threat-detection:$IMAGE_TAG_3 . --push

- name: Configure AWS Credentials for ECR
uses: aws-actions/configure-aws-credentials@v1
Expand All @@ -108,7 +113,7 @@ jobs:
run: |
docker buildx create --use
# Build a docker container and push it to DockerHub
cd apps/api-threat-detection
cd apps/threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/$REGISTRY_ALIAS/akto-api-protection:$IMAGE_TAG -t $ECR_REGISTRY/$REGISTRY_ALIAS/akto-api-protection:$IMAGE_TAG2 . --push
echo "::set-output name=image::$ECR_REGISTRY/akto-api-protection:$IMAGE_TAG"

Expand Down
15 changes: 11 additions & 4 deletions .github/workflows/staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Staging
# Controls when the workflow will run
on:
push:
branches: [master]
branches: [ master ]
pull_request:
workflow_dispatch:

Expand All @@ -25,7 +25,12 @@ jobs:
architecture: x64
- uses: actions/setup-node@v2
with:
node-version: "17"
node-version: '17'
- uses: bufbuild/buf-action@v1
with:
setup_only: true
- name: Generate Proto files
run: make proto-gen
- name: Convert github branch name to be compatible with docker tag name convention and generate tag name
id: docker_tag
run: echo "IMAGE_TAG=a-$(echo ${{ github.ref_name }} | sed 's/[^a-zA-Z0-9]/-/g')" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -79,8 +84,10 @@ jobs:
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-billing:$IMAGE_TAG . --push
cd ../internal
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-internal:$IMAGE_TAG . --push
cd ../api-threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-api-protection:$IMAGE_TAG . --push
cd ../threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-detection:$IMAGE_TAG . --push
cd ../threat-detection-backend
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-detection-backend:$IMAGE_TAG . --push
cd ../source-code-analyser
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/source-code-analyser:$IMAGE_TAG . --push

Expand Down
8 changes: 8 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
proto-gen:
sh ./scripts/proto-gen.sh

build: proto-gen
mvn install -DskipTests

build-clean: proto-gen
mvn clean install -DskipTests
Loading
Loading