Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Qacc-Backend Soft Release #140

Merged
merged 23 commits into from
Nov 6, 2024
Merged

Qacc-Backend Soft Release #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
5861170
add mongo to docker-compose-local.yml
ae2079 Nov 3, 2024
f96da0f
add a service to save and restore reports with mongo db
ae2079 Nov 3, 2024
5461e0e
save and restore reports in batch minting script
ae2079 Nov 3, 2024
42564e2
Adding high availability Setup, Proxy, health-checks & auto-healing
mhmdksh Nov 5, 2024
cc8562d
Adding Zero Downtime deployments
mhmdksh Nov 5, 2024
a4efe7e
Merge pull request #136 from GeneralMagicio/add/HA_health-check_auto-…
mhmdksh Nov 5, 2024
24e5c71
Excluding .env file for main repo
mhmdksh Nov 5, 2024
93e1f17
Updating the Zero-Deployment Steps in Pipelines
mhmdksh Nov 5, 2024
0ff615c
Updating the Zero-Deployment Steps in Pipelines adding consistency
mhmdksh Nov 5, 2024
e38b01a
Updating the Zero-Deployment Steps in Pipelines, fixing restart all
mhmdksh Nov 5, 2024
fde1ff6
Updating the Zero-Deployment Steps in Pipelines, fixing restart all
mhmdksh Nov 5, 2024
a98964c
fix issue in redundant transactions and bug in skipping other donatio…
ae2079 Nov 5, 2024
0225089
Merge pull request #137 from GeneralMagicio/FixMinorIssuesInBatchMinting
ae2079 Nov 5, 2024
abf6657
remove containers for job2 and job3, rename job1 to job
geleeroyale Nov 5, 2024
9a71560
add migration to insert EA rounds
ae2079 Nov 6, 2024
a23bd83
add migration to insert QAcc round
ae2079 Nov 6, 2024
db24d99
Merge pull request #138 from GeneralMagicio/high-availability
aminlatifi Nov 6, 2024
1330948
Merge pull request #130 from GeneralMagicio/saveReportsInMongo
aminlatifi Nov 6, 2024
e98251d
Merge pull request #139 from GeneralMagicio/addEARoundsMigration
aminlatifi Nov 6, 2024
22c329d
Updated health check ports
aminlatifi Nov 6, 2024
635fe21
Revert "Updated health check ports"
aminlatifi Nov 6, 2024
20f1dae
set different command for running job and graphql
aminlatifi Nov 6, 2024
902e3fe
Hotfix to Deploy Only Pipelines
mhmdksh Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
MY_URL="https://mydomain.com"
IP_WHITELIST="0.0.0.0/0"
MY_URL="https://mydomain.com" # Backend URL
IP_WHITELIST="0.0.0.0/0" # Backend Whitelisted IP Addresses
DOCKER_IMAGE="ghcr.io/generalmagicio/qacc-be:staging" # For Production: ghcr.io/generalmagicio/qacc-be:main
107 changes: 104 additions & 3 deletions .github/workflows/main-deploy-only.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,109 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout main
git pull
docker compose -f docker-compose-production.yml pull
docker compose -f docker-compose-production.yml up --force-recreate -d
git pull origin main
docker image prune -a --force
docker compose pull

rollout-deploy-1:
needs: deploy
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Update each backend service one by one
## First Deployment
docker compose down qacc-be-graph-ql1
docker compose down qacc-be-job
docker compose up --force-recreate -d qacc-be-graph-ql1
docker compose up --force-recreate -d qacc-be-job

# Wait for qacc-be-graph-ql1 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql1 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql1)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql1 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql1 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql1 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql1)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql1 is not healthy, stopping deployment"
exit 1
fi
# Check if qacc-be-job is running
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job)" != "\"running\"" ]; then
echo "qacc-be-job is not running, stopping deployment"
exit 1
fi
echo "First deployment phase completed successfully"

rollout-deploy-2:
needs: rollout-deploy-1
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Second Deployment
docker compose down qacc-be-graph-ql2
docker compose up --force-recreate -d qacc-be-graph-ql2

# Wait for qacc-be-graph-ql2 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql2 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql2)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql2 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql2 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql2 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql2)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql2 is not healthy, stopping deployment"
exit 1
fi
echo "Second deployment phase completed successfully"

rollout-deploy-3:
needs: rollout-deploy-2
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Third Deployment
docker compose down qacc-be-graph-ql3
docker compose up --force-recreate -d qacc-be-graph-ql3

# Wait for qacc-be-graph-ql3 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql3 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql3)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql3 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql3 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql3 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql3)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql3 is not healthy, stopping deployment"
exit 1
fi
echo "First deployment phase completed successfully"
107 changes: 104 additions & 3 deletions .github/workflows/main-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,109 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout main
git pull
docker compose -f docker-compose-production.yml pull
docker compose -f docker-compose-production.yml up -d
git pull origin main
docker image prune -a --force
docker compose pull

rollout-deploy-1:
needs: deploy
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Update each backend service one by one
## First Deployment
docker compose down qacc-be-graph-ql1
docker compose down qacc-be-job
docker compose up --force-recreate -d qacc-be-graph-ql1
docker compose up --force-recreate -d qacc-be-job

# Wait for qacc-be-graph-ql1 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql1 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql1)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql1 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql1 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql1 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql1)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql1 is not healthy, stopping deployment"
exit 1
fi
# Check if qacc-be-job is running
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job)" != "\"running\"" ]; then
echo "qacc-be-job is not running, stopping deployment"
exit 1
fi
echo "First deployment phase completed successfully"

rollout-deploy-2:
needs: rollout-deploy-1
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Second Deployment
docker compose down qacc-be-graph-ql2
docker compose up --force-recreate -d qacc-be-graph-ql2

# Wait for qacc-be-graph-ql2 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql2 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql2)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql2 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql2 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql2 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql2)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql2 is not healthy, stopping deployment"
exit 1
fi
echo "Second deployment phase completed successfully"

rollout-deploy-3:
needs: rollout-deploy-2
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Third Deployment
docker compose down qacc-be-graph-ql3
docker compose up --force-recreate -d qacc-be-graph-ql3

# Wait for qacc-be-graph-ql3 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql3 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql3)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql3 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql3 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql3 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql3)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql3 is not healthy, stopping deployment"
exit 1
fi
echo "First deployment phase completed successfully"
107 changes: 104 additions & 3 deletions .github/workflows/main-publish-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,109 @@ jobs:
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
git reset --hard HEAD~1
git checkout main
git pull
docker compose -f docker-compose-production.yml pull
docker compose -f docker-compose-production.yml up --force-recreate -d
git pull origin main
docker image prune -a --force
docker compose pull

rollout-deploy-1:
needs: deploy
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Update each backend service one by one
## First Deployment
docker compose down qacc-be-graph-ql1
docker compose down qacc-be-job
docker compose up --force-recreate -d qacc-be-graph-ql1
docker compose up --force-recreate -d qacc-be-job

# Wait for qacc-be-graph-ql1 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql1 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql1)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql1 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql1 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql1 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql1)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql1 is not healthy, stopping deployment"
exit 1
fi
# Check if qacc-be-job is running
if [ "$(docker inspect --format='{{json .State.Status}}' qacc-be-job)" != "\"running\"" ]; then
echo "qacc-be-job is not running, stopping deployment"
exit 1
fi
echo "First deployment phase completed successfully"

rollout-deploy-2:
needs: rollout-deploy-1
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Second Deployment
docker compose down qacc-be-graph-ql2
docker compose up --force-recreate -d qacc-be-graph-ql2

# Wait for qacc-be-graph-ql2 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql2 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql2)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql2 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql2 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql2 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql2)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql2 is not healthy, stopping deployment"
exit 1
fi
echo "Second deployment phase completed successfully"

rollout-deploy-3:
needs: rollout-deploy-2
runs-on: ubuntu-latest
steps:
- name: SSH and Redeploy
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.PRODUCTION_HOST }}
username: ${{ secrets.PRODUCTION_USERNAME }}
key: ${{ secrets.PRODUCTION_PRIVATE_KEY }}
port: ${{ secrets.SSH_PORT }}
script: |
cd QAcc-BE
## Third Deployment
docker compose down qacc-be-graph-ql3
docker compose up --force-recreate -d qacc-be-graph-ql3

# Wait for qacc-be-graph-ql3 to be healthy (timeout after 5 minutes)
echo "Waiting for qacc-be-graph-ql3 to become healthy..."
timeout 300 bash -c 'until [ "$(docker inspect --format="{{json .State.Health.Status}}" qacc-be-graph-ql3)" == "\"healthy\"" ]; do echo "Waiting for qacc-be-graph-ql3 to be healthy..."; sleep 5; done'
if [ $? -eq 124 ]; then
echo "Timeout waiting for qacc-be-graph-ql3 to become healthy"
exit 1
fi
# Check if qacc-be-graph-ql3 is healthy
if [ "$(docker inspect --format='{{json .State.Health.Status}}' qacc-be-graph-ql3)" != "\"healthy\"" ]; then
echo "qacc-be-graph-ql3 is not healthy, stopping deployment"
exit 1
fi
echo "First deployment phase completed successfully"
Loading
Loading