Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor backend for monorepo move #1527

Merged
merged 1 commit into from
Feb 6, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
27 changes: 0 additions & 27 deletions .dockerignore

This file was deleted.

4 changes: 3 additions & 1 deletion .github/actions/poetry/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,17 @@ runs:
shell: bash

- name: "Configure poetry"
working-directory: backend
run: poetry config virtualenvs.in-project true
shell: bash

- name: "Load cached poetry environment"
uses: actions/cache@v4
with:
path: .venv
path: backend/.venv
key: cache-venv-python${{ inputs.poetry-version }}-${{ hashFiles('**/poetry.lock') }}-v1

- name: "Install dependencies"
working-directory: backend
run: poetry install
shell: bash
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ name: "Build"

on:
push:
paths:
- backend/**
branches:
- main
tags:
Expand Down Expand Up @@ -70,8 +72,8 @@ jobs:
- name: "Build backend container"
uses: docker/build-push-action@v6
with:
context: .
file: docker/Dockerfile
context: backend/
file: backend/docker/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ name: "Tests"

on:
push:
paths:
- backend/**
branches:
- main
pull_request:
Expand Down Expand Up @@ -66,12 +68,14 @@ jobs:
python-version: ${{ steps.setup-python.outputs.python-version }}

- name: "Test for migration issues"
working-directory: backend
run: poetry run python manage.py makemigrations --check --no-color --no-input --dry-run
env:
DJANGO_SETTINGS_ENVIRONMENT: CI
DATABASE_URL: postgis://tvp:tvp@localhost:5432/tvp

- name: "Test for missing translation"
working-directory: backend
run: poetry run python -m config.hooks.translations_done
env:
DJANGO_SETTINGS_ENVIRONMENT: CI
Expand Down Expand Up @@ -137,6 +141,7 @@ jobs:
name: test_durations

- name: "Run pytest with coverage"
working-directory: backend
run: |
poetry run coverage run \
--branch \
Expand All @@ -161,7 +166,7 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: cov-${{ steps.artifact.outputs.artifact }}-${{ matrix.group }}
path: .coverage.${{ matrix.group }}
path: ./backend/.coverage.${{ matrix.group }}
if-no-files-found: error
retention-days: 1
include-hidden-files: true
Expand Down Expand Up @@ -190,6 +195,7 @@ jobs:
- name: "Download coverage artefacts"
uses: actions/download-artifact@v4
with:
path: ./backend
pattern: cov-${{ steps.artifact.outputs.artifact }}-*
merge-multiple: true

Expand All @@ -200,15 +206,20 @@ jobs:
python-version: ${{ env.PYTHON_VERSION }}

- name: "Install coverage"
working-directory: backend
run: pip install coverage

- name: "Combine coverage reports and generate XML report"
working-directory: backend
run: |
coverage combine
coverage xml
sed -i 's@<source>@<source>backend/@g' coverage.xml

- name: "SonarCloud Scan"
uses: SonarSource/sonarqube-scan-action@v4
with:
projectBaseDir: ./backend
# Skip analysis for all bots
if: ${{ ! contains(fromJSON(vars.BOTS), github.event.pull_request.user.login) }}
env:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/update_test_durations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ jobs:
python-version: ${{ steps.setup-python.outputs.python-version }}

- name: "Run pytest with coverage"
working-directory: backend
run: poetry run pytest --disable-warnings --store-durations
env:
DJANGO_SETTINGS_ENVIRONMENT: AutomatedTests
Expand All @@ -66,7 +67,7 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: test_durations
path: .test_durations
path: ./backend/.test_durations
if-no-files-found: error
retention-days: 90
include-hidden-files: true
Expand Down
88 changes: 57 additions & 31 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,36 +1,62 @@
# environments
**/node_modules
**/.pnpm
**/.pnp.js

# yarn
**/.yarn
**/.yarnrc.yml

# debug
**/npm-debug.log*
**/yarn-debug.log*
**/yarn-error.log*

# local certs
**/certificates

# caches
**/.turbo
**/.next
**/__pycache__
**/.cache
**/.pytest_cache
**/.ruff_cache

# builds
**/dist
**/build
**/out

# local config
**/.env.local
**/.local
**/.idea
**/.vscode
backend/local_settings.py
backend/.env
backend/pytest.ini

# misc
**/.DS_Store
**/*.pem
**/.eslintcache
**/*.tsbuildinfo
*~
__pycache__
db.sqlite3

# translations
*.mo
*.sqlite
data/
exports/
media/
static/
.django_secret
*local_settings.py
/venv*
/.idea

# backend generated static
backend/media/
backend/static/
backend/staticroot/

# coverage
.coverage
htmlcov
.cache
perf*csv
/*hack*py
*.egg-info
/build
/dist
/src
.env
/.vscode
.pytest_cache
.DS_Store
/staticroot/
/broker
celerybeat-schedule
.python-version
.tox
coverage.xml
control/*
.ruff_cache
pytest.ini
dump.rdb

# celery
backend/broker*
backend/control*
27 changes: 5 additions & 22 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
ci:
skip:
# Cannot access poetry on pre-commit.ci
- python-safety-dependencies-check
- poetry-lock

repos:
Expand All @@ -16,33 +15,17 @@ repos:
args: ["--markdown-linebreak-ext=md"]

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.9.4
rev: v0.9.3
hooks:
- id: ruff
entry: bash -c 'cd backend && ruff check --force-exclude'
- id: ruff-format

- repo: https://github.com/pycqa/bandit
rev: 1.8.2
hooks:
- id: bandit
args: ["-c", "pyproject.toml"]
additional_dependencies: ["bandit[toml]"]

- repo: https://github.com/Lucas-C/pre-commit-hooks-safety
rev: v1.3.3
hooks:
- id: python-safety-dependencies-check
files: pyproject.toml
# Ignore some issues in sub-dependencies:
# - https://data.safetycli.com/v/64396/97c/ | django-helusers -> python-jose -> ecdsa
# - https://data.safetycli.com/v/64459/97c/ | django-helusers -> python-jose -> ecdsa
# - https://data.safetycli.com/v/70612/97c/ | django-jinja -> jinja2
# - https://data.safetycli.com/v/70715/97c/ | django-helusers -> python-jose
# - https://data.safetycli.com/v/70716/97c/ | django-helusers -> python-jose
args: ["--ignore=64396,64459,70612,70715,70716"]
entry: bash -c 'cd backend && ruff format --force-exclude'

- repo: https://github.com/python-poetry/poetry
rev: 2.0.1
hooks:
- id: poetry-check
args: ["--project=backend"]
- id: poetry-lock
args: ["--project=backend"]
72 changes: 1 addition & 71 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,23 +1,10 @@
.PHONY: Makefile
.PHONY: bash
.PHONY: celery
.PHONY: clear-db
.PHONY: db-reset
.PHONY: dev
.PHONY: flush
.PHONY: generate
.PHONY: help
.PHONY: hooks
.PHONY: indices
.PHONY: lint
.PHONY: Makefile
.PHONY: migrate
.PHONY: migrations
.PHONY: run
.PHONY: services
.PHONY: stop
.PHONY: check-translations
.PHONY: translations
.PHONY: translate

# Trick to allow passing commands to make
# Use quotes (" ") if command contains flags (-h / --help)
Expand All @@ -33,22 +20,9 @@ define helptext
Commands:

bash Open bash in backend container.
celery Run a local celery worker.
clear-db Reset database.
dev Run local backend.
flush-db Flush database.
generate Generate test data via a script.
hooks Add pre-commit hooks.
indices Rebuild search indices.
lint Run pre-commit hooks.
migrate Run database migrations.
migrations Compile database migrations.
run Start docker containers for frontend development.
services Run required services in docker.
stop Stop running containers.
check-translations Check if translations are up to date.
translations Fetch all translation strings under ./locale/.
translate Compile translation strings.

endef

Expand All @@ -61,33 +35,6 @@ help:
bash:
@docker exec -it tvp-core bash

celery:
@celery -A config worker --beat --loglevel=INFO --scheduler=django

clear-db:
@python manage.py dbshell -- -c "DROP SCHEMA $(call args, 'public') CASCADE;CREATE SCHEMA $(call args, 'public');"

dev:
@python manage.py runserver localhost:8000

flush-db:
@python manage.py flush --no-input

generate:
@python manage.py create_test_data

hooks:
@pre-commit install

lint:
@pre-commit run --all-files

migrate:
@python manage.py migrate

migrations:
@python manage.py makemigrations

run:
@docker compose up --detach --build

Expand All @@ -96,20 +43,3 @@ services:

stop:
@docker compose stop

check-translations:
@python -m config.hooks.translations_done

translations:
@echo ""
@echo Making translations...
@python manage.py maketranslations -l fi -l sv --no-obsolete --omit-header --add-location file
@echo ""
@echo Done!

translate:
@echo ""
@echo Compiling translations...
@python manage.py compilemessages
@echo ""
@echo Done!
Loading
Loading