added lots of technical documentation about devops. Renamed few commands

.github/workflows/api-tests.yaml

@@ -23,7 +23,7 @@ jobs:
       - name: Create env-files
         env:
           GEONAMES_USERNAME: ${{ secrets.GEONAMES_USERNAME }}
-        run: GEONAMES_USERNAME=$GEONAMES_USERNAME npm run config:all
+        run: GEONAMES_USERNAME=$GEONAMES_USERNAME npm run config
 
       - name: 'Start containers'
         run: npm run start:anon:api -- --wait -d

.github/workflows/e2e.yaml

@@ -23,7 +23,7 @@ jobs:
       - name: Create env-files
         env:
           GEONAMES_USERNAME: ${{ secrets.GEONAMES_USERNAME }}
-        run: GEONAMES_USERNAME=$GEONAMES_USERNAME npm run config:all
+        run: GEONAMES_USERNAME=$GEONAMES_USERNAME npm run config
 
       - name: Install root dependencies
         run: npm ci

.github/workflows/unit-tests.yaml

@@ -15,8 +15,10 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
-      - name: Copy .anon.env
-        run: cp .template.env .anon.env
+      - name: Create env-files
+        env:
+          GEONAMES_USERNAME: ${{ secrets.GEONAMES_USERNAME }}
+        run: GEONAMES_USERNAME=$GEONAMES_USERNAME npm run config
 
       - name: Setup node
         uses: actions/setup-node@v4
@@ -26,5 +28,8 @@ jobs:
       - name: 'Install dependencies'
         run: npm run setup
 
+      - name: Run backend unit-tests
+        run: npm run test:unit:backend
+
       - name: Run frontend unit-tests
         run: npm run test:unit:frontend

README.md

@@ -13,80 +13,23 @@ The new version is not deployed yet.
 **Basic commands** :computer:
 
 - `npm run setup`
-  - Installs all node modules, and generates prisma client
+  - Creates all files and directories required to run the project
 - `npm run dev`
-  - Uses `docker-compose.dev.yml`
-  - Run in development mode: with hot reload
-  - To create test users visit `http://localhost:4000/test/create-test-users`
-
-Testing:
-
-- `npm run check`
-  - Runs `tsc` and `eslint` for frontend and backend: Use this to check for errors before committing.
+  - Runs the application in development mode
+  - Supports hot reloading
+  - Serves frontend by default at `http://localhost:5173`
 - `npm run test:api`
-  - Runs the api-tests in a docker container
-  - Requires database to be running. Can be achieved by `npm run start:anon -- db-anon` or just `npm run start:anon:api`.
-  - Uses `test_data`
-  - Uses `.test.env`
-  - `npm run test:api:build` builds the container used to run the tests.
-  - `npm run test:api:run` runs the previously built container.
+  - Runs api-tests inside Docker
+  - Requires database to be running (start one with correct data: `npm run start:anon:api`)
 - `npm run test:e2e`
-  - Runs e2e-tests inside a docker container.
-  - Requires `frontend`, `backend` and `database` to be running. Can be achieved by running `npm run start:anon`.
-  - Uses `test_data`
-- `npm run test:e2e:local`
-  - Runs cypress-tests locally (not in docker) in headless mode.
-  - Requires `npm run start:anon` to be running first.
-- `npm run test:api:local`
-  - Runs backend api tests locally (not in docker).
-  - Requires `db-anon` to be running first. (Note: backend doesn't need to be running.)
-  - Requires `.test.env`
-- `npm run cypress`
-  - Opens cypress GUI. You can run tests and see what they do.
-
-Windows testing:
-
-- Both `e2e` and `api-tests` have a Windows specific command.
-  - `e2e` is `npm run test:e2e:windows`
-  - `api-tests` is `npm run test:api:windows`
-  - For `api-tests`: inside `.test.env`, you need to change all instances of `localhost` to `host.docker.internal`.
-    - These include `MARIADB_HOST`, `DATABASE_URL` and `LOG_DATABASE_URL`.
-  - Note: these are not tested yet.
-
-Test env-file:
-
-- `.test.env`
-  - `api-tests` requires `.test.env` to set the correct environment variables.
-  - This is created by copying `.anon.env` and changing all instances of `nowdb-db-anon` to `localhost`.
-    - These include `MARIADB_HOST`, `DATABASE_URL` and `LOG_DATABASE_URL`.
-
-Coverage:
-
-- Coverage opening
-  - Coverage can be opened by opening the `coverage/lcov-report/index.html` with your browser of choice.
-- `coverage:sed`
-  - Fixes issues with `.nyc_output` paths. Replaces the paths inside Docker to `$PWD` (current directory).
-- `coverage`
-  - First fixes paths with `coverage:sed` and then recreates `coverage/lcov-report` to match the new path.
-- `coverage:report`
-  - Alias to `npm run coverage`. Used by cypress e2e tests so that `coverage:sed` gets executed automatically.
-
-Run with the same image that is used in production:
-
-- `npm start`
-  - Uses `docker-compose.yml`
-  - This builds the code and runs it. Mostly you'll want to use this after major technical changes to check the code works also in production mode, before deploying. And then you'll likely want to do: `npm start -- --build` to force rebuild of containers.
-
-Docker:
-
-- `npm run dev:down`
-  - Stops and removes the dev container and it's volumes.
-- `-- --build`
-  - Useful flag to force docker to rebuild the containers. Example usage for dev: `npm run dev -- --build`
+  - Runs api-tests inside Docker
+  - Requires frontend, backend and database to be running (start all with `npm run start:anon`)
 
 **Further documentation**
 
-:rocket: [Initialize app & restore db](documentation/init.md) How to get the app up & running, and initialize database from sql-dumps
+:rocket: [Setup app & restore db](documentation/setup.md) How to get the app up & running, and initialize database from sql-dumps
+
+:keyboard: [List of all commands](documentation/commands.md) A comprehensive list of all of the most important commands
 
 :wrench: [Technical explanations](documentation/guides/technical_explanations.md) Explanations of how things work, and guides for some more rare tasks like updating database-schema
 

backend/Dockerfile

@@ -49,16 +49,3 @@ EXPOSE 4000
 ENV NODE_ENV=production
 
 CMD ["node", "src/index.js"]
-
-
-FROM node:22-alpine3.20 AS api-tests
-
-USER node
-COPY --chown=node --from=backend-build /usr/src/app /usr/src/app
-
-WORKDIR /usr/src/app/backend
-
-RUN npm ci
-RUN npm run prisma
-
-CMD ["npm", "run", "test:api"]

backend/dev.Dockerfile

@@ -1,4 +1,4 @@
-FROM node:22-alpine3.20
+FROM node:22-alpine3.20 as dev
 
 ENV TZ="Europe/Helsinki"
 
@@ -10,6 +10,10 @@ WORKDIR /usr/src/app
 
 RUN mkdir -p /usr/src/app/frontend/src/shared/validators
 
+COPY --chown=node ./frontend/src/backendTypes.d.ts /usr/src/app/frontend/src/backendTypes.d.ts
+COPY --chown=node ./frontend/src/types.ts /usr/src/app/frontend/src/types.ts
+COPY --chown=node ./frontend/src/validators /usr/src/app/frontend/src/validators
+
 COPY --chown=node ./backend/package.json backend/
 COPY --chown=node ./backend/package-lock.json backend/
 COPY --chown=node ./backend/prisma/schema.prisma backend
@@ -26,3 +30,8 @@ COPY --chown=node ./backend .
 RUN npm run prisma
 
 CMD ["npm", "run", "dev"]
+
+
+FROM dev as api-tests
+
+CMD ["npm", "run", "test:api"]

backend/package.json

@@ -12,7 +12,7 @@
     "start": "node build/index.js",
     "lint": "eslint --report-unused-disable-directives --max-warnings 0",
     "prisma": "prisma generate --schema prisma/schema.prisma && prisma generate --schema prisma/schema_log.prisma",
-    "clean": "rm -rf node_modules && rm -rf prisma/generated && rm -rf build"
+    "clean": "rm -rf node_modules && rm -rf prisma/generated && rm -rf build && rm -rf coverage"
   },
   "author": "",
   "devDependencies": {

docker-compose.anon.yml

@@ -22,6 +22,7 @@ services:
     build:
       context: .
       dockerfile: ./backend/dev.Dockerfile
+      target: dev
     volumes:
       - ./backend/src:/usr/src/app/backend/src
       - ./frontend/src/:/usr/src/app/frontend/src/

docker-compose.dev.yml

@@ -16,6 +16,7 @@ services:
     build:
       context: .
       dockerfile: ./backend/dev.Dockerfile
+      target: dev
     volumes:
       - ./backend/src:/usr/src/app/backend/src
       - ./frontend/src/:/usr/src/app/frontend/src/
@@ -30,7 +31,7 @@ services:
     env_file: .dev.env
     volumes:
       - ./data/:/docker-entrypoint-initdb.d
-      - nowdb-db:/var/lib/mysql/data
+      - nowdb-db-dev:/var/lib/mysql/data
     ports:
       - 127.0.0.1:3306:3306
 
@@ -46,4 +47,4 @@ services:
       - "9090:443"
 
 volumes:
-  nowdb-db:
+  nowdb-db-dev:

documentation/commands.md

@@ -0,0 +1,91 @@
+# npm commands
+
+## Initialization
+* `setup`
+    * Installs node_modules to project root, backend and frontend
+    * Sets up Prisma in backend
+    * Creates all config files
+* `config:all`
+    * Creates all config files: `.dev.env`, `.anon.env` and `.test.env`
+
+## Running the application
+* `start`
+    * Runs the application in the production environment
+    * Opens backend, frontend and a MariaDB database (initialized from `data` directory)
+    * Utilizes `.env`
+* `start:anon`
+    * Runs the application in dev environment
+    * Opens backend, frontend, phpMyAdmin and a MariaDB database (initialized from `test_data` directory)
+    * Utilizes `.anon.env`
+* `dev`
+    * Runs the application in development environment
+    * Enables hotswap (docker compose doesn't need to be restarted on changes inside `backend/src` or `frontend/src`)
+        * Note: When editing for example `package.json` the docker needs to be manually restarted.
+    * Opens backend, frontend, phpMyAdmin, and a MariaDB database (initialized from `data` directory)
+    * Utilizes `.dev.env`
+    * To create test users visit `http://localhost:4000/test/create-test-users`
+
+## Testing
+* `test:api`
+    * Runs tests for backend API
+    * Tests are located in `backend/src/api-tests`
+    * Tests are executed inside Docker
+    * Requires a database running with (preferrably, see last point) `test_data`
+    * Utilizes `.test.env`
+    * Anon version of the database is recommended be used by these tests
+    * Resets the database with `test_data` multiple times!
+* `test:api:local`
+    * Same as `test:api` but tests are executed locally
+* `test:e2e`
+    * Runs e2e tests with cypress
+    * Tests are located in `cypress/e2e`
+    * Tests are executed inside Docker
+    * Requires backend, frontend and database (with preferably `test_data`, see last point) to be running
+        * Expects to find frontend at `<baseUrl>` defined in `cypress/config.js`
+        * Expects to find database reset API-point at `<databaseResetUrl>` defined in `cypress.config.js`
+        * Both can be also defined as a environment variable
+    * Resets the database with `test_data` multiple times!
+* `test:e2e:local`
+    * Same as `test:e2e` but tests are executed locally
+    * Change the browser by running `npm run test:e2e:local -- --browser <YOUR_BROWSER_NAME>`
+* `test:unit`
+    * Runs all unit tests defined in `backend/src/unit-tests` and `frontend/src/tests`
+    * Doesn't require containers running
+    * Executed locally
+* `test:{api,e2e}:windows`
+    * Executes api or e2e command with (supposed) Windows support
+    * Not actually tested on Windows: "should work"
+    * For `api-tests` to work with Windows: inside `.test.env`, you need to change all instances of `localhost` to `host.docker.internal`.
+        * These include `MARIADB_HOST`, `DATABASE_URL` and `LOG_DATABASE_URL`.
+    * A short script that should do the same: `sed -i -e s|localhost|host.docker.internal|g .test.env`
+
+
+## Utilities
+* `check`
+    * Runs linting and typescript checking for backend, frontend and cypress
+* `anon:down`
+    * Closes all docker containers running anon versions
+    * REMOVES `nowdb-db-anon` VOLUME PERMANENTLY
+        * This is usually not a problem as the normal development environment uses a different volume
+* `dev:down`
+    * Closes all docker containers running dev versions
+    * REMOVES `nowdb-db-dev` VOLUME PERMANENTLY
+* `clean`
+    * Removes all node_modules, build directories, prisma's autogenerated files, .nyc_output and coverage directories
+* `coverage`
+    * Creates coverage report based on `.nyc_output` 
+    * Replaces the incorrect paths of the files from the path inside Docker container to the current working directory (`$PWD`)
+    * Automatically run by `e2e` each time
+* `coverage:report`
+    * Alias to `coverage` for cypress to create coverage while running tests
+    * https://github.com/cypress-io/code-coverage?tab=readme-ov-file#custom-nyc-command
+
+
+### Be aware
+* Weird problems with docker?
+    * Add `-- --build` after for example `npm run start:anon` to force rebuilding all containers
+    * Sometimes docker doesn't rebuild containers when needed and it leads to hard-to-find bugs
+
+* Difference between `cmd1 && cmd2` and `cmd1; cmd2`
+    * `&&`: `cmd2` is executed only when `cmd1` returns 0 (=is succesful)
+    * `;`: `cmd2` is always executed (after `cmd1`)