Merge torrust#105: E2E testing scaffolding

eedb0ce tests: e2e tests scaffolding (Jose Celano)
6ecde1d feat: add docker support (Jose Celano)

Pull request description:

  Hi @da2ce7 @WarmBeer. I want to add some tests for the API before changing things, for example, updating dependencies.

  I wanted to implement something like what we did for the tracker ([integration tests](https://github.com/torrust/torrust-tracker/blob/develop/tests/integration.rs)). But If I want to do the same:

  - I need to change the application bootstrapping. I need to start/stop the backend programmatically.
  - I also need to mock or enable/disable the tracker.

  I started following that path, but it requires a lot of changes, and I do not want to make big changes without tests :-/. So I decided to try a different approach:

  - Run the backend and its dependencies (tracker and DB) with docker-compose.
  - Implement real E2D tests without changing anything (black box).

  I'm going to use the same instance. Steps:

  - Docker-compose up
  - Run the tests
  - Docker-compose down

  I will need an option to reset the DB. Not implemented yet.

  This PR is just the scaffolding with one test for the API entrypoint (about).

  I think I can add some tests for most endpoints and then switch to integration tests (like the Tracker). They should be faster and more reliable. I think I can reuse the same tests. The only difference should be the bootstrapping for the app.

  Once we have the integration tests, we can start adding the unit tests.

  In conclusion, the steps would be:

  - E2E test (minimum changes on production code).
  - Migrate to integrations test (apply changes needed for integrations tests).
  - Start writing unit tests.

  This way we always are safe while refactoring.

  The only problem I see is we could have a problem with only one app running. But I guess we can avoid test conflicts if we use unique IDs, torrents, users, etcetera, for each test.

  What do you think?

ACKs for top commit:
  josecelano:
    ACK eedb0ce

Tree-SHA512: 161c022a0a29c74f264e979501ddc27cff00cd2d564f0ceb72fa9a5ab039a06a87d0074a34a0183dd9d510fb803edc6ca49ab44574c8d7a3edb8ba599be9f857

.cargo/config.toml

@@ -0,0 +1,7 @@
+[alias]
+cov = "llvm-cov"
+cov-lcov = "llvm-cov --lcov --output-path=./.coverage/lcov.info"
+cov-html = "llvm-cov --html"
+time = "build --timings --all-targets"
+e2e = "test --features e2e-tests"
+

.dockerignore

@@ -0,0 +1,22 @@
+/.env
+/.env.local
+/.git
+/.git-blame-ignore
+/.github
+/.gitignore
+/.vscode
+/data_v2.db*
+/data.db*
+/bin/
+/config-idx-back.toml.local
+/config-tracker.toml.local
+/config.toml
+/config.toml.local
+/cspell.json
+/data.db
+/docker/
+/project-words.txt
+/README.md
+/rustfmt.toml
+/storage/
+/target/

.env.local

@@ -0,0 +1,6 @@
+DATABASE_URL=sqlite://storage/database/data.db?mode=rwc
+TORRUST_IDX_BACK_CONFIG=
+TORRUST_IDX_BACK_USER_UID=1000
+TORRUST_TRACKER_CONFIG=
+TORRUST_TRACKER_USER_UID=1000
+TORRUST_TRACKER_API_TOKEN=MyAccessToken

.github/workflows/develop.yml

@@ -1,6 +1,6 @@
 name: Development Checks
 
-on: [push,pull_request]
+on: [push, pull_request]
 
 jobs:
   run:
@@ -14,27 +14,16 @@ jobs:
           components: rustfmt, clippy
       - uses: Swatinem/rust-cache@v2
       - name: Format
-        uses: ClementTsang/cargo-action@main
-        with:
-          command: fmt
-          args: --all --check
+        run: cargo fmt --all --check
       - name: Check
-        uses: ClementTsang/cargo-action@main
-        with:
-          command: check
-          args: --all-targets
+        run: cargo check --all-targets
       - name: Clippy
-        uses: ClementTsang/cargo-action@main
-        with:
-          command: clippy
-          args: --all-targets
-      - name: Build
-        uses: ClementTsang/cargo-action@main
-        with:
-          command: build
-          args: --all-targets
-      - name: Test
-        uses: ClementTsang/cargo-action@main
-        with:
-          command: test
-          args: --all-targets
+        run: cargo clippy --all-targets
+      - name: Unit and integration tests
+        run: cargo test --all-targets
+      - uses: taiki-e/install-action@cargo-llvm-cov
+      - uses: taiki-e/install-action@nextest
+      - name: Test Coverage
+        run: cargo llvm-cov nextest        
+      - name: E2E Tests
+        run: ./docker/bin/run-e2e-tests.sh

.github/workflows/publish_docker_image.yml

@@ -0,0 +1,83 @@
+name: Publish Docker Image
+
+on:
+  push:
+    branches:
+      - "main"
+      - "develop"
+    tags:
+      - "v*"
+
+env:
+  TORRUST_IDX_BACK_RUN_AS_USER: appuser
+
+jobs:
+  check-secret:
+    runs-on: ubuntu-latest
+    environment: dockerhub-torrust
+    outputs:
+      publish: ${{ steps.check.outputs.publish }}
+    steps:
+      - id: check
+        env:
+          DOCKER_HUB_USERNAME: "${{ secrets.DOCKER_HUB_USERNAME }}"
+        if: "${{ env.DOCKER_HUB_USERNAME != '' }}"
+        run: echo "publish=true" >> $GITHUB_OUTPUT
+
+  test:
+    needs: check-secret
+    if: needs.check-secret.outputs.publish == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: stable
+          components: llvm-tools-preview
+      - uses: Swatinem/rust-cache@v2
+      - name: Run Tests
+        run: cargo test
+
+  dockerhub:
+    needs: test
+    if: needs.check-secret.outputs.publish == 'true'
+    runs-on: ubuntu-latest
+    environment: dockerhub-torrust
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: |
+            # For example: torrust/index-backend
+            "${{ secrets.DOCKER_HUB_USERNAME }}/${{secrets.DOCKER_HUB_REPOSITORY_NAME }}"
+          tags: |
+            type=ref,event=branch
+            type=ref,event=pr
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKER_HUB_USERNAME }}
+          password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Build and push
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          file: ./Dockerfile
+          build-args: |
+            RUN_AS_USER=${{ env.TORRUST_IDX_BACK_RUN_AS_USER }}
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

.github/workflows/release.yml

@@ -1,3 +1,5 @@
+name: Publish Github Release
+
 on:
   push:
     branches:
@@ -23,6 +25,9 @@ jobs:
       - uses: Swatinem/rust-cache@v1
       - name: Run tests
         run: cargo test
+      - name: Stop databases
+        working-directory: ./tests
+        run: docker-compose down
 
   tag:
     needs: test

.github/workflows/test_docker.yml

@@ -0,0 +1,26 @@
+name: Test Docker build
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Build docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          file: ./Dockerfile
+          push: false
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Build docker-compose images
+        run: docker compose build

.gitignore

@@ -1,6 +1,9 @@
+/.coverage/
 /.env
 /config.toml
-/data.db*
 /data_v2.db*
+/data.db*
+/storage/
 /target
-/uploads/
+/uploads/
+

Cargo.toml

@@ -8,6 +8,9 @@ default-run = "main"
 [profile.dev.package.sqlx-macros]
 opt-level = 3
 
+[features]
+e2e-tests = []
+
 [dependencies]
 actix-web = "4.0.0-beta.8"
 actix-multipart = "0.4.0-beta.5"

Dockerfile

@@ -0,0 +1,72 @@
+FROM clux/muslrust:stable AS chef
+WORKDIR /app
+RUN cargo install cargo-chef
+
+
+FROM chef AS planner
+WORKDIR /app
+COPY . .
+RUN cargo chef prepare --recipe-path recipe.json
+
+
+FROM chef as development
+WORKDIR /app
+ARG UID=1000
+ARG RUN_AS_USER=appuser
+ARG IDX_BACK_API_PORT=3000
+# Add the app user for development
+ENV USER=appuser
+ENV UID=$UID
+RUN adduser --uid "${UID}" "${USER}"
+# Build dependencies
+COPY --from=planner /app/recipe.json recipe.json
+RUN cargo chef cook --recipe-path recipe.json
+# Build the application
+COPY . .
+RUN cargo build --bin main
+USER $RUN_AS_USER:$RUN_AS_USER
+EXPOSE $IDX_BACK_API_PORT/tcp  
+CMD ["cargo", "run"]
+
+
+FROM chef AS builder
+WORKDIR /app
+ARG UID=1000
+# Add the app user for production
+ENV USER=appuser
+ENV UID=$UID
+RUN adduser \
+  --disabled-password \
+  --gecos "" \
+  --home "/nonexistent" \
+  --shell "/sbin/nologin" \
+  --no-create-home \
+  --uid "${UID}" \
+  "${USER}"  
+# Build dependencies
+COPY --from=planner /app/recipe.json recipe.json
+RUN cargo chef cook --release --target x86_64-unknown-linux-musl --recipe-path recipe.json
+# Build the application
+COPY . .
+RUN cargo build --release --target x86_64-unknown-linux-musl --bin main
+# Strip the binary
+# More info: https://github.com/LukeMathWalker/cargo-chef/issues/149
+RUN strip /app/target/x86_64-unknown-linux-musl/release/main
+
+
+FROM alpine:latest
+WORKDIR /app
+ARG RUN_AS_USER=appuser
+ARG IDX_BACK_API_PORT=3000
+RUN apk --no-cache add ca-certificates
+ENV TZ=Etc/UTC
+ENV RUN_AS_USER=$RUN_AS_USER
+COPY --from=builder /etc/passwd /etc/passwd
+COPY --from=builder /etc/group /etc/group
+COPY --from=builder --chown=$RUN_AS_USER \
+  /app/target/x86_64-unknown-linux-musl/release/main \
+  /app/main
+RUN chown -R $RUN_AS_USER:$RUN_AS_USER /app
+USER $RUN_AS_USER:$RUN_AS_USER
+EXPOSE $IDX_BACK_API_PORT/tcp
+ENTRYPOINT ["/app/main"]

bin/install.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Generate the default settings file if it does not exist
+if ! [ -f "./config.toml" ]; then
+    cp ./config.toml.local ./config.toml
+fi
+
+# Generate storage directory if it does not exist
+mkdir -p "./storage/database"
+
+# Generate the sqlite database for the index baclend if it does not exist
+if ! [ -f "./storage/database/data.db" ]; then
+    # todo: it should get the path from config.toml and only do it when we use sqlite
+    touch ./storage/database/data.db
+    echo ";" | sqlite3 ./storage/database/data.db
+fi
+
+# Generate the sqlite database for the tracker if it does not exist
+if ! [ -f "./storage/database/tracker.db" ]; then
+    touch ./storage/database/tracker.db
+    echo ";" | sqlite3 ./storage/database/tracker.db
+fi