Seeking help migrating from separate repos to monorepo

[ThomasBaruzier]

Hello,

It has been a few months since I have been hosting my own Revolt instance, and to be honest, I've been really liking it so far. So, thanks a lot!

I made my own docker compose and dockerfiles to be able to compile and run on ARM, because there was no other solution in the past (i don't know the current status of Revolt on ARM today tbh). The whole process relies on the backend being delta and bonfire and using autumn and january from their own separate repos.

However, on a random day, after updating, everything broke. Looking into the problem, I see that the backend became a monorepo with everything we needed, which is great!

I get to work and update my build/run setup. Everything seems to work, except autumn. It cannot retrieve the previous files and won't allow to upload new files:

XHRPOST https://website.example.com/autumn/attachments [HTTP/1.1 405 Method Not Allowed 13ms]

    POST
    https://website.example.com/autumn/attachments
    Status
    405
    Method Not Allowed
    VersionHTTP/1.1
    Transferred2.14 kB (0 B size)
    Referrer Policystrict-origin-when-cross-origin
    Request PriorityHighest
    DNS ResolutionSystem

    allow
        GET,HEAD
    Connection
        keep-alive
    Content-Length
        0
    Date
        Tue, 10 Sep 2024 09:30:46 GMT
    Server
        nginx
 
    Accept
        application/json, text/plain, */*
    Accept-Encoding
        gzip, deflate, br, zstd
    Accept-Language
        en-US,en;q=0.5
    Connection
        keep-alive
    Content-Length
        1386
    Content-Type
        multipart/form-data; boundary=---------------------------262816025840990021613783940425
    DNT
        1
    Host
        website.example.com
    Origin
        https://website.example.com
    Priority
        u=0
    Referer
         https://website.example.com/channel/01HZYD0FQGKR2S4N5NR8A492K2
    Sec-Fetch-Dest
        empty
    Sec-Fetch-Mode
        cors
    Sec-Fetch-Site
        same-origin
    Sec-GPC
        1
    User-Agent
        Mozilla/5.0 (Windows NT 10.0; rv:130.0) Gecko/20100101 Firefox/130.0

nginx.conf

    server {
        server_name revolt.example.com;

        root /home/user/docker/revolt/revite/dist;
        index index.html;

        gzip on;
        gzip_vary on;
        gzip_comp_level 6;
        gzip_min_length 1024;
        gzip_proxied expired no-cache no-store private auth;
        gzip_types text/plain text/css text/xml text/javascript application/x-javascript;

        location / {
            try_files $uri $uri/ /index.html;
        }

        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
            expires max;
            add_header Cache-Control "public, max-age=31536000";
        }

        location /delta/ {
            proxy_pass http://127.0.0.1:3010;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        location /bonfire/ {
            proxy_pass http://127.0.0.1:3011;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        location /autumn/ {
            proxy_pass http://127.0.0.1:3012;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        location /january/ {
            proxy_pass http://127.0.0.1:3013;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }

    server {
        server_name revolt-delta.example.com;

        location / {
            proxy_pass http://127.0.0.1:3010;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }

    server {
        server_name revolt-bonfire.example.com;

        location / {
            proxy_pass http://127.0.0.1:3011;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }

    server {
        server_name revolt-autumn.example.com;

        location / {
            proxy_pass http://127.0.0.1:3012;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;

            # CORS headers
            add_header 'Access-Control-Allow-Origin' 'https://revolt.example.com' always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range' always;
            add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range' always;

            # Handle preflight requests
            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' 'https://revolt.example.com';
                add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
                add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
                add_header 'Access-Control-Max-Age' 1728000;
                add_header 'Content-Type' 'text/plain; charset=utf-8';
                add_header 'Content-Length' 0;
                return 204;
            }
        }

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }

    server {
        server_name revolt-january.example.com;

        location / {
            proxy_pass http://127.0.0.1:3013;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }

    server {
        if ($host = revolt.example.com) {
            return 301 https://$host$request_uri;
        }
        server_name revolt.example.com;
        listen 80;
        return 404;
    }

    server {
        if ($host = revolt-api.example.com) {
            return 301 https://$host$request_uri;
        }
        server_name revolt-api.example.com;
        listen 80;
        return 404;
    }

    server {
        if ($host = revolt-bonfire.example.com) {
            return 301 https://$host$request_uri;
        }
        server_name revolt-bonfire.example.com;
        listen 80;
        return 404;
    }

    server {
        if ($host = revolt-autumn.example.com) {
            return 301 https://$host$request_uri;
        }
        server_name revolt-autumn.example.com;
        listen 80;
        return 404;
    }

    server {
        if ($host = revolt-january.example.com) {
            return 301 https://$host$request_uri;
        }
        server_name revolt-january.example.com;
        listen 80;
        return 404;
    }

compose.yml

services:
  redis:
    image: eqalpha/keydb
    networks:
      - backend-network

  database:
    image: mongo
    networks:
      - backend-network
    volumes:
      - ./.data/db:/data/db

  minio:
    image: minio/minio
    command: server /data
    environment:
      MINIO_ROOT_USER: minioautumn
      MINIO_ROOT_PASSWORD: minioautumn
    networks:
      - backend-network
    volumes:
      - ./.data/minio:/data

  createbuckets:
    image: minio/mc
    depends_on:
      - minio
    networks:
      - backend-network
    environment:
      MINIO_ROOT_USER: minioautumn
      MINIO_ROOT_PASSWORD: minioautumn
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add minio http://minio:9000 minioautumn minioautumn;
      while ! /usr/bin/mc ready minio; do echo 'Waiting minio...' && sleep 1; done;
      /usr/bin/mc mb minio/revolt-uploads;
      exit 0;
      "

  # i am aware of the distroless containers, i'll update this soon enough
  revolt-backend:
    build:
      context: .
      dockerfile: Dockerfile.useCurrentArch
    networks:
      - backend-network
    depends_on:
      - redis
      - database
      - minio
    ports:
      - "3010:14702"  # Delta
      - "3011:14703"  # Bonfire
      - "3012:14704"  # Autumn
      - "3013:14705"  # January
    volumes:
      - type: bind
        source: ./Revolt.override.toml
        target: /Revolt.toml
    command: >
      /bin/sh -c "
      ./target/release/revolt-january &
      ./target/release/revolt-bonfire &
      ./target/release/revolt-autumn &
      ./target/release/revolt-delta &
      wait
      "

networks:
  backend-network:
    driver: bridge

Revolt.override.toml

[database]
# MongoDB connection URL
# Use the service name from docker-compose
mongodb = "mongodb://database:27017"
# Redis connection URL
# Use the service name from docker-compose
redis = "redis://redis:6379"

[hosts]
# Web locations of various services
# Defaults assume all services are reverse-proxied
# See https://github.com/revoltchat/self-hosted/blob/master/Caddyfile
#
# Remember to change these to https/wss where appropriate in production!
app = "https://revolt.example.com"
api = "https://revolt.example.com/api"
events = "wss://revolt-bonfire.example.com"
autumn = "https://revolt-autumn.example.com"
january = "https://revolt-january.example.com"
voso_legacy = ""
voso_legacy_ws = ""

[api]

[api.registration]
# Whether an invite should be required for registration
# See https://github.com/revoltchat/self-hosted#making-your-instance-invite-only
invite_only = true

[api.smtp]
# Email server configuration for verification
# Defaults to no email verification (host field is empty)
host = ""
username = ""
password = ""
from_address = ""
reply_to = ""
port = 14025
use_tls = false

[api.vapid]
# Generate your own keys:
# 1. Run `openssl ecparam -name prime256v1 -genkey -noout -out vapid_private.pem`
# 2. Find `private_key` using `base64 vapid_private.pem`
# 3. Find `public_key` using `openssl ec -in vapid_private.pem -outform DER|tail -c 65|base64|tr '/+' '_-'|tr -d '\n'`
[REMOVED FOR PRIVACY]

[api.fcm]
# Google Firebase Cloud Messaging Service Account Key
# Obtained from the cloud messaging console
[REMOVED FOR PRIVACY]

[api.apn]
# Apple Push Notifications keys for sending notifications
sandbox = false
pkcs8 = ""
key_id = ""
team_id = ""

[api.security]
# Authifier Shield API key
authifier_shield_key = ""
# Legacy voice server management token
voso_legacy_token = ""
# Whether services are behind the Cloudflare network
trust_cloudflare = false

[api.security.captcha]
# hCaptcha configuration
hcaptcha_key = ""
hcaptcha_sitekey = ""

[api.workers]
# Maximum concurrent connections (to proxy server)
max_concurrent_connections = 100

[files]
# Encryption key for stored files
# Generate your own key using `openssl rand -base64 32`
encryption_key = "placeholder"
# Quality used for lossy WebP previews (set to 100 for lossless)
webp_quality = 70.0
# Mime types that cannot be uploaded or served
#
# Example for Windows executables and Android installation files:
# ["application/vnd.microsoft.portable-executable", "application/vnd.android.package-archive"]
blocked_mime_types = []
# ClamAV service
# hostname:port
clamd_host = ""

[files.limit]
# Minimum image resolution
min_resolution = [1, 1]
# Maximum MP of images
max_mega_pixels = 100
# Maximum pixel side of an image
max_pixel_side = 16_000

[files.preview]
# Maximum image resolution
attachments = [1280, 1280]
avatars = [256, 256]
backgrounds = [1280, 720]
icons = [128, 128]
banners = [480, 480]
emojis = [128, 128]

[files.s3]
# Configuration for S3
# Defaults included for MinIO + self-hosted setup
#
# Backblaze B2:
# - endpoint is listed on the "Buckets" page
# - region is `eu-central-003` string from endpoint URL
# - access_key_id is keyID generated on the "Application Keys" page
# - secret_access_key is token generated on the "Application Keys" page
# - default_bucket matches the name of the bucket you've created

# S3 protocol endpoint
endpoint = "http://minio:9000"
# S3 region name
region = "minio"
# S3 protocol key ID
access_key_id = "minioautumn"
# S3 protocol access key
secret_access_key = "minioautumn"
# Bucket to upload to by default
default_bucket = "revolt-uploads"

[features]
# Feature gate options
webhooks_enabled = false

[features.limits]

[features.limits.global]
group_size = 1000
message_embeds = 100
message_replies = 100
message_reactions = 100
server_emoji = 10000
server_roles = 1000
server_channels = 1000

# How many days since creation a user is considered new
new_user_days = 3

# Maximum permissible body size in bytes for uploads
# (should be greater than any one file upload limit)
body_limit_size = 20_000_000_000

[features.limits.new_user]
outgoing_friend_requests = 100

bots = 100
message_length = 100000
message_attachments = 100
servers = 1000

attachment_size = 20_000_000_000
avatar_size = 4_000_000
background_size = 6_000_000
icon_size = 2_500_000
banner_size = 6_000_000
emoji_size = 500_000

[features.limits.default]
outgoing_friend_requests = 100

bots = 100
message_length = 100000
message_attachments = 100
servers = 100

attachment_size = 20_000_000
avatar_size = 4_000_000
background_size = 6_000_000
icon_size = 2_500_000
banner_size = 6_000_000
emoji_size = 500_000

[sentry]
# Configuration for Sentry error reporting
api = ""
events = ""

Another question I have is: How do we migrate the previous autumn db from the separate repo to the new backend backed one? I see in the docker compose files that the previous version created multiple various storage buckets, but the newer docker compose only creates revolt-uploads. While migrating the previous mongo db was as straight as a single directory copy, I am scratching my head on this one.

I would be grateful for any help, thanks!

[insertish]

Hi the services in the monorepo are still under development and not ready for use, autumn in particular is nearly done and I'm now updating the rest of the backend to work with it.

Migration information will be provided on the self hosted repository once it's complete.

[ThomasBaruzier]

Alright. Thanks!

[0ktav1us]

+++ After the upgrade, my avatars are not loading, I can't see other attachments, please fix it
I get {"type":"NotFound","location":"crates/services/autumn/src/api.rs:372:20"}

[ThomasBaruzier]

I haven't been able to use the autumn server since it was merged in the monorepo, but i haven't taken the time to properly troubleshoot what's going on tbh

[insertish]

Please refer to migration information in notices: https://github.com/revoltchat/self-hosted?tab=readme-ov-file#notices

[ThomasBaruzier]

Please refer to migration information in notices: https://github.com/revoltchat/self-hosted?tab=readme-ov-file#notices

Following these instructions did not work on my end. Here's what I did and what I got: revoltchat/self-hosted#119 (comment)