From 6da204467d160d29cf1ad853656008eea5bac9b1 Mon Sep 17 00:00:00 2001 From: Prosper Otemuyiwa Date: Fri, 15 Nov 2024 07:42:55 +0100 Subject: [PATCH] Update Django Guide to use Celery Beat and Celery (#609) * Update django guide * Add step to add Celery and Celery Beat * change image to django image * fix * fix * Fix * Add community template * Update where to find name of app --- src/docs/guides/django.md | 106 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 102 insertions(+), 4 deletions(-) diff --git a/src/docs/guides/django.md b/src/docs/guides/django.md index 6b8082dc7..a83862c5d 100644 --- a/src/docs/guides/django.md +++ b/src/docs/guides/django.md @@ -12,7 +12,7 @@ It’s free, open-source, and comes with a range of features to streamline tasks To create a new Django app, ensure that you have [Python](https://www.python.org/downloads/) and [Django](https://docs.djangoproject.com/en/5.1/intro/install/) installed on your machine. -Follow the steps blow to set up the project in a directory. +Follow the steps below to set up the project in a directory. Create a virtual environment ```bash @@ -73,15 +73,26 @@ import os from pathlib import Path ``` -3. Configure the database: +3. Configure the database and run migrations: -A fresh Django project uses SQLite by default, but we need to switch to PostgreSQL. +A fresh Django project uses SQLite by default, but we need to switch to PostgreSQL. + +Create a database named `liftoff_dev` in your local Postgres instance. Open the `liftoff/settings.py` file. In the Database section, replace the existing configuration with: ```python + # Database # https://docs.djangoproject.com/en/5.1/ref/settings/#databases + +# Set default values for the environment variables if they’re not already set +os.environ.setdefault("PGDATABASE", "liftoff_dev") +os.environ.setdefault("PGUSER", "username") +os.environ.setdefault("PGPASSWORD", "") +os.environ.setdefault("PGHOST", "localhost") +os.environ.setdefault("PGPORT", "5432") + DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', @@ -94,6 +105,10 @@ DATABASES = { } ``` +Replace the values of `PGUSER`, `PGPASSWORD` with your local credentials. + +Run `python manage.py migrate` in your terminal to apply the database migrations. Once it completes successfully, check your database. You should see the auth and other Django tables created. + 4. Static files configuration: We'll configure Django to serve static files using [WhiteNoise](https://whitenoise.readthedocs.io/en/stable/index.html). @@ -246,7 +261,90 @@ To deploy the Django app to Railway, start by pushing the app to a GitHub repo. - Click [Generate Domain](/guides/public-networking#railway-provided-domain) to create a public URL for your app. This guide covers the main deployment options on Railway. Choose the approach that suits your setup, and start deploying your Django apps effortlessly! - + +**Note:** The next step shows how to configure and run your Django app along with Celery and Celery beat. + +## Set Up Database, Migrations, Celery Beat and Celery + +This setup deploys your Django app on Railway, ensuring that your database, scheduled tasks (crons)--Celery Beat, and queue workers (Celery) are all fully operational. + +The deployment structure follows a "majestic monolith" architecture, where the entire Django app is managed as a single codebase but split into four separate services on Railway: +- **App Service**: Handles HTTP requests and user interactions. +- **Cron Service**: Manages scheduled tasks (e.g., sending emails or running reports) using Celery Beat. +- **Worker Service**: Processes background jobs from the queue using Celery. +- **Database Service**: Stores and retrieves your application's data. + + +_My Monolith Django app_ + + +**Note:** This guide follows the assumption that you have installed Celery and Celery Beat in your app, the broker uses Redis and you already have a Postgres database service provisioned for your app as shown earlier. + +Please follow these steps to get it setup on Railway: + +1. Create a Redis Database service on the Project Canvas by clicking the **Create** button. Then select **Database** and choose **Add Redis**. + - Click on **Deploy**. +3. Create a new service on the Project Canvas by clicking the **Create** button. Then select **Empty service**. + - Name the service **App Service**, and click on **Settings** to configure it. + - **Note:** If you followed the guide from the beginning, simply rename the existing service to **App Service**. + - Connect your GitHub repo to the **Source Repo** in the **Source** section. + - Go to the top of the service and click on **Variables**. + - Add all the necessary environment variables required for the Django app especially the ones listed below. + - `REDIS_URL`: Set the value to `${{Postgres.REDIS_URL}}` + - `PGUSER`: Set the value to `${{Postgres.PGUSER}}` + - `PGPASSWORD`: Set the value to `${{Postgres.PGPASSWORD}}` + - `PGHOST`: Set the value to `${{Postgres.PGHOST}}` + - `PGPORT`: Set the value to `${{Postgres.PGPORT}}` + - `PGDATABASE`: Set the value to `${{Postgres.PGDATABASE}}` (this references the Postgres database name). Learn more about [referencing service variables](/guides/variables#referencing-another-services-variable). + - Click **Deploy**. +4. Create a new service on the Project Canvas by clicking the **Create** button. Then select **Empty service**. + - Name the service **Cron Service**, and click on **Settings**. + - Connect your GitHub repo to the **Source Repo** in the **Source** section. + - Add `celery -A liftoff beat -l info --concurrency=3` to the **Custom Start Command** in the **Deploy** section. + - _Note:_ `liftoff` is the name of the app. You can find the app name in your Django project’s main folder, typically in the directory containing `settings.py`. + - The `--concurrency=3` option here means it can process up to 3 tasks in parallel. You can adjust the [concurrency level](https://docs.celeryq.dev/en/latest/userguide/workers.html#concurrency) based on your system resources. The higher the level, the more memory and resources it consumes. + - Head back to the top of the service and click on **Variables**. + - Add all the necessary environment variables especially those highlighted already in step 2. + - Click **Deploy**. +5. Create a new service again on the Project Canvas. + - Name the service **Worker Service**, and click on **Settings**. + - Connect your GitHub repo to the **Source Repo** in the **Source** section. + - Add `celery -A liftoff worker -l info --concurrency=3` to the **Custom Start Command** in the **Deploy** section. + - _Note:_ `liftoff` is the name of the app. You can find the app name in your Django project’s main folder, typically in the directory containing `settings.py`. + - The `--concurrency=3` option here means it can process up to 3 tasks in parallel. You can adjust the [concurrency level](https://docs.celeryq.dev/en/latest/userguide/workers.html#concurrency) based on your system resources. The higher the level, the more memory and resources it consumes. + - Head back to the top of the service and click on **Variables**. + - Add all the necessary environment variables especially those highlighted already in step 2. + - Click **Deploy**. + +At this point, you should have all services deployed and connected to the Postgres and Redis Database service: + + + +- **Cron Service**: This service should run Celery Beat Scheduler to manage scheduled tasks. + + + +- **Worker Service**: This service should be running Celery and ready to process jobs from the queue. + + + +- **App Service**: This service should be running and is the only one that should have a public domain, allowing users to access your application. + +**Note:** There is a [community template](https://railway.app/template/yZDfUu) available that demonstrates this deployment approach. You can easily deploy this template and then connect it to your own GitHub repository for your application. + ## Next Steps Explore these resources to learn how you can maximize your experience with Railway: