V2 - Disable Services #140
Conversation
| env_file: | ||
| - .env | ||
| environment: | ||
| AIRFLOW__CORE__EXECUTOR: LocalExecutor | ||
| AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://postgres:$PG_PASSWORD@db:5432/postgres |
There was a problem hiding this comment.
Can we actually do this part without impacting airflow? I thought this refers to airflow using the database to store information about jobs. It's possible this isn't actually happening so everything will be fine though.
There was a problem hiding this comment.
Ah this makes sense, I thought it was using this only for access inside the container network. I'll have a look at the airflow docs tomorrow and see what's the minimum we need for it to work well in production.
docker-compose.yml
Outdated
| db: | ||
| image: ghcr.io/synthetixio/data/postgres:${VERSION} | ||
| indexer: | ||
| build: | ||
| context: ./postgres | ||
| dockerfile: Dockerfile | ||
| restart: always | ||
| networks: | ||
| - data | ||
| environment: | ||
| POSTGRES_PASSWORD: $PG_PASSWORD | ||
| PGDATA: /var/lib/postgresql/data | ||
| POSTGRES_SHARED_BUFFERS: "4GB" | ||
| POSTGRES_WORK_MEM: "100MB" | ||
| POSTGRES_MAINTENANCE_WORK_MEM: "1GB" | ||
| POSTGRES_EFFECTIVE_CACHE_SIZE: "8GB" | ||
| volumes: | ||
| - ./postgres/initdb:/docker-entrypoint-initdb.d | ||
| - ./postgres-data:/var/lib/postgresql/data | ||
| - ./parquet-data:/parquet-data | ||
| ports: | ||
| - "${DB_PORT}:5432" | ||
| deploy: | ||
| resources: | ||
| limits: | ||
| cpus: "4.0" | ||
| memory: 8192M | ||
| shm_size: 4gb | ||
| tmpfs: | ||
| - /dev/shm:size=4g |
There was a problem hiding this comment.
Need to check if this impacts airflow. If it does, I think we can scale it down significantly but may need to keep some of this.
|
@Tburm I've added the postgres service again, will test it out and see how much we can downscale it once I have the pipeline setup done for the scheduler part |
this is my favorite part
|
docker-compose.yaml