Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] MODSECURITY_CRS_VERSION 4 with 50+ virtual hosts causes API reload timeout #1525

Open
2 tasks done
jeremyj opened this issue Sep 28, 2024 · 2 comments
Open
2 tasks done

[BUG] MODSECURITY_CRS_VERSION 4 with 50+ virtual hosts causes API reload timeout #1525

jeremyj opened this issue Sep 28, 2024 · 2 comments
Assignees
@fl0ppy-d1sk @TheophileDiot
Labels
bug Something isn't working

Comments

@jeremyj
Copy link

jeremyj commented Sep 28, 2024 •
edited
Loading

What happened?

When enabling Modsecurity CRS v4 the bunkerweb container takes time to test the nginx config and the reload API call from the scheduler times out.
This is tested on a 32GB 4 vCPU AWS machine.

How to reproduce?

Deploy a Bunkerweb stack on a host that has 50+ virtual hosts attached to the bw_services network and labeled with the following:

bunkerweb.REVERSE_PROXY_URL=/
bunkerweb.REVERSE_PROXY_HOST=http://container:80
bunkerweb.SERVER_NAME=virtualhost
bunkerweb.USE_REVERSE_PROXY=yes

Try changing the Modsecurity CRS from v3 to v4 either with an env variable or via the Web UI

Configuration file(s) (yaml or .env)

services:
  bunkerweb:
    image: bunkerity/bunkerweb:1.5.10
    ports:
      - 80:8080
      - 443:8443
    labels:
      - "bunkerweb.INSTANCE=yes"
    environment:
      - SERVER_NAME=
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - AUTOCONF_MODE=yes
      - MULTISITE=yes
      - API_WHITELIST_IP=127.0.0.0/8 10.20.30.0/24 
      - AUTO_LETS_ENCRYPT=yes
      - EMAIL_LETS_ENCRYPT=xxxxxxxxxxxx
      - USE_LIMIT_REQ=no
      - WHITELIST_IP=
      - USE_CROWDSEC=yes
      - CROWDSEC_API=http://10.10.10.253:8080 # This is the address of the CrowdSec container API in the same network
      - CROWDSEC_API_KEY=xxxxxxxxxxxx # Remember to set a stronger key for the bouncer
      - MODSECURITY_CRS_VERSION=3
      - USE_MODSECURITY=yes
      - USE_MODSECURITY_CRS=yes
      - EXTERNAL_PLUGIN_URLS=https://github.com/bunkerity/bunkerweb-plugins/archive/refs/tags/v1.6.zip      
      - LISTEN_STREAM=no
      - SERVE_FILES=no
      - USE_BROTLI=yes
      - USE_GZIP=yes
      - USE_CLIENT_CACHE=yes
    networks:
      - bw-universe
      - bw-services
      - bw-plugins
    logging:
      driver: syslog
      options:
        syslog-address: "udp://10.10.10.254:514"

  bw-autoconf:
    image: bunkerity/bunkerweb-autoconf:1.5.10
    depends_on:
      - bunkerweb
      - bw-docker
    environment:
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - AUTOCONF_MODE=yes
      - DOCKER_HOST=tcp://bw-docker:2375
    networks:
      - bw-universe
      - bw-docker

  bw-scheduler:
    image: bunkerity/bunkerweb-scheduler:1.5.10
    depends_on:
      - bunkerweb
      - bw-docker
    environment:
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - DOCKER_HOST=tcp://bw-docker:2375
      - AUTOCONF_MODE=yes
      - LOG_LEVEL=info
    networks:
      - bw-universe
      - bw-docker

  bw-docker:
    image: tecnativa/docker-socket-proxy:nightly
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - CONTAINERS=1
      - LOG_LEVEL=warning
    networks:
      - bw-docker

  bw-db:
    image: mariadb:10.10
    environment:
      - MYSQL_RANDOM_ROOT_PASSWORD=yes
      - MYSQL_DATABASE=db
      - MYSQL_USER=bunkerweb
      - MYSQL_PASSWORD=xxxxxxxxxxxx
    volumes:
      - bw-data:/var/lib/mysql
    networks:
      - bw-docker

  bw-ui:
    image: bunkerity/bunkerweb-ui:1.5.10
    networks:
      bw-docker:
      bw-universe:
        aliases:
          - bw-ui
    environment:
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - DOCKER_HOST=tcp://bw-docker:2375
      - AUTOCONF_MODE=yes
      - ADMIN_USERNAME=xxxxxxxxxxxx
      - ADMIN_PASSWORD=xxxxxxxxxxxx
    labels:
      - "bunkerweb.SERVER_NAME=xxxxxxxxxxxx"
      - "bunkerweb.USE_UI=yes"
      - "bunkerweb.USE_REVERSE_PROXY=yes"
      - "bunkerweb.REVERSE_PROXY_URL=/path"
      - "bunkerweb.REVERSE_PROXY_HOST=http://bw-ui:7000"
      - "bunkerweb.MAX_CLIENT_SIZE=50m"

  crowdsec:
    image: crowdsecurity/crowdsec:v1.6.3 # Use the latest version but always pin the version for a better stability/security
    volumes:
      - cs-data:/var/lib/crowdsec/data # To persist the CrowdSec data
      - bw-logs:/var/log:ro # The logs of BunkerWeb for CrowdSec to parse
      - /docker/conf/bw-conf/acquis.yaml:/etc/crowdsec/acquis.yaml # The acquisition file for BunkerWeb logs
    environment:
      BOUNCER_KEY_bunkerweb: "xxxxxxxxxxxx" # Remember to set a stronger key for the bouncer
      COLLECTIONS: "crowdsecurity/nginx crowdsecurity/wordpress crowdsecurity/base-http-scenarios" # If you don't want to use the AppSec Component use this line instead
      LEVEL_INFO: true
      ENROLL_INSTANCE_NAME: "xxxxxxxxxxxx"
    networks:
      bw-plugins:
          ipv4_address: 10.10.10.253

  syslog:
    image: balabit/syslog-ng:4.8.0 # Use the latest version but always pin the version for a better stability/security
    command: --no-caps
    volumes:
      - bw-logs:/var/log # The logs of BunkerWeb for syslog-ng to store
      - /docker/conf/bw-conf/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf # The syslog-ng configuration file
    networks:
      bw-plugins:
        ipv4_address: 10.10.10.254 # The IP address of the syslog service so BunkerWeb can send logs to it

volumes:
  bw-data:
  bw-logs:
  cs-data:

networks:
  bw-universe:
    name: bw-universe
    ipam:
      driver: default
      config:
        - subnet: 10.20.30.0/24
  bw-plugins:
    ipam:
      driver: default
      config:
        - subnet: 10.10.10.0/24
  bw-services:
    name: bw-services
  bw-docker:
    name: bw-docker

Relevant log output

[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Instances changed, generating ...
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [🐛] - Changes: {'custom_configs_changed': False, 'external_plugins_changed': False, 'pro_plugins_changed': False, 'instances_changed': True, 'last_custom_configs_change': datetime.datetime(2024, 9, 27, 17, 12, 31), 'last_external_plugins_change': datetime.datetime(2024, 9, 27, 17, 11, 22), 'last_pro_plugins_change': datetime.datetime(2024, 9, 27, 17, 11, 22), 'last_instances_change': datetime.datetime(2024, 9, 27, 17, 18), 'plugins_config_changed': {}}
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [🐛] - Testing write access to the database ...
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Removing old custom configs files ...
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Sending /etc/bunkerweb/configs folder ...
[2024-09-27 17:18:01 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/custom_configs
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Successfully sent /etc/bunkerweb/configs folder
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Generator started ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Settings : /usr/share/bunkerweb/settings.json
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Templates : /usr/share/bunkerweb/confs
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Core : /usr/share/bunkerweb/core
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Plugins : /etc/bunkerweb/plugins
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Pro plugins : /etc/bunkerweb/pro/plugins
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Output : /etc/nginx
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Target : /etc/nginx
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Variables : /var/tmp/bunkerweb/scheduler.env
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Checking arguments ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Computing config ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Removing old files ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Rendering templates ...
[2024-09-27 17:18:09 +0000] [GENERATOR] [691] [ℹ ] - Generator successfully executed !
[2024-09-27 17:18:09 +0000] [SCHEDULER] [130] [ℹ ] - Sending /etc/nginx folder ...
[2024-09-27 17:18:09 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/cache/bunkerweb folder ...
[2024-09-27 17:18:10 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/confs
[2024-09-27 17:18:10 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/cache
[2024-09-27 17:18:10 +0000] [SCHEDULER] [130] [ℹ ] - Successfully sent /var/cache/bunkerweb folder
[2024-09-27 17:18:40 +0000] [API] [130] [❌] - Can't send API request to http://bunkerweb-bunkerweb-1:5000/reload : Request failed: HTTPConnectionPool(host='bunkerweb-bunkerweb-1', port=5000): Read timed out. (read timeout=30)
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [❌] - Error while reloading bunkerweb, failing over to last working configuration ...
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/tmp/bunkerweb/failover/config folder ...
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/tmp/bunkerweb/failover/cache folder ...
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/tmp/bunkerweb/failover/custom_configs folder ...
[2024-09-27 17:18:40 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/custom_configs
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Successfully sent /var/tmp/bunkerweb/failover/custom_configs folder
[2024-09-27 17:18:41 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/confs
[2024-09-27 17:18:51 +0000] [API] [130] [❌] - Can't send API request to http://bunkerweb-bunkerweb-1:5000/cache : Request failed: ('Connection aborted.', TimeoutError('timed out'))
[2024-09-27 17:18:51 +0000] [SCHEDULER] [130] [❌] - Error while sending /var/tmp/bunkerweb/failover/cache folder
[2024-09-27 17:19:16 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/reload
[2024-09-27 17:19:16 +0000] [SCHEDULER] [130] [ℹ ] - Executing job scheduler ...


bwapi 10.20.30.4 - - [27/Sep/2024:17:18:01 +0000] "POST /custom_configs HTTP/1.1" 200 80 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:09 +0000] "POST /confs HTTP/1.1" 200 67 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:10 +0000] "POST /cache HTTP/1.1" 200 78 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:40 +0000] "POST /custom_configs HTTP/1.1" 200 80 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:41 +0000] "POST /confs HTTP/1.1" 200 67 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:10 +0000] "POST /reload HTTP/1.1" 200 58 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:56 +0000] "POST /reload HTTP/1.1" 200 58 "-" "bwapi"
2024/09/27 17:19:16 [warn] 6633#6633: *938 [API] call from 10.20.30.4 on /cache failed : client aborted, client: 10.20.30.4, server: bwapi, request: "POST /cache HTTP/1.1", host: "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:56 +0000] "POST /cache HTTP/1.1" 400 53 "-" "bwapi"

BunkerWeb version

1.5.10

What integration are you using?

Docker

Linux distribution (if applicable)

No response

Removed private data

  • I have removed all private data from the configuration file and the logs

Code of Conduct

  • I agree to follow this project's Code of Conduct
@jeremyj jeremyj added the bug Something isn't working label Sep 28, 2024
@TheophileDiot
Copy link
Member

Hi, thank you for opening this Issue and sorry for the late response. Indeed the problem is present and is caused by ModSecurity. We'll dig into it and let you know about our findings.
AFAIK it's because of ModSecurity but we'll have a deeper look!

TheophileDiot added a commit that referenced this issue Oct 23, 2024
@TheophileDiot
[#1525] Add the possibility to defined a custom timeout to api calls …
4abb20b 
…and set a linear one depending on the number of services upon reload
@TheophileDiot
Copy link
Member

TheophileDiot commented Oct 23, 2024 •
edited
Loading

Hi @jeremyj, this fixes the issue while we dig a little deeper on how we can optimize NGINX's tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fl0ppy-d1sk fl0ppy-d1sk

@TheophileDiot TheophileDiot

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jeremyj @fl0ppy-d1sk @TheophileDiot