Some services return a blank 200 page with Coraza enabled

[ericswpark]

Some services that I use through the reverse proxy feature on Caddy break when I enable Coraza. One example that I've run into is with Overseerr

Reproduction steps

• Set up Overseerr
• Reverse proxy to Overseerr with Caddy
• Verify site works without Coraza enabled
• Enable Coraza with Caddyfile configuration

Expected behavior

Site loads properly and redirects to /login endpoint

Actual behavior

Site returns a blank 200 page. Developer tools say that no response was returned from the server.

No logs are generated to indicate why the request has failed.

Access logs say that a 307 response code was returned, but browsers report a 200 response code (tested with Firefox and Edge).

Configuration used

Caddyfile:

{
	order coraza_waf first
}

*.example.com {
	# Comment out this entire block to make Overseerr load properly
	coraza_waf {
		load_owasp_crs
		directives `
			Include @coraza.conf-recommended
			Include @crs-setup.conf.example
			Include @owasp_crs/*.conf
			SecRuleEngine On
		`
	}

	@overseerr host overseerr.example.com
	handle @overseerr {
		reverse_proxy overseerr:5055
	}
}

Logs

Caddy stdout:

{"level":"warn","ts":1686497529.0466342,"logger":"http.handlers.waf","msg":"failed to parse server name","tx_id":"__________","error":"failed to parse server name from authority \"overseerr.example.com\", address overseerr.example.com: missing port in address"}

Caddy access.log:

{
  "level": "info",
  "ts": 1686498117.092308,
  "logger": "http.log.access.log1",
  "msg": "handled request",
  "request": {
    "remote_ip": "[redacted]",
    "remote_port": "18232",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "overseerr.example.com",
    "uri": "/login",
    "headers": {
      "User-Agent": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
      ],
// --snip--
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "overseerr.example.com"
    }
  },
  "user_id": "",
  "duration": 0.029880206,
  "size": 0,
  "status": 307,
  "resp_headers": {
    "Date": [
      "Sun, 11 Jun 2023 15:41:57 GMT"
    ],
    "Server": [
      "Caddy"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "X-Powered-By": [
      "Express"
    ],
    "Location": [
      "/"
    ]
  }
}

[jcchavezs]

Could you include in the directives and tell us about the logs?

			SecDebugLog /dev/stdout
			SecDebugLogLevel 9

[ericswpark]

@jcchavezs Just tried adding those lines in the directive, but I didn't see any additional lines in either the Docker logs (stdout) or the access log.

[jcchavezs]

Could you also I crease the log level in caddy? Coraza uses underneath the caddy log level so although you can set coraza log level to trace it depends on caddy's log level (we need to document this). Maybe set to debug?

[ericswpark]

@jcchavezs the only log level I found was inside the log block, which I tried to increase from INFO (the default) to DEBUG. However there was no difference and nothing else was printed to the logs, other than what I posted above.

One interesting thing I found is that only the /login endpoint seems to be "blocked" by Coraza. I initially thought it was working again and then realized Overseerr had just remembered my session. As far as I can tell the rest of the endpoints work without any issues. Are there any rules in the default ruleset that overly scrutinize endpoints with the word "login" in it?

[jcchavezs]

Interesting, any clue @M4tteoP @dune73

[dune73]

No clue. But you may want to test with curl to be really sure what is sent across the wire.

[ericswpark]

OK, I checked a bit more with curl on Windows (which seems to be a PowerShell wrapper of some kind). At first I misread the output and thought everything was working as it should. So I tested some more and got the following behavior from the browsers:

• With Coraza enabled
  • Logged in
    • / and all other subpaths -> Loads fine
    • /login -> returns blank 0B page with 200 status code
  • Logged out
    • / and all other subpaths -> returns blank 0B page with 200 status code
    • /login -> Loads fine
• With Coraza disabled -> all endpoints work properly

So it seems like redirection is broken with Coraza, as Overseerr fails to redirect from / to /login if not logged in, and vice versa. I'm wondering if something in Coraza (or some ruleset) is rewriting something so that browsers get confused.

In fact, this is exactly what I was misreading from the curl output from before. When requesting the root / endpoint, curl returned the following:

StatusCode        : 200
StatusDescription : OK
Content           : {}
RawContent        : HTTP/1.1 200 OK
                    Alt-Svc: h3=":443"; ma=2592000
                    Transfer-Encoding: chunked
                    Date: Mon, 12 Jun 2023 12:37:21 GMT
                    Location: /login
                    Server: Caddy
                    X-Powered-By: Express


Headers           : {[Alt-Svc, h3=":443"; ma=2592000], [Transfer-Encoding, chunked], [Date, Mon, 12 Jun 2023 12:37:21
                    GMT], [Location, /login]...}
RawContentLength  : 0

We see that the 307 response sent by Caddy has been changed to 200, and the redirect location has been written to the Location: part of RawContent. I am assuming (but am not sure) that since Caddy responds with 200 the browsers don't attempt to follow this redirect location (or maybe this is not what is meant to be followed if Caddy does return a 307? Again, not sure). What I can tell is that the Coraza plugin is modifying what Caddy has initially sent, which breaks things down once Coraza is enabled.

[dune73]

Funny output for curl, but I confirm that a browser is very likely to ignore a Location header if it comes with status code 200.

Also, I do not like the Transfer-Encoding header instead of Content-Length. Not illegal per se, but unusual. Apparently depends on the application or Caddy or what not.

[jcchavezs]

@ericswpark could you please try to reproduce the error using the example? https://github.com/corazawaf/coraza-caddy/blob/main/example/docker-compose.yml it uses httpbin which would help you to replicate whatever response the /login endpoint returns.

[ericswpark]

@jcchavezs sorry, but the server that I'm running the services on doesn't have Docker Compose support.

I can try and replicate it on my local machine but that might take a while. If you guys want to test on your end, it's really easy as the Overseerr service is available as a Docker image and configuration required to reach the login/main page is pretty minimal.

[jcchavezs]

I will try with https://docs.overseerr.dev/getting-started/installation#docker

[jcchavezs]

I think this is related to #10 cc @jptosso

[KorvinSzanto]

Definitely seems like this is an issue in Coraza. If I comment out all Coraza related stuff from my caddyfile I start to see my 302's flow out properly.
One bizarre thing is HEAD requests flow through and get the proper 302 while GET requests get a 200 which is what causes the curl -I confusion.

[jcchavezs]

@ericswpark @KorvinSzanto could you please try this branch #85 and check if this solves your problem?

[ericswpark]

@jcchavezs unfortunately I am unable to test this because I moved away from Caddy altogether (I also had some other issues unrelated to Coraza that I was unable to solve in time). Sorry :(

I will keep the issue open for @KorvinSzanto and others that are still running into the bug.

[KorvinSzanto]

It doesn't seem to be working but perhaps we're not running xcaddy properly. Can you should the xcaddy command you'd expect us to use to test out that branch?

[M4tteoP]

Directly pointing to the specific commit should work fine: xcaddy build --with github.com/corazawaf/coraza-caddy/v2@5626d6c9562a5da600568acadc740b1427f043ed

[KorvinSzanto]

Using that commit we still see the 302 redirects flowing out as 200s so unfortunately still not working.

[rorylshanks]

Hi, we have also experienced this issue, and it appears to be related to the processResponse call. I have forked this repo with response processing disabled and this issue is "fixed". You can test it with

xcaddy build --with github.com/corazawaf/coraza-caddy/v2=github.com/rorylshanks/coraza-caddy/v2@main

Of course, this is not a real "fix", but rather a stopgap until this issue is fixed

[jcchavezs]

Hi @KorvinSzanto and @megalan247 I just tried redirection following this local env https://github.com/corazawaf/coraza-caddy#local and calling and following the redirect locations as described in https://github.com/ahmetb/go-httpbin

• curl -i localhost:8080/redirect/3
• curl -i localhost:8080/absolute-redirect/3
• curl -i "localhost:8080/redirect-to?url=https://google.com"

In the three of them I got this response from curl:

HTTP/1.1 302 Found
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Wed, 19 Jul 2023 18:59:59 GMT
Location: [redirect location]
Server: Caddy

But of course if fails if I use the reverse_proxy functionality. The problem here is that https://github.com/corazawaf/coraza-caddy/blob/main/interceptor.go#L121 this status code is never changed, meaning the Write(statusCode int) method is never been called from the reverse_proxy handler which makes me thing there is a mutation of the response somewhere directly rather than calling the http.ResponseWriter methods. Any thoughts @mholt?

UPDATE: after further investigations I realized I was doing the reverse proxy incorrectly. Now it works as expected.

[mholt]

@jcchavezs Caddy wouldn't happen to be redirecting to https:// would it?

[jcchavezs]

@jcchavezs Caddy wouldn't happen to be redirecting to https:// would it?

You mean because of curl -i "localhost:8080/redirect-to?url=https://google.com"? It is just an example.

[jcchavezs]

I just tried the following:

{
	order coraza_waf first
}

:8082 {
    log {
        output stdout
    }

	# Comment out this entire block to make Overseerr load properly
	coraza_waf {
		load_owasp_crs
		directives `
			Include @coraza.conf-recommended
			Include @crs-setup.conf.example
			Include @owasp_crs/*.conf
			SecRuleEngine On
            SecDebugLogLevel 9
		`
	}

	@overseerr host *
	handle @overseerr {
		reverse_proxy :8083
	}
}

where :8083 is just an http server that returns a 302 and a Location and it did work correctly. Could any of you @KorvinSzanto or @megalan247 come up with a caddy file and a setup where I can reproduce the error?

[jptosso]

This url simulates the same scenario and it works https://tosso.io/redirect?type=publication&id=2

[mholt]

@jcchavezs In general, Caddy redirects HTTP to HTTPS when auto-HTTPS is engaged. This happens when a site is served with a domain name, for example. So in your reproducer config above, you use the site name of :8082 which doesn't have a domain, so auto HTTPS doesn't activate. Thus there will be no redirect to HTTPS.

I'm just guessing though, since the redirect location was redacted.

[rorylshanks]

An example of a broken config is

{
    order coraza_waf before reverse_proxy
    auto_https off
    admin   off
    http_port 2080
    servers :2080 {
        
        trusted_proxies static private_ranges
        
    }
}

:2080 {

log {
    output stdout
    format json
    level DEBUG
}

handle * {
    coraza_waf {
        directives `
            SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"
            SecRuleEngine DetectionOnly
        `
    }
    
    reverse_proxy https://mail.google.com {
        header_up X-Forwarded-For {header.X-Forwarded-For}
        header_down -Server
    }
}


}

And this returns for me

curl localhost:2080 -v -H "Host: mail.google.com"
*   Trying 127.0.0.1:2080...
* Connected to localhost (127.0.0.1) port 2080 (#0)
> GET / HTTP/1.1
> Host: mail.google.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=7776000
< Content-Security-Policy: frame-ancestors 'self'
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 20 Jul 2023 12:49:33 GMT
< Expires: Thu, 20 Jul 2023 12:49:33 GMT
< Location: /mail/
< Server: Caddy
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Transfer-Encoding: chunked
< 
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="/mail/">here</A>.
</BODY>
</HTML>
* Connection #0 to host localhost left intact

Notice the 200 OK with the Location header. Of course, curling https://mail.google.com directly, returns the same HTML and Location header, but with a 301 status set, which is correct.

[jptosso]

I'm starting to believe that as @mholt mentioned, this is a TLS issue, I added the following to tosso.io:

tosso.io www.tosso.io {
    log {
        format console
        output file /logs/access.log
        level info
    }
    handle /experiment/redirect* {
        rewrite * /{uri}
        coraza_waf {
            directives `
                SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"
                SecRuleEngine DetectionOnly
            `
        }
        
        reverse_proxy https://mail.google.com {
            header_up X-Forwarded-For {header.X-Forwarded-For}
            header_down -Server
            header_up Host mail.google.com
        }
    }  
}

You can try here: https://tosso.io/experiment/redirect

Curl result:

* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 301 
...........
< content-length: 244
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/experiment/redirect/mail">here</A>.
</BODY></HTML>
* Connection #0 to host tosso.io left intact

This is my Dockerfile, updated today to latest:

FROM caddy:2.6-builder-alpine AS builder
RUN xcaddy build \
    --with github.com/corazawaf/coraza-caddy/v2@latest

FROM caddy:2.6-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy