Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some services return a blank 200 page with Coraza enabled #71

Open
ericswpark opened this issue Jun 11, 2023 · 27 comments
Open

Some services return a blank 200 page with Coraza enabled #71

ericswpark opened this issue Jun 11, 2023 · 27 comments
Labels

Comments

@ericswpark
Copy link

ericswpark commented Jun 11, 2023

Some services that I use through the reverse proxy feature on Caddy break when I enable Coraza. One example that I've run into is with Overseerr

Reproduction steps

  • Set up Overseerr
  • Reverse proxy to Overseerr with Caddy
  • Verify site works without Coraza enabled
  • Enable Coraza with Caddyfile configuration

Expected behavior

Site loads properly and redirects to /login endpoint

Actual behavior

Site returns a blank 200 page. Developer tools say that no response was returned from the server.

No logs are generated to indicate why the request has failed.

Access logs say that a 307 response code was returned, but browsers report a 200 response code (tested with Firefox and Edge).

Configuration used

Caddyfile:

{
	order coraza_waf first
}

*.example.com {
	# Comment out this entire block to make Overseerr load properly
	coraza_waf {
		load_owasp_crs
		directives `
			Include @coraza.conf-recommended
			Include @crs-setup.conf.example
			Include @owasp_crs/*.conf
			SecRuleEngine On
		`
	}

	@overseerr host overseerr.example.com
	handle @overseerr {
		reverse_proxy overseerr:5055
	}
}

Logs

Caddy stdout:

{"level":"warn","ts":1686497529.0466342,"logger":"http.handlers.waf","msg":"failed to parse server name","tx_id":"__________","error":"failed to parse server name from authority \"overseerr.example.com\", address overseerr.example.com: missing port in address"}

Caddy access.log:

{
  "level": "info",
  "ts": 1686498117.092308,
  "logger": "http.log.access.log1",
  "msg": "handled request",
  "request": {
    "remote_ip": "[redacted]",
    "remote_port": "18232",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "overseerr.example.com",
    "uri": "/login",
    "headers": {
      "User-Agent": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
      ],
// --snip--
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "overseerr.example.com"
    }
  },
  "user_id": "",
  "duration": 0.029880206,
  "size": 0,
  "status": 307,
  "resp_headers": {
    "Date": [
      "Sun, 11 Jun 2023 15:41:57 GMT"
    ],
    "Server": [
      "Caddy"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "X-Powered-By": [
      "Express"
    ],
    "Location": [
      "/"
    ]
  }
}
@jcchavezs
Copy link
Member

jcchavezs commented Jun 11, 2023

Could you include in the directives and tell us about the logs?

			SecDebugLog /dev/stdout
			SecDebugLogLevel 9

@ericswpark
Copy link
Author

@jcchavezs Just tried adding those lines in the directive, but I didn't see any additional lines in either the Docker logs (stdout) or the access log.

@jcchavezs
Copy link
Member

Could you also I crease the log level in caddy? Coraza uses underneath the caddy log level so although you can set coraza log level to trace it depends on caddy's log level (we need to document this). Maybe set to debug?

@ericswpark
Copy link
Author

@jcchavezs the only log level I found was inside the log block, which I tried to increase from INFO (the default) to DEBUG. However there was no difference and nothing else was printed to the logs, other than what I posted above.

One interesting thing I found is that only the /login endpoint seems to be "blocked" by Coraza. I initially thought it was working again and then realized Overseerr had just remembered my session. As far as I can tell the rest of the endpoints work without any issues. Are there any rules in the default ruleset that overly scrutinize endpoints with the word "login" in it?

@jcchavezs
Copy link
Member

Interesting, any clue @M4tteoP @dune73

@dune73
Copy link

dune73 commented Jun 12, 2023

No clue. But you may want to test with curl to be really sure what is sent across the wire.

@ericswpark
Copy link
Author

OK, I checked a bit more with curl on Windows (which seems to be a PowerShell wrapper of some kind). At first I misread the output and thought everything was working as it should. So I tested some more and got the following behavior from the browsers:

  • With Coraza enabled
    • Logged in
      • / and all other subpaths -> Loads fine
      • /login -> returns blank 0B page with 200 status code
    • Logged out
      • / and all other subpaths -> returns blank 0B page with 200 status code
      • /login -> Loads fine
  • With Coraza disabled -> all endpoints work properly

So it seems like redirection is broken with Coraza, as Overseerr fails to redirect from / to /login if not logged in, and vice versa. I'm wondering if something in Coraza (or some ruleset) is rewriting something so that browsers get confused.

In fact, this is exactly what I was misreading from the curl output from before. When requesting the root / endpoint, curl returned the following:

StatusCode        : 200
StatusDescription : OK
Content           : {}
RawContent        : HTTP/1.1 200 OK
                    Alt-Svc: h3=":443"; ma=2592000
                    Transfer-Encoding: chunked
                    Date: Mon, 12 Jun 2023 12:37:21 GMT
                    Location: /login
                    Server: Caddy
                    X-Powered-By: Express


Headers           : {[Alt-Svc, h3=":443"; ma=2592000], [Transfer-Encoding, chunked], [Date, Mon, 12 Jun 2023 12:37:21
                    GMT], [Location, /login]...}
RawContentLength  : 0

We see that the 307 response sent by Caddy has been changed to 200, and the redirect location has been written to the Location: part of RawContent. I am assuming (but am not sure) that since Caddy responds with 200 the browsers don't attempt to follow this redirect location (or maybe this is not what is meant to be followed if Caddy does return a 307? Again, not sure). What I can tell is that the Coraza plugin is modifying what Caddy has initially sent, which breaks things down once Coraza is enabled.

@dune73
Copy link

dune73 commented Jun 12, 2023

Funny output for curl, but I confirm that a browser is very likely to ignore a Location header if it comes with status code 200.

Also, I do not like the Transfer-Encoding header instead of Content-Length. Not illegal per se, but unusual. Apparently depends on the application or Caddy or what not.

@jcchavezs
Copy link
Member

@ericswpark could you please try to reproduce the error using the example? https://github.com/corazawaf/coraza-caddy/blob/main/example/docker-compose.yml it uses httpbin which would help you to replicate whatever response the /login endpoint returns.

@ericswpark
Copy link
Author

@jcchavezs sorry, but the server that I'm running the services on doesn't have Docker Compose support.

I can try and replicate it on my local machine but that might take a while. If you guys want to test on your end, it's really easy as the Overseerr service is available as a Docker image and configuration required to reach the login/main page is pretty minimal.

@jcchavezs
Copy link
Member

@jcchavezs
Copy link
Member

jcchavezs commented Jun 12, 2023

I think this is related to #10 cc @jptosso

@KorvinSzanto
Copy link

KorvinSzanto commented Jun 23, 2023

Definitely seems like this is an issue in Coraza. If I comment out all Coraza related stuff from my caddyfile I start to see my 302's flow out properly.
One bizarre thing is HEAD requests flow through and get the proper 302 while GET requests get a 200 which is what causes the curl -I confusion.

@jcchavezs
Copy link
Member

@ericswpark @KorvinSzanto could you please try this branch #85 and check if this solves your problem?

@ericswpark
Copy link
Author

ericswpark commented Jul 8, 2023

@jcchavezs unfortunately I am unable to test this because I moved away from Caddy altogether (I also had some other issues unrelated to Coraza that I was unable to solve in time). Sorry :(

I will keep the issue open for @KorvinSzanto and others that are still running into the bug.

@KorvinSzanto
Copy link

It doesn't seem to be working but perhaps we're not running xcaddy properly. Can you should the xcaddy command you'd expect us to use to test out that branch?

@M4tteoP
Copy link
Member

M4tteoP commented Jul 10, 2023

Directly pointing to the specific commit should work fine: xcaddy build --with github.com/corazawaf/coraza-caddy/v2@5626d6c9562a5da600568acadc740b1427f043ed

@KorvinSzanto
Copy link

Using that commit we still see the 302 redirects flowing out as 200s so unfortunately still not working.

@rorylshanks
Copy link

rorylshanks commented Jul 19, 2023

Hi, we have also experienced this issue, and it appears to be related to the processResponse call. I have forked this repo with response processing disabled and this issue is "fixed". You can test it with

xcaddy build --with github.com/corazawaf/coraza-caddy/v2=github.com/rorylshanks/coraza-caddy/v2@main

Of course, this is not a real "fix", but rather a stopgap until this issue is fixed

@jcchavezs
Copy link
Member

jcchavezs commented Jul 19, 2023

Hi @KorvinSzanto and @megalan247 I just tried redirection following this local env https://github.com/corazawaf/coraza-caddy#local and calling and following the redirect locations as described in https://github.com/ahmetb/go-httpbin

  • curl -i localhost:8080/redirect/3
  • curl -i localhost:8080/absolute-redirect/3
  • curl -i "localhost:8080/redirect-to?url=https://google.com"

In the three of them I got this response from curl:

HTTP/1.1 302 Found
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Wed, 19 Jul 2023 18:59:59 GMT
Location: [redirect location]
Server: Caddy

But of course if fails if I use the reverse_proxy functionality. The problem here is that https://github.com/corazawaf/coraza-caddy/blob/main/interceptor.go#L121 this status code is never changed, meaning the Write(statusCode int) method is never been called from the reverse_proxy handler which makes me thing there is a mutation of the response somewhere directly rather than calling the http.ResponseWriter methods. Any thoughts @mholt?

UPDATE: after further investigations I realized I was doing the reverse proxy incorrectly. Now it works as expected.

@mholt
Copy link

mholt commented Jul 19, 2023

@jcchavezs Caddy wouldn't happen to be redirecting to https:// would it?

@jcchavezs
Copy link
Member

@jcchavezs Caddy wouldn't happen to be redirecting to https:// would it?

You mean because of curl -i "localhost:8080/redirect-to?url=https://google.com"? It is just an example.

@jcchavezs
Copy link
Member

I just tried the following:

{
	order coraza_waf first
}

:8082 {
    log {
        output stdout
    }

	# Comment out this entire block to make Overseerr load properly
	coraza_waf {
		load_owasp_crs
		directives `
			Include @coraza.conf-recommended
			Include @crs-setup.conf.example
			Include @owasp_crs/*.conf
			SecRuleEngine On
            SecDebugLogLevel 9
		`
	}

	@overseerr host *
	handle @overseerr {
		reverse_proxy :8083
	}
}

where :8083 is just an http server that returns a 302 and a Location and it did work correctly. Could any of you @KorvinSzanto or @megalan247 come up with a caddy file and a setup where I can reproduce the error?

@jptosso
Copy link
Member

jptosso commented Jul 19, 2023

This url simulates the same scenario and it works https://tosso.io/redirect?type=publication&id=2

@mholt
Copy link

mholt commented Jul 19, 2023

@jcchavezs In general, Caddy redirects HTTP to HTTPS when auto-HTTPS is engaged. This happens when a site is served with a domain name, for example. So in your reproducer config above, you use the site name of :8082 which doesn't have a domain, so auto HTTPS doesn't activate. Thus there will be no redirect to HTTPS.

I'm just guessing though, since the redirect location was redacted.

@rorylshanks
Copy link

rorylshanks commented Jul 20, 2023

An example of a broken config is

{
    order coraza_waf before reverse_proxy
    auto_https off
    admin   off
    http_port 2080
    servers :2080 {
        
        trusted_proxies static private_ranges
        
    }
}

:2080 {

log {
    output stdout
    format json
    level DEBUG
}

handle * {
    coraza_waf {
        directives `
            SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"
            SecRuleEngine DetectionOnly
        `
    }
    
    reverse_proxy https://mail.google.com {
        header_up X-Forwarded-For {header.X-Forwarded-For}
        header_down -Server
    }
}


}

And this returns for me

curl localhost:2080 -v -H "Host: mail.google.com"
*   Trying 127.0.0.1:2080...
* Connected to localhost (127.0.0.1) port 2080 (#0)
> GET / HTTP/1.1
> Host: mail.google.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=7776000
< Content-Security-Policy: frame-ancestors 'self'
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 20 Jul 2023 12:49:33 GMT
< Expires: Thu, 20 Jul 2023 12:49:33 GMT
< Location: /mail/
< Server: Caddy
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Transfer-Encoding: chunked
< 
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="/mail/">here</A>.
</BODY>
</HTML>
* Connection #0 to host localhost left intact

Notice the 200 OK with the Location header. Of course, curling https://mail.google.com directly, returns the same HTML and Location header, but with a 301 status set, which is correct.

@jptosso
Copy link
Member

jptosso commented Jul 20, 2023

I'm starting to believe that as @mholt mentioned, this is a TLS issue, I added the following to tosso.io:

tosso.io www.tosso.io {
    log {
        format console
        output file /logs/access.log
        level info
    }
    handle /experiment/redirect* {
        rewrite * /{uri}
        coraza_waf {
            directives `
                SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"
                SecRuleEngine DetectionOnly
            `
        }
        
        reverse_proxy https://mail.google.com {
            header_up X-Forwarded-For {header.X-Forwarded-For}
            header_down -Server
            header_up Host mail.google.com
        }
    }  
}

You can try here: https://tosso.io/experiment/redirect

Curl result:

* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 301 
...........
< content-length: 244
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/experiment/redirect/mail">here</A>.
</BODY></HTML>
* Connection #0 to host tosso.io left intact

This is my Dockerfile, updated today to latest:

FROM caddy:2.6-builder-alpine AS builder
RUN xcaddy build \
    --with github.com/corazawaf/coraza-caddy/v2@latest

FROM caddy:2.6-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants