Upload stalling with HTTPS

[oliverpool]

Probably similar to #1147

1. What version of Caddy are you running (caddy -version)?

0.9.5

2. What are you trying to do?

Upload large file (with caddy acting as reverse proxy in front of tus)

3. What is your entire Caddyfile?

tus.example.com {
  proxy localhost:3001 { #adress of the tus server
    transparent
  }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy.exe
on windows 10

5. What did you expect to see?

Upload of 10MB file coimpleting fine over https

6. What did you see instead (give full error messages and/or log)?

When trying on http, the upload works fine.
On https, the upload stalls at about 60%. From the client side, the JS doesn't send (no more progress events)

I don't know how I could dig deeper to debug this situation: any input is appreciated!

[oliverpool]

If I pause the upload, tus is capable of writing what has been sent until now: if I then resume the upload, everything works fine (the last 4MB are uploaded successfully)

[tobya]

@oliverpool This may be related to default timeouts set in 0.9.5 (they will be disabled in next release)
try

tus.example.com {
  timeouts 0
  proxy localhost:3001 { #adress of the tus server
   transparent
  }
}

[oliverpool]

This may be related to default timeouts set in 0.9.5 (they will be disabled in next release)
try

But then it should also fail on http (not only on https), no?

Anyway, I will try it tomorrow

[tobya]

@oliverpool ohh sorry didnt notice that, but it may be worth a try anyway.

[unknwon]

I have the exact same problem.

If I set timeouts none my client just hangs forever. Then if I upload with raw http://IP:port, everything goes fine. Also using 0.9.5, linux amd64, HTTPS.

[mholt]

@unknwon Is timeouts 0 or timeouts none set?

[unknwon]

@mholt yes,

domain.com {
        timeouts {
                read none
        }
        proxy / localhost:8086
}

[unknwon]

And just realize same for download, hangs on about 16.1MB.

[mholt]

@unknwon You've only set the read timeout to 0, set all of them to 0 and try again. Is the HTTPS upload any slower than the HTTP upload?

[unknwon]

Sorry about multiple comments so quickly, roll back to 0.9.4 immediately solved the problem.

You've only set the read timeout to 0, set all of them to 0 and try again.

At first I didn't set any timeouts, and it just hangs on upload about 13.8MB until it timeout.

[mholt]

roll back to 0.9.4 immediately solved the problem.

That's interesting. Would you be able to spare a few minutes to help debug this since I have nothing to reproduce this with from this issue so far?

Just brainstorming here... if you could try before these commits/PRs (like bisect?) and see if any of them are the culprit:

• proxy: use a new context for the outgoing request #1358
• ae10122
• proxy: Unbuffered request optimization #1314
• 4babe4b
• 153d4a5

[unknwon]

@mholt sure, but probably half an hour later...

[unknwon]

I can't enable HTTPS for local fake domain for all sorts of reason, it is giving me hard time to debug. And HTTP seems working fine (master).

Update: only happens for large content transfer with proxy directive, browse/root at 0.9.5 works fine.

[mholt]

@unknwon Even with tls self_signed for localhost? Should make it much easier.

@oliverpool Would you please be able to try Caddy at those commits/PRs I linked to above and see which one causes it?

[oliverpool]

@mholt & @unknwon : I found a preciser cause !

It seems to be the write timeout (tests with version 0.9.5, self-signed tls, read timeout set to none):

• when the write timeouts is set to none, everything works fine (http & https)
• when the write timeouts is unset (or set to some value smaller than my upload duration) strange things happen:
  • https: the upload simply stalls (no more progress whatsoever). When I interrupt the upload, the end of the request is sent to my tus server and everything seems fine
  • http: the upload goes to the end, but the tus server doubly receives the request and only the last one is sent back to my browser (which is a failure because the first one was properly proceeded by the tus server)

So a simple workaround is to correctly set the timeouts 😄

But there might still be a bug regarding what happens when a timeout occurs.

[oliverpool]

According to https://blog.cloudflare.com/exposing-go-on-the-internet/, there is a bug regarding readTimeout for http/2: golang/go#16450 in Go 1.7.

There is a similar problem with WriteTimeout. The example program will hang after WriteTimeout.

I will try to compile caddy with go 1.8 and update my bug report.

[oliverpool]

Compiled with go 1.8 (go version go1.8 windows/amd64):

The write timeout seems to be ignored (I set it to 5s and my upload is about 30s): everything works fine for http and for https (no stall / no double request) 🎉

[mholt]

@oliverpool Ah, excellent. I forgot about that bug mentioned in the blog post! 😁 Oops.

Okay, so for now, disable timeouts completely on Caddy 0.9.5 if you're having this problem. The next version will be built on Go 1.8 so that should resolve the bug (and default timeouts will be disabled anyway).

Thanks for figuring it out! 🎉

[oliverpool]

I will just use the one I compiled it with go 1.8 ;-)

Thank you for your support!

[michalmedvecky]

I can confirm this issue with 0.9.5. with https, upload fails after some time (502), with http, it just infinitely waits.

Downgrading to 0.9.4 helped.

Can I help debugging this issue?

[oliverpool]

Can I help debugging this issue?

Did you try with Go 1.8 ?

[michalmedvecky]

I'm using abiosoft/caddy docker image. No clue which go is it using, but Dockerfile looks like it's downloading the official binary from caddyserver.com.

[oliverpool]

Well then you will have to wait a bit (#1480 (comment) from mholt):

Okay, so for now, disable timeouts completely on Caddy 0.9.5 if you're having this problem. The next version will be built on Go 1.8 so that should resolve the bug (and default timeouts will be disabled anyway).

[iskrisis]

Is this something ongoing? I cant upload files above certain size. Also 'limits' directive Parse error: Unknown directive 'limits'.

Is there any way how to go around this?

[oliverpool]

@krisak which version of Caddy are you using ?

If this is happening with version 0.10.3, it might be a good idea to open a new issue.

[iskrisis]

Oh damn i am still 0.10.2. I will try to update and see whats going on.

Edit: Worked. Funny i thought that 0.10.2 was pretty bleeding edge and that limits directive is something loong in caddy because it is in docs :))