Bottle app running on Paste or CherryPy or any other server cannot handle chunked input

[asrivast13]

I am trying to run this curl command:
curl -H'Transfer-Encoding: chunked' -F'audio=@bar.wav' -F'text={"key": "value"}' -F'image=@foo.jpg' localhost:8085/upload

against this application:

---

import paste
from bottle import Bottle, route, error, post, get, response, request, template
app = Bottle()
@app.post('/upload')
def upload():
print "Files: ", request.files['image'].filename, "|", request.files['audio'].filename
print "Forms: ", request.forms['text']
print "Content-Length: ", request.headers['Content-Length']
print "Content-Type: ", request.headers['Content-Type']
print "Contents of File [", request.files['image'].name, "]: ", request.files['image'].file.readlines()
print "Contents of File [", request.files['audio'].name, "]: ", request.files['audio'].file.readlines()
return {"message": "OK"}

---

When I use the internal wsgiref server in Bottle to run this code as:
app.run(host='localhost', port=8085, reloader=True)
the files upload OK and the output gets printed but if I use the third-part server ('paste' for example) as
app.run(server='paste', host='localhost', port=8085, reloader=True)
then I get an error saying "Error while parsing chunked transfer body."

What am I doing wrong?

Thanks,
Amit

[pepelisu]

Hi,
I have tested the code and the problem is there. I suspect that it is due to wsgi server, with cherrypy bottle uses CherryPyWSGIServer that is a subclass of HTTPServer, not sure if this server can manage chunked transfers. According to this issue in flask (pallets/flask#367) WSGI cannot process chunked encoding. So some trick has to be performed.
Cherrypy has a class for deal with chunked encoding : class cherrypy.wsgiserver.ChunkedRFile(rfile, maxlen, bufsize=8192)https://cherrypy.readthedocs.io/en/3.2.6/refman/wsgiserver/init.html
But not sure if can be used in bottle.
Here more info about how Content-Length is managed in the default WSGI server used by bottle: https://www.python.org/dev/peps/pep-3333/#handling-the-content-length-header
Here more about transfer encoding (w3c)https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1

[defnull]

You are not doing anything wrong. See #700

WSGI requires servers to parse chunked transfers and provide the WSGI application with a continuous stream of data. Most WSGI servers do that, but some (e.g. paste and gevent) fail to remove the Transfer-Encoding header afterwards. Bottle sees the header and tries to parse the chunked data a second time, which fails because the data is no longer chunked.

In short: Paste is lying to bottle about the transfer encoding. The fault is still on our side: The WSGI spec does not require a server to remove these headers. Bottle should not try to parse chunked transfers in WSGI environments and completely ignore the header.