dummy example?

[vsoch]

hey @jkeifer ! I'm implementing a restful chunked upload, and hoping to give your module a try! Do you have a dummy example to go from?

[jkeifer]

@Vosch I don't have anything like that prepared at the moment, but I can certainly understand how that could be handy.

I'd be hard pressed to pull something together at the moment, just due to the time I have available right now. I can point you towards the project for which I made this module: https://github.com/PSU-CSAR/django-ebagis. You'll see it in use mainly under ebagis/models/upload.py and ebagis/views/upload.py. But I know that's FAR from a minimal example, and may not help you much.

I'll definitely add it to my todo list to get better documentation put together. Unfortunately, I don't have any idea when I will be able to get to that.

[vsoch]

hey are you still not able to help? I'm still struggling, and I am trying to reproduce your demo, and I'm stuck with this error:

uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django_filters/filterset.py", line 90, in __new__
uwsgi_1   |     new_class.base_filters = new_class.get_filters()
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django_filters/filterset.py", line 328, in get_filters
uwsgi_1   |     filters[filter_name] = cls.filter_for_field(field, field_name, lookup_expr)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django_filters/filterset.py", line 345, in filter_for_field
uwsgi_1   |     f, lookup_type = resolve_field(f, lookup_expr)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django_filters/utils.py", line 116, in resolve_field
uwsgi_1   |     query = model_field.model._default_manager.all().query
uwsgi_1   | AttributeError: 'NoneType' object has no attribute 'all'

[vsoch]

I just completely removed that attribute, and it seemed to restart the server without error, and I'm finally getting a response! It looks like it needs credentials?

 curl -v -X PUT -d "my_file=home/vanessa/e279432e6d3962777bb7b5e8d54f30f4347d867e.simg" "http://127.0.0.1/api/upload/chunked/2/"*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> PUT /api/upload/chunked/2/ HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 66
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 66 out of 66 bytes
< HTTP/1.1 401 Unauthorized
< Server: nginx/1.13.6
< Date: Thu, 14 Jun 2018 02:57:07 GMT
< Content-Type: application/json
< Content-Length: 58
< Connection: keep-alive
< WWW-Authenticate: Basic realm="api"
< Allow: GET, POST, PUT, HEAD, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Vary: Accept, Cookie
< 
* Connection #0 to host 127.0.0.1 left intact
{"detail":"Authentication credentials were not provided."}

is this possibly coming from the restful api? Do you have a sense of how to debug / fix?

[jkeifer]

Based on the error and your curl command, it looks like you need to be providing a username and token to authenticate your access to that upload endpoint. You may also be able to change your django-rest-framework permissions settings to not have user authentication globally or for this specific endpoint. Please see http://www.django-rest-framework.org/api-guide/permissions/ and http://www.django-rest-framework.org/api-guide/authentication/ for information on drf permissions and user authentication.

[vsoch]

Okay cool! I figured out that part, and now I'm authenticated with my token (I had done this before in python with requests, I am just using curl now and didn't add the header). Now it looks like I probably have an issue with specifying the file?

$ curl -v -X PUT --header "Transfer-Encoding: chunked" -d "my_file=/home/vanessa/ed9755a08ization: Token xxxxxxxxxxxxxxxx" "http://127.0.0.1/api/upload/chunked/2/"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> PUT /api/upload/chunked/2/ HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
> Transfer-Encoding: chunked
> Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Content-Type: application/x-www-form-urlencoded
> 
> 3b
* upload completely sent off: 66 out of 59 bytes
< HTTP/1.1 400 Bad Request
< Server: nginx/1.13.6
< Date: Thu, 14 Jun 2018 19:53:56 GMT
< Content-Type: application/json
< Content-Length: 40
< Connection: keep-alive
< Vary: Accept, Cookie
< Allow: GET, POST, PUT, HEAD, OPTIONS
< X-Frame-Options: SAMEORIGIN
< 
* Connection #0 to host 127.0.0.1 left intact
{"detail":"No chunk file was submitted"}

I also tried without the transfer encoding chunked bit.

[vsoch]

okay here is something! I triggered a 415 error:

curl -v -X PUT -H "Transfer-Encoding: chunked" -T /home/vanessa/ed9755a0871f04db3e14971bec56a33f.simg -H "Authorization: Token xxxxxxxxxxxxxxxxxxx" "http://127.0.0.1/api/upload/chunked/2/"

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> PUT /api/upload/chunked/2/ed9755a0871f04db3e14971bec56a33f.simg HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
> Transfer-Encoding: chunked
> Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 415 Unsupported Media Type
< Server: nginx/1.13.6
< Date: Thu, 14 Jun 2018 19:57:58 GMT
< Content-Type: application/json
< Content-Length: 52
< Connection: keep-alive
< Vary: Accept, Cookie
< Allow: GET, POST, PUT, HEAD, OPTIONS
< X-Frame-Options: SAMEORIGIN
< 
* transfer closed with 52 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 52 bytes remaining to read

[vsoch]

415 looks like content type... I wonder what it's expecting?

The HTTP 415 Unsupported Media Type client error response code indicates that the server refuses to accept the request because the payload format is in an unsupported format.

oh derp, it says it right above in the response, I need to read things :)

[vsoch]

Okay, not sure if this is progress:

$ curl -v -H "filename: /home/vanessa/e \755a0871f04db3e14971bec56a33f.simg" 
>         -H "Content-Type: multipart/form-data" \
>         -H "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxx" \
>         -F "file=@/home/vanessa/ed9755a0871f04db3e14971bec56a33f.simg;type=application/octet-stream" \
> -X PUT http://127.0.0.1/api/upload/chunked/2/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> PUT /api/upload/chunked/2/ HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
> filename: /home/vanessa/ed9755a0871f04db3e14971bec56a33f.simg
> Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxx
> Content-Length: 65347992
> Content-Type: multipart/form-data; boundary=------------------------00c775c2bb290050
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 400 Bad Request
< Server: nginx/1.13.6
< Date: Thu, 14 Jun 2018 20:12:47 GMT
< Content-Type: application/json
< Content-Length: 37
< Connection: keep-alive
< Vary: Accept, Cookie
< Allow: GET, POST, PUT, HEAD, OPTIONS
< X-Frame-Options: SAMEORIGIN
* HTTP error before end of send, stop sending
< 
* Closing connection 0
{"detail":"Error in request headers"}

I'm not sure where my_file needs to be designated?

[vsoch]

The only instruction I have to go by is very general:

An initial PUT request is sent to the url linked to ChunkedUploadView (or any subclass) with the first chunk of the file. The name of the chunk file can be overriden in the view (class attribute field_name).

Example:
{"my_file": file}

[vsoch]

:(

curl -v -H "Content-Type: multipart/form-data;" -H "Authorization: Token 2c9738d6eaa9802a7b7278deef5d5bacdd53fbdf" -H "Transfer-Encoding: chunked;" -F "filename=/home/vanessa/ed9755a0871f04db3e14971bec56a33f.simg"  -F "file=@/home/vanessa/ed9755a0871f04db3e14971bec56a33f.simg;type=application/octet-stream" -X PUT "http://127.0.0.1/api/upload/chunked/2/"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> PUT /api/upload/chunked/2/ HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
> Authorization: Token 2c9738d6eaa9802a7b7278deef5d5bacdd53fbdf
> Transfer-Encoding: chunked;
> Content-Type: multipart/form-data;; boundary=------------------------74c5f7a623b4a1d6
> Expect: 100-continue
> 
< HTTP/1.1 501 Not Implemented
< Server: nginx/1.13.6
< Date: Thu, 14 Jun 2018 20:28:31 GMT
< Content-Type: text/html
< Content-Length: 181
< Connection: close
< 
<html>
<head><title>501 Not Implemented</title></head>
<body bgcolor="white">
<center><h1>501 Not Implemented</h1></center>
<hr><center>nginx/1.13.6</center>
</body>
</html>
* Closing connection 0

[vsoch]

Thanks anyway, I wound up not using this in favor of nginx upload module that uploads directly via nginx for much faster uploads. I do appreciate the help that you gave, and hope that if you get a chance you do eventually get a dummy example. I'm happy to close the issue.