Uploading a large file to a writeable gateway using POST does not work

[rklaehn]

Version information:

go-ipfs version: 0.4.13-cc01b7f
Repo version: 6
System version: amd64/darwin
Golang version: go1.9.2

Type:

Bug

Severity:

High

Description:

Adding a large file to a writeable IPFS gateway using HTTP POST does not work. This seems to be due to the fact that curl is chunking the upload. Either curl is not properly producing a chunked upload, or ipfs is not properly handling it. Given that curl is very widely used and as old as the hills, I suspect that ipfs is to blame.

From the ipfs error log, we get this error: ERROR commands/h: err: multipart: NextPart: unexpected EOF handler.go:285

Steps to reproduce:

• create a file with random data:

dd if=/dev/urandom of=example bs=1024 count=1024

• add the file

$ curl -v -d @example http://localhost:8080/ipfs/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /ipfs/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 510626
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Ipfs-Hash: QmUyyoux19fLWoPrNqkoJUJdjYrx1g1AoRAWLFAdReq41t
< Location: /ipfs/QmUyyoux19fLWoPrNqkoJUJdjYrx1g1AoRAWLFAdReq41t
< Date: Mon, 22 Jan 2018 12:35:34 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact

• get the data for the returned hash

$ ipfs get QmUyyoux19fLWoPrNqkoJUJdjYrx1g1AoRAWLFAdReq41t
Saving file(s) to QmUyyoux19fLWoPrNqkoJUJdjYrx1g1AoRAWLFAdReq41t
 498.79 KB / 498.79 KB [======================================================================================================================================================================] 100.00% 0s

• compare with original => file is truncated somehow

$ ls -l QmUyyoux19fLWoPrNqkoJUJdjYrx1g1AoRAWLFAdReq41t example 
-rw-r--r--  1 rklaehn  staff   510626 Jan 22 13:37 QmUyyoux19fLWoPrNqkoJUJdjYrx1g1AoRAWLFAdReq41t
-rw-r--r--  1 rklaehn  staff  1048576 Jan 22 13:35 example

[gosubpl]

Also, in the ipfs daemon log you get:

ERROR commands/h: err: multipart: NextPart: unexpected EOF handler.go:285

[gosubpl]

Also, using the multipart/form-data according to RFC2388 (option -F in curl) also does not work, as then the contents of the form are stored together with the file.

[magik6k]

I'm able to reproduce this, it looks like the sharness tests for writable gateway don't check if the hash matches with what ipfs add would produce.

[OriginLegend]

I appear to be seeing the same issue still when using libcurl through a C++ program.
I notice that this is a pretty old issue, is there any plans to fix it soon? Any work-arounds?

[eingenito]

Has this already been addressed? Is the issue here the handling of binary data by curl? I think you have to specify --data-binary instead of just plain -d to prevent curl's interpretation of data in the file:

% curl -v -d @example http://127.0.0.1:8080/ipfs/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> POST /ipfs/ HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 519956
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output
< Ipfs-Hash: QmXktdbBLY3s41WkLQFmKVoxuftQQf3G8dfrm6ZuzzY3vT
< Location: /ipfs/QmXktdbBLY3s41WkLQFmKVoxuftQQf3G8dfrm6ZuzzY3vT
< Date: Fri, 22 Feb 2019 22:11:25 GMT
< Content-Length: 0
<
* Connection #0 to host 127.0.0.1 left intact

Note the incorrect Content-Length header and compare with:

% curl -v --data-binary @example http://127.0.0.1:8080/ipfs/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> POST /ipfs/ HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 1048576
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output
< Ipfs-Hash: QmdoKUL6FLmoJWUi8bMENdi8YAjWfUJ1daVvsNGEBSFC4n
< Location: /ipfs/QmdoKUL6FLmoJWUi8bMENdi8YAjWfUJ1daVvsNGEBSFC4n
< Date: Fri, 22 Feb 2019 22:12:49 GMT
< Content-Length: 0
<
* Connection #0 to host 127.0.0.1 left intact

which is the correct hash:

% ipfs add -n example
added QmdoKUL6FLmoJWUi8bMENdi8YAjWfUJ1daVvsNGEBSFC4n example

I think the sharness test is correct incidentally.

[Stebalien]

@eingenito is correct. Encoding is the issue.

[Stebalien]

Note: I'm improving the tests in #6539.