HTTP Response Json error #3036

sachitsac · 2012-11-21T06:10:42Z

Hi,

I am using ZF2.0.3.

I am working with a webservice that return a json response. I have used this webservice successfully using curl and i seem to get the correct response. But when i use zend HTTP Client and try to do a json decode on response->getContent, i get json error 4 (syntax error).

when i print the response, it looks like this:

1890e
{json......}
0

For some reason, the valid json is being wrapped around content that i have no idea where its getting. I checked in the the Zend\Http\Response and tried to dump what's held in the string that is passed in the fromString static method and the output looks like this:

Date: Wed, 21 Nov 2012 05:54:58 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-1ubuntu4.10
Content-Language: en
Content-Encoding: identity
Connection: close
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8

1890e
{"coreEntity": ........}
0

I am not sure if this is a bug with zend http client response or there is something i am doing wrong?

The json is build using associative array and then doing a json_encode on the associative array. Also, this only occurs for large json data. If the json data is not large, it works ok. But if the json data is large, it gets those unwanted items in the response content.

Any help is much appreciated.

Thanks
Sachit

sachitsac · 2012-12-04T23:51:31Z

Hi Guys,

Though i left out an important information. This error occurs when i use the default adapter. i.e. Zend\Http\Client\Adapter\Socket. IT works with curl adapter. I haven't tried other adapters as curl adapter works for me.

Cheers.

jcrawford · 2012-12-05T00:56:42Z

@sachitsac Could you provide the full output from an HTTP request to the API which you are trying to consume? I have not been able to replicate locally as the following code works fine for me.

  <?php
  date_default_timezone_set('America/New_York');
  require_once('vendor/autoload.php');

  $options = array(
     'maxredirects' => 10,
     'timeout' => 60,
 );

 $client = new Zend\Http\Client();

 $client->setUri('http://home.josephcrawford.com/');
 $client->setMethod('GET');
 $response = $client->send($client->getRequest());
 var_dump(json_decode($response->getContent())); 
 exit;

http://home.josephcrawford.com/ outputs the following

{"foo":"faz","0":{"bar":"baz"}}

If you could provide more information I could diagnose and fix the issue.

Here is the output of me running the test script

jcrawford@Workstation:~ # php test2.php 
object(stdClass)#10 (2) {
  ["foo"]=>
  string(3) "faz"
  ["0"]=>
  object(stdClass)#11 (1) {
    ["bar"]=>
    string(3) "baz"
  }
}

sachitsac · 2012-12-05T01:19:23Z

Hi @jcrawford ,

It works fine for small json response. But as the response gets larger, the embeds those extra unwanted strings.

Here is the full response from the webservice.

sachitsac · 2012-12-05T01:21:59Z

The comment was probably too long so some of the response is chopped off. Can i attach it as a file somewhere ??

jcrawford · 2012-12-05T01:34:59Z

if you could pastebin.com, pastie.org it and set never expire and post the link here please.

jcrawford · 2012-12-05T01:35:37Z

@sachitsac please do not post the Zend Http Response, rather please use Live HTTP Headers to capture the headers and post the body content as well, the zend http response will not provide me with a json structure to test with :)

jcrawford · 2012-12-05T01:36:51Z

@sachitsac is this a public api I could test against?

sachitsac · 2012-12-05T01:52:30Z

Hi @jcrawford

Here is the link to pastebin containing the json response.

http://pastebin.com/hYM6iWBY

No, the version of api is not public yet so i can't provide any links to it.

Let me know if you need anything else.

Thanks for your help.

jcrawford · 2012-12-05T01:55:04Z

@sachitsac that output will work, can you also by chance get the headers for me so I can replicate those as well?

sachitsac · 2012-12-05T02:13:40Z

Hi @jcrawford

Is this what you are after ??

Cheers.

Response Headers

Connection  Keep-Alive
Content-Encoding    gzip
Content-Length  320
Content-Type    text/html
Date    Wed, 05 Dec 2012 02:07:58 GMT
Keep-Alive  timeout=15, max=99
Server  Apache/2.2.22 (Ubuntu)
Vary    Accept-Encoding
X-Powered-By    PHP/5.3.10-1ubuntu3.4

Request Headers

Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection  keep-alive
Cookie  PHPSESSID=o9um7h9en44ua42jok37dtoq66
DNT 1
User-Agent  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/20100101 Firefox/17.0

jcrawford · 2012-12-05T03:42:37Z

@sachitsac @weierophinney This has been resolved in PR #3169

sachitsac · 2012-12-05T03:46:11Z

Thanks @jcrawford for the fix. Appreciate the help.

Cheers.

Close #3169 Fixes #3036

DrBeza · 2013-01-03T12:46:23Z

I have just upgraded from 2.0.3 to 2.0.6 and this change has caused an issue with my API calls.

I've been looking into this and it seems that the change $chunk = ''; causes the returned $response to omit a single line above each chunk of data that contains the chunk size and CRLF.

This seems to cause decodeChunkedBody() to malform the returned data (it seems to be losing chunks of data). The $body value is as expected and $decBody is malformed.

If I revert the line $chunk = $line; in Socket.php the API calls work as expected.

Looking at the initial issue by Sachit it seems the response object's content still contains the chunk size and CRLF - so I'm wondering if the problem was in decodeChunkedBody() and not in Socket.php.

iwalz · 2013-01-03T13:17:23Z

Same for me after updating to 2.0.6. Before, I had the chunk data as a string in the responses, now I don't have that anymore, but got the following error:

PHP Fatal error:  Uncaught exception 'Zend\Http\Exception\RuntimeException' with message 'Error parsing body - doesn't seem to be a chunked message' in /var/www/zendserverapi/vendor/zendframework/zend-http/Zend/Http/Response.php:449

Stack trace:

#0 /var/www/zendserverapi/vendor/zendframework/zend-http/Zend/Http/Response.php(303): Zend\Http\Response->decodeChunkedBody('<zendServerAPIR...')

#1 /var/www/zendserverapi/library/ZendService/ZendServerAPI/Adapter/DumpParser.php(32): Zend\Http\Response->getBody()

#2 /var/www/zendserverapi/library/ZendService/ZendServerAPI/Method/Method.php(146): ZendService\ZendServerAPI\Adapter\DumpParser->parse(NULL)

#3 /var/www/zendserverapi/library/ZendService/ZendServerAPI/Request.php(296): ZendService\ZendServerAPI\Method\Method->parseResponse()

#4 /var/www/zendserverapi/library/ZendService/ZendServerAPI/Jobqueue.php(60): ZendService\ZendServerAPI\Request->send()

#5 /var/www/zendserverapi/debug.php(9): ZendService\ZendServerAPI\Jobqueue->jobqueueJobsList()

#6 {main}

thrown in /var/www/zendserverapi/vendor/zendframework/zend-http/Zend/Http/Response.php on line 449

jcrawford · 2013-01-03T13:31:37Z

I will see if i can resolve this issue this evening

iwalz · 2013-01-03T13:40:09Z

Reverting this line solves my issue too. The reason seem to be, that deocdeChunkedBody is called via getBody in the response. At that time, it needs the chunk line to determinate the size. But if this line is removed via the socket adapter, that will always fail (decodeChunkedBody in \Zend\Http\Response):

while (trim($body)) {
   if (! preg_match("/^([\da-fA-F]+)[^\r\n]*\r\n/sm", $body, $m)) {
        throw new Exception\RuntimeException(
           "Error parsing body - doesn't seem to be a chunked message"
       );
   }

  $length   = hexdec(trim($m[1]));
  $cut      = strlen($m[0]);
  $decBody .= substr($body, $cut, $length);
  $body     = substr($body, $cut + $length + 2);
}

dkidd · 2013-01-09T16:46:34Z

I came across the following in Zend\Http\Client\Adapter\Curl::write this morning:

// cURL automatically decodes chunked-messages, this means we have to disallow the Zend\Http\Response to do it again
        if (stripos($this->response, "Transfer-Encoding: chunked\r\n")) {
            $this->response = str_ireplace("Transfer-Encoding: chunked\r\n", '', $this->response);
        }

If Zend\Http\Client\Adapter\Curl automatically decodes chunked responses, why not have Zend\Http\Client\Adapter\Socket behave the same way (i.e., the way it has since 2.0.6)?

It seems to me that there are two possible solutions:

have Zend\Http\Client\Adapter\Socket remove the "Transfer-Encoding" header (like Zend\Http\Client\Adapter\Curl does), or
have both adapters leave the Transfer-Encoding header alone, not attempt a second decoding in Zend\Http\Response::getBody(), and deprecate Zend\Http\Response::deocdeChunkedBody().

BinaryKitten · 2013-01-09T16:58:48Z

There is a fix for the Zend\Http\Client\Adapter\Socket

has also been reported #3267
The Error occurred with @66f269f - reverting this commit "fixes" the error in regards to chunked body.

Not sure what the fix was meant for or if this was intended or not.

dkidd · 2013-01-09T21:05:04Z

@BinaryKitten @weierophinney The fix (i.e., #3169) was meant to solve this issue (#3036) as well as #3143. In both those cases, Zend\Http\Response::getContent() was invoked when Zend\Http\Response::getBody() would have been more appropriate.

When special encoding ("chunked", "gzip", or "deflate") is not used, Zend\Http\Response::getContent() and Zend\Http\Response::getBody() return the same result. Compounding the risk of confusion, the Zend\Http\Client\Adapter\Curl automatically decodes chunked responses and strips out the "Transfer-Encoding" header, making it even more likely that getContent and getBody will return the same result.

My earlier suggestions notwithstanding, I think reverting to the pre-PR #3169 behavior is probably the best course of action. @sachitsac and I will just have to update our code to use getBody like we should have in the first place.

runphp · 2013-01-11T07:33:02Z

throw Error parsing body - doesn't seem to be a chunked message

I test change code Zend/Http/Response.php

protected function decodeChunkedBody($body)
{ return $body

was ok!

I think it was already decoded,

my server is nginx php-fpm

DrBeza · 2013-01-11T10:56:42Z

@runphp it is not decoded, it just looks okay - an earlier commit 66f269f has caused each chunk recieved to lose the first line that specifies the length of the chunk and CRLF (carriage return followed by line feed).

weierophinney · 2013-01-11T17:44:46Z

Can somebody -- @DrBeza @BinaryKitten @dkidd -- give me a PR that resolves the issue? Ideally, one that also solves the original issue @jcrawford was trying to solve...

stoefln · 2013-05-13T13:26:06Z

In 2.2 RC3 i am still/again experiencing something like this:
The content looks like this:

10db
{ "foo": "bar" }
0

Here is my code:

        $client = new Client();
        $client->setUri("http://cutoutcam.com/jsontest.php");
        $client->setOptions(array(
            'maxredirects' => 2,
            'timeout' => 30
        ));
        $response = $client->send();
        print_r($response);

weierophinney · 2013-05-13T14:10:32Z

@stoefln - I cannot reproduce the error using code from current master (which is unchanged since RC3). Most likely it's an issue of environment -- perhaps a proxy?

dkidd · 2013-05-13T14:11:15Z

@stoefln: I'm not in a position to test this right now, but if 2.2 behaves like 2.0 did, your need to call the getBody method of $response in order to get the decoded response string.

E.g.:

$json = $response->getBody();
print_r($json);

should give you

{ "foo": "bar"}

stoefln · 2013-05-13T14:23:37Z

That did the job! So is it a bug then, or not?

weierophinney · 2013-05-13T14:30:29Z

@stoefln Not a bug -- getBody() is the correct, documented way to use the response and get the decoded body.

stoefln · 2013-05-13T14:39:11Z

OK, thanks!

thadudexx · 2013-09-05T09:28:37Z

If its not a bug the getBody() function should be documented for Zend/Http/Response, its not even mentioned at the moment. I am sure that would save a lot of time to newcomers.

weierophinney · 2013-09-05T14:40:48Z

@thadudexx You can edit the relevant documentation here: https://github.com/zendframework/zf2-documentation/edit/master/docs/languages/en/modules/zend.http.response.rst

eddiejaoude · 2014-03-25T15:35:19Z

I am running ZF v2.2.* "zendframework/zendframework" : "2.2.*" & I get this issue with the Http Socket Adapter with Nginx where the body content being prepended with 44e ... json. With Apache or with another Adapter (Curl) I do not get this issue.

On Nginx I was able to fix it by turning off chunked_transfer_encoding

…length of chunk was added to a variable which was re-used. I removed the connection between variables and the issue went away

Close zendframework/zendframework#3169 Fixes zendframework/zendframework#3036

Close zendframework/zendframework#3412 Fixes zendframework/zendframework#3036 Fixes zendframework/zendframework#3267

sachitsac closed this as completed Dec 5, 2012

sachitsac reopened this Dec 5, 2012

sachitsac closed this as completed Dec 5, 2012

jcrawford mentioned this issue Dec 7, 2012

Zend\Http\Client\Adaper\Socket does not remove chunk size lines from chunked response #3143

Closed

weierophinney added a commit that referenced this issue Dec 11, 2012

Merge branch 'hotfix/3169'

a755309

Close #3169 Fixes #3036

weierophinney reopened this Jan 3, 2013

dkidd mentioned this issue Jan 11, 2013

Restored chunk delimiters to chunked response received by Zend\H... #3412

Closed

weierophinney closed this as completed in df38c0e Jan 11, 2013

thadudexx mentioned this issue Oct 8, 2013

Added getBody description zendframework/zf2-documentation#1055

Merged

weierophinney added a commit to zendframework/zend-http that referenced this issue May 15, 2015

Merge branch 'hotfix/3169'

e39ea8a

Close zendframework/zendframework#3169 Fixes zendframework/zendframework#3036

weierophinney added a commit to zendframework/zend-http that referenced this issue May 15, 2015

Merge branch 'hotfix/3412'

ddc8029

Close zendframework/zendframework#3412 Fixes zendframework/zendframework#3036 Fixes zendframework/zendframework#3267

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP Response Json error #3036

HTTP Response Json error #3036

sachitsac commented Nov 21, 2012

sachitsac commented Dec 4, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

sachitsac commented Dec 5, 2012

jcrawford commented Dec 5, 2012

jcrawford commented Dec 5, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

DrBeza commented Jan 3, 2013

iwalz commented Jan 3, 2013

jcrawford commented Jan 3, 2013

iwalz commented Jan 3, 2013

dkidd commented Jan 9, 2013

BinaryKitten commented Jan 9, 2013

dkidd commented Jan 9, 2013

runphp commented Jan 11, 2013

DrBeza commented Jan 11, 2013

weierophinney commented Jan 11, 2013

stoefln commented May 13, 2013

weierophinney commented May 13, 2013

dkidd commented May 13, 2013

stoefln commented May 13, 2013

weierophinney commented May 13, 2013

stoefln commented May 13, 2013

thadudexx commented Sep 5, 2013

weierophinney commented Sep 5, 2013

eddiejaoude commented Mar 25, 2014

HTTP Response Json error #3036

HTTP Response Json error #3036

Comments

sachitsac commented Nov 21, 2012

sachitsac commented Dec 4, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

sachitsac commented Dec 5, 2012

jcrawford commented Dec 5, 2012

jcrawford commented Dec 5, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

jcrawford commented Dec 5, 2012

sachitsac commented Dec 5, 2012

DrBeza commented Jan 3, 2013

iwalz commented Jan 3, 2013

jcrawford commented Jan 3, 2013

iwalz commented Jan 3, 2013

dkidd commented Jan 9, 2013

BinaryKitten commented Jan 9, 2013

dkidd commented Jan 9, 2013

runphp commented Jan 11, 2013

DrBeza commented Jan 11, 2013

weierophinney commented Jan 11, 2013

stoefln commented May 13, 2013

weierophinney commented May 13, 2013

dkidd commented May 13, 2013

stoefln commented May 13, 2013

weierophinney commented May 13, 2013

stoefln commented May 13, 2013

thadudexx commented Sep 5, 2013

weierophinney commented Sep 5, 2013

eddiejaoude commented Mar 25, 2014