Option to allow pretty print without alpha sorting

[patcon]

Right now, the options are colors, format, and all, but often the order of the json returned is intentional and for clarify. It would be nice to be able to pretty print the json without having it's keys sorted :)

Awesome tool, and thanks very much for everything!

[kracekumar]

Results from my introspection.

➜  httpie git:(issue-128) http http://headers.jsontest.com/
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: private
Content-Type: application/json; charset=ISO-8859-1
Date: Sat, 27 Jul 2013 07:01:24 GMT
Server: Google Frontend
Transfer-Encoding: chunked

{
    "Accept": "*/*",
    "Host": "headers.jsontest.com",
    "User-Agent": "HTTPie/0.6.0"
}

➜  httpie git:(issue-128) curl http://headers.jsontest.com/
{
   "Host": "headers.jsontest.com",
   "User-Agent": "curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5",
   "Accept": "*/*"
}

Since httpie uses requests guessing r.json() is responsible for this, trying requests.get.

In [36]: r = requests.get("http://headers.jsontest.com/")

In [37]: r.json()
Out[37]:
{u'Accept': u'*/*',
 u'Host': u'headers.jsontest.com',
 u'User-Agent': u'python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0'}

Same result. Now looking intor.json code.

In [38]: r.json??
Type:       instancemethod
String Form:<bound method Response.json of <Response [200]>>
File:       /Library/Python/2.7/site-packages/requests/models.py
Definition: r.json(self, **kwargs)
Source:
    def json(self, **kwargs):
        """Returns the json-encoded content of a response, if any.

        :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
        """

        if not self.encoding and len(self.content) > 3:
            # No encoding set. JSON RFC 4627 section 3 states we should expect
            # UTF-8, -16 or -32. Detect which one to use; If the detection or
            # decoding fails, fall back to `self.text` (using chardet to make
            # a best guess).
            encoding = guess_json_utf(self.content)
            if encoding is not None:
                return json.loads(self.content.decode(encoding), **kwargs)
        return json.loads(self.text or self.content, **kwargs)

Look into requests code json is imported from compat.py, the exact code is

try:
    import simplejson as json
except ImportError:
    import json

Trying other library like ujson provides same result

n [43]: ujson.loads("""{
   "Host": "headers.jsontest.com",
   "User-Agent": "curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5",
   "Accept": "*/*"
}""")
Out[43]:
{u'Accept': u'*/*',
 u'Host': u'headers.jsontest.com',
 u'User-Agent': u'curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5'}

Since the the returned object is dictionary it isn't possible to maintain the order due to hashing. If it is possible to maintain the order it would be great.

[kracekumar]

Seems like I made a mistake. There is a JSONProcessor but still it might not solve the problem.

In [48]: r.content
Out[48]: '{\n   "Host": "headers.jsontest.com",\n   "User-Agent": "python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0",\n   "Accept": "*/*"\n}\n'

In [49]: json.loads(r.content)
Out[49]:
{u'Accept': u'*/*',
 u'Host': u'headers.jsontest.com',
 u'User-Agent': u'python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0'}

Trying with JSONProcessor

In [61]: j = JSONProcessor()

In [62]: j.process_body(r.content, r.headers['Content-Type'], subtype='json', encoding='utf8')
Out[62]: u'{\n    "Accept": "*/*", \n    "Host": "headers.jsontest.com", \n    "User-Agent": "python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0"\n}'

[kracekumar]

One way to get around the problem is use PygmentsProcessor.

In [64]: p = PygmentsProcessor()

In [65]: p.process_body(r.content, r.headers['Content-Type'], subtype='json', encoding='utf8')
Out[65]: u'\x1b[38;5;245m{\x1b[39m\n\x1b[38;5;245m   \x1b[39m\x1b[38;5;33m"Host"\x1b[39m\x1b[38;5;245m:\x1b[39m\x1b[38;5;245m \x1b[39m\x1b[38;5;37m"headers.jsontest.com"\x1b[39m\x1b[38;5;245m,\x1b[39m\n\x1b[38;5;245m   \x1b[39m\x1b[38;5;33m"User-Agent"\x1b[39m\x1b[38;5;245m:\x1b[39m\x1b[38;5;245m \x1b[39m\x1b[38;5;37m"python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0"\x1b[39m\x1b[38;5;245m,\x1b[39m\n\x1b[38;5;245m   \x1b[39m\x1b[38;5;33m"Accept"\x1b[39m\x1b[38;5;245m:\x1b[39m\x1b[38;5;245m \x1b[39m\x1b[38;5;37m"*/*"\x1b[39m\n\x1b[38;5;245m}\x1b[39m'

In [66]: print(p.process_body(r.content, r.headers['Content-Type'], subtype='json', encoding='utf8'))
{
   "Host": "headers.jsontest.com",
   "User-Agent": "python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0",
   "Accept": "*/*"
}

[patcon]

Nice. I recently used something called an OrderedHash is ruby that solved a similar problem. Thinking that OrderedDict might be the solution in python...
http://stackoverflow.com/questions/6921699/can-i-get-json-to-load-into-an-ordereddict-in-python

[kracekumar]

@patcon That was easier solution than mine. I will modify the code accordingly and update the pull request.

[patcon]

👍 Closing this one :)

[kracekumar]

@patcon Why?, pull request isn't merged though.

[patcon]

Ah. Oops. Misunderstood the context of the referring issue above.

[carueda]

👍 veeery useful option, please merge/fix at the next convenient opportunity. If not "reasonably" soon, any idea of by when approximately?

[pontuspalmenas]

+1
Please add this option. Sorting is a deal breaker for me.

[altfatterz]

+1
Yes, please add this option. I would like to have pretty print but without sorting the keys

[rawaludin]

Any news about this?

[v-kolesnikov]

+1

[tedmiston]

+1

[hartym]

This problem is indeed misleading. I struggled a bit to understand why my (server side) OrderedDict was not correctly serialised in my app, then I finally understood that the culprit was not the server but indeed the client re-sorting the keys. It is sometimes useful to have the keys sorted, but for an HTTP client, having a default that changes the response content does not sound like the right thing.

Thanks for the amazing tool anyway ;)

[msabramo]

@jkbrzt: You mention a while back in #151 (comment) that you wanted a more generic solution to this problem; was any progress made on that?

[jkbrzt]

@msabramo it's still on my to do list but not progress yet

[NN88]

+1 this really needs to become a feature.

Thanks for this amazing tool!

[kylebebak]

I wrote an HTTP client for Sublime Text that's also built on top of Requests, called Requester. I also wanted to allow users to choose whether or not keys for JSON responses are sorted, so I added a 'fmt' argument to calls to requests.

I solved it like this. If the response body is JSON, calling res.json() on the response object res returns a Python dictionary. At this point you can pass the dict to json.dumps to specify indentation and whether keys are sorted. However, if you just return res.text (equivalent to returning the raw response body) the keys won't be sorted.

[opensas]

Has this feature been added? I tried with --pretty=unsorted but I get

http: error: argument --pretty: invalid choice: 'unsorted' (choose from 'all', 'colors', 'format', 'none')

Anyway, just passing --pretty=colors did the trick, the output is not sorted (I already format the output), but I found no wayt to specify something like --pretty=colors,format

$ http --version
0.9.9

[virgo47]

This feature should be default. I love HTTPie, but resorting the keys - while convenient for some usage - is confusing if you don't realize, that client tool does this. Piping http to jq -C is a workaround, but it should be default behavior. While the order is not probably that important for a machine, it may be for a human observer.

[nmtitov]

Knock-knock... 7 years after... any updates? jq -C is good but making sorting disabled by default would be much better

[opensas]

Anybody know of an httpie fork that took care of this annoying issue?

[opensas]

I found the following workaround, instead of using httpie I found curlie, which seems to

Like curl but unlike httpie, headers are written on stderr instead of stdout.

So I can use it like this:

$ curlie GET localhost:3000/tasks | jq -C
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 304
ETag: W/"130-ED1W4hQo1i7na7wy5Ewc7iKdoJc"
Date: Wed, 27 May 2020 06:28:26 GMT
Connection: keep-alive

[
  {
    "id": 2,
    "title": "new task2",
    "description": "description2",
    "status": "OPEN"
  },
  {
    "id": 3,
    "title": "new task3",
    "description": "description3",
    "status": "OPEN"
  }
]

Where as with httpie I would the headers

BTW, I created this convenient script:

$ cat ~/bin/c
curlie "$@" | jq -C

[jkbrzt]

How about this?

Disable the default header and JSON key sorting, and specify a custom JSON indent size:

$ http --format-options headers.sort=false,json.sort_keys=false,json.indent=2 httpbin.org/get

  --format-options FORMAT_OPTIONS
      Controls output formatting. Only relevant when formatting is enabled
      through (explicit or implied) --pretty=all or --pretty=format.
      The following are the default options:

          headers.sort=true
          json.format=true
          json.indent=4
          json.sort_keys=true

      You can specify multiple comma-separated options. For example, this modifies
      the settings to disable the sorting of headers and JSON keys:

          --format-options headers.sort=false,json.sort_keys=false

      This is something you will typically put into your config file.

It’s already master & feedback would be appreciated! (Install HTTPie from master.)

[jkbrzt]

The idea behind the syntax is to allow more options to be easily added in the future. But I do realise that it’s quite a verbose one. On the other hand, I would expect people to set it once in the config file and not type it manually. Thoughts?

[opensas]

Thanks a lot for the reply, being able to configure those options for format output would be great, speacially if you can just put it on a conf file.

I tried to install from master but I have the following issue:

$ sudo apt install python-pip
[...]

$ pip install --upgrade https://github.com/jakubroztocil/httpie/archive/master.tar.gz
Collecting https://github.com/jakubroztocil/httpie/archive/master.tar.gz
  Downloading https://github.com/jakubroztocil/httpie/archive/master.tar.gz
     | 2.2MB 34.5MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-9g7C_j-build/setup.py", line 74
        download_url=f'https://github.com/jakubroztocil/httpie/archive/{httpie.__version__}.tar.gz',
                                                                                                  ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-9g7C_j-build/

$ pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

$ python --version
Python 2.7.17

$ uname -a
Linux xps 5.0.0-1052-oem-osp1 #57-Ubuntu SMP Mon May 4 11:39:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Do I need python 3?

[jkbrzt]

@opensas thanks for testing it out! Yes, you need Python 3.6+ (Python 2.7 is dead, and since v2.0.0 HTTPie does not support it).

[jkbrzt]

@opensas if you have Docker installed, you can test it in a container:

docker run --rm -it python:3.8-slim

Then inside the container:

pip install --upgrade https://github.com/jakubroztocil/httpie/archive/master.tar.gz

$ http --version
2.2.0-dev

[opensas]

I could install it with python3 -m pip install --upgrade https://github.com/jakubroztocil/httpie/archive/master.tar.gz

I tested it and it works ok with http --format-options headers.sort=false,json.sort_keys=false,json.indent=2

I'm trying to set those options in my config file, but I guess I have a trouble with the xx=xx options, I'm trying like this:

$ cat ~/.config/httpie/config.json
{
    "default_options": [
      "--format-options=headers.sort=false,json.sort_keys=false,json.indent=2"
    ]
}

What should be the correct syntax for my config.json file? DO I have to escape the = signs?

[jkbrzt]

@opensas what output do you get when you run http --debug?

[jkbrzt]

@opensas the syntax is correct (it should be the same like on the CLI, there nothing special about the config options).

[opensas]

Sorry, I found the issue, I had a ~/.httpie/config.json from previous installation, so it was bypassing the file I created at ~/.config/httpie/config.json

just removed the .httpie folder and now everything works as expected

thanks a lot! I hope you can release this changes in the near future!

Some feedback:

• I would consider setting headers.sort and json.sort to false by default, I think that changing the output order should be the exception and not the rule

• perhaps also add a --format-options sort=[true|false] shortcut to set both headers.sort and json.sort at the same time

anyway, congrats on the excellent job you are doing

[opensas]

more feedback:

I think the easiest thing to use from the command line would be to allow something like:

--pretty=colors,format,unsorted

--pretty=all,unsorted

[jkbrzt]

Sorry, I found the issue, I had a ~/.httpie/config.json from previous installation, so it was bypassing the file I created at ~/.config/httpie/config.json
just removed the .httpie folder and now everything works as expected

The $XDG_CONFIG_HOME-based ~/.config/httpie location is also new in the yet-to-be-released version. Looks like a warning if both the locations exist would be helpful.

thanks a lot! I hope you can release this changes in the near future!

Yes, it should be out pretty soon.

I would consider setting headers.sort and json.sort to false by default, I think that changing the output order should be the exception and not the rule

I agree that silently changing the output order is generally not a good idea. Here, however, it’s part of the "prettification" meant for human consumption on the terminal, and the HTTP header and JSON key order is insignificant. Also, this behaviour has been the default for a long time, so a lot of people would have to change their config files to get back what they’re used to.

perhaps also add a --format-options sort=[true|false] shortcut to set both headers.sort and json.sort at the same time

more feedback:
I think the easiest thing to use from the command line would be to allow something like:

--pretty=colors,format,unsorted
--pretty=all,unsorted

Good point, a shortcut for toggling both the sortings might be handy.

Thanks for the feedback!

[jkbrzt]

Just released v2.2.0 that addresses this issue. Learn about about the new --unsorted, --sorted and --format-options here:

https://httpie.org/docs#colors-and-formatting