[python] Binary string response encoding with Python 3

[jqr]

When using binary string responses under Python 3, the encoding defaults to UTF8 which is incompatible with binary data. This looks to stem from Python 3 having str default to UTF8 and a new type called bytes which represents binary data. This is a significant departure from functionality in Python 2 and you can read more about it.

This issue does not exist in Python 2.

Swagger Revision: 18262c1 (master as of right now)

Response schema:

schema:
  type: string
  format: binary

Example Error:

Traceback (most recent call last):
  File "./dr.py", line 13, in <module>
    "document_type": "pdf",                                         # pdf or xls or xlsx
  File "/usr/local/lib/python3.5/site-packages/docraptor/apis/doc_api.py", line 203, in create_doc
    callback=params.get('callback'))
  File "/usr/local/lib/python3.5/site-packages/docraptor/api_client.py", line 322, in call_api
    response_type, auth_settings, callback)
  File "/usr/local/lib/python3.5/site-packages/docraptor/api_client.py", line 149, in __call_api
    post_params=post_params, body=body)
  File "/usr/local/lib/python3.5/site-packages/docraptor/api_client.py", line 358, in request
    body=body)
  File "/usr/local/lib/python3.5/site-packages/docraptor/rest.py", line 208, in POST
    body=body)
  File "/usr/local/lib/python3.5/site-packages/docraptor/rest.py", line 171, in request
    r.data = r.data.decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 10: invalid continuation byte

I'm not sure what a good general fix would look like but I have a work-around for our use by delaying byte decoding until right before JSON deserialization and short-circuiting str. You can see my fix directly on generated code.

Related Issues and PRs:

• Python 3 Support DocRaptor/docraptor-python#2
• Python 3 fixes DocRaptor/docraptor-python#3

[wing328]

@jqr thanks for the details. May I know if you've cycle to contribute the fix with a test case using this endpoint (the method name is automatically generated by codegen since it's empty)?

[jqr]

I'm not entirely sure the fix is generic enough to be used everywhere, as I only tested our use cases.

If someone else wants to run with this, they should as I won't have the time to address this in a generic fashion for a few weeks at least.

[wing328]

@jqr please take your time. We've not heard anyone else requesting this feature in Python yet.

[JHibbard]

Binary file responses are also being parsed as utf-8, causing errors. Does anyone know a temporary workaround for this?

schema:
    type: file
    format: binary

[wing328]

@JHibbard What about simply model the response as just type: file without format: binary?

[microo8]

The server sends bytes and client wants to decode it with utf8.
I've resolved it with removing the line: r.data = r.data.decode('utf8') in the rest.py file.

But then in the api_client.py file, in function __deserialize_file(self, response) it wants to write the response.data in an text file.
So I've edited this also, so the function opens the file as "wb":

with open(path, "wb") as f:
    f.write(response.data)

But I don't know that what will now not work, but at least the file download works for me.

[g-bon]

@wing328 I see the same happening in python 2.7, and I think is a more generic problem that binary files are not written to disk correctly because they may be corrupted by the "w" mode instead of "wb"
as pointed out by @microo8 .

I think it would be formally more correct to have wb instead since both text and binary file are expected to be deserialized in here.