client: posting multipart-formdata and integer

[pylover]

the multipart-formdata is not working with integer values, i spend many times to find-out what is the problem:

Actual behaviour

Traceback (most recent call last):
  File "/home/vahid/workspace/aiohttp/aiohttp/client_reqrep.py", line 390, in write_bytes
    result = stream.send(value)
  File "/home/vahid/workspace/aiohttp/aiohttp/helpers.py", line 188, in _gen_form_data
    yield from self._writer.serialize()
  File "/home/vahid/workspace/aiohttp/aiohttp/multipart.py", line 958, in serialize
    yield from part.serialize()
  File "/home/vahid/workspace/aiohttp/aiohttp/multipart.py", line 759, in serialize
    yield from self._maybe_encode_stream(self._serialize_obj())
  File "/home/vahid/workspace/aiohttp/aiohttp/multipart.py", line 772, in _serialize_obj
    return self._serialize_default(obj)
  File "/home/vahid/workspace/aiohttp/aiohttp/multipart.py", line 804, in _serialize_default
    raise TypeError('unknown body part type %r' % type(obj))
TypeError: unknown body part type <class 'int'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/pycharm-2016.1.1/helpers/pydev/pydevd.py", line 1531, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/opt/pycharm-2016.1.1/helpers/pydev/pydevd.py", line 938, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/pycharm-2016.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/vahid/workspace/hazelnut/practice/formdata.py", line 26, in <module>
    loop.run_until_complete(main())
  File "/usr/local/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/vahid/workspace/hazelnut/practice/formdata.py", line 21, in main
    await session.post(url, data=data)
  File "/home/vahid/workspace/aiohttp/aiohttp/client.py", line 537, in __await__
    resp = yield from self._coro
  File "/home/vahid/workspace/aiohttp/aiohttp/client.py", line 191, in _request
    yield from resp.start(conn, read_until_eof)
  File "/home/vahid/workspace/aiohttp/aiohttp/client_reqrep.py", line 619, in start
    message = yield from httpstream.read()
  File "/home/vahid/workspace/aiohttp/aiohttp/streams.py", line 592, in read
    result = yield from super().read()
  File "/home/vahid/workspace/aiohttp/aiohttp/streams.py", line 447, in read
    yield from self._waiter
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
aiohttp.errors.ClientRequestError: Can not write request body for http://example.org/

Steps to reproduce

import asyncio
import aiohttp
import io


sample_data = {
    'host': 'localhost',
    'label': 'DefaultNode',
    'quality': 100,
    'capacity': 100000,
}

async def main():
    url = 'http://example.org'
    data = aiohttp.FormData()
    data.add_field('file', io.StringIO('Sample text file'), content_type='text/plain')
    for k, v in sample_data.items():
        data.add_field(k, v)

    with aiohttp.ClientSession() as session:
        await session.post(url, data=data)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

My environment

os: Ubuntu 14.04
python 3.5.1
aiohttp: master: f180199

[asvetlov]

See also #761

@kxepal we can add more obvious error into _serialize_default just for integer/float case.
Opinions?

[kxepal]

@asvetlov
I think the error itself is pretty clear, but general issue is that serialization is too much lazy which makes hard to catch the moment when issue actually happens. For instance, if an exception was raised on add_field call it would be much easy to find out why and for what exactly field it happened.

Let me rethink for MultipartWriter works, may be we can make it more friendly.

[asvetlov]

Exception in add_field work for me

[pylover]

Thanks a lot. but some frameworks trying to convert the given value to str

[kxepal]

@pylover
It's tricky. If we do convert everything to string, we can produce unwanted results. Let's pick int. It looks pretty safe to do just str(intval), but how to be with bool type? They are int-like, but it depends on server in which way to handle them: True or true or 1 or something else?

With other types there is more or less same story. And if we add support of one Python base type, why not to add the rest? But how?

For now we expect value as str and only and that's gives pretty consistent behavior and expectations (except the moment when you get the error about bad value - that's really unfriendly).

[asvetlov]

@kxepal do you have a time for refactoring or prefer to close the issue without a fix?

[kxepal]

@asvetlov Yes, I do, will take care about this.

[pylover]

Thanks a lot.

[fafhrd91]

@kxepal any news?

[pylover]

@kxepal, Currently i completely agree with you, so i think the python's __repr__ is enought, This is what the other frameworks do.

[kxepal]

@pylover You mean apply repr() as default serializer?

[pylover]

@kxepal , Yes, but applying an IOC mechanism to allow users to register their own preferred callable as serializer is usefull

[kxepal]

@pylover
This worries me since you will notice the problem only after you start to send requests with something like 'ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:423'. Lucky you if backend will not accept anything of such, but what if it will?

[pylover]

@kxepal , What about using __str__ ( str() )?

I think the str() function is better choice.

[kxepal]

@pylover Yes, str() is a right choice, but it doesn't solves general issue I previously mentioned.

[pylover]

I don't get your meaning from previous comment: 'parallel......'. Why backend should accept such thing? And please mention a real world example to demonstrate the problem

[kxepal]

@pylover
Actually, now there is no any realworld problem, because multipart will not implicitly serialize any values to string (: I'm trying say that such default serialization will cause the problems. One of them I demostrated: implicit serialization in the way you don't actually want.

[pylover]

@kxepal , So, raising ValueError for any non-string value?

[kxepal]

@pylover We already have TypeError for that case - a bit more specific.

[pylover]

@kxepal , Yes, you are true. You can replace it to ValueError, to tell the user to serialize their data. Or wathever you want. At all, i prefer to stringify whatever received.

[pylover]

Thank's a lot

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.