Bizarre issue with JSON request bodies getting mutated

[fiatjaf]

I don't know if this is an issue with Lambda or Zappa, but since I couldn't find anyone complaining from this strange bug that would affect anyone using JSON to send data to Lambda functions, I guess it is more likely that the problem is here? (Maybe it is a Flask bug also, but I couldn't find anything there also and it doesn't make sense that Flask works differently locally and in the Lambda environment -- or maybe does?)

Here's my Flask app (full code):

import os
from flask import Flask, request

app = Flask(__name__)

@app.route('/test', methods=['POST'])
def test():
    print('request.get_data():', request.get_data())
    print('request.data:', request.data)
    return request.data

if __name__ == '__main__':
    app.run('0.0.0.0', int(os.getenv('PORT', 8787)), debug=os.getenv('DEBUG', True))

Here's what happens when I test it locally:

fiatjaf@cantillon ~/c/webhooks> curl -X POST http://localhost:8787/test -d '{"x":22}' -H 'application/json'
{"x":22}⏎                                                                                       
fiatjaf@cantillon ~/c/webhooks> curl -X POST http://localhost:8787/test -d '{"x":22}'
{"x":22}⏎      

Then I deploy (everything looks normal for me):

(venv) fiatjaf@cantillon ~/c/webhooks> zappa deploy test
Packaging project as zip...
Uploading zip (6.7MiB)...
100%|█████████████████████████████████████████████████████| 7.08M/7.08M [00:02<00:00, 2.66Mit/s]Creating API Gateway routes..
512it [00:48, 10.64it/s]                                                                        Deploying API Gateway..
Your Zappa deployment is live!: https://zr8az5jgn5.execute-api.us-east-1.amazonaws.com/test

Testing the "live" version (here's where the problem appears):

fiatjaf@cantillon ~/c/webhooks> curl -X POST https://zr8az5jgn5.execute-api.us-east-1.amazonaws.com/test/test -d '{"x":22}'
{x=22}⏎                                                                                         
fiatjaf@cantillon ~/c/webhooks> curl -X POST https://zr8az5jgn5.execute-api.us-east-1.amazonaws.com/test/test -d '{"x":22}' -d 'Content-Type: application/json'
{x=22}⏎         

Why is my {"x": 22} JSON getting mutated to {x=22}? This is odd, very odd. These logs don't show it, but I have also tried to log type(data) and it is <str> ({key=value} is not the standard way Python prints objects anyway).

The lambda logs show the following:

(venv) fiatjaf@cantillon ~/c/webhooks> zappa tail test
[1464094582372] [INFO]  2016-05-24T12:56:22.372Z        e9defd0f-21ae-11e6-b5f6-ebc4d39e71b3   162.243.220.95 - - [24/May/2016:12:56:22 +0000] "GET / HTTP/1.1" 404 233 "" "python-requests/2.10.0" 0/468.216
[1464094582372] PCFET0NUWVBFIGh0bWw+NDA0PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDMuMiBGaW5hbC8vRU4iPgo8dGl0bGU+NDA0IE5vdCBGb3VuZDwvdGl0bGU+CjxoMT5Ob3QgRm91bmQ8L2gxPgo8cD5UaGUgcmVxdWVzdGVkIFVSTCB3YXMgbm90IGZvdW5kIG9uIHRoZSBzZXJ2ZXIuICBJZiB5b3UgZW50ZXJlZCB0aGUgVVJMIG1hbnVhbGx5IHBsZWFzZSBjaGVjayB5b3VyIHNwZWxsaW5nIGFuZCB0cnkgYWdhaW4uPC9wPgo=: LambdaException
Traceback (most recent call last):
  File "/var/task/handler.py", line 161, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 55, in lambda_handler
    return cls().handler(event, context)
  File "/var/task/handler.py", line 143, in handler
    raise e
LambdaException: PCFET0NUWVBFIGh0bWw+NDA0PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDMuMiBGaW5hbC8vRU4iPgo8dGl0bGU+NDA0IE5vdCBGb3VuZDwvdGl0bGU+CjxoMT5Ob3QgRm91bmQ8L2gxPgo8cD5UaGUgcmVxdWVzdGVkIFVSTCB3YXMgbm90IGZvdW5kIG9uIHRoZSBzZXJ2ZXIuICBJZiB5b3UgZW50ZXJlZCB0aGUgVVJMIG1hbnVhbGx5IHBsZWFzZSBjaGVjayB5b3VyIHNwZWxsaW5nIGFuZCB0cnkgYWdhaW4uPC9wPgo=
[1464094675182] ('request.get_data():', '{x=22}')
[1464094675182] ('request.data:', '{x=22}')
[1464094675183] [INFO]  2016-05-24T12:57:55.183Z        21cb3e15-21af-11e6-9d0c-abfebd2c382c   162.243.220.95 - - [24/May/2016:12:57:55 +0000] "POST /test HTTP/1.1" 200 6 "" "curl/7.43.0" 0/0.97
[1464094720613] ('request.get_data():', '{x=22}')
[1464094720613] ('request.data:', '{x=22}')
[1464094720613] [INFO]  2016-05-24T12:58:40.613Z        3cdfc4f5-21af-11e6-92b2-97032d56bec9   162.243.220.95 - - [24/May/2016:12:58:40 +0000] "POST /test HTTP/1.1" 200 6 "" "curl/7.43.0" 0/0.931

I'm using Python 2.7.10, a virtualenv and running zappa from inside the virtualenv, so I think I'm doing everything right as explained in the books. I just don't know what is that Exception that appears on the beggining of Lambda logs right there. Could it be the cause? I doubt it, but who knows?

[Miserlou]

Thanks for reporting! Especially for the repro. Investigating..

[Miserlou]

I have a committed a fix to master that I THINK will fix this, but needs code review and more testing. 8f47569

[Miserlou]

(You will need to 'undeploy' and 'redeploy' for the changes to take affect, I think.)

[Miserlou]

The type will still be a string, but I think that's actually the correct behavior - it's the job of the application do the the parsing properly, Zappa's job is just the scurry the data.

[fiatjaf]

Right, I don't quite understand the fix, but I think it is it. I was seeing different behavior when specifying Content-Type and when not, but at some point when writing the reproducible example I lost the point where the result differed (things get confused when you're getting errors).

I'll test the new code later today. Thank you.

[fiatjaf]

It has kind of fixed it, but there's a different issue now:

fiatjaf@cantillon ~/c/webhooks> curl -X POST https://fj9czanxrk.execute-api.us-east-1.amazonaws.com/test/test -d '{"x":22}'
{"x":22}⏎                                                                                                        
fiatjaf@cantillon ~/c/webhooks> curl -X POST https://fj9czanxrk.execute-api.us-east-1.amazonaws.com/test/test -d '{"x":22}' -H 'Content-Type: application/json'
<!DOCTYPE html>500.
From Zappa: <pre>must be string or buffer, not dict</pre><br />
<pre>Traceback (most recent call last):<br />
File "/var/task/handler.py", line 122, in handler<br /> 
trailing_slash=False<br />
File "/home/fiatjaf/comp/webhooks/venv/lib/python2.7/site-packages/zappa/wsgi.py", line 25, in create_wsgi_request<br />
File "/usr/lib64/python2.7/base64.py", line 73, in b64decode<br />
return binascii.a2b_base64(s)<br />
TypeError: must be string or buffer, not dict<br /></pre>⏎                                                               

[Miserlou]

Good catch, I see the issue, needs thought. More soon.

[Miserlou]

Pushed another fix, although I fear I may have gone too far in the other direction..

[fiatjaf]

Now it is working for me, although I don't quite understand all the changes involved, thank you very much.

Am I expecting what is the correct expected behavior, or will this change in the future?

[Miserlou]

The correct behavior should be to get the same inputs/outputs across Zappa as you would in any traditional deployment, so yes. Leaving this ticket open until this is tested and published.

[Miserlou]

Published.