Intermittent 500 for /api/issues/category/new

[miketaylr]

STR:

Fire up local server
Visit localhost:5000/

127.0.0.1 - - [15/Dec/2015 14:52:50] "GET /api/issues/category/new HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/miket/dev/compat/webcompat.com/webcompat/helpers.py", line 360, in wrapped_func
    return func(*args, **kwargs)
  File "/Users/miket/dev/compat/webcompat.com/webcompat/api/endpoints.py", line 136, in get_issue_category
    'content': filter_new(json.loads(issues[0])),
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

It doesn't seem to happen every time, if I refresh I sometimes get a 200. That's weird.

[miketaylr]

Ooo. I just added print(issues) to https://github.com/webcompat/webcompat.com/blob/master/webcompat/api/endpoints.py#L131 and got the following:

('', 304, {'etag': '"14d9337fa97f2774fe9379bb7949ec6a"', 'content-type': 'application/json', 'cache-control': 'private, max-age=60, s-maxage=60'})
127.0.0.1 - - [15/Dec/2015 15:01:24] "GET /api/issues/category/new HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/miket/dev/compat/webcompat.com/env/lib/python2.7/site-package
...

[miketaylr]

Hmm, thinking of the best way to handle this.

The problem is if we get a 304 from GitHub when we request all open issues, we get an empty body (which is what should happen). However, we try to convert that to json before passing to filter_new.

We could kill filter_new entirely and use the Search API with the is:open -label:status-contactready -label:status-sitewait -label:status-needscontact -label:status-needsdiagnosis syntax. That would be cool, except remember that there's a 30 requests/min limit for proxy requests, or 10 a minute for unauthed (client-side) requests.

OR, we could filter client-side and avoid the whole problem.

OR... something I'm not thinking of.

Thoughts @karlcow @hallvors?

[miketaylr]

Actually, I think this is a bug that @karlcow fixed a long time ago and I re-introduced it.

I have a fix to just pass back the 304 with an empty body and the browser does the right thing™.

[karlcow]

💡 hehe