Nested fields not working as expected

[gasparuben]

Dear All,
I am trying to use response marshaling but I get an error. I managed to make the fields/marshal work as indicate in the example - Nested fields

Code

import hashlib, hmac, time
try:
    from urllib.parse import urlencode
except ImportError:
    from urllib import urlencode
import requests
from flask_restx import Namespace, Resource, fields
from dataenhancer.secret.config import ENDPOINT_ROOMS, INDICO_SERVER, API_KEY, SECRET_KEY

namespace = Namespace('indico', description="Contrast information with Indico endpoint")

timing_fields = {}
timing_fields['date'] = fields.String(readOnly=True, description='aa')
timing_fields['tz'] = fields.String(readOnly=True, description='aa')
timing_fields['time'] = fields.String(readOnly=True, description='aa')

booking_fields = {}
booking_fields['url'] = fields.String(readOnly=True, description='aa') 
booking_fields['id'] = fields.String(readOnly=True, description='aa') 
booking_fields['creator'] = fields.String(readOnly=True, description='aa') 
booking_fields['start_dt'] = fields.Nested(timing_fields)
booking_fields['end_dt'] = fields.Nested(timing_fields)

resource_fields = {'vc_room': fields.String(readOnly=True, description='aa')}
resource_fields['event'] = fields.String(readOnly=True, description='aa')
resource_fields['booking'] = fields.Nested(booking_fields)

indico_model = namespace.model('indicoevents', resource_fields)

def _build_indico_request(path, params, api_key=None, secret_key=None, only_public=False, persistent=False):
    items = list(params.items()) if hasattr(params, 'items') else list(params)
    if api_key:
        items.append(('apikey', api_key))
    if only_public:
        items.append(('onlypublic', 'yes'))
    if secret_key:
        if not persistent:
            items.append(('timestamp', str(int(time.time()))))
        items = sorted(items, key=lambda x: x[0].lower())
        url = '%s?%s' % (path, urlencode(items))
        signature = hmac.new(secret_key.encode('utf-8'), url.encode('utf-8'),
                             hashlib.sha1).hexdigest()
        items.append(('signature', signature))
    if not items:
        return path
    return '%s?%s' % (path, urlencode(items))

@namespace.doc('class')
@namespace.route('/<string:office>')
class IndicoEventsEndpoint(Resource):
    #decorators = [require_token]

    @namespace.doc('Get_all_possible_events')
    @namespace.marshal_with(indico_model)
    def get(self, office):
        params = {
            'from': 'today',
            'to': 'today'
        }
        query = _build_indico_request('{}{}.json'.format(ENDPOINT_ROOMS, office), params, api_key=API_KEY, secret_key=SECRET_key,
                                persistent=True)   
        res = requests.get('{}/{}'.format(INDICO_SERVER, query))
        res.raise_for_status()
        return res.json()

I get always the following exception:

Unable to render schema
Traceback (most recent call last):
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/api.py", line 500, in __schema__
    self._schema = Swagger(self).as_dict()
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 202, in as_dict
    **kwargs
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 391, in serialize_resource
    path[method] = self.serialize_operation(doc, method)
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 397, in serialize_operation
    'responses': self.responses_for(doc, method) or None,
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 510, in responses_for
    schema = self.serialize_schema(model)
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 563, in serialize_schema
    self.register_model(model)
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 592, in register_model
    self.register_field(field)
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 600, in register_field
    self.register_model(field.nested)
  File "/Users/r/.pyenv/versions/3.6.5/lib/python3.6/site-packages/flask_restx/swagger.py", line 583, in register_model
    if name not in self.api.models:
TypeError: unhashable type: 'dict'
127.0.0.1 - - [04/Feb/2020 23:07:01] "GET /api/v1/swagger.json HTTP/1.1" 500 -
127.0.0.1 - - [04/Feb/2020 23:07:01] "GET /swaggerui/favicon-32x32.png HTTP/1.1" 200 -

By doing:

$ python
Python 3.6.5 (default, Apr 28 2019, 21:42:51) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from flask_restx import fields, marshal
>>> import json
>>> timing_fields = {}
timing_fields['date'] = fields.String(readOnly=True, description='aa')
timing_fields['tz'] = fields.String(readOnly=True, description='aa')
timing_fields['time'] = fields.String(readOnly=True, description='aa')

booking_fields = {}
booking_fields['url'] = fields.String(readOnly=True, description='aa') 
booking_fields['id'] = fields.String(readOnly=True, description='aa') 
booking_fields['creator'] = fields.String(readOnly=True, description='aa') 
booking_fields['start_dt'] = fields.Nested(timing_fields)
booking_fields['end_dt'] = fields.Nested(timing_fields)

resource_fields = {'vc_room': fields.String(readOnly=True, description='aa')}
resource_fields['event'] = fields.String(readOnly=True, description='aa')
resource_fields['booking'] = fields.Nested(booking_fields)>>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> 
>>> 
>>> data = {
            "vc_room": "dafdafds",
            "event": "afdasfdsfda",
            "booking": {
                "url": "https://indico-staging.xx.xx/rooms/booking/402627",
                "start_dt": {
                    "date": "2020-02-03",
                    "tz": "None",
                    "time": "13:00:00"
                },
                "end_dt": {
                    "date": "2020-02-03",
                    "tz": "None",
                    "time": "17:00:00"
                },
                "id": 402627,
                "creator": "R"
            }
}... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 
>>> 
>>> 
>>> json.dumps(marshal(data, resource_fields))
'{"vc_room": "dafdafds", "event": "afdasfdsfda", "booking": {"url": "https://indico-staging.cern.ch/rooms/booking/402627", "id": "402627", "creator": "R", "start_dt": {"date": "2020-02-03", "tz": "None", "time": "13:00:00"}, "end_dt": {"date": "2020-02-03", "tz": "None", "time": "17:00:00"}}}'
>>> ^C

Is this a bug? I can provide further evidences or reduce the code to reach this issue, but it looks to me quite a general one.

Running python 3.6.5:
Flask==1.1.1
Flask-Cors==3.0.8
flask-restplus==0.13.0
flask-restx==0.1.0

Apologies if I am missing something trivial.
Thank you.

[SVilgelm]

I think that there is an error in the documentation, because the Nested accepts a model, not just a dictionary with the fields.

So, in your case just replace:

booking_fields['end_dt'] = fields.Nested(timing_fields)
resource_fields['booking'] = fields.Nested(booking_fields)

with

booking_fields['end_dt'] = fields.Nested(namespace.model('timing_fields', timing_fields))
resource_fields['booking'] = fields.Nested(namespace.model('booking_fields', booking_fields))

[gasparuben]

Thank you @SVilgelm, indeed the documentation is a bit confusing about fields.Nested. As you wrote it down it does work.

I close the issue. Thank you for your help.

[SVilgelm]

Let's keep the issue open, we need to fix the documentation.

[tjysdsg]

@SVilgelm I think allowing fields.Nested to accept a dict is possibly a good feature to implement. For example, if a dict input is detected, fields.Nested update its value by wrapping it with namespace.model.

I like feature because it allows me to generate models without knowing the namespace, therefore I don't need to pass in an extra argument.

[SVilgelm]

It can be useful, but in this case we have we have a discrepancy between the documentation and the actual behavior.

[synchronizing]

@SVilgelm I think allowing fields.Nested to accept a dict is possibly a good feature to implement. For example, if a dict input is detected, fields.Nested update its value by wrapping it with namespace.model.

I like feature because it allows me to generate models without knowing the namespace, therefore I don't need to pass in an extra argument.

+1 for this behavior. As it is currently implemented, it's confusing to quickly architect a nested dictionary model.

[b33rcity]

Also getting this error, in my case while trying to use it with fields.Wildcard. The thing that's driving me bonkers is that it works when integrated into another Model, but not when I try to use it on its own:

# serializers.py
inner = fields.Wilcard(fields.Nested({
    ...
})
outter = Model('outter', {
    'inner': fields.Nested({'*': inner})
    ...
})
# Previously tried `outter = fields.Wildcard(fields.Nested...` and `outter = fields.Nested(...`

inner_and_outter_model = Model('InnerNOutter', {
    '*': fields.Wildcard(fields.Nested(outter))
})

other_model = Model('OtherModel', {
    'list1': fields.List(fields.Nested(inner_and_outter_model)),
    'list2': fields.List(fields.Nested(inner_and_outter_model))
})

# api.py
from serializers import *
namespace = Namespace(...)
namespace.model(inner_and_outter_model.name, inner_and_outter_model)
namespace.model(other_model.name, other_model)

@namespace.route('/<int:id>')
class Specific(Resource):
    @namespace.marshal_with(inner_and_outter_model)
    def get(self, id):
        return some_func(id)

@namespace.route('')
class Other(Resource):
    @namespace.marshal_with(other_model)
    def get(self):
        return some_other_func

For some reason, Other.get will work just fine, marshal correctly and everything. I was so excited when it worked the first time.

But, as soon as I set the marshal_with to inner_and_outter_model in Specific.get, Swagger breaks. I get the TypeError: unhashable type: 'dict' error when I reload the page.

I've spent a couple of hours going around in circles, moving the serializer declaration into the api module so I can call namespace.model() instead of Model(), refactoring the serializer in various ways.... I tried writing outter as a bare dict, as a fields.Nested without a Wildcard, as a fields.Wildcard without a Nested... If it broke on other_model as well as inner_and_outter_model, I'd chalk it up to my code being fundamentally wrong, but as it is I can't fathom any reason why this model works in one case but not the other. I've been through the docs and even the source code for Wildcard, Nested, and Raw and it's just not making sense.

[b33rcity]

(Edit: Most of this overly long post is me working through the problem and discovering what others in this thread already knew. I'll leave it for anyone else coming along who's in the same position. For those that already understand the problem, I invite you to skip to the bottom of this post.)

I just realized I didn't include a stack trace yesterday--shame on me.

08/07/2020 08:46:57 PM - ERROR - Unable to render schema
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask_restx/api.py", line 538, in __schema__
    self._schema = Swagger(self).as_dict()
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 239, in as_dict
    serialized = self.serialize_resource(
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 441, in serialize_resource
    path[method] = self.serialize_operation(doc, method)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 447, in serialize_operation
    "responses": self.responses_for(doc, method) or None,
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 567, in responses_for
    schema = self.serialize_schema(model)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 632, in serialize_schema
    self.register_model(model)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 661, in register_model
    self.register_field(field)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 671, in register_field
    self.register_field(field.container)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 669, in register_field
    self.register_model(field.nested)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 652, in register_model
    if name not in self.api.models:
TypeError: unhashable type: 'dict'

I've been digging through this further and I've found something that I'm very tempted to call a bug.

swagger.py:650-673:

  def register_model(self, model):
        name = model.name if isinstance(model, ModelBase) else model
        if name not in self.api.models:
            raise ValueError("Model {0} not registered".format(name))
        specs = self.api.models[name]
        if name in self._registered_models:
            return ref(model)
        self._registered_models[name] = specs
        if isinstance(specs, ModelBase):
            for parent in specs.__parents__:
                self.register_model(parent)
        if isinstance(specs, Model):
            for field in itervalues(specs):
                self.register_field(field)
        return ref(model)

    def register_field(self, field):
        if isinstance(field, fields.Polymorph):
            for model in itervalues(field.mapping):
                self.register_model(model)
        elif isinstance(field, fields.Nested):
            self.register_model(field.nested)
        elif isinstance(field, (fields.List, fields.Wildcard)):
            self.register_field(field.container)

For a Model which contains a Wildcard which contains a Nested, our data bounces between these two methods twice, including one recursive call to register_field() when the field argument is a Wildcard. On this second call to register_field(), the Wildcard has been stripped away by passing its container attribute, which is a Nested in my case.

At this point, we're bounced back to register_model(), being called against <Nested>.nested this time. This object is a dict. Since dict is not a ModelBase, name is set to this dict object on 651, which is then membership-checked on 652 against another dict, self.api.models. This, in case you're like me and had never pondered if this is a valid operation in Python, is not a valid operation in Python:

>>> foo = {'one': 1}
>>> boo = {'two': 2}
>>> boo in foo
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: unhashable type: 'dict'

A second membership check would occur a couple lines later, this time against self._registered_models, and one could expect the exact same error to occur.

As a serializer, the model works. I've confirmed this in flask shell, and as I mentioned before I am able to use it from the Swagger UI page and get correct output from it when it's wrapped in other_model and a fields.List. Swagger itself is what breaks when I attempt to pass this model to @marshal_with().

I'm not sure how the workaround given in #31 (comment) would even work, seeing as the Namespace.model() method seems to be acting on the same dicts as the register_model() method that's failing. Even if it worked for OP, I did try that approach in my own app and it did not work for me (obligatory caveat: it's always possible I did it wrong).

In any case, it seems like it should be an easy fix: instead of

elif isinstance(field, fields.Nested):
            self.register_model(field.nested)

perhaps

elif isinstance(field, fields.Nested):
            self.register_model(field)

would work? This would preserve the .name attribute, which would be picked up correctly on 651. (I'm ready to be told it's not that simple ;) .)

Update:

Well, I went ahead and tried that, along with a bit of refactoring of how the final Model gets assembled, and adding a few things to what gets output by the raise on line 253. I got a new error:

  File "/usr/local/lib/python3.8/site-packages/flask_restx/fields.py", line 270, in schema
    ref = "#/definitions/{0}".format(self.nested.name)
AttributeError: 'dict' object has no attribute 'name'

A call to <Nested>.schema() is among the things I added to the raise output. What's really throwing me off here is the reference to self.nested.name. For the Nested class, nested is a property that only does return getattr(self.model, "resolved", self.model). self.model is also a dict, of course.

>>> labsf
<flask_restx.fields.Nested object at 0x7fb8de8db6a0>
>>> labsf.nested
{'*': <flask_restx.fields.Wildcard object at 0x7fb8de8db550>}
>>> labsf.schema()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/flask_restx/fields.py", line 270, in schema
    ref = "#/definitions/{0}".format(self.nested.name)
AttributeError: 'dict' object has no attribute 'name'

Update part 2:
As I was about to reaffirm my hypothesis that this is a bug, I decided to refactor so that every object being assembled into the final model is itself a Model object. This seems to go against the documentation's recommendation to not declare Wildcard fields inside the model, which is why I waited so long to try it this way.

By passing in an actual Model and not a dict, <Nested>.nested does indeed return an object which has a name attribute. So, now I can sheepishly say that I finally understand the proposal in #31 (comment), and agree that wrapping a dict in a Model should give the behavior that we all seemed to expect.

[cisaacson]

The documentation is still not fixed here.

This is the offending line (692 in swagger.py in register_model):

if name not in self.api.models:

The recommended fix above does work by wrapping the fields that are being Nested in a model, but the documentation for Nested doesn't say anything about that.

Hope that helps to get this resolved.