Insert to collection fails when using dot notation and pymongo c extensions

[dlbeswick]

MongoKit documents with nested structure elements and dot notation activated fail to save when pymongo c extensions are present.

I think that there's a difference the pymongo C and Python code paths, with the C codepaths failing to recognise mongokit's "DotedDict" as a dictionary.

One instance of the problem is most likely in the bson package's function "_write_element_to_buffer" in "_cbsonmodule.c".

The code runs without issue after deleting "site-packages/pymongo/_cmessage.so" and "site-packages/bson/_cbson.so".

I realise that this is possibly a pymongo bug rather than a mongokit bug, but perhaps there's something to be done in mongokit to mitigate this while the bug is fixed upstream (if it is considered a bug.)

from mongokit import *

connection = Connection()

@connection.register
class Test(Document):
    __collection__ = 'test'
    __database__ = 'mongokit_use_c_test'
    structure = {
        'nested': {
            'value': int
        }
    }

@connection.register
class TestDot(Document):
    __collection__ = 'test_dot'
    __database__ = 'mongokit_use_c_test'
    structure = {
        'nested': {
            'value': int
        }
    }

    use_dot_notation = True


def main():
    # Ok
    test = connection.Test()
    test.save()

    # Failure when pymongo c extensions are present
    test = connection.TestDot()
    test.save()

main()

With pymongo c extensions:

Traceback (most recent call last):
  File "test.py", line 37, in <module>
    main()
  File "test.py", line 35, in main
    test.save()
  File "/home/davidb/devel/mongokit-use-c-bug/ve/local/lib/python2.7/site-packages/mongokit/document.py", line 410, in save
    self.validate(auto_migrate=False)
  File "/home/davidb/devel/mongokit-use-c-bug/ve/local/lib/python2.7/site-packages/mongokit/document.py", line 229, in validate
    size = self.get_size()
  File "/home/davidb/devel/mongokit-use-c-bug/ve/local/lib/python2.7/site-packages/mongokit/document.py", line 263, in get_size
    size = len(BSON.encode(self))
  File "/home/davidb/devel/mongokit-use-c-bug/ve/local/lib/python2.7/site-packages/bson/__init__.py", line 590, in encode
    return cls(_dict_to_bson(document, check_keys, uuid_subtype))
bson.errors.InvalidDocument: Cannot encode object: {'value': None}

[hmenager]

thanks a lot for your analysis of this issue. I can confirm that we are also impacted by this issue, and that the following workarounds temporarily solve this issue:
1- use version 2.6 of pymongo instead of 2.7
2- remove the abovementioned c extensions...
Is it a mongokit or a pymongo issue though? one can argue that the python and the c implementation in pymongo should behave identically...

[GitFree]

@hmenager good advice!

[ThiefMaster]

+1, very annoying bug

[manneorama]

I can just quickly confirm that Linux (Ubuntu 12.04) is also affected by this issue. The workaround mentioned above works here as well, pymongo 2.6.3 does not have this issue.

[last-g]

I've got fix for this problem with #179 or you can use my master branch which also contains some other fixes for i18n

That's real bug, but it caused by bug in MongoKit not pymongo. i18nDotedDict and i18nDotedDict classes violate __getattr__ procotol and __hasattr__ always returns True. And pymongo C extension need to have correct __hasattr__ implementation. There are good explanation why http://emptysqua.re/blog/python-c-extensions-and-mod-wsgi/#how-should-c-extensions-handle-multiple-sub-interpreters

[namlook]

I merged the patch from last-g that fix #179. Please, can you test the HEAD ? If everything is all right, I'll release a new version.

[sylvinus]

I can confirm pymongo 2.7.1 fixes this with mongokit 0.9.1.1