UpdateDoesNotApply when deleting a field which does not exist

[levofski]

Hi I am using mongo-connector 2.5.2 with ElasticSearch2 doc manager.

I was getting floods of the following error in my log :-

2017-08-01 00:00:18,129 [ERROR] mongo_connector.oplog_manager:288 - Unable to process oplog document {u'h': 3765553650119896258L, u'ts': Timestamp(1501252773, 83), u'o': {u'$set': {u'datasheets': [{u'url': u'/download/53c3b896e34e240705e236c9.pdf?format=pdf', u'source': u'http://drawings.amphenolrf.com/ARFA_CUSTOMER_OUTLINE/C31-6528.PDF', u'id': u'53c3b896e34e240705e236c9'}]}}, u't': 2L, u'v': 2, u'ns': u'eds2.part', u'o2': {u'_id': ObjectId('546e1445e34e2431397eec78')}, u'op': u'u'}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/oplog_manager.py", line 274, in run
    ns, timestamp)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/util.py", line 35, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 291, in update
    self.upsert(updated, namespace, timestamp, update_spec)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/util.py", line 35, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 323, in upsert
    self.index(action, meta_action, doc, update_spec)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 473, in index
    self.commit()
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 495, in commit
    self.send_buffered_operations()
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 482, in send_buffered_operations
    action_buffer = self.BulkBuffer.get_buffer()
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 693, in get_buffer
    self.update_sources()
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/util.py", line 35, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 648, in update_sources
    updated = self.docman.apply_update(source, update_spec)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 224, in apply_update
    return super(DocManager, self).apply_update(doc, update_spec)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/doc_managers/doc_manager_base.py", line 120, in apply_update
    exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/doc_managers/doc_manager_base.py", line 114, in apply_update
    _unset_field(doc, to_unset)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.5.2.dev0-py2.7.egg/mongo_connector/doc_managers/doc_manager_base.py", line 89, in _unset_field
    del doc[to_unset]
UpdateDoesNotApply: Cannot apply update {u'$unset': {u'datasheets': True}} to {u'updated': datetime.datetime(2017, 7, 28, 11, 58, 25, 115000), u'manufacturerref': u"DBRef(u'manufacturer', ObjectId('58d4d4615a8ec9796c8b45e4'))", u'familyPartsCount': 0.0, u'partno': u'ADAS1126', u'desc': u'Analog to Digital Converters', u'changed': datetime.datetime(2017, 7, 28, 11, 58, 25), u'stateref': u"DBRef(u'part_state', ObjectId('57c71322a32430d252bfcf19'))", u'pricerefs': [u"DBRef(u'distributorpart', ObjectId('58ff1b528322bedce104e217'))"], u'url': u'adas1126', u'state': {u'id': u'57c71322a32430d252bfcf19', u'name': u'Active'}, u'equivalentPartrefs': [u"DBRef(u'part', ObjectId('5317a4c2e34e24960251be07'))"], u'refdesignCount': 0.0, u'priceCount': 1, u'equivalentParts': [{u'partno': u'DDC232', u'_id': u'5317a4c2e34e24960251be07', u'replacementType': u'P', u'manufacturer': {u'url': u'texas-instruments', u'_id': u'517a3c31e34e24f40f04d681'}, u'desc': u'DDC232 32 Channel Current-Input 20-bit ADC'}], u'partnolc': u'adas1126', u'pricesUpdated': datetime.datetime(2017, 7, 28, 11, 58, 25), u'manufacturer': {u'url': u'analog-devices', u'trackingParams': [], u'name': u'Analog Devices', u'eiTrackingParams': []}, u'isFamilyPart': False, u'name': u'ADAS1126'}

I am not very experienced with Python but I had a look myself, and I believe the issue is that Python is throwing an error due to the attempt to remove a dictionary element that does not exist :-

del doc[to_unset]

I did some googling and I think the solution is to use

doc.pop(to_unset, None)

I made the change to my version and it has stopped the errors, however I would like a real Python developer to review. I will submit a change request, so that the change can be reviewed and corrected if needed.

[SebC99]

any news on this? I do have the exact same issue

[levofski]

No responses, I have just kept using my altered version and it is working fine.

[radzhome]

The PR to replace del with pop seems to be failing. Also getting this issue now all of a sudden.