Serialization: store_user_data is ignored in DocBin constructor

[nrodnova]

How to reproduce the behaviour

When I try to create a DocBin out of documents that have Spans in custom attributes, I get an exception, even though I set store_user_data = False. The exception is thrown in DocBin.add() function, line 113 of _serialize.py file:

self.user_data.append(srsly.msgpack_dumps(doc.user_data))

The exception is thrown because srsly.msgpack_dumps(doc.user_data) cannot serialize Spans. By mistake, it is called even when self.store_user_data is set to False

The following code reproduces the behavior:

import spacy
from spacy.tokens import Doc
from spacy.tokens import DocBin
nlp = spacy.load('en_core_web_sm')
Doc.set_extension(name = 'test', default = None, force = True)

text = "Hello, world!"
doc = nlp(text)
doc._.test = doc[0].sent # Assign a span to the doc's custom attribute
docs = [doc]

## Create DocBin
attributes = ("ORTH", "TAG", "HEAD", "DEP", "ENT_IOB", "ENT_TYPE", "ENT_ID", "ENT_KB_ID", "LEMMA", "MORPH", "POS", "LOWER", "SENT_START", "SENT_END", "IDX", 'IS_PUNCT', 'LIKE_NUM', 'IS_BRACKET', "IS_LEFT_PUNCT", "IS_RIGHT_PUNCT")
doc_bin = DocBin(attrs = attributes, store_user_data = False, docs = docs)

The error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-5796ac30343d> in <module>
      1 ## Store
      2 attributes = ("ORTH", "TAG", "HEAD", "DEP", "ENT_IOB", "ENT_TYPE", "ENT_ID", "ENT_KB_ID", "LEMMA", "MORPH", "POS", "LOWER", "SENT_START", "SENT_END", "IDX", 'IS_PUNCT', 'LIKE_NUM', 'IS_BRACKET', "IS_LEFT_PUNCT", "IS_RIGHT_PUNCT")
----> 3 doc_bin = DocBin(attrs = attributes, store_user_data = False, docs = docs)
      4 #serialized_docs = doc_bin.to_bytes()

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/spacy/tokens/_serialize.py in __init__(self, attrs, store_user_data, docs)
     78         self.store_user_data = store_user_data
     79         for doc in docs:
---> 80             self.add(doc)
     81 
     82     def __len__(self) -> int:

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/spacy/tokens/_serialize.py in add(self, doc)
    109             self.strings.add(token.ent_kb_id_)
    110         self.cats.append(doc.cats)
--> 111         self.user_data.append(srsly.msgpack_dumps(doc.user_data))
    112         self.span_groups.append(doc.spans.to_bytes())
    113         for key, group in doc.spans.items():

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/_msgpack_api.py in msgpack_dumps(data)
     12     RETURNS (bytes): The serialized bytes.
     13     """
---> 14     return msgpack.dumps(data, use_bin_type=True)
     15 
     16 

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/msgpack/__init__.py in packb(o, **kwargs)
     53     Pack an object and return the packed bytes.
     54     """
---> 55     return Packer(**kwargs).pack(o)
     56 
     57 

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/msgpack/_packer.pyx in srsly.msgpack._packer.Packer.pack()

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/msgpack/_packer.pyx in srsly.msgpack._packer.Packer.pack()

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/msgpack/_packer.pyx in srsly.msgpack._packer.Packer.pack()

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/msgpack/_packer.pyx in srsly.msgpack._packer.Packer._pack()

~/python-virtualenv/eon-spacy-3/lib/python3.7/site-packages/srsly/msgpack/_packer.pyx in srsly.msgpack._packer.Packer._pack()

TypeError: can not serialize 'spacy.tokens.span.Span' object

Your Environment

• spaCy version: 3.0.6
• Platform: Darwin-20.3.0-x86_64-i386-64bit
• Python version: 3.7.9
• Pipelines: en_core_web_lg (3.0.0), en_core_web_md (3.0.0), en_core_web_sm (3.0.0)

[polm]

Thanks for the report, I confirmed this is still an issue with 3.1.2. (I also got some separate errors related to the attributes, though I need to look at that more.)

We'll take a look at fixing this.

[polm]

This should be fixed by #9226. Thanks again for the report!

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.