Can´t use Extension attributes in Tok2Vec

[bratao]

First of all, thanks for this awesome library.

I´m trying to use a custom attr in the Tok2Vec. I registered it according to https://nightly.spacy.io/usage/processing-pipelines#custom-components-attributes , but I can´t use it from tok2vec.

Did I did something wrong? I

How to reproduce the behaviour

@registry.architectures.register("my_tok2vec")
def build_hash_embed_cnn_tok2vec(
) -> Model[List[Doc], List[Floats2d]]:
    return build_Tok2Vec_model(
        embed=MultiHashEmbed(
            width=32,
            rows=[500],
            #attrs=["NORM", "PREFIX", "SHAPE"],
            attrs=["CUSTOM_ATTRIB"],
            include_static_vectors=False,
        ),
        encode=MaxoutWindowEncoder(
            width=32, depth=2, maxout_pieces=2, window_size=1
        ),
    )


Token.set_extension("CUSTOM_ATTRIB", default=None)
Doc.set_extension("CUSTOM_ATTRIB", default=None)

nlp = English()

config = {"model": {"@architectures": "my_tok2vec"}}
tok = nlp.add_pipe("tok2vec", config=config)

ner = nlp.add_pipe("beam_ner")
train_examples = []
for text, annotations in TRAIN_DATA:
    doc = nlp.make_doc(text)

    for pos, token_spacy in enumerate(doc):
        token_spacy._.set("CUSTOM_ATTRIB", "test")

    example = Example.from_dict(doc, annotations)
    train_examples.append(example)
    for ent in annotations.get("entities"):
        ner.add_label(ent[2])
optimizer = nlp.initialize()
print("training")

Error traceback

Traceback (most recent call last):
  File "E:/Projetos/spaCy/ner.py", line 84, in <module>
    optimizer = nlp.initialize()
  File "E:\Projetos\spaCy\spacy\language.py", line 1232, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "E:\Projetos\spaCy\spacy\pipeline\tok2vec.py", line 226, in initialize
    self.model.initialize(X=doc_sample)
  File "C:\Python38\lib\site-packages\thinc\model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "C:\Python38\lib\site-packages\thinc\layers\chain.py", line 86, in init
    layer.initialize(X=curr_input, Y=Y)
  File "C:\Python38\lib\site-packages\thinc\model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "C:\Python38\lib\site-packages\thinc\layers\chain.py", line 90, in init
    curr_input = layer.predict(curr_input)
  File "C:\Python38\lib\site-packages\thinc\model.py", line 312, in predict
    return self._func(self, X, is_train=False)[0]
  File "E:\Projetos\spaCy\spacy\ml\featureextractor.py", line 20, in forward
    attrs = doc.to_array(columns)
  File "spacy\tokens\doc.pyx", line 890, in spacy.tokens.doc.Doc.to_array
    cpdef np.ndarray to_array(self, object py_attr_ids):
  File "spacy\tokens\doc.pyx", line 926, in spacy.tokens.doc.Doc.to_array
    raise KeyError(Errors.E983.format(dict="IDS", key=msg, keys=keys)) from None
KeyError: "[E983] Invalid key for 'IDS': 'CUSTOM_ATTRIB'. Available keys: ['', 'IS_ALPHA', 'IS_ASCII', 'IS_DIGIT', 'IS_LOWER', 'IS_PUNCT', 'IS_SPACE', 'IS_TITLE', 'IS_UPPER', 'LIKE_URL', 'LIKE_NUM', 'LIKE_EMAIL', 'IS_STOP', 'IS_OOV_DEPRECATED', 'IS_BRACKET', 'IS_QUOTE', 'IS_LEFT_PUNCT', 'IS_RIGHT_PUNCT', 'IS_CURRENCY', 'ID', 'ORTH', 'LOWER', 'NORM', 'SHAPE', 'PREFIX', 'SUFFIX', 'LENGTH', 'CLUSTER', 'LEMMA', 'POS', 'TAG', 'DEP', 'ENT_IOB', 'ENT_TYPE', 'ENT_ID', 'ENT_KB_ID', 'HEAD', 'SENT_START', 'SENT_END', 'SPACY', 'PROB', 'LANG', 'MORPH', 'IDX']"

Process finished with exit code 1

Your Environment

Python 3.8 with Spacy compiled from Today (12/08/2020) v3 branch version.

[bratao]

This problem is related to #5382

The problem I´m trying to solve is that my NER need to have some external features from the document layout. Such as is_bold, is_italic, ....

[adrianeboyd]

Yes, that's the same underlying reason. In order to be fast, the provided spacy.Tok2Vec.v1 uses doc.to_array to convert the token attributes to a numpy array for the feature extraction step. doc.to_array can only access the Cython data, which are the attributes saved in the TokenC and LexemeC structs underneath: https://nightly.spacy.io/api/cython-structs , the list of attributes is in spacy.attrs (similar to what's available for the Matcher):

https://github.com/explosion/spaCy/blob/8921364579fa9285e7051c115f8babab47b359e0/spacy/attrs.pyx

You could implement an alternate hash embed with a custom feature extractor that can access custom attributes (the main drawback is that it will be slower), but we haven't provided this with the library (...yet?). It could look similar to how the Matcher uses the _ key to match on custom attributes. You can see what the feature extractor looks like that uses the attrs argument with doc.to_array in the end:

spaCy/spacy/ml/featureextractor.py

Lines 8 to 28 in 8921364

	@registry.layers("spacy.FeatureExtractor.v1")
	def FeatureExtractor(columns: List[Union[int, str]]) -> Model[List[Doc], List[Ints2d]]:
	return Model("extract_features", forward, attrs={"columns": columns})
	def forward(
	model: Model[List[Doc], List[Ints2d]], docs, is_train: bool
	) -> Tuple[List[Ints2d], Callable]:
	columns = model.attrs["columns"]
	features: List[Ints2d] = []
	for doc in docs:
	if hasattr(doc, "to_array"):
	attrs = doc.to_array(columns)
	else:
	attrs = doc.doc.to_array(columns)[doc.start : doc.end]
	if attrs.ndim == 1:
	attrs = attrs.reshape((attrs.shape[0], 1))
	features.append(model.ops.asarray2i(attrs, dtype="uint64"))
	backprop: Callable[[List[Ints2d]], List] = lambda d_features: []
	return features, backprop

One of the tricky parts would be dealing with arbitrary data types in custom extensions. For doc.to_array() we know that all the features are integers, which you don't know for the custom extensions.

[bratao]

@adrianeboyd I just did this, created a custom feature extractor, and it is working great.
For future users, here is the Custom feature extractor code:

def forward(
    model: Model[List[Doc], List[Ints2d]], docs, is_train: bool
) -> Tuple[List[Ints2d], Callable]:
    columns = model.attrs["columns"]
    possible_extra_columns = ["IS_BOLD", "IS_ITALIC"]
    extra_columns = [x for x in columns if x in possible_extra_columns]
    columns = [x for x in columns if x not in possible_extra_columns]
    features: List[Ints2d] = []
    for doc in docs:
        if hasattr(doc, "to_array"):
            attrs = doc.to_array(columns)
        else:
            attrs = doc.doc.to_array(columns)[doc.start : doc.end]
        if attrs.ndim == 1:
            attrs = attrs.reshape((attrs.shape[0], 1))

        # Lets add the extra_columns here
        extra_features = []
        for token in doc:
            token_features = []
            for extra_column in extra_columns:
                token_features.append(doc.vocab.strings[token._.get(extra_column)])
            extra_features.append(token_features)
        output_extra = numpy.array(extra_features)
        attrs = numpy.concatenate((attrs, output_extra), axis=1)

        features.append(model.ops.asarray2i(attrs, dtype="uint64"))

    backprop: Callable[[List[Ints2d]], List] = lambda d_features: []
    return features, backprop

[marcuscollins]

@bratao @adrianeboyd I tried this, but I have two problems:

1. It doesn't seem to produce the desired feature,
2. The required annotations from previous components don't seem to be set

Do you have a more complete example? I assume you have to create a new FeatureExtractor component using @Language.component so that those extra columns are available, and you need to use that forward function in some model, like so:

@registry.layers("spacy.FeatureExtractor.custom")
def ExtraFeatureExtractor(columns: List[Union[int, str]]) -> Model[List[Doc], List[Ints2d]]:
    return Model("extract_features", forward, attrs={"columns": columns})

and then write a very slightly different MultiHashEmbed function/model that uses the new feature extractor model. Am I missing something?

I opened a new discussion as well #10654

Thanks.

[adrianeboyd]

Great, it's nice to hear that it was easy to extend and customize! It's ones of the things we wanted to make a lot easier in v3.

[beatanyari]

hi @adrianeboyd,

For the custom attributes to be seen during training with the SpaCy project config, the corpus reader also has to be customized because the custom token attributes don't get copied over to the predicted doc. But even with the custom reader, the evaluate CLI doesn't work because it uses the standard corpus reader. Is there a better solution?

[adrianeboyd]

Not out-of-the-box at the moment, no. The easiest solution would be create a custom evaluate script. One option is to copy spacy/cli/evaluate.py, change the relative imports to absolute imports, and then customize it however you like. It should work fine as a standalone script once you modify the imports.

We don't want to use the corpus readers in the config because they're often customized for training and may even refer to resources / readers that you don't currently have, but maybe we can think of a way to support other corpus options here, since this is a reasonable thing to want to do.

[panamantis]

Was there a clean implementation of this in v3? ie/ I want to include a CUSTOM_ATTRIBUTE in my MultiHashEmbed in the tok2vec such that I can get more accurate NER classifications (ie/ include html tag info). (I prefer the statistical hashes rather then direct matches).
But it sounds like I need to manually adjust: the corpus reader, evaluate.py, featureextractor.py, architecture registries and extensions !! Now I'm confused. Maybe I should start another discussion?

[marzooq-unbxd]

I wanted to use custom attributes,how to use https://spacy.io/usage/processing-pipelines#custom-components-attributes and train using the CLI ?If possible can you give us a small example
As @panamantis asked, do we still have to make changes in so many files or only 'the config file and code functions file' in this case?