annotating_components don't seem to annotate

[marcuscollins]

I am playing around with building a custom parser, for which I want to add some custom features derived from dependencies and lemmas. I'm using Spacy 3.2 with python 3.8.11. Here's the top of the config:

[paths]
train = null
dev = null
vectors = "en_core_web_lg"
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec", "tagger", "parser", "lemmatizer", "extra_features", "ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

the "extra_features" part is borrowed from #6527, and the "ner" part is based on the rel_components project/tutorial from Explosion, but is heavily modified. That part works, but previously only used features that were from tok2vec. The first four parts of my pipeline are all taken from en_core_web_lg, like so:

[components.tok2vec]
source = "en_core_web_lg"
component = "tok2vec"

[components.tagger]
source = "en_core_web_lg"
component = "tagger"

[components.parser]
source = "en_core_web_lg"
component = "parser"

[components.lemmatizer]
source = "en_core_web_lg"
component = "lemmatizer"

While debugging the "extra_features" pipe, I discovered that

1. it didn't seem to be doing anything, as it was only producing the default value for the extension (doc._.extra_features).
2. the values for the tagger, parser, and lemmatized were all also the defaults, which is to say, 0 or an empty string.

Here, for instance, are the first four rows of the usual features (these happen to be NORM, PREFIX, SUFFIX, POS):

ipdb> attrs
array([[ 7368681034670656470, 16900879642891266615,  7409412530419520829, 0],
           [18194338103975822726,  2985121464356781022, 11402152577528684049, 0],
           [ 2104994216896503478,  8861071527689086543, 11446655443681503830, 0],
           [  602994839685422785,   602994839685422785,   602994839685422785, 0],

and the extra_features, which are just the default value of np.ndarray(size=(10)):

ipdb> doc._.get(extra_column)
array([ 2.68156159e+154,  1.29074022e-231,  2.47032823e-323,
        0.00000000e+000,  2.68156159e+154, -2.68678733e+154,
        1.48219694e-323,  0.00000000e+000,  0.00000000e+000,
        6.95335581e-309])

If the system was producing the extra features correctly, it should be a 1 x 17 array of ones and zeros. I chose the default so that the system would break if it didn't produce the right output on test sentences of any other length. But then I noticed that all the tags were 0, and started digging around to find that none of .tag, .dep, and .lemma were actually set.

It was my understanding that adding tagger, parser, lemmatizer to the annotating_components would cause them to produce the required values, but that does not seem to be happening. Here the the training section of my config:

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = ["tok2vec", "tagger", "parser", "lemmatizer"]
annotating_components = ["tok2vec", "tagger", "parser", "lemmatizer"]
before_to_disk = null

And finally, the pipe definition for extra_features:

@Language.component(
        "extra_features",
        requires=["token.tag", "token.dep", "token.lemma"],
        assigns=["doc._.extra_features"]
        )
def compute_extra_features(doc):
    features = []
    for tk in doc:
        tk_features = [
            # Compute binary dependency type features, i.e. is it a dobj child of "has"
            has_ancestor(tk, 'dobj', 'be'),
            has_ancestor(tk, 'nsubj', 'be')
            # compute binary features for whether it matches any of our attribute name/value token matching rules
            ]
        features.append(np.array(tk_features))
    doc._.extra_features = np.array(features)
    return doc

Any idea what I'm doing incorrectly? It's really quite frustrating that this doesn't seem to be easy, and that there isn't better documentation. I have considered the possibility that by changing the FeatureExtractor and MultiHashEmbed that I'm using, maybe I'm messing up the original Tok2Vec pipeline, but mine are registered under different names and the functions have different names too, so I don't see how that would happen. It would be nice to figure out where I'm at in the pipeline when this happens.

[adrianeboyd]

Sorry you're having trouble with this.

annotating_components is pretty simple under the hood and I doubt that it's the source of the problem. You can try out the unit tests (https://github.com/explosion/spaCy/blob/caf8528af79dd062f30bdcb54ee4be561063c995/spacy/tests/pipeline/test_annotates_on_update.py) or a demo project (https://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_predicted_annotations) to double-check that it's working how you expect for simpler examples.

If you need POS or LEMMA from en_core_web_lg, you also need to include attribute_ruler. More details on the pipeline design: https://spacy.io/models#design

From the info above, it looks like you might also need to add extra_features as an annotating component so that its annotation is set for the following component?

Developing a new tok2vec embed architecture and feature extractor that handle features outside of the TokenC features from doc.to_array is often the trickier part of this.