Only for doc classification models

[cayorodriguez]

Hi,
It looks like this will only work for doc classification tasks, not for token classification fine tunning, like NER, right?

[KennethEnevoldsen]

Indeed. At the model it does not work for token prediction, nor span prediction like NER (see also #36). This should be implementable though.

Do you have a specific model in mind?

[cayorodriguez]

Hi, Yes, we were thinking about showcasing in Spacy our BioNER models in Spanish, for example: https://huggingface.co/PlanTL-GOB-ES/bsc-bio-ehr-es-pharmaconer Without training a new one specifically with Spacy Thanks ============== Carlos Rodriguez ==============

…

On Mon, Nov 7, 2022 at 1:22 PM Kenneth Enevoldsen ***@***.***> wrote: Indeed. At the model it does not work for token prediction, nor span prediction like NER (see also #36 <#36>). This should be implementable though. Do you have a specific model in mind? — Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2FKPRETKPCLLLIVBG2KQDWHDYAXANCNFSM6AAAAAARZBXK7Q> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[KennethEnevoldsen]

Sure I get that. Probably not that big of an issue to add. If have can send me the code for the forward pass using transformers I will try to implement it during the weekend (currently at a conference).

[cayorodriguez]

Hi Kenneth, that would be great. I am not clear about what you mean about the code for the forward pass ... The idea is to use your wrapper to add a component to a Spacy pipeline that can detect entities in clinical text, such as proteins, drugs, etc.
let us know how we can help ...

[cayorodriguez]

Sorry about the delay, our models use the RobertaForTokenClassification forward pass

[cayorodriguez]

This is the code:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/roberta/modeling_roberta.py#L1384

[KennethEnevoldsen]

Sorry for the misunderstanding. I was referring to code for using the model on an example. I would use it to ensure my implementation works as intended.

I can also reconstruct it from the example on the API site.

[cayorodriguez]

No problem, thanks! Once we have it we'll add the component to a standard Spacy pipeline, and apply it as NER. ============== Carlos Rodriguez ==============

…

On Wed, Nov 9, 2022 at 9:54 AM Kenneth Enevoldsen ***@***.***> wrote: Sorry for the misunderstanding. I was referring to code for using the model on an example. I would use it to ensure my implementation works as intended. I can also reconstruct it from the example on the API site. — Reply to this email directly, view it on GitHub <#74 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2FKPWAPIZDKTCHBZHN63DWHNRCZANCNFSM6AAAAAARZBXK7Q> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[KennethEnevoldsen]

Hi @cayorodriguez, just letting you know that I started working on this, but it might take a bit longer than expected

[cayorodriguez]

No worries, thanks for the update! ============== Carlos Rodriguez ==============

…

On Sun, Nov 13, 2022 at 4:41 PM Kenneth Enevoldsen ***@***.***> wrote: Hi @cayorodriguez <https://github.com/cayorodriguez>, just letting you know that I started working on this, but it might take a bit longer than expected — Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2FKPWCQ2V7MO7EUMLRFJ3WIED3TANCNFSM6AAAAAARZBXK7Q> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[KennethEnevoldsen]

Still working on this, but I assume an update will be out soon.

It is taking more time than expected since I need to handle both NER and token predictions (pos, tag) in a single pipeline as they both use the huggingface TokenClassificationPipeline (I don't use that, though, but try to model the spacy component after that such that the predictions should be the same across the two frameworks).

[cayorodriguez]

Hi Kenneth, We appreciate the effort, but don't worry about it. We don't have any deadline for this. We think it would be an amazing way to make our models (and indeed any HF models) available for anyone, without having to know how to train a Spacy pipeline, which is not trivial at all. Cheers! ============== Carlos Rodriguez ==============

…

On Tue, Nov 22, 2022 at 3:29 PM Kenneth Enevoldsen ***@***.***> wrote: Still working on this, but I assume an update will be out soon. It is taking more time than expected since I need to handle both NER and token predictions (pos, tag) in a single pipeline as they both use the huggingface TokenClassificationPipeline (I don't use that, though, but try to model the spacy component after that such that the predictions should be the same across the two frameworks). — Reply to this email directly, view it on GitHub <#74 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2FKPVHSTBQHRTW3AXS3XTWJTKGHANCNFSM6AAAAAARZBXK7Q> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[KennethEnevoldsen]

No worries. I simply wanted to let you know that progress was being made - and indeed, it would be great not to have to retrain models - sometimes, that is not even possible due to a lack of data

[KennethEnevoldsen]

So progress is being made. It is almost solved for all cases.

Currently one of the only problems I have left is an edge case related to the spacy tokenizer. Using your example:

Se realizó estudio analítico destacando incremento de niveles de PTH y vitamina D (103,7 pg/ml y 272 ng/ml, respectivamente), atribuidos al exceso de suplementación de vitamina D.

Where I extract the entities:

# word, token id, label
("PTH", (9, 10), "PROTEINAS"),
("vitamina D", (11, 13), "NORMALIZABLES"),
("vitamina", (33, 34), "NORMALIZABLES")

Note the missing D in the second Vitamina D. It is caused by the D. (note the punctuation) is considered as one token by spaCy. So as with any word consisting of multiple word pieces, I aggregate the logits (a similar strategy as in the HF TokenClassificationPipeline). The only way around that is to replace the tokenizer (it is essentially the component which makes the error). Tried out the stanza tokenizer as well (from spacy_stanza), which is learned, but it had the same error.

nlp = spacy.blank("es")
nlp.add_pipe("token_classification_transformer", config=config)
doc = nlp(TEXT)
doc.ents # just as any other ner pipeline in spacy

# additionally you can access:
doc.tok_clf_trf_data  # model output, along w. token alignment etc.
doc.tok_clf_predictions  # a list of dicts containing probabilities for each token

Where the config looks like

config = {"model": {"name": "PlanTL-GOB-ES/bsc-bio-ehr-es-pharmaconer"}}

Or more explicitly:

        {
            "doc_extension_trf_data": "tok_clf_trf_data",
            "doc_extension_prediction": "tok_clf_predictions",
            "labels": None,  # infer from model
            "predictions_to": None # in which case it infers from iob labels, or ["ents"] in which case it is added to doc.ents as well.
            "aggregation_strategy": "average" # how tokens should be aggregated to match w. spacy tokens.
            "model": {
                "name": "PlanTL-GOB-ES/bsc-bio-ehr-es-pharmaconer",  # hub model id 
                "@architectures": "spacy-wrap.TokenClassificationTransformerModel.v1",
            },
        },

Let me know if there is anything missing. Will finalize the PR (#75) in the coming days.