Luke NER Fine Tuning on Custom Entities.

[ahmadaii]

Hi, I am trying to fine tune luke base model and prepared conll like dataset with two columns (one for tokens and other for labels). The training runs smoothly but asserts no entity label when trying to make predictions.
Can I really know what changes in the code are supposed to be making luke ner solution useful for custom ner with different number of classes.

[elonmusk-01]

[elonmusk-01]

That's how my dataset looks like;

[elonmusk-01]

I have made one change in the code according to my entity numbers:

[elonmusk-01]

So can I get what I have to do additionally? Is my dataset structure CORRECT?

[ikuyamada]

Hi @ahmadaii,
If I correctly remember, the assertion (assert not entity_labels) aims to confirm that all gold entity spans in the dataset are detected after preprocessing (i.e., sentence and subword tokenization). entity_labels should be empty if all gold entity spans are detected.
Also, there is another implementation of our NER model based on AllenNLP, which may be easier to work with.

[elonmusk-01]

Alright. Is there any separate script for converting luke ner model to transformers or I have to use same script provided for converting re model??

[ikuyamada]

Actually, LUKE was added to the Transformers library after the experiments conducted in our paper. Therefore, the examples available in this repository do not support the Transformers library.

[elonmusk-01]

@ikuyamada I am referring to this script that was provided to convert luke re model to be used in transformers. Now I have fine tuned luke large model for NER task on my custom data. Is there any similar script for ner?

[binhna]

@elonmusk-01 Check out this script, although you need to modify few things to make it work

[NiteshMethani]

@binhna Were you able to run that script on custom dataset?
I tried but I am getting this error:
TypeError: forward() got an unexpected keyword argument 'ner_tags'

Any idea on how to fix it?

[binhna]

@NiteshMethani I managed to get it running, but not this script though. You could try this one, but you still need to edit some code to get it running.
The result in my language is quite bad though 😕

[NiteshMethani]

@binhna Thanks for sharing the script. I too was able to run it but with no luck. My performance even on conll2003 is:
{'precision': 0.8916275430359938, 'recall': 0.4036486007793128, 'f1': 0.5557181175323092, 'accuracy': 0.8966561566544972}

These were my hyperparameters for training:
python run_luke_ner_no_trainer.py
--model_name_or_path studio-ousia/luke-base
--dataset_name conll2003
--task_name $TASK_NAME
--max_length 128
--per_device_train_batch_size 32
--learning_rate 2e-5
--num_train_epochs 3
--output_dir /tmp/$TASK_NAME/

I wonder what's the reason for such poor F1 score on conll.
Would appreciate any suggestions @Ryou0634 @binhna @ikuyamada