Using jiant to make BioBERT and SciBERT to run superGLUE tasks

[shi-kejian]

Hi,
As always, thank you for this amazing contribution.

I am taking prof.bowman's class and attempting to run BioBERT https://github.com/dmis-lab/biobert-pytorch and SciBERT https://github.com/allenai/scibert on jiant. One of our objectives for the course project is to run BioBERT and SciBERT on common NLU tasks.

As far as I understand, it should be possible to add any Transformers encoder model to jiant, but both of those models will probably require a bit of code?

I am sketching out what I'd have to do in jiant vs. plain transformers. Will using jiant create more overheads (by trying to support those models) than just following the standard fine-tuning processes in transformers?

Any suggestions and pointers will be helpful. Thank you!

[jeswan]

I think either fine-tuning with transformers or using jiant is suitable for your task. If you decide to use jiant, I will guide you through the process. BioBERT and SciBERT are available as pertained models here: https://huggingface.co/models. Both of the models you mentioned appear to have a BERT architecture with different pretraining objectives. I am currently working on a feature branch (https://github.com/nyu-mll/jiant/tree/js/feature/easy_add_model - should be merged to main in the next week) that is suitable for your goal. Working off the feature branch would be a good place to start. You can simply use the model name from the Transformers Model Hub as an argument(hf_pretrained_model_name_or_path="allenai/scibert_scivocab_cased") in the simple API to fine-tune the models.

What tasks are you planning on running? The following notebook is an example of how to fine tune tasks currently supported by jiant: https://colab.research.google.com/github/nyu-mll/jiant/blob/master/examples/notebooks/simple_api_fine_tuning.ipynb

List of supported tasks: https://github.com/nyu-mll/jiant/blob/master/guides/tasks/supported_tasks.md

[shi-kejian]

Thank you very much Jesse for the prompt reply!

Yeah, I think using jiant will definitely save us lots of code and time.

For the mllu course project, we aim to measure the degree to which using a different pre-training corpus would change that affect both in terms of the amount of bias learned and used in downstream applications but also what is the tradeoff with actual pure NLU performance. I am planning to run BERT, RoBERTa, BioBERT, and SciBERT on ALL SuperGLUE tasks, because I don't think there is a way to clearly say apriori which tasks would be best at discriminating the models.

Back to my question. I guess the main issue here is that I'd better finish the tasks by the end of this week because there will be a partial draft due next Thursday, so I am afraid I will not be able to use your amazing new feature :(. However, by looking at your notebook examples, I think I can also use the current main branch of jiant right? by loading different models in line say MODEL_TYPE = "roberta-base". Or did I miss anything?

Again, thank you very much. And it would be much appreciated if I could ask more follow-up questions during the process (I am planning to install jiant on NYU greene and run the tasks). @jeswan

edit: sorry I accidentally removed myself as an assignee for this thread. If you need it, please reassign me (I guess that's how you keep track of the issues?) Thank you so much. @jeswan

[jeswan]

jiant would be well-suited for what you described. You could use the main branch of jiant but may need to make some modifications for jiant to recognize SciBERT as a BERT model. As a starting point, I would use this notebook: https://colab.research.google.com/github/nyu-mll/jiant/blob/master/examples/notebooks/jiant_Basic_Example.ipynb. You will need to change the following:

export_model.export_model(
    hf_pretrained_model_name_or_path="allenai/scibert_scivocab_cased",
    output_base_path="./models/scibert",
)

In the subsequent steps, you will need to modify jiant to treat allenai/scibert_scivocab_cased the same as bert (this is not required in the feature branch mentioned above). Feel free to ask any additional questions during the process.

I previously assigned the issue to you to notify you that I required your feedback. For now, I'll close this issue but please reopen the issue and comment if you have more questions.

[shi-kejian]

Thank you very much for the resource.

I would like to quickly follow up tho. When you mentioned "to modify jiant", did you mean we need to somehow modify the source code of jiant, after we install jiant on Greene HPC? Or do we just need to modify, say, the example notebook to make sure all configs are correct for each model and task?
Right now I am setting up jiant (master branch) on HPC.
Thank you for your time.

edit: Or maybe I just install the feature branch?

[jeswan]

Modify the source code. However, if you use the feature branch above (should be merged into master within a week), you should not need to modify the source code.

[shi-kejian]

Thank you.
I think I can first run BERT and RoBERTa (already supported) using the simple fine-tuning API and get the results.
Since that's just a partial draft due next week plus we had other finished bias detecting experiments, so I think I can leave running BioBERT and SciBERT on SuperGLUE later next week, using your amazing new feature.

Thanks again.

[shi-kejian]

Hello, I actually run into a problem. I installed jiant v2 on HPC Greene. And I am running the simple API using python. I am running roberta-large on superglue tasks. However, it says that superglue_axg and superglue_axb are not supported? Here is the screenshot:

Thank you very much.

[jeswan]

Please use superglue_broadcoverage_diagnostics and superglue_winogender_diagnostics. The following discussion may also be relevant: #1255

[shi-kejian]

Thanks for the prompt reply!

[shi-kejian]

Dear Jesse,
Just checking in -- will the easy_add_model branch be merged to master recently as expected?
We were simply using the Simple API.
.....
..... (previous information deprecated)
Thank you for the amazing tool!! @jeswan

Update: Sorry I think I was confused! So previously you actually meant the feature branch IS WORKING right? I just need to download that branch not the master... So I don't need to wait till it merged to main.

Update: I was working on the easy_add_model feature branch just using Simple API. However, it seems like the data path was not compatible with the current master branch.

Here are the details:
I have a directory called "biobert_base" inside /jiant.
And here is my python code using Simple API to run Biobert on BoolQ.

Downloading the data and model seems good:

However, right before it starts to tokenize: I have an error:

So it seems like somewhere the path gets concatenated twice? I am trying to locate which files are deprecated. If this sounds/looks familiar to you, any pointer would be appreciated. Thank you so much!

[shi-kejian]

Hey! I just saw the easy_add_model branch is in the PR. Congrats! @jeswan
I think my error described above may help you resolve some conflicts? Also, I will be more than happy to help you test the latest updates. I will use several other models to run both Simple and Main API.

Thank you so much!

[jeswan]

@gokejian I will try BioBert with the PR branch to try and replicate the above error!

[jeswan]

@gokejian Fixed the issue and will push the fix to the easy_add_model branch. If you want to fix the issue locally use the following for RUN_NAME in the simple api notebook:

# Remove forward slashes so RUN_NAME can be used as path
MODEL_NAME = HF_PRETRAINED_MODEL_NAME.split("/")[-1]
RUN_NAME = f"simple_{TASK_NAME}_{MODEL_NAME}"

[shi-kejian]

Thank you. Now the path issue is fixed.

However, I am still getting some errors:

This is the code for BioBERT to run BoolQ:

Error file:

So I think there might be a few commits behind the master branch.

Hope this helps! Thank you.

[shi-kejian]

Yeah I think this error applies to other tasks also:
Here is the CB task:

[jeswan]

Ah, I know what is going on here. I'll push a fix for this shortly (need to add num_labels property to the Task object or slightly change how that label is used).

[shi-kejian]

Cool thanks! BTW all tasks inherit the Task object right? Is it possible for a local fix? I am trying to download the updated feature branch of jiant

[jeswan]

Local fix: Change line 89 in heads.py to self.out_proj = nn.Linear(hidden_size, len(task.LABELS))

[shi-kejian]

Thank you. Here is the update:

1. no errors for BoolQ, CB, RTE, MultiRC, COPA

2. Running WIC task yields such error, and WSC also has a similar error.

.

[jeswan]

I'll also fix this :) Another task required the switch back to slow tokenizers but it looks like fast tokenizers are still referenced.

[shi-kejian]

Thanks! so if local fixes are possible please let me know. Otherwise, I will download the latest checkpoint.

[jeswan]

Can you try the HEAD of the feature branch? I was not able to replicate your issue with the latest commit.

[shi-kejian]

Oh my bad. Yes, the current HEAD is working fine. Thanks.