Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Multitask Learning in Flair #2910

Merged
merged 23 commits into from
Aug 17, 2022
Merged

Support for Multitask Learning in Flair #2910

merged 23 commits into from
Aug 17, 2022

Conversation

alanakbik
Copy link
Collaborator

@alanakbik alanakbik commented Aug 17, 2022

This PR adds support for multitask learning in Flair (closes #2508 and closes #1260) with hopefully a simple syntax to define multiple tasks that share parts of the model.

The most common part to share is the transformer, which you might want to fine-tune across several tasks. Instantiate a transformer embedding and pass it to two separate models that you instantiate as before:

# --- Embeddings that are shared by both models --- #
shared_embedding = TransformerDocumentEmbeddings("distilbert-base-uncased", fine_tune=True)

# --- Task 1: Sentiment Analysis (5-class) --- #
corpus_1 = SENTEVAL_SST_GRANULAR()

model_1 = TextClassifier(shared_embedding,
                         label_dictionary=corpus_1.make_label_dictionary("class"),
                         label_type="class")

# -- Task 2: Binary Sentiment Analysis on Customer Reviews -- #
corpus_2 = SENTEVAL_CR()

model_2 = TextClassifier(shared_embedding,
                         label_dictionary=corpus_2.make_label_dictionary("sentiment"),
                         label_type="sentiment",
                         )

# -- Define mapping (which tagger should train on which model) -- #
multitask_model, multicorpus = make_multitask_model_and_corpus(
    [
        (model_1, corpus_1),
        (model_2, corpus_2),
    ]
)

# -- Create model trainer and train -- #
trainer = ModelTrainer(multitask_model, multicorpus)
trainer.fine_tune(f"resources/taggers/multitask_test")

The mapping part here defines which tagger should be trained on which corpus. By calling make_multitask_model_and_corpus with a mapping, you get a corpus and model object that you can train as before.

In addition, this PR makes some experimental changes that will likely be adapted further:

  • adds support for gradient reversal to all models inheriting from DefaultClassifier
  • changes loss reduction from 'mean' to 'sum' in DefaultClassifier
  • removes the loss calculation in the final test in training

Edit: In addition, this PR removes some model classes that were very beta: the DependencyParser, the DistancePredictor and the SimilarityLearner. These may get added back when the model interfaces are finalized.

@alanakbik alanakbik merged commit 4045942 into master Aug 17, 2022
@alanakbik alanakbik deleted the multitask_mod branch August 17, 2022 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Multitask Learning in flair Multitask learning
2 participants