Dependency parser

[73minerva]

This PR adds dependency parser model based on Deep biaffine attention for neural dependency parsing. The main model in paper uses a static word embedding and POS tags vector as input, while it only uses StackedEmbeddings.
The first trained model can be downloaded from here. It was trained with Glove and Flair embeddings on flair.datasets.UD_ENGLISH dataset. If I understand correctly, the UD_ENGLISH dataset in flair library, only contains the "EWT" version, not the other ones mentioned in this. In the end, the following results were obtained:
UAS : 0.9032 - LAS : 0.9243
In the near future, I will train the model on PTB dataset to have a comparison with paper results.

For better printing of parsed sentences, I add the "print_tree" param to predict method:

sentence = Sentence('I prefer the morning flight through Denver.')
dep_parser_model: DependencyParser = DependencyParser.load(path_to_model)
dep_parser_model.predict(sentence, print_tree=True)

This should print something like this:

It is not done yet and needs some refactoring. Until then, it will be great if you share your comments and suggestions.

[alanakbik]

@73minerva really cool, thanks for adding this and sorry for not getting around to reviewing this sooner.

I'm trying to train a model, but its throwing errors if the mini-batch size is higher than 1. See script below - am I instantiating it wrong?

corpus = UD_ENGLISH()

dictionary = corpus.make_label_dictionary("dependency")

model = DependencyParser(token_embeddings=WordEmbeddings("turian"), relations_dictionary=dictionary)

trainer = ModelTrainer(model, corpus)

trainer.train("resources/taggers/dependency", mini_batch_size=2)

[73minerva]

@alanakbik, The recent commit should fix the mentioned mini-batch error. I also added word dropout.
Here is a trained dependency parser model with bert that achieved UAS : 0.9210 - LAS : 0.9354.

[stefan-it]

Hi @73minerva ,

thanks for that PR! I wanted to test it with Transformer-based models, but this failed due to an error that was I think fixed a while ago on Flair master. So could you please rebase your PR on latest master branch 🤔 Many thanks!

[alanakbik]

A few minor comments and some questions on parameter choices.

If you prefer, I can merge this PR now and do the rebase to master @stefan-it mentioned and we can do further improvements in more PRs.

[alanakbik]

what does this line do?

[alanakbik]

This layer does not seem to be used anywhere

[alanakbik]

Curious: Why not use the default implementation of LSTM bei PyTorch? Is the variational aspect important for performance?

[alanakbik]

These default dropout values seem quite high. Have you tried lower values?

[73minerva]

All default hyperparameters including dropout, were set according to the paper. As mentioned in the paper, to reduce label classifier overfitting, increasing dropout is necessary. More details and why label classifier suffers from overfitting are explained in the 4.2.1 section of the paper. I haven't tried lower values and unfortunately, there is no experiment about it.

[alanakbik]

this is sometimes a tensor and sometimes an int, leading to uneven printouts

[73minerva]

Thanks, It's fixed on the recent commit. Also, I add CSV header to prevent confusion. it's better now.

[alanakbik]

@73minerva thanks a lot for adding this! And sorry for taking so long to review!

[alanakbik]

@73minerva another thing: the LAS should only count positives if both attachment and deprel are correctly predicted. So the LAS should always be lower than the UAS. I have it fixed in my local branch and will do a PR soon!