Not able to prune the BERT model

[ishita1995]

Hi,
I am running the command

bash experiments/BERT/heads_ablation.sh MNLI

I am getting the following error

Traceback (most recent call last):
  File "pytorch-pretrained-BERT/examples/run_classifier.py", line 578, in <module>
    main()
  File "pytorch-pretrained-BERT/examples/run_classifier.py", line 275, in main
    model.bert.mask_heads(to_prune)
  File "/home/pdguest/ishita/py-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 535, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertModel' object has no attribute 'mask_heads'

[pmichel31415]

Hi, did you make sure to

1. clone my fork of pytorch-pretrained-BERT
2. checkout the paul branch?

[ishita1995]

Yes, I cloned your fork and checked out to paul branch

[pmichel31415]

Ok that is weird... Is it possible that you have another installation of the original pytorch-pretrained-BERT package that overrides this one?

For instance

python -c "import pytorch_pretrained_bert as ppb; print(ppb.__version__)"

Should print 0.4.0

[ishita1995]

Yes, it is indeed 0.4.0

[pmichel31415]

OK this is weird... I just checked again and it's working on my end. Did you check that you don't have pre-existing version of pytorch-pretrained-bert? It might be the same version (0.4.0). One way to check is to change the version number under pytorch-pretrained-BERT/pytorch_pretrained_bert/__init__.py (in this repo) to something like 0.4.1 and see whether

python -c "import pytorch_pretrained_bert as ppb; print(ppb.__version__)"

returns 0.4.0 or 0.4.1

Another workaround would be to create a new virtual environment and reinstall pytorch and my fork of pytorch-pretrained-bert

[insop]

I had a similar issue, since I have pytorch_pretrained_bert installed in system with different version.
So I use this way to specify installed directory, and the issue is fixed.

• https://stackoverflow.com/questions/4383571/importing-files-from-different-folder

Add the following to pytorch-pretrained-BERT/examples/run_classifier.py

+++ b/examples/run_classifier.py
@@ -24,6 +24,10 @@ import numpy as np
 import torch
 from torch.optim import SGD
 
+import sys
+sys.path.insert(0, '/PATH_TO/are-16-heads-really-better-than-1/pytorch-pretrained-BERT')
+

[pmichel31415]

Thanks @insop , I'll mark this issue as resolved and update the README with a link to your answer.