Skip to content

Sentence Embeddings using Supervised Contrastive Learning. Danqi Liao.

Notifications You must be signed in to change notification settings

BoilerToad/584-final

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

29 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Sentence Embeddings using Supervised Contrastive Learning

This code accompanies the paper: Sentence Embeddings using Supervised Contrastive Learning

Danqi Liao.

This is a final project for COS 584.

scl

To download data(SNLI) from HuggingFace and train:

python bert_sent_embed.py --pos_num -1 --neg_num -1 --use_SCL

To load data from local disk and train:

python bert_sent_embed.py --load_data_from_disk --pos_num -1 --neg_num -1 --use_SCL

To evaluate trained model on downstream sentence tasks through SentEval

  1. cd SentEval-master/examples/
  2. Modify 'sentbert_eval.py' to change $MODEL_PATH to your model
  3. Modify 'sentbert_eval.py' to change $transfer_tasks to the tasks you want to evaluate
  4. Run the evaluation script
python sentbert_eval.py

To generate sentence embedding using trained model

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = SentBert(512*3, 3, tokenizer)
model.load_state_dict(torch.load(model_path, map_location=device))

embedding = model.encode("hello world.")

Evaluation results on Avg Glove embeddings, our SBERT baseline, all positive/negative SCL model with lambda 0.3

Tables STS(12-16) AVG Sentence Transfer Tasks AVG
Avg. GloVe Embeddings 44.98 74.27
Our SBERT baseline 67.61 75.56
allpalln-lambda0.3-SCL 70.44 76.16

Note, our SBERT baseline is not the full scale SBERT model from SBERT, but rather our own replementation using only SNLI data, medium-sized 8/512 bert, and the same hyperparameters with the SCL models. The reason for using a smaller sized bert and only SNLI data is simply computation constraints.

Example trained sentence encoding model

sent-bert8

About

Sentence Embeddings using Supervised Contrastive Learning. Danqi Liao.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 93.8%
  • Shell 6.2%