Overall Tasks on Repair

[yogeswarl]

No context

1. Transformer Fine-Tuning
   • #24
   • Bert (or other one)
2. Choice of pairing
   • query.docs
   • query.doc
   • docs.query
   • doc.query
3. Query Set
   • msmarco.passage
   • msmarco.document
   • Aol-title
   • Aol-tile-url
   • Aol-text
   • Yahoo Q & A

Information Retrieval

1. Sparse Retrieval
   • BM25
   • qld
2. Dense Retrieval
   • Colbert -- > Next step #20

• Hybrid Retrieval

Evaluation

• MAP
• MRR
• Success
• nDCG

Stats

• BLEU, ROUGE-L and F1 score between original and refined queries #21

Supervised Query Refinement

• Acg
• Anmt
• Hredqs
• transformer

Context

• Context aware creation of large scale datasets with user as context #23
• Time
• QueryType

Paper Writeup

• Related Work
• Benchmark on RePair
• ...

[hosseinfani]

@yogeswarl
any update?

[yogeswarl]

Sorry @hosseinfani, Forgot to update this issue page. I have completed the MRR and boxes computation of title ,title-url . I am keeping them in a separate folder from the current publicly available data.
I have finished computing TCT Colbert for MSMarco originals. I still have to do the 25 predicted queries. The only problem is the tct index is 31GB with encoders of 441MB. The search is relatively slow when compared to BM25. But I am hoping to give you the results of MSMARCO's dense by this weekend. If this works. I will start with Hybrid retrieval as well. Once these are done, I will move on to AOL context version and run the whole results in the pipeline!

[hosseinfani]

@yogeswarl
These are the remaining items. Please read and come up with timeline. I need you to drop by during office every week and report on your progress.

sparse experiment:

• result of mrr, files, ...

colbert experiment:

• Details of training, indexing, ... in readme
• Details of subset of queries we did the retrieval and explain why? Selection criteria??
• results of mrr, map, files, ...

Benchmarks

• bm25 on acg, hredqs, ...
• colbert on acg, hredqs, ...

On paper:

• Benchmarks on RePair’s Gold Standard Datasets
• RELATED WORK

[yogeswarl]

Hello @DelaramRajaei and @ZahraTaherikhonakdar, This is the main tasks list of completed works on Repair. Can you please add the query sets and variants to this task to keep track of all the works done on repair.

Thanks