Add Regression Test for Car17 (#379)

README.md

@@ -50,6 +50,10 @@ Anserini is designed to support experiments on various standard TREC collections
 + [Experiments on Tweets2011 (MB11 & MB12)](docs/experiments-mb11.md)
 + [Experiments on Tweets2013 (MB13 & MB14)](docs/experiments-mb13.md)
 
+### Wikipedia
+
++ [Experiments on CAR17](docs/experiments-car17.md)
+
 ## Tools
 
 + `IndexUtils` is a powerful utility to interact with an index using the command line, e.g. print index statistics. Refer to `target/appassembler/bin/IndexUtils -h` for more details.

docs/experiments-car17.md

@@ -0,0 +1,73 @@
+# Anserini: Experiments on [Car17](http://trec-car.cs.unh.edu/)
+
+## Indexing
+
+Typical indexing command:
+
+```
+nohup sh target/appassembler/bin/IndexCollection -collection CarCollection \
+-generator LuceneDocumentGenerator -threads 40 -input /path/to/car17 -index \
+lucene-index.car17.pos+docvectors -storeRawDocs -storePositions -storeDocvectors \
+-optimize >& log.car17.pos+docvectors+rawdocs &
+```
+
+The directory `/path/to/Car17` should be the root directory of Car17 collection, i.e., `ls /path/to/Car17` should bring up a list of `.cbor` files.
+
+For additional details, see explanation of [common indexing options](common-indexing-options.md).
+
+## Retrieval
+
+Topics and qrels are stored in `src/main/resources/topics-and-qrels/`, downloaded from NIST:
+
++ `topics.car17.test200.txt`: [Topics for the test200 subset (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
++ `qrel: qrels.car17.test200.hierarchical.txt`: [adhoc qrels (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
+
+
+After indexing has completed, you should be able to perform retrieval as follows:
+
+```
+nohup target/appassembler/bin/SearchCollection -topicreader Car -index lucene-index.car17.pos+docvectors -topic src/main/resources/topics-and-qrels/topics.car17.test200.txt -output run.car17.bm25.topics.car17.test200.txt -bm25 &
+
+nohup target/appassembler/bin/SearchCollection -topicreader Car -index lucene-index.car17.pos+docvectors -topic src/main/resources/topics-and-qrels/topics.car17.test200.txt -output run.car17.bm25+rm3.topics.car17.test200.txt -bm25 -rm3 &
+
+nohup target/appassembler/bin/SearchCollection -topicreader Car -index lucene-index.car17.pos+docvectors -topic src/main/resources/topics-and-qrels/topics.car17.test200.txt -output run.car17.bm25+ax.topics.car17.test200.txt -bm25 -axiom -rerankCutoff 20 -axiom.deterministic &
+
+nohup target/appassembler/bin/SearchCollection -topicreader Car -index lucene-index.car17.pos+docvectors -topic src/main/resources/topics-and-qrels/topics.car17.test200.txt -output run.car17.ql.topics.car17.test200.txt -ql &
+
+nohup target/appassembler/bin/SearchCollection -topicreader Car -index lucene-index.car17.pos+docvectors -topic src/main/resources/topics-and-qrels/topics.car17.test200.txt -output run.car17.ql+rm3.topics.car17.test200.txt -ql -rm3 &
+
+nohup target/appassembler/bin/SearchCollection -topicreader Car -index lucene-index.car17.pos+docvectors -topic src/main/resources/topics-and-qrels/topics.car17.test200.txt -output run.car17.ql+ax.topics.car17.test200.txt -ql -axiom -rerankCutoff 20 -axiom.deterministic &
+
+```
+
+Evaluation can be performed using `trec_eval`:
+
+```
+eval/trec_eval.9.0/trec_eval -m map -m recip_rank src/main/resources/topics-and-qrels/qrels.car17.test200.hierarchical.txt -output run.car17.bm25.topics.car17.test200.txt
+
+eval/trec_eval.9.0/trec_eval -m map -m recip_rank src/main/resources/topics-and-qrels/qrels.car17.test200.hierarchical.txt -output run.car17.bm25+rm3.topics.car17.test200.txt
+
+eval/trec_eval.9.0/trec_eval -m map -m recip_rank src/main/resources/topics-and-qrels/qrels.car17.test200.hierarchical.txt -output run.car17.bm25+ax.topics.car17.test200.txt
+
+eval/trec_eval.9.0/trec_eval -m map -m recip_rank src/main/resources/topics-and-qrels/qrels.car17.test200.hierarchical.txt -output run.car17.ql.topics.car17.test200.txt
+
+eval/trec_eval.9.0/trec_eval -m map -m recip_rank src/main/resources/topics-and-qrels/qrels.car17.test200.hierarchical.txt -output run.car17.ql+rm3.topics.car17.test200.txt
+
+eval/trec_eval.9.0/trec_eval -m map -m recip_rank src/main/resources/topics-and-qrels/qrels.car17.test200.hierarchical.txt -output run.car17.ql+ax.topics.car17.test200.txt
+
+```
+
+## Effectiveness
+
+With the above commands, you should be able to replicate the following results:
+
+MAP                                     | BM25      | BM25+RM3  | BM25+AX   | QL        | QL+RM3    | QL+AX     |
+:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
+All Topics                              | 0.1650    | 0.1354    | 0.1318    | 0.1515    | 0.1220    | 0.1083    |
+
+
+RECIP_RANK                              | BM25      | BM25+RM3  | BM25+AX   | QL        | QL+RM3    | QL+AX     |
+:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
+All Topics                              | 0.2270    | 0.1860    | 0.1817    | 0.2085    | 0.1687    | 0.1503    |
+
+

docs/regression-tuna.md

@@ -39,6 +39,10 @@ Here's how to run the end-to-end experiments:
 + `python src/main/python/run_regression.py --collection mb11`: [Experiments on Tweets2011 (MB11 & MB12)](experiments-mb11.md)
 + `python src/main/python/run_regression.py --collection mb13`: [Experiments on Tweets2013 (MB13 & MB14)](experiments-mb13.md)
 
+### Wikipedia
+
++ `python src/main/python/run_regression.py --collection car17`: [Experiments on Car17](experiments-car17.md)
+
 ### JDIQ2018 Effectiveness
 
 Please see doc [here](experiments-jdiq2018.md)

pom.xml

@@ -282,7 +282,7 @@
     <dependency>
       <groupId>com.github.TREMA-UNH</groupId>
       <artifactId>trec-car-tools-java</artifactId>
-      <version>2.0</version>
+      <version>13</version>
     </dependency>
   </dependencies>
 </project>

src/main/java/io/anserini/search/query/CarTopicReader.java

@@ -18,6 +18,7 @@
 
 import java.io.BufferedReader;
 import java.io.IOException;
+import java.io.UnsupportedEncodingException;
 import java.nio.file.Path;
 import java.util.HashMap;
 import java.util.Map;
@@ -43,13 +44,33 @@ public SortedMap<String, Map<String, String>> read(BufferedReader bRdr) throws I
     while ((line = bRdr.readLine()) != null) {
       Map<String,String> fields = new HashMap<>();
       line = line.trim();
-      if (line.startsWith("enwiki:")) {
+      // topic file
+      if (line.indexOf('%') > -1 || line.indexOf('/') > -1) {
         String id = line;
-        String title = java.net.URLDecoder.decode(line.substring(7).replace("%20", " ")
-            .replace("%2", " ").replace("/", " "), "utf-8");
-        fields.put("title", title);
-        map.put(id, fields);
+        String title = null;
+        try {
+          String title_url;
+          if (line.startsWith("enwiki:")) {
+            title_url = line.substring(7);
+          }
+          else {
+            title_url = line;
+          }
+          title_url = title_url.replaceAll("%(?![0-9a-fA-F]{2})", "%25");
+          title_url = title_url.replaceAll("\\+", "%2B");
+          title = java.net.URLDecoder.decode(title_url, "utf-8")
+              .replace("/", " ").replace("(", " ").replace(")", " ");
+          fields.put("title", title);
+          map.put(id, fields);
+        } catch (UnsupportedEncodingException e) {
+          System.out.println(line);
+          e.printStackTrace();
+        } catch (IllegalArgumentException e) {
+          System.out.println(line);
+          e.printStackTrace();
+        }
       }
+      // title file
       else if (line.length() != 0) {
         String title = line;
         String id = "enwiki:" + java.net.URLEncoder.encode(line, "utf-8")

src/main/resources/docgen/templates/car17.template

@@ -0,0 +1,39 @@
+# Anserini: Experiments on [Car17](http://trec-car.cs.unh.edu/)
+
+## Indexing
+
+Typical indexing command:
+
+```
+${index_cmds}
+```
+
+The directory `/path/to/Car17` should be the root directory of Car17 collection, i.e., `ls /path/to/Car17` should bring up a list of `.cbor` files.
+
+For additional details, see explanation of [common indexing options](common-indexing-options.md).
+
+## Retrieval
+
+Topics and qrels are stored in `src/main/resources/topics-and-qrels/`, downloaded from NIST:
+
++ `topics.car17.test200.txt`: [Topics for the test200 subset (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
++ `qrel: qrels.car17.test200.hierarchical.txt`: [adhoc qrels (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
+
+
+After indexing has completed, you should be able to perform retrieval as follows:
+
+```
+${ranking_cmds}
+```
+
+Evaluation can be performed using `trec_eval`:
+
+```
+${eval_cmds}
+```
+
+## Effectiveness
+
+With the above commands, you should be able to replicate the following results:
+
+${effectiveness}

src/main/resources/regression/car17.yaml

@@ -0,0 +1,101 @@
+---
+name: car17
+index_command: target/appassembler/bin/IndexCollection
+index_utils_command: target/appassembler/bin/IndexUtils
+search_command: target/appassembler/bin/SearchCollection
+topic_root: src/main/resources/topics-and-qrels/
+qrels_root: src/main/resources/topics-and-qrels/
+index_root:
+ranking_root:
+generator: LuceneDocumentGenerator
+threads: 40
+index_options:
+  - -storePositions
+  - -storeDocvectors
+  - -storeRawDocs
+topic_reader: Car
+input: /tuna1/collections/car17/paragraphcorpus/
+index_path: "/tuna1/indexes/lucene-index.car17.pos+docvectors+rawdocs" # path to the existing index, used in regression test if `--index` option is absent
+collection: CarCollection
+index_stats:
+  documents: 29678360
+  documents (non-empty): 29674409
+  total terms: 1257896158
+topics:
+  - name: "All Topics"
+    path: topics.car17.test200.txt
+    qrel: qrels.car17.test200.hierarchical.txt
+evals:
+  - command: eval/trec_eval.9.0/trec_eval
+    params:
+      - -m map
+    separator: "\t"
+    parse_index: 2
+    metric: map
+    metric_precision: 4
+    can_combine: true
+  - command: eval/trec_eval.9.0/trec_eval
+    params:
+      - -m recip_rank
+    separator: "\t"
+    parse_index: 2
+    metric: recip_rank
+    metric_precision: 4
+    can_combine: true
+models:
+  - name: bm25
+    params:
+      - -bm25
+    results:
+      map:
+        - 0.1650
+      recip_rank:
+        - 0.2270
+  - name: bm25+rm3
+    params:
+      - -bm25
+      - -rm3
+    results:
+      map:
+        - 0.1354
+      recip_rank:
+        - 0.1860
+  - name: bm25+ax
+    params:
+      - -bm25
+      - -axiom
+      - -rerankCutoff 20
+      - -axiom.deterministic
+    results:
+      map:
+        - 0.1318 
+      recip_rank:
+        - 0.1817
+  - name: ql
+    params:
+      - -ql
+    results:
+      map:
+        - 0.1515
+      recip_rank:
+        - 0.2085
+  - name: ql+rm3
+    params:
+      - -ql
+      - -rm3
+    results:
+      map:
+        - 0.1220
+      recip_rank:
+        - 0.1687
+  - name: ql+ax
+    params:
+      - -ql
+      - -axiom
+      - -rerankCutoff 20
+      - -axiom.deterministic
+    results:
+      map:
+        - 0.1083
+      recip_rank:
+        - 0.1503