Refactoring of WashingtonPost collection for Core18

[lintool]

Refactoring of WashingtonPost collection based on the clarified definition of contents() and raw() in SourceDocument, per #1048.

Before, both contents() and raw() returned the raw JSON, and WashingtonPostGenerator extracted the article text for indexing.

Now, raw() returns the raw JSON, and contents() returns the extracted document contents. In other words, the logic for parsing the JSON has been moved from WashingtonPostGenerator into the collection itself. As a result, the generator has been simplified. It inherits from the DefaultLuceneDocumentGenerator, which creates the default fields, and then adds additional specific ones.

Regression values went down slightly for Ax, which seems very sensitive to tiny differences. The difference here is that, before, the "empty document" check was performed on the JSON, so it never triggered (since the JSON was never empty). With the new processing logic, the "empty document" check is performed on contents(), and so the number of empty documents is now accurate (there are 6). Apparently, this is enough to change regression numbers for Ax.

[codecov]

Codecov Report

Merging #1092 into master will increase coverage by 0.47%.
The diff coverage is 51.78%.

@@             Coverage Diff              @@
##             master    #1092      +/-   ##
============================================
+ Coverage     44.81%   45.29%   +0.47%     
- Complexity      664      673       +9     
============================================
  Files           141      141              
  Lines          8184     8188       +4     
  Branches       1166     1167       +1     
============================================
+ Hits           3668     3709      +41     
+ Misses         4187     4147      -40     
- Partials        329      332       +3     

Impacted Files	Coverage Δ	Complexity Δ
...ndex/generator/DefaultLuceneDocumentGenerator.java	54.83% <0.00%> (-3.79%)	2.00 <0.00> (ø)
...erini/index/generator/WashingtonPostGenerator.java	34.48% <0.00%> (+18.60%)	0.00 <0.00> (ø)
...arch/topicreader/BackgroundLinkingTopicReader.java	19.08% <0.00%> (-0.15%)	9.00 <0.00> (ø)
.../main/java/io/anserini/index/IndexReaderUtils.java	45.94% <55.55%> (+0.49%)	21.00 <1.00> (+1.00)
.../anserini/collection/WashingtonPostCollection.java	78.40% <76.66%> (+6.40%)	2.00 <0.00> (ø)
...c/main/java/io/anserini/search/SimpleSearcher.java	53.44% <100.00%> (+0.26%)	26.00 <1.00> (+1.00)
...anserini/ltr/feature/base/PMIFeatureExtractor.java	86.53% <0.00%> (+1.92%)	13.00% <0.00%> (+1.00%)
...java/io/anserini/ltr/feature/CountBigramPairs.java	93.50% <0.00%> (+10.38%)	35.00% <0.00%> (+6.00%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 585229f...e2e9df8. Read the comment docs.

[edwinzhng]

LGTM! Surprised the regression numbers dropped from such a small change

[edwinzhng]

What's a kicker?

[lintool]

It's one of the fields in WashingtonPost corpus: http://trec-news.org/guidelines-2020.pdf

... we decree that articles from the "Opinion", "Letters to the Editor", or "The Post's View" sections, as labeled in the "kicker" field, are not relevant