Better jump sync and run-on

[mattdowle]

Closes #2267
Closes #2561

nextGoodLine() simplified to always return a good guess rather than potentially fail with false. This removes the fail points such as "No good line" and "did not finish exactly".

Only the first sample of 100 lines can now bump the quoteRule. Later sample jumps skip rather than bumping quoteRule because it's more likely that sample jump point was to inside a quoted field. The thread at headPos when reading (which has full lineage from the sof) will bump the quoteRule with certainty then, if needed. This prepares for #2265.

Dirty jump sweeping implemented. Thread at headPos now knows if it finished after nextJumpStart; which will happen if nextJumpStart landed awkwardly inside a quoted field with many newlines causing nextGoodLine to pick a false newline. In this case, jump0 is set to reread the dirty chunk and the team restarted. If too many dirty jumps are encountered, a single thread with full lineage from sof will be needed anyway, so the team is downgraded to single-threaded from that point on. In most cases of a few dirty jumps, they will be swept by the thread at headPos and the remainder of the file continued to be read in parallel.

• simulate bad_fill.csv from fread fails on file with inconsistent # columns #2267. At 35MB it's too big for CRAN but works now.
• simulate issue682.txt from Jump 0 did not finish counting rows exactly where jump 1 found its first good line start #2561. At 2.1MB it's too big for CRAN but works now.
• simulate bad jump point by constructing an embedded field with 5 newlines

[codecov-io]

Codecov Report

Merging #2627 into master will increase coverage by 0.07%.
The diff coverage is 95.65%.

@@            Coverage Diff             @@
##           master    #2627      +/-   ##
==========================================
+ Coverage   92.95%   93.02%   +0.07%     
==========================================
  Files          61       61              
  Lines       12129    12123       -6     
==========================================
+ Hits        11275    11278       +3     
+ Misses        854      845       -9

Impacted Files	Coverage Δ
R/print.data.table.R	98.13% <100%> (ø)	⬆️
src/fread.c	97.11% <95.6%> (+0.72%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5c0eda...3eec5a8. Read the comment docs.

[st-pasha]

Superb! These "invalid chunks" errors were some of the most annoying issues in fread, and now it seems you solved them.

[st-pasha]

Could we perhaps split this into multiple lines (one per issue)? It's already grew so big that it's impossible to know what was added/deleted, and it will probably get even bigger in the future...

[mattdowle]

Could do. I'm not quite sure how to split it though. The idea of this item in NEWS is to just merely to convey to lay readers of NEWS on release to CRAN that lots of people have helped to find and fix lots of problems. I'm not expecting anyone at all to click through them all. If we want to know what's been done and when in more detail, we'd use the milestone tag instead. I'm only adding to the end, so the diff is only ever at the end (or adding new people to the end of the list at the beginning of this item). Also, many of these items are fixes to new problems that have been created by going parallel in dev. NEWS is only really supposed to cover user changes from the last released version to CRAN.

[HughParsonage]

You could just put every pull request link on a new line. I believe markdown ignores single new lines (like TeX).

[MichaelChirico]

I'm fine with it as it is, but, it could be worth writing a short blog-post-type document with release to highlight all the work that has been done, and refer in NEWS to that ("see [this]() post for more")