Speed up tokenizing of a row in csv and xstrtod parsing

[vnlitvinov]

• closes: N/A
• tests passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

I will update the PR when CI finishes running, as I locally tested io.parser.test_common only.

[jreback]

can you run the asv benchmarks for csv and report the changes. we may not have benchmarks that are specificying hitting the code you changes; if so can you add one.

[jreback]

also say :meth:`read_csv` a user has no idea what any of the other things are

[vnlitvinov]

asv continuous -f 1.05 origin/master HEAD -b io.csv results:

before	after	ratio	test name
[e8d951d]	[a4f6dcd]
master	speed-up-tokenizer
1.58±0.02ms	1.49±0.02ms	0.94	io.csv.ReadCSVFloatPrecision.time_read_csv(',', '.', 'high')
5.23±0.1ms	4.87±0.03ms	0.93	io.csv.ReadUint64Integers.time_read_uint64_na_values
10.1±0.2ms	9.15±0.3ms	0.91	io.csv.ReadCSVSkipRows.time_skipprows(10000)
13.0±0.3ms	11.6±0.2ms	0.89	io.csv.ReadCSVThousands.time_thousands('|', None)

So this basically gives from 5% to 10% increase on mini-benchmarks. Note that this scales quite well, and on cases where csv files are million lines long it still gives same 5-10%.

[vnlitvinov]

The results are even more promising if I allow more warmup and more sampling time so Turboboost and frequency scaling don't impact the performance too much.

Running asv continuous -f 1.05 origin/master HEAD -b io.csv -a sample_time=2 -a warmup_time=2 yields:

before	after	ratio	test name
[e8d951d]	[a4f6dcd]
master	speed-up-tokenizer
34.9±0.2ms	32.9±1ms	0.94	io.csv.ReadCSVCategorical.time_convert_direct
13.5±0.02ms	12.6±0.08ms	0.93	io.csv.ReadCSVThousands.time_thousands(',', ',')
5.10±0.07ms	4.71±0.09ms	0.92	io.csv.ReadUint64Integers.time_read_uint64_neg_values
14.6±0.06ms	13.0±0.04ms	0.89	io.csv.ReadCSVThousands.time_thousands('|', ',')
16.0±0.3ms	13.8±0.09ms	0.86	io.csv.ReadCSVSkipRows.time_skipprows(None)
10.3±0.1ms	8.81±0.1ms	0.86	io.csv.ReadCSVSkipRows.time_skipprows(10000)
12.9±0.1ms	10.7±0.09ms	0.84	io.csv.ReadCSVThousands.time_thousands('|', None)
13.0±0.05ms	10.8±0.08ms	0.83	io.csv.ReadCSVThousands.time_thousands(',', None)

[vnlitvinov]

@jreback I've fixed whatsnew entry per your comment and rebased to latest master for clean history.

[codecov]

Codecov Report

Merging #25784 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25784      +/-   ##
==========================================
- Coverage   91.26%   91.26%   -0.01%     
==========================================
  Files         173      173              
  Lines       52982    52982              
==========================================
- Hits        48356    48355       -1     
- Misses       4626     4627       +1

Flag	Coverage Δ
#multiple	89.83% <ø> (ø)	⬆️
#single	41.76% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/util/testing.py	89.3% <0%> (-0.11%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4663951...12a6df9. Read the comment docs.

[codecov]

Codecov Report

Merging #25784 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25784      +/-   ##
==========================================
+ Coverage   91.26%   91.27%   +<.01%     
==========================================
  Files         173      173              
  Lines       52982    53002      +20     
==========================================
+ Hits        48356    48375      +19     
- Misses       4626     4627       +1

Flag	Coverage Δ
#multiple	89.83% <ø> (ø)	⬆️
#single	41.77% <ø> (+0.01%)	⬆️

Impacted Files	Coverage Δ
pandas/util/testing.py	89.3% <0%> (-0.11%)	⬇️
pandas/core/series.py	93.67% <0%> (-0.01%)	⬇️
pandas/core/ops.py	91.74% <0%> (ø)	⬆️
pandas/core/frame.py	96.79% <0%> (ø)	⬆️
pandas/core/generic.py	93.52% <0%> (ø)	⬆️
pandas/core/computation/expr.py	88.52% <0%> (+0.35%)	⬆️
pandas/core/computation/common.py	89.47% <0%> (+3.75%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4663951...72c7570. Read the comment docs.

[jreback]

comment on whatsnew, ping on green.

[jreback]

lgtm.

[vnlitvinov]

ping on green

@jreback all tests are green

[jreback]

thanks @vnlitvin nice patch!