Speed up scrapers

[smcgivern]

This significantly speeds up the scrapers.

Perform fewer Pandas appends

pandas-dev/pandas#35407 says:

Unless I'm mistaken, users are always better off building up a list of values and passing them to the constructor, or building up a list of NDFrames followed by a single concat.

So let's do that. It is significantly faster: https://stackoverflow.com/a/56746204

Using py-spy (https://github.com/benfred/py-spy) for flamegraphs and good old time for overall timings, I created this comparison.

Before (uncached, for women's Test team records only):

$ rm -r data/*
$ sudo time py-spy record -o before.svg -- python update_data.py
py-spy> Sampling process 100 times a second. Press Control-C to exit.

# snip
Scraping page 4
https://stats.espncricinfo.com/ci/engine/stats/index.html?class=8;filter=advanced;orderby=start;page=4;size=200;template=results;type=team;view=innings
All done!

py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'before.svg'. Samples: 1235 Errors: 0
       28.53 real        10.93 user         1.65 sys

After:

$ rm -r data/*
$ sudo time py-spy record -o list.svg -- python update_data.py
py-spy> Sampling process 100 times a second. Press Control-C to exit.

# snip
Scraping page 4
https://stats.espncricinfo.com/ci/engine/stats/index.html?class=8;filter=advanced;orderby=start;page=4;size=200;template=results;type=team;view=innings
All done!

py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'list.svg'. Samples: 395 Errors: 0
       20.91 real         4.06 user         1.41 sys

For a full update, before:

$ sudo time py-spy record -o before-2.svg -- python update_data.py
py-spy> Sampling process 100 times a second. Press Control-C to exit.
# snip
py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'before-2.svg'. Samples: 21656 Errors: 0
      626.02 real       194.27 user        31.50 sys

And after:

$ sudo time py-spy record -o list-2.svg -- python update_data.py
py-spy> Sampling process 100 times a second. Press Control-C to exit.
# snip
py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'list-2.svg'. Samples: 8958 Errors: 0
      492.66 real        85.26 user        26.58 sys

If we then add the cache (#6) into the equation:

      226.39 real        73.39 user        11.82 sys

This needed an upgrade to Pandas 1.4 in order to work with the nullable integer types. This was to take pandas-dev/pandas#43949 which fixed pandas-dev/pandas#25472.

Selectolax

The second commit uses Selectolax to speed up HTML parsing. https://rushter.com/blog/python-fast-html-parser/ has a good overview, and this is very straightforward.

Combining all these changes gives a big improvement: with caching, more efficient dataframe construction, and a faster HTML parser, we go from 626 seconds (just over 10 minutes) to 155 seconds (just over 2.5 minutes)!

$ sudo time py-spy record -o selectolax-cached.svg -- python update_data.py
py-spy> Sampling process 100 times a second. Press Control-C to exit.
# snip
py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'selectolax-cached.svg'. Samples: 2618 Errors: 0
      155.07 real        21.05 user         8.74 sys

We now spend about half our time just writing the CSVs!