Set usedforsecurity=False in hashlib methods (FIPS compliance)

[Wauplin]

Related to huggingface/transformers#27034 and huggingface/huggingface_hub#1782.

TL;DR: hashlib is not a secure library for cryptography-related stuff. We are only using hashlib for non-security-related purposes in datasets so it's fine. From Python 3.9 we set can usedforsecurity=False in any hashlib method which is mandatory for companies that forbid the use of hashlib for security purposes. This PR fixes that.

Note: before merging this we need to release a new tokenizers version that would allow the newest huggingface_hub version (see huggingface/tokenizers#1385). Otherwise it might create friction to users that want to install datasets + tokenizers at the same time.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008434 / 0.011353 (-0.002919)	0.006755 / 0.011008 (-0.004253)	0.106169 / 0.038508 (0.067661)	0.049329 / 0.023109 (0.026220)	0.433610 / 0.275898 (0.157712)	0.441993 / 0.323480 (0.118513)	0.004703 / 0.007986 (-0.003282)	0.006996 / 0.004328 (0.002667)	0.080330 / 0.004250 (0.076080)	0.066098 / 0.037052 (0.029045)	0.435444 / 0.258489 (0.176955)	0.490442 / 0.293841 (0.196601)	0.047050 / 0.128546 (-0.081496)	0.014520 / 0.075646 (-0.061127)	0.339805 / 0.419271 (-0.079467)	0.101161 / 0.043533 (0.057629)	0.423236 / 0.255139 (0.168097)	0.455627 / 0.283200 (0.172427)	0.036218 / 0.141683 (-0.105465)	1.766128 / 1.452155 (0.313973)	1.923919 / 1.492716 (0.431203)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.242939 / 0.018006 (0.224933)	0.515582 / 0.000490 (0.515093)	0.020271 / 0.000200 (0.020071)	0.000383 / 0.000054 (0.000328)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030927 / 0.037411 (-0.006484)	0.093951 / 0.014526 (0.079425)	0.109028 / 0.176557 (-0.067529)	0.174947 / 0.737135 (-0.562188)	0.120538 / 0.296338 (-0.175800)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.553884 / 0.215209 (0.338675)	5.424566 / 2.077655 (3.346911)	2.439420 / 1.504120 (0.935301)	2.019324 / 1.541195 (0.478129)	2.170781 / 1.468490 (0.702290)	0.924424 / 4.584777 (-3.660353)	5.706029 / 3.745712 (1.960317)	5.096911 / 5.269862 (-0.172951)	3.168261 / 4.565676 (-1.397416)	0.094336 / 0.424275 (-0.329940)	0.015899 / 0.007607 (0.008292)	0.709684 / 0.226044 (0.483639)	7.476865 / 2.268929 (5.207936)	3.350983 / 55.444624 (-52.093641)	2.653419 / 6.876477 (-4.223058)	2.802201 / 2.142072 (0.660129)	1.081442 / 4.805227 (-3.723785)	0.217025 / 6.500664 (-6.283639)	0.077248 / 0.075469 (0.001779)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.598621 / 1.841788 (-0.243167)	23.490338 / 8.074308 (15.416030)	21.853488 / 10.191392 (11.662096)	0.209625 / 0.680424 (-0.470799)	0.028166 / 0.534201 (-0.506035)	0.473883 / 0.579283 (-0.105400)	0.584226 / 0.434364 (0.149862)	0.538605 / 0.540337 (-0.001732)	0.837060 / 1.386936 (-0.549876)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009029 / 0.011353 (-0.002324)	0.004945 / 0.011008 (-0.006063)	0.084539 / 0.038508 (0.046031)	0.081014 / 0.023109 (0.057905)	0.431291 / 0.275898 (0.155393)	0.478913 / 0.323480 (0.155433)	0.006107 / 0.007986 (-0.001879)	0.003939 / 0.004328 (-0.000390)	0.079932 / 0.004250 (0.075682)	0.057936 / 0.037052 (0.020884)	0.437295 / 0.258489 (0.178806)	0.489790 / 0.293841 (0.195949)	0.049544 / 0.128546 (-0.079003)	0.013675 / 0.075646 (-0.061972)	0.093143 / 0.419271 (-0.326128)	0.064104 / 0.043533 (0.020571)	0.444699 / 0.255139 (0.189560)	0.443688 / 0.283200 (0.160489)	0.034331 / 0.141683 (-0.107352)	1.753014 / 1.452155 (0.300859)	1.877274 / 1.492716 (0.384558)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.250460 / 0.018006 (0.232454)	0.527241 / 0.000490 (0.526752)	0.007679 / 0.000200 (0.007479)	0.000115 / 0.000054 (0.000061)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.033269 / 0.037411 (-0.004142)	0.111262 / 0.014526 (0.096736)	0.133503 / 0.176557 (-0.043053)	0.177998 / 0.737135 (-0.559137)	0.117899 / 0.296338 (-0.178440)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.633588 / 0.215209 (0.418379)	6.105283 / 2.077655 (4.027628)	2.779309 / 1.504120 (1.275189)	2.445788 / 1.541195 (0.904594)	2.396443 / 1.468490 (0.927953)	0.925928 / 4.584777 (-3.658849)	5.266142 / 3.745712 (1.520430)	4.868830 / 5.269862 (-0.401031)	2.998768 / 4.565676 (-1.566909)	0.103135 / 0.424275 (-0.321140)	0.008059 / 0.007607 (0.000452)	0.753159 / 0.226044 (0.527115)	7.532170 / 2.268929 (5.263242)	3.563941 / 55.444624 (-51.880683)	2.829208 / 6.876477 (-4.047269)	2.913954 / 2.142072 (0.771881)	1.085843 / 4.805227 (-3.719384)	0.214195 / 6.500664 (-6.286469)	0.071509 / 0.075469 (-0.003960)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.544819 / 1.841788 (-0.296968)	23.790149 / 8.074308 (15.715841)	23.086019 / 10.191392 (12.894627)	0.242695 / 0.680424 (-0.437729)	0.041706 / 0.534201 (-0.492495)	0.552402 / 0.579283 (-0.026881)	0.652518 / 0.434364 (0.218154)	0.581876 / 0.540337 (0.041539)	0.795425 / 1.386936 (-0.591511)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004573 / 0.011353 (-0.006780)	0.002965 / 0.011008 (-0.008043)	0.061913 / 0.038508 (0.023405)	0.029474 / 0.023109 (0.006365)	0.258117 / 0.275898 (-0.017781)	0.279854 / 0.323480 (-0.043626)	0.003954 / 0.007986 (-0.004031)	0.002479 / 0.004328 (-0.001850)	0.048685 / 0.004250 (0.044434)	0.044733 / 0.037052 (0.007681)	0.256659 / 0.258489 (-0.001830)	0.285235 / 0.293841 (-0.008606)	0.023566 / 0.128546 (-0.104981)	0.007291 / 0.075646 (-0.068355)	0.202701 / 0.419271 (-0.216570)	0.055706 / 0.043533 (0.012173)	0.258790 / 0.255139 (0.003651)	0.278675 / 0.283200 (-0.004525)	0.018574 / 0.141683 (-0.123109)	1.109359 / 1.452155 (-0.342796)	1.184434 / 1.492716 (-0.308282)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.095048 / 0.018006 (0.077042)	0.305027 / 0.000490 (0.304537)	0.000310 / 0.000200 (0.000110)	0.000066 / 0.000054 (0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018183 / 0.037411 (-0.019228)	0.066130 / 0.014526 (0.051604)	0.073948 / 0.176557 (-0.102608)	0.120458 / 0.737135 (-0.616678)	0.075995 / 0.296338 (-0.220343)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.279419 / 0.215209 (0.064210)	2.728591 / 2.077655 (0.650936)	1.439016 / 1.504120 (-0.065104)	1.325798 / 1.541195 (-0.215397)	1.352050 / 1.468490 (-0.116440)	0.395041 / 4.584777 (-4.189736)	2.377651 / 3.745712 (-1.368061)	2.618473 / 5.269862 (-2.651389)	1.587580 / 4.565676 (-2.978096)	0.045910 / 0.424275 (-0.378365)	0.004843 / 0.007607 (-0.002764)	0.335491 / 0.226044 (0.109447)	3.378441 / 2.268929 (1.109512)	1.827757 / 55.444624 (-53.616868)	1.502360 / 6.876477 (-5.374117)	1.508460 / 2.142072 (-0.633612)	0.471309 / 4.805227 (-4.333918)	0.098934 / 6.500664 (-6.401730)	0.041705 / 0.075469 (-0.033764)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.945067 / 1.841788 (-0.896720)	11.548209 / 8.074308 (3.473900)	10.422628 / 10.191392 (0.231236)	0.141494 / 0.680424 (-0.538929)	0.014345 / 0.534201 (-0.519856)	0.267750 / 0.579283 (-0.311533)	0.261488 / 0.434364 (-0.172876)	0.307192 / 0.540337 (-0.233145)	0.427926 / 1.386936 (-0.959010)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004831 / 0.011353 (-0.006522)	0.002876 / 0.011008 (-0.008132)	0.048629 / 0.038508 (0.010121)	0.055090 / 0.023109 (0.031981)	0.271381 / 0.275898 (-0.004517)	0.292350 / 0.323480 (-0.031130)	0.004001 / 0.007986 (-0.003985)	0.002389 / 0.004328 (-0.001939)	0.047527 / 0.004250 (0.043277)	0.038065 / 0.037052 (0.001012)	0.277387 / 0.258489 (0.018898)	0.307209 / 0.293841 (0.013368)	0.025136 / 0.128546 (-0.103411)	0.007309 / 0.075646 (-0.068338)	0.054483 / 0.419271 (-0.364789)	0.032807 / 0.043533 (-0.010726)	0.274364 / 0.255139 (0.019225)	0.290280 / 0.283200 (0.007080)	0.017855 / 0.141683 (-0.123828)	1.185912 / 1.452155 (-0.266243)	1.228141 / 1.492716 (-0.264576)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094787 / 0.018006 (0.076781)	0.314191 / 0.000490 (0.313701)	0.000217 / 0.000200 (0.000017)	0.000058 / 0.000054 (0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.020920 / 0.037411 (-0.016491)	0.070446 / 0.014526 (0.055920)	0.081371 / 0.176557 (-0.095186)	0.119127 / 0.737135 (-0.618009)	0.085658 / 0.296338 (-0.210680)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.290601 / 0.215209 (0.075392)	2.874091 / 2.077655 (0.796436)	1.598934 / 1.504120 (0.094814)	1.464329 / 1.541195 (-0.076866)	1.504943 / 1.468490 (0.036453)	0.410457 / 4.584777 (-4.174320)	2.428706 / 3.745712 (-1.317006)	2.596510 / 5.269862 (-2.673352)	1.547084 / 4.565676 (-3.018592)	0.047546 / 0.424275 (-0.376729)	0.004740 / 0.007607 (-0.002867)	0.351168 / 0.226044 (0.125123)	3.424554 / 2.268929 (1.155626)	1.969792 / 55.444624 (-53.474832)	1.676731 / 6.876477 (-5.199745)	1.668769 / 2.142072 (-0.473304)	0.482486 / 4.805227 (-4.322741)	0.100018 / 6.500664 (-6.400646)	0.040956 / 0.075469 (-0.034513)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.966306 / 1.841788 (-0.875482)	12.158909 / 8.074308 (4.084601)	10.926447 / 10.191392 (0.735055)	0.130359 / 0.680424 (-0.550065)	0.016162 / 0.534201 (-0.518039)	0.269977 / 0.579283 (-0.309306)	0.283366 / 0.434364 (-0.150997)	0.304517 / 0.540337 (-0.235821)	0.410398 / 1.386936 (-0.976539)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004686 / 0.011353 (-0.006667)	0.002764 / 0.011008 (-0.008244)	0.061411 / 0.038508 (0.022902)	0.030450 / 0.023109 (0.007341)	0.247648 / 0.275898 (-0.028250)	0.278033 / 0.323480 (-0.045447)	0.002903 / 0.007986 (-0.005082)	0.002350 / 0.004328 (-0.001979)	0.047514 / 0.004250 (0.043264)	0.044446 / 0.037052 (0.007393)	0.256170 / 0.258489 (-0.002319)	0.285977 / 0.293841 (-0.007864)	0.023407 / 0.128546 (-0.105139)	0.007223 / 0.075646 (-0.068423)	0.201274 / 0.419271 (-0.217997)	0.054022 / 0.043533 (0.010489)	0.253841 / 0.255139 (-0.001298)	0.278219 / 0.283200 (-0.004980)	0.017796 / 0.141683 (-0.123886)	1.105950 / 1.452155 (-0.346205)	1.182021 / 1.492716 (-0.310695)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.089584 / 0.018006 (0.071578)	0.299338 / 0.000490 (0.298849)	0.000202 / 0.000200 (0.000003)	0.000050 / 0.000054 (-0.000005)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018974 / 0.037411 (-0.018437)	0.062352 / 0.014526 (0.047826)	0.073667 / 0.176557 (-0.102889)	0.119225 / 0.737135 (-0.617911)	0.075393 / 0.296338 (-0.220945)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.282749 / 0.215209 (0.067540)	2.795822 / 2.077655 (0.718167)	1.492946 / 1.504120 (-0.011174)	1.382340 / 1.541195 (-0.158855)	1.377281 / 1.468490 (-0.091209)	0.397361 / 4.584777 (-4.187415)	2.379416 / 3.745712 (-1.366296)	2.552967 / 5.269862 (-2.716895)	1.546347 / 4.565676 (-3.019330)	0.045851 / 0.424275 (-0.378424)	0.004830 / 0.007607 (-0.002777)	0.351194 / 0.226044 (0.125150)	3.407406 / 2.268929 (1.138478)	1.852983 / 55.444624 (-53.591641)	1.536381 / 6.876477 (-5.340095)	1.542786 / 2.142072 (-0.599287)	0.471960 / 4.805227 (-4.333267)	0.098336 / 6.500664 (-6.402328)	0.041569 / 0.075469 (-0.033900)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.912718 / 1.841788 (-0.929070)	11.339404 / 8.074308 (3.265095)	10.480593 / 10.191392 (0.289201)	0.139508 / 0.680424 (-0.540916)	0.014210 / 0.534201 (-0.519991)	0.268152 / 0.579283 (-0.311131)	0.260503 / 0.434364 (-0.173860)	0.304735 / 0.540337 (-0.235602)	0.422155 / 1.386936 (-0.964781)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004714 / 0.011353 (-0.006638)	0.002638 / 0.011008 (-0.008370)	0.047967 / 0.038508 (0.009459)	0.050758 / 0.023109 (0.027649)	0.265619 / 0.275898 (-0.010279)	0.286920 / 0.323480 (-0.036560)	0.003936 / 0.007986 (-0.004050)	0.002351 / 0.004328 (-0.001977)	0.047642 / 0.004250 (0.043392)	0.038412 / 0.037052 (0.001360)	0.269561 / 0.258489 (0.011072)	0.302057 / 0.293841 (0.008216)	0.023893 / 0.128546 (-0.104653)	0.006793 / 0.075646 (-0.068854)	0.053091 / 0.419271 (-0.366180)	0.032228 / 0.043533 (-0.011305)	0.267110 / 0.255139 (0.011971)	0.287211 / 0.283200 (0.004011)	0.017945 / 0.141683 (-0.123738)	1.191770 / 1.452155 (-0.260384)	1.269644 / 1.492716 (-0.223072)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.088067 / 0.018006 (0.070061)	0.298383 / 0.000490 (0.297893)	0.000202 / 0.000200 (0.000002)	0.000048 / 0.000054 (-0.000007)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.020685 / 0.037411 (-0.016726)	0.069883 / 0.014526 (0.055357)	0.080107 / 0.176557 (-0.096450)	0.119311 / 0.737135 (-0.617825)	0.080791 / 0.296338 (-0.215548)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.295781 / 0.215209 (0.080572)	2.905536 / 2.077655 (0.827881)	1.579184 / 1.504120 (0.075064)	1.475937 / 1.541195 (-0.065258)	1.533708 / 1.468490 (0.065218)	0.409851 / 4.584777 (-4.174926)	2.443217 / 3.745712 (-1.302496)	2.543980 / 5.269862 (-2.725882)	1.512187 / 4.565676 (-3.053489)	0.046390 / 0.424275 (-0.377885)	0.004762 / 0.007607 (-0.002845)	0.345066 / 0.226044 (0.119021)	3.485133 / 2.268929 (1.216204)	1.954690 / 55.444624 (-53.489934)	1.671104 / 6.876477 (-5.205372)	1.655330 / 2.142072 (-0.486743)	0.487910 / 4.805227 (-4.317317)	0.097707 / 6.500664 (-6.402957)	0.040379 / 0.075469 (-0.035090)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.981620 / 1.841788 (-0.860168)	11.806530 / 8.074308 (3.732222)	10.868275 / 10.191392 (0.676883)	0.141230 / 0.680424 (-0.539194)	0.015785 / 0.534201 (-0.518416)	0.271416 / 0.579283 (-0.307867)	0.276048 / 0.434364 (-0.158316)	0.310988 / 0.540337 (-0.229349)	0.410078 / 1.386936 (-0.976858)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lhoestq]

Feel free to merge after the releases

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004803 / 0.011353 (-0.006550)	0.002961 / 0.011008 (-0.008047)	0.061431 / 0.038508 (0.022923)	0.030189 / 0.023109 (0.007080)	0.255755 / 0.275898 (-0.020143)	0.277841 / 0.323480 (-0.045639)	0.003083 / 0.007986 (-0.004902)	0.002432 / 0.004328 (-0.001896)	0.047674 / 0.004250 (0.043424)	0.045066 / 0.037052 (0.008014)	0.268701 / 0.258489 (0.010211)	0.286673 / 0.293841 (-0.007168)	0.023663 / 0.128546 (-0.104883)	0.007148 / 0.075646 (-0.068499)	0.201962 / 0.419271 (-0.217310)	0.054953 / 0.043533 (0.011420)	0.257155 / 0.255139 (0.002016)	0.277769 / 0.283200 (-0.005431)	0.017803 / 0.141683 (-0.123880)	1.100270 / 1.452155 (-0.351884)	1.146975 / 1.492716 (-0.345741)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092776 / 0.018006 (0.074770)	0.303786 / 0.000490 (0.303296)	0.000237 / 0.000200 (0.000037)	0.000055 / 0.000054 (0.000000)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019647 / 0.037411 (-0.017765)	0.063211 / 0.014526 (0.048686)	0.076684 / 0.176557 (-0.099873)	0.121952 / 0.737135 (-0.615184)	0.077202 / 0.296338 (-0.219137)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.282087 / 0.215209 (0.066878)	2.789204 / 2.077655 (0.711550)	1.510376 / 1.504120 (0.006256)	1.384241 / 1.541195 (-0.156954)	1.414949 / 1.468490 (-0.053541)	0.402206 / 4.584777 (-4.182570)	2.377601 / 3.745712 (-1.368111)	2.585354 / 5.269862 (-2.684508)	1.592937 / 4.565676 (-2.972740)	0.045217 / 0.424275 (-0.379058)	0.004772 / 0.007607 (-0.002835)	0.339584 / 0.226044 (0.113539)	3.373184 / 2.268929 (1.104256)	1.855196 / 55.444624 (-53.589428)	1.599559 / 6.876477 (-5.276918)	1.604421 / 2.142072 (-0.537651)	0.467754 / 4.805227 (-4.337474)	0.098244 / 6.500664 (-6.402420)	0.042631 / 0.075469 (-0.032838)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.947680 / 1.841788 (-0.894108)	11.539875 / 8.074308 (3.465567)	10.340830 / 10.191392 (0.149438)	0.145591 / 0.680424 (-0.534833)	0.014367 / 0.534201 (-0.519834)	0.270506 / 0.579283 (-0.308777)	0.268825 / 0.434364 (-0.165539)	0.308372 / 0.540337 (-0.231966)	0.425039 / 1.386936 (-0.961897)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004813 / 0.011353 (-0.006540)	0.002931 / 0.011008 (-0.008078)	0.047997 / 0.038508 (0.009489)	0.050753 / 0.023109 (0.027644)	0.272704 / 0.275898 (-0.003194)	0.294045 / 0.323480 (-0.029435)	0.004059 / 0.007986 (-0.003927)	0.002491 / 0.004328 (-0.001838)	0.047621 / 0.004250 (0.043371)	0.038824 / 0.037052 (0.001772)	0.275322 / 0.258489 (0.016833)	0.306447 / 0.293841 (0.012606)	0.024402 / 0.128546 (-0.104145)	0.007252 / 0.075646 (-0.068394)	0.053346 / 0.419271 (-0.365925)	0.032224 / 0.043533 (-0.011309)	0.271468 / 0.255139 (0.016329)	0.289429 / 0.283200 (0.006229)	0.018285 / 0.141683 (-0.123398)	1.116743 / 1.452155 (-0.335412)	1.182724 / 1.492716 (-0.309993)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091899 / 0.018006 (0.073893)	0.299161 / 0.000490 (0.298671)	0.000224 / 0.000200 (0.000024)	0.000053 / 0.000054 (-0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021823 / 0.037411 (-0.015588)	0.071227 / 0.014526 (0.056701)	0.080503 / 0.176557 (-0.096053)	0.120243 / 0.737135 (-0.616892)	0.082328 / 0.296338 (-0.214010)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.324951 / 0.215209 (0.109742)	2.842358 / 2.077655 (0.764703)	1.602317 / 1.504120 (0.098197)	1.481103 / 1.541195 (-0.060091)	1.497557 / 1.468490 (0.029067)	0.406523 / 4.584777 (-4.178254)	2.402743 / 3.745712 (-1.342970)	2.545435 / 5.269862 (-2.724427)	1.534071 / 4.565676 (-3.031605)	0.046914 / 0.424275 (-0.377361)	0.004728 / 0.007607 (-0.002879)	0.341544 / 0.226044 (0.115499)	3.412017 / 2.268929 (1.143089)	1.937442 / 55.444624 (-53.507182)	1.668774 / 6.876477 (-5.207703)	1.668908 / 2.142072 (-0.473165)	0.477398 / 4.805227 (-4.327829)	0.098531 / 6.500664 (-6.402133)	0.041077 / 0.075469 (-0.034392)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.983888 / 1.841788 (-0.857900)	12.072703 / 8.074308 (3.998395)	11.028622 / 10.191392 (0.837230)	0.148097 / 0.680424 (-0.532327)	0.015869 / 0.534201 (-0.518332)	0.267609 / 0.579283 (-0.311674)	0.272345 / 0.434364 (-0.162019)	0.303840 / 0.540337 (-0.236497)	0.409199 / 1.386936 (-0.977737)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005016 / 0.011353 (-0.006337)	0.002931 / 0.011008 (-0.008077)	0.062142 / 0.038508 (0.023634)	0.030758 / 0.023109 (0.007648)	0.251689 / 0.275898 (-0.024209)	0.272114 / 0.323480 (-0.051366)	0.004102 / 0.007986 (-0.003884)	0.002500 / 0.004328 (-0.001828)	0.049187 / 0.004250 (0.044937)	0.047150 / 0.037052 (0.010098)	0.256497 / 0.258489 (-0.001992)	0.288069 / 0.293841 (-0.005772)	0.023915 / 0.128546 (-0.104632)	0.007204 / 0.075646 (-0.068442)	0.204257 / 0.419271 (-0.215015)	0.063879 / 0.043533 (0.020346)	0.253008 / 0.255139 (-0.002131)	0.266554 / 0.283200 (-0.016645)	0.018929 / 0.141683 (-0.122754)	1.140547 / 1.452155 (-0.311608)	1.197049 / 1.492716 (-0.295668)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094111 / 0.018006 (0.076105)	0.301618 / 0.000490 (0.301128)	0.000219 / 0.000200 (0.000019)	0.000042 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018614 / 0.037411 (-0.018797)	0.062426 / 0.014526 (0.047900)	0.073079 / 0.176557 (-0.103477)	0.120313 / 0.737135 (-0.616823)	0.076445 / 0.296338 (-0.219894)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.285151 / 0.215209 (0.069942)	2.754272 / 2.077655 (0.676617)	1.485254 / 1.504120 (-0.018866)	1.368412 / 1.541195 (-0.172783)	1.402819 / 1.468490 (-0.065671)	0.396561 / 4.584777 (-4.188216)	2.375708 / 3.745712 (-1.370004)	2.656088 / 5.269862 (-2.613773)	1.588676 / 4.565676 (-2.977001)	0.048662 / 0.424275 (-0.375613)	0.004963 / 0.007607 (-0.002644)	0.339747 / 0.226044 (0.113702)	3.315841 / 2.268929 (1.046912)	1.841439 / 55.444624 (-53.603186)	1.547803 / 6.876477 (-5.328674)	1.601872 / 2.142072 (-0.540200)	0.468637 / 4.805227 (-4.336591)	0.099423 / 6.500664 (-6.401241)	0.041926 / 0.075469 (-0.033543)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.933058 / 1.841788 (-0.908730)	11.680870 / 8.074308 (3.606561)	10.239009 / 10.191392 (0.047617)	0.129974 / 0.680424 (-0.550450)	0.014081 / 0.534201 (-0.520120)	0.273076 / 0.579283 (-0.306207)	0.261914 / 0.434364 (-0.172450)	0.305982 / 0.540337 (-0.234356)	0.430623 / 1.386936 (-0.956313)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004969 / 0.011353 (-0.006384)	0.003084 / 0.011008 (-0.007924)	0.048686 / 0.038508 (0.010178)	0.057234 / 0.023109 (0.034125)	0.295408 / 0.275898 (0.019510)	0.323774 / 0.323480 (0.000294)	0.004014 / 0.007986 (-0.003972)	0.002423 / 0.004328 (-0.001905)	0.048000 / 0.004250 (0.043749)	0.039872 / 0.037052 (0.002820)	0.294717 / 0.258489 (0.036228)	0.331149 / 0.293841 (0.037309)	0.027884 / 0.128546 (-0.100662)	0.007155 / 0.075646 (-0.068491)	0.053812 / 0.419271 (-0.365460)	0.032483 / 0.043533 (-0.011050)	0.293402 / 0.255139 (0.038263)	0.312553 / 0.283200 (0.029354)	0.017848 / 0.141683 (-0.123835)	1.125600 / 1.452155 (-0.326554)	1.189469 / 1.492716 (-0.303248)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.096198 / 0.018006 (0.078191)	0.305096 / 0.000490 (0.304607)	0.000229 / 0.000200 (0.000029)	0.000045 / 0.000054 (-0.000010)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021992 / 0.037411 (-0.015419)	0.072082 / 0.014526 (0.057556)	0.082704 / 0.176557 (-0.093853)	0.124512 / 0.737135 (-0.612624)	0.084541 / 0.296338 (-0.211797)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.296440 / 0.215209 (0.081231)	2.923392 / 2.077655 (0.845738)	1.599057 / 1.504120 (0.094937)	1.480473 / 1.541195 (-0.060722)	1.551837 / 1.468490 (0.083347)	0.418618 / 4.584777 (-4.166159)	2.472727 / 3.745712 (-1.272985)	2.796141 / 5.269862 (-2.473721)	1.629139 / 4.565676 (-2.936538)	0.047703 / 0.424275 (-0.376572)	0.004971 / 0.007607 (-0.002636)	0.354453 / 0.226044 (0.128408)	3.514861 / 2.268929 (1.245932)	1.993597 / 55.444624 (-53.451028)	1.694386 / 6.876477 (-5.182090)	1.748562 / 2.142072 (-0.393510)	0.487158 / 4.805227 (-4.318070)	0.102021 / 6.500664 (-6.398643)	0.042648 / 0.075469 (-0.032821)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.974950 / 1.841788 (-0.866837)	13.391204 / 8.074308 (5.316896)	11.474696 / 10.191392 (1.283304)	0.142618 / 0.680424 (-0.537806)	0.016163 / 0.534201 (-0.518038)	0.271453 / 0.579283 (-0.307830)	0.287049 / 0.434364 (-0.147315)	0.309069 / 0.540337 (-0.231268)	0.417117 / 1.386936 (-0.969819)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004974 / 0.011353 (-0.006379)	0.002950 / 0.011008 (-0.008058)	0.061856 / 0.038508 (0.023348)	0.030539 / 0.023109 (0.007429)	0.250105 / 0.275898 (-0.025793)	0.276687 / 0.323480 (-0.046793)	0.003077 / 0.007986 (-0.004908)	0.002412 / 0.004328 (-0.001916)	0.048336 / 0.004250 (0.044086)	0.045849 / 0.037052 (0.008797)	0.251757 / 0.258489 (-0.006732)	0.284914 / 0.293841 (-0.008927)	0.024033 / 0.128546 (-0.104513)	0.007343 / 0.075646 (-0.068303)	0.202867 / 0.419271 (-0.216405)	0.061294 / 0.043533 (0.017762)	0.263590 / 0.255139 (0.008451)	0.272744 / 0.283200 (-0.010455)	0.019613 / 0.141683 (-0.122070)	1.104263 / 1.452155 (-0.347892)	1.164128 / 1.492716 (-0.328588)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094261 / 0.018006 (0.076255)	0.303340 / 0.000490 (0.302850)	0.000215 / 0.000200 (0.000015)	0.000057 / 0.000054 (0.000003)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018381 / 0.037411 (-0.019030)	0.062727 / 0.014526 (0.048201)	0.074955 / 0.176557 (-0.101602)	0.124810 / 0.737135 (-0.612326)	0.074335 / 0.296338 (-0.222004)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.279368 / 0.215209 (0.064159)	2.721641 / 2.077655 (0.643986)	1.510773 / 1.504120 (0.006653)	1.364349 / 1.541195 (-0.176845)	1.386044 / 1.468490 (-0.082446)	0.403051 / 4.584777 (-4.181726)	2.416525 / 3.745712 (-1.329187)	2.623198 / 5.269862 (-2.646663)	1.560869 / 4.565676 (-3.004808)	0.046613 / 0.424275 (-0.377662)	0.004861 / 0.007607 (-0.002746)	0.337875 / 0.226044 (0.111830)	3.289956 / 2.268929 (1.021028)	1.851707 / 55.444624 (-53.592917)	1.571092 / 6.876477 (-5.305385)	1.600328 / 2.142072 (-0.541745)	0.480766 / 4.805227 (-4.324461)	0.099138 / 6.500664 (-6.401526)	0.041691 / 0.075469 (-0.033779)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.941162 / 1.841788 (-0.900626)	11.745335 / 8.074308 (3.671027)	10.645509 / 10.191392 (0.454117)	0.132506 / 0.680424 (-0.547918)	0.015192 / 0.534201 (-0.519009)	0.272483 / 0.579283 (-0.306800)	0.270269 / 0.434364 (-0.164094)	0.309580 / 0.540337 (-0.230758)	0.431513 / 1.386936 (-0.955423)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005068 / 0.011353 (-0.006285)	0.003069 / 0.011008 (-0.007939)	0.048605 / 0.038508 (0.010097)	0.059557 / 0.023109 (0.036448)	0.275092 / 0.275898 (-0.000806)	0.298910 / 0.323480 (-0.024570)	0.004198 / 0.007986 (-0.003788)	0.002499 / 0.004328 (-0.001830)	0.048248 / 0.004250 (0.043997)	0.040302 / 0.037052 (0.003249)	0.279539 / 0.258489 (0.021050)	0.312500 / 0.293841 (0.018659)	0.025407 / 0.128546 (-0.103140)	0.007364 / 0.075646 (-0.068282)	0.053086 / 0.419271 (-0.366186)	0.033291 / 0.043533 (-0.010242)	0.276521 / 0.255139 (0.021382)	0.292943 / 0.283200 (0.009743)	0.019416 / 0.141683 (-0.122267)	1.151734 / 1.452155 (-0.300421)	1.205021 / 1.492716 (-0.287695)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094112 / 0.018006 (0.076106)	0.309534 / 0.000490 (0.309044)	0.000219 / 0.000200 (0.000019)	0.000052 / 0.000054 (-0.000003)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021539 / 0.037411 (-0.015872)	0.070325 / 0.014526 (0.055799)	0.080468 / 0.176557 (-0.096089)	0.121095 / 0.737135 (-0.616040)	0.082008 / 0.296338 (-0.214331)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.302591 / 0.215209 (0.087382)	2.943475 / 2.077655 (0.865820)	1.597970 / 1.504120 (0.093850)	1.468774 / 1.541195 (-0.072421)	1.504812 / 1.468490 (0.036322)	0.413715 / 4.584777 (-4.171062)	2.418319 / 3.745712 (-1.327393)	2.616656 / 5.269862 (-2.653206)	1.558165 / 4.565676 (-3.007512)	0.047169 / 0.424275 (-0.377106)	0.004761 / 0.007607 (-0.002846)	0.347225 / 0.226044 (0.121180)	3.479624 / 2.268929 (1.210696)	1.961253 / 55.444624 (-53.483371)	1.673532 / 6.876477 (-5.202944)	1.698900 / 2.142072 (-0.443172)	0.488373 / 4.805227 (-4.316855)	0.098322 / 6.500664 (-6.402342)	0.040832 / 0.075469 (-0.034637)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.009133 / 1.841788 (-0.832655)	13.373258 / 8.074308 (5.298949)	11.327360 / 10.191392 (1.135968)	0.135778 / 0.680424 (-0.544646)	0.015813 / 0.534201 (-0.518388)	0.275404 / 0.579283 (-0.303879)	0.282564 / 0.434364 (-0.151799)	0.311830 / 0.540337 (-0.228507)	0.419008 / 1.386936 (-0.967928)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004899 / 0.011353 (-0.006454)	0.002780 / 0.011008 (-0.008229)	0.061997 / 0.038508 (0.023489)	0.029909 / 0.023109 (0.006800)	0.233445 / 0.275898 (-0.042453)	0.254128 / 0.323480 (-0.069351)	0.002927 / 0.007986 (-0.005058)	0.002396 / 0.004328 (-0.001932)	0.048118 / 0.004250 (0.043868)	0.044520 / 0.037052 (0.007468)	0.237594 / 0.258489 (-0.020895)	0.268407 / 0.293841 (-0.025434)	0.023517 / 0.128546 (-0.105029)	0.007035 / 0.075646 (-0.068612)	0.202803 / 0.419271 (-0.216469)	0.057692 / 0.043533 (0.014159)	0.237058 / 0.255139 (-0.018081)	0.252966 / 0.283200 (-0.030233)	0.017934 / 0.141683 (-0.123748)	1.096406 / 1.452155 (-0.355749)	1.153509 / 1.492716 (-0.339207)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091812 / 0.018006 (0.073806)	0.298410 / 0.000490 (0.297920)	0.000228 / 0.000200 (0.000028)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018333 / 0.037411 (-0.019078)	0.062685 / 0.014526 (0.048159)	0.073295 / 0.176557 (-0.103261)	0.119234 / 0.737135 (-0.617901)	0.074603 / 0.296338 (-0.221736)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.279078 / 0.215209 (0.063869)	2.768535 / 2.077655 (0.690880)	1.457049 / 1.504120 (-0.047071)	1.326870 / 1.541195 (-0.214325)	1.349657 / 1.468490 (-0.118833)	0.405003 / 4.584777 (-4.179774)	2.428726 / 3.745712 (-1.316986)	2.595776 / 5.269862 (-2.674086)	1.557879 / 4.565676 (-3.007797)	0.045985 / 0.424275 (-0.378291)	0.004854 / 0.007607 (-0.002753)	0.336437 / 0.226044 (0.110392)	3.317330 / 2.268929 (1.048401)	1.784525 / 55.444624 (-53.660100)	1.500295 / 6.876477 (-5.376182)	1.529869 / 2.142072 (-0.612203)	0.473426 / 4.805227 (-4.331801)	0.099609 / 6.500664 (-6.401055)	0.042054 / 0.075469 (-0.033415)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.937154 / 1.841788 (-0.904633)	11.482383 / 8.074308 (3.408075)	10.468769 / 10.191392 (0.277377)	0.132724 / 0.680424 (-0.547700)	0.015242 / 0.534201 (-0.518959)	0.281124 / 0.579283 (-0.298159)	0.268603 / 0.434364 (-0.165761)	0.311410 / 0.540337 (-0.228928)	0.431817 / 1.386936 (-0.955119)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004695 / 0.011353 (-0.006658)	0.002873 / 0.011008 (-0.008135)	0.048133 / 0.038508 (0.009625)	0.052505 / 0.023109 (0.029396)	0.271679 / 0.275898 (-0.004219)	0.292530 / 0.323480 (-0.030950)	0.003844 / 0.007986 (-0.004142)	0.002417 / 0.004328 (-0.001912)	0.048619 / 0.004250 (0.044369)	0.039152 / 0.037052 (0.002100)	0.276575 / 0.258489 (0.018086)	0.307836 / 0.293841 (0.013995)	0.023877 / 0.128546 (-0.104669)	0.006897 / 0.075646 (-0.068749)	0.053241 / 0.419271 (-0.366031)	0.032487 / 0.043533 (-0.011046)	0.274205 / 0.255139 (0.019066)	0.289701 / 0.283200 (0.006502)	0.018250 / 0.141683 (-0.123432)	1.137902 / 1.452155 (-0.314253)	1.202043 / 1.492716 (-0.290673)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091453 / 0.018006 (0.073446)	0.297032 / 0.000490 (0.296543)	0.000224 / 0.000200 (0.000024)	0.000056 / 0.000054 (0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021062 / 0.037411 (-0.016349)	0.069848 / 0.014526 (0.055322)	0.084337 / 0.176557 (-0.092219)	0.119951 / 0.737135 (-0.617184)	0.082805 / 0.296338 (-0.213533)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.297056 / 0.215209 (0.081846)	2.890110 / 2.077655 (0.812456)	1.609918 / 1.504120 (0.105798)	1.491184 / 1.541195 (-0.050011)	1.529433 / 1.468490 (0.060943)	0.396081 / 4.584777 (-4.188696)	2.408310 / 3.745712 (-1.337402)	2.567905 / 5.269862 (-2.701957)	1.514465 / 4.565676 (-3.051212)	0.045329 / 0.424275 (-0.378946)	0.004738 / 0.007607 (-0.002869)	0.344373 / 0.226044 (0.118328)	3.428333 / 2.268929 (1.159404)	1.981401 / 55.444624 (-53.463223)	1.688007 / 6.876477 (-5.188470)	1.685542 / 2.142072 (-0.456531)	0.478045 / 4.805227 (-4.327182)	0.096664 / 6.500664 (-6.404001)	0.040335 / 0.075469 (-0.035135)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.972912 / 1.841788 (-0.868876)	12.055045 / 8.074308 (3.980737)	10.821073 / 10.191392 (0.629681)	0.139177 / 0.680424 (-0.541247)	0.015046 / 0.534201 (-0.519155)	0.275670 / 0.579283 (-0.303613)	0.280366 / 0.434364 (-0.153998)	0.315781 / 0.540337 (-0.224556)	0.424536 / 1.386936 (-0.962400)

[Wauplin]

I'm finally merging as transformers/tokenizers dependency pins have been removed + huggingface_hub 0.19.4 has fixed the deps incompatibility issue. All good now :)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004435 / 0.011353 (-0.006918)	0.002924 / 0.011008 (-0.008084)	0.062159 / 0.038508 (0.023651)	0.029639 / 0.023109 (0.006529)	0.237470 / 0.275898 (-0.038428)	0.269641 / 0.323480 (-0.053839)	0.004124 / 0.007986 (-0.003862)	0.002528 / 0.004328 (-0.001800)	0.048114 / 0.004250 (0.043864)	0.046055 / 0.037052 (0.009002)	0.245844 / 0.258489 (-0.012645)	0.278085 / 0.293841 (-0.015756)	0.023152 / 0.128546 (-0.105394)	0.007194 / 0.075646 (-0.068452)	0.206493 / 0.419271 (-0.212778)	0.055687 / 0.043533 (0.012155)	0.243301 / 0.255139 (-0.011838)	0.267645 / 0.283200 (-0.015555)	0.017413 / 0.141683 (-0.124270)	1.113071 / 1.452155 (-0.339083)	1.201436 / 1.492716 (-0.291280)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092576 / 0.018006 (0.074570)	0.303516 / 0.000490 (0.303027)	0.000213 / 0.000200 (0.000013)	0.000043 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019108 / 0.037411 (-0.018303)	0.062326 / 0.014526 (0.047800)	0.073711 / 0.176557 (-0.102846)	0.120414 / 0.737135 (-0.616721)	0.075837 / 0.296338 (-0.220501)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.278267 / 0.215209 (0.063058)	2.766231 / 2.077655 (0.688576)	1.455613 / 1.504120 (-0.048507)	1.337128 / 1.541195 (-0.204066)	1.357659 / 1.468490 (-0.110831)	0.404549 / 4.584777 (-4.180228)	2.409084 / 3.745712 (-1.336628)	2.645000 / 5.269862 (-2.624861)	1.600475 / 4.565676 (-2.965201)	0.046680 / 0.424275 (-0.377595)	0.004887 / 0.007607 (-0.002720)	0.340338 / 0.226044 (0.114294)	3.332647 / 2.268929 (1.063719)	1.852529 / 55.444624 (-53.592096)	1.532442 / 6.876477 (-5.344035)	1.550383 / 2.142072 (-0.591689)	0.482702 / 4.805227 (-4.322525)	0.101067 / 6.500664 (-6.399597)	0.042132 / 0.075469 (-0.033337)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.945481 / 1.841788 (-0.896307)	11.886240 / 8.074308 (3.811932)	10.484620 / 10.191392 (0.293228)	0.130906 / 0.680424 (-0.549518)	0.014880 / 0.534201 (-0.519321)	0.268836 / 0.579283 (-0.310447)	0.268112 / 0.434364 (-0.166251)	0.304300 / 0.540337 (-0.236038)	0.440262 / 1.386936 (-0.946674)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005028 / 0.011353 (-0.006325)	0.002937 / 0.011008 (-0.008071)	0.049038 / 0.038508 (0.010530)	0.057763 / 0.023109 (0.034653)	0.273196 / 0.275898 (-0.002702)	0.295519 / 0.323480 (-0.027961)	0.004102 / 0.007986 (-0.003883)	0.002487 / 0.004328 (-0.001841)	0.049148 / 0.004250 (0.044898)	0.040303 / 0.037052 (0.003251)	0.279187 / 0.258489 (0.020698)	0.311086 / 0.293841 (0.017245)	0.024961 / 0.128546 (-0.103585)	0.007264 / 0.075646 (-0.068382)	0.055711 / 0.419271 (-0.363561)	0.032355 / 0.043533 (-0.011178)	0.274304 / 0.255139 (0.019165)	0.290953 / 0.283200 (0.007753)	0.018358 / 0.141683 (-0.123325)	1.115984 / 1.452155 (-0.336170)	1.190409 / 1.492716 (-0.302308)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.095765 / 0.018006 (0.077759)	0.287947 / 0.000490 (0.287457)	0.000242 / 0.000200 (0.000042)	0.000047 / 0.000054 (-0.000007)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022165 / 0.037411 (-0.015246)	0.070465 / 0.014526 (0.055940)	0.082078 / 0.176557 (-0.094479)	0.120209 / 0.737135 (-0.616926)	0.084573 / 0.296338 (-0.211765)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.298492 / 0.215209 (0.083283)	2.924981 / 2.077655 (0.847327)	1.597326 / 1.504120 (0.093206)	1.459132 / 1.541195 (-0.082062)	1.511471 / 1.468490 (0.042981)	0.406671 / 4.584777 (-4.178106)	2.443154 / 3.745712 (-1.302558)	2.591131 / 5.269862 (-2.678731)	1.549931 / 4.565676 (-3.015745)	0.047042 / 0.424275 (-0.377234)	0.004891 / 0.007607 (-0.002716)	0.346274 / 0.226044 (0.120230)	3.456050 / 2.268929 (1.187121)	1.959328 / 55.444624 (-53.485296)	1.647631 / 6.876477 (-5.228845)	1.692024 / 2.142072 (-0.450049)	0.478307 / 4.805227 (-4.326920)	0.098738 / 6.500664 (-6.401926)	0.041743 / 0.075469 (-0.033726)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.984619 / 1.841788 (-0.857168)	12.403984 / 8.074308 (4.329676)	10.974347 / 10.191392 (0.782955)	0.132893 / 0.680424 (-0.547530)	0.015504 / 0.534201 (-0.518697)	0.275354 / 0.579283 (-0.303929)	0.283312 / 0.434364 (-0.151052)	0.313661 / 0.540337 (-0.226677)	0.419065 / 1.386936 (-0.967871)