Speedup package:crypto (with a focus on md5) #892

jensjoha · 2025-05-22T13:41:06Z

TL;DR
This makes it faster and use less ram.

Details
Tested on out/ReleaseX64/wasm-opt from a dart sdk build (167MB):

JIT, dart --deterministic:

Difference at 95.0% confidence
        -313.4 +/- 44.2824
        -22.8359% +/- 3.22664%
        (Student's t, pooled s = 30.3628)

JIT, dart:

Difference at 95.0% confidence
        -275.6 +/- 43.5498
        -20.1905% +/- 3.19046%
        (Student's t, pooled s = 29.8605)

AOT, dartaotruntime --deterministic:

Difference at 95.0% confidence
        -499 +/- 11.5944
        -40.3135% +/- 0.936693%
        (Student's t, pooled s = 7.94984)

AOT, dartaotruntime:

Difference at 95.0% confidence
        -506.8 +/- 39.0824
        -41.304% +/- 3.1852%
        (Student's t, pooled s = 26.7974)

Running the AOT snapshots through the "benchmarker" utility from dart front_end I get these results (with 25 runs of each version):

msec task-clock:u: -41.1276% +/- 0.3390% (-515.19 +/- 4.25) (1252.67 -> 737.48)
page-faults:u: -55.7760% +/- 0.1266% (-655.48 +/- 1.49) (1175.20 -> 519.72)
cycles:u: -41.7236% +/- 0.3140% (-2185317986.60 +/- 16443529.36) (5237610655.96 -> 3052292669.36)
instructions:u: -32.5192% +/- 0.0000% (-5346525546.76 +/- 2905.52) (16441121298.68 -> 11094595751.92)
branch-misses:u: -97.0502% +/- 0.3379% (-8025400.44 +/- 27942.52) (8269332.32 -> 243931.88)
seconds time elapsed: -41.0827% +/- 0.3407% (-0.52 +/- 0.00) (1.25 -> 0.74)
seconds user: -41.7571% +/- 0.4373% (-0.50 +/- 0.01) (1.19 -> 0.70)
seconds sys: -27.8724% +/- 4.1716% (-0.02 +/- 0.00) (0.06 -> 0.04)
MarkSweep(   old space) goes from 2 to 1

Running it through /usr/bin/time -v reports the "Maximum resident set size (kbytes)" going from around ~346k to ~176k (i.e. uses about half the ram).

MD5 calculations is used internally in the analyzer, and patching this in, for an AOT compile of the analyzer we see changes like this (when run through the benchmarker with parameters like --train-using pkg/front_end/lib/ in the dart sdk):

msec task-clock:u: -1.2865% +/- 0.2432% (-105.99 +/- 20.03) (8238.61 -> 8132.62)
page-faults:u: -6.7334% +/- 0.0969% (-10705.66 +/- 154.06) (158992.30 -> 148286.64)
cycles:u: -1.1201% +/- 0.2487% (-384785056.06 +/- 85428229.56) (34351469228.82 -> 33966684172.76)
instructions:u: -2.6679% +/- 0.0191% (-1227909335.74 +/- 8796243.14) (46025301866.14 -> 44797392530.40)
branch-misses:u: -1.5364% +/- 0.9908% (-1886562.18 +/- 1216638.62) (122788764.20 -> 120902202.02)
seconds time elapsed: -1.3342% +/- 0.2478% (-0.11 +/- 0.02) (8.27 -> 8.16)
seconds user: -1.0854% +/- 0.2985% (-0.09 +/- 0.02) (7.88 -> 7.79)
seconds sys: -5.8743% +/- 3.7392% (-0.02 +/- 0.01) (0.36 -> 0.34)
Scavenge(   new space) goes from 134 to 131
MarkSweep(   old space) goes from 3 to 2

An aot-compile, hashing a big file says ``` Difference at 95.0% confidence -140.4 +/- 35.1436 -11.2825% +/- 2.82414% (Student's t, pooled s = 24.0967) ```

github-actions · 2025-05-22T13:43:30Z

PR Health

Breaking changes ✔️

Package	Change	Current Version	New Version	Needed Version	Looking good?
crypto	None	3.0.6	3.0.7-wip	3.0.6	✔️

Changelog Entry ❗

Package	Changed Files
package:crypto	pkgs/crypto/lib/src/hash_sink.dart pkgs/crypto/lib/src/md5.dart

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Coverage

⚠️

File	Coverage
pkgs/crypto/benchmark/benchmark.dart	💔 Not covered
pkgs/crypto/lib/src/hash_sink.dart	💚 98 % ⬆️ 0 %
pkgs/crypto/lib/src/md5.dart	💚 100 %
pkgs/crypto/tool/md5sum.dart	💔 Not covered

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbols

License Headers ✔️

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

pkgs/crypto/lib/src/hash_sink.dart

pkgs/crypto/utils/md5sum.dart

pkgs/crypto/lib/src/hash_sink.dart

lrhn

LGTM.
Feel free to ignore my style comments if it works and is faster, and you can't be bothered.
We can always improve more later.

pkgs/crypto/lib/src/hash_sink.dart

pkgs/crypto/lib/src/md5.dart

pkgs/crypto/utils/md5sum.dart

pkgs/crypto/lib/src/md5.dart

pkgs/crypto/lib/src/hash_sink.dart

lrhn · 2025-05-23T14:40:04Z

pkgs/crypto/benchmark/benchmark.dart

+    } else if (arg == 'sha512256') {
+      setFunction(sha512256, 'sha512/256');
+    } else {
+      throw ArgumentError('Unknown argument: $arg');


Maybe print an error on stderr and exit. This is a command line usage mistake, not a Dart argument error.

lrhn · 2025-05-23T14:43:32Z

pkgs/crypto/benchmark/benchmark.dart

+    print('Using hash function $message');
+  }
+
+  for (var arg in args) {


Seems like you only need to look at args.first. If there are two, either one is an error, or the second will find the function already set.

jensjoha added 6 commits May 19, 2025 12:29

hash sink copies less

627e433

An aot-compile, hashing a big file says ``` Difference at 95.0% confidence -140.4 +/- 35.1436 -11.2825% +/- 2.82414% (Student's t, pooled s = 24.0967) ```

slightly better still

f731eb0

split for loop

b22e39e

Get rid of some bound checks

ae5fcff

Add util, remove comments referencing old implementation

78c723e

Add copyright header to new file

51bf2d3

jensjoha requested a review from a team as a code owner May 22, 2025 13:41

github-actions bot added the package:crypto label May 22, 2025

mosuem reviewed May 22, 2025

View reviewed changes

pkgs/crypto/lib/src/hash_sink.dart Outdated Show resolved Hide resolved

mosuem requested a review from lrhn May 22, 2025 14:18

kevmoo reviewed May 22, 2025

View reviewed changes

pkgs/crypto/utils/md5sum.dart Outdated Show resolved Hide resolved

kevmoo reviewed May 22, 2025

View reviewed changes

pkgs/crypto/lib/src/hash_sink.dart Show resolved Hide resolved

kevmoo reviewed May 22, 2025

View reviewed changes

pkgs/crypto/lib/src/hash_sink.dart Outdated Show resolved Hide resolved

jensjoha added 2 commits May 23, 2025 10:53

Add benchmark

c17890e

Added comments to fields; renamed fields.

64e7ef6

lrhn approved these changes May 23, 2025

View reviewed changes

Feedback

61cde33

lrhn reviewed May 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Speedup package:crypto (with a focus on md5) #892

Speedup package:crypto (with a focus on md5) #892

Uh oh!

jensjoha commented May 22, 2025

Uh oh!

github-actions bot commented May 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lrhn left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lrhn May 23, 2025 •

edited

Loading

Uh oh!

lrhn May 23, 2025

Uh oh!

Uh oh!

Speedup package:crypto (with a focus on md5) #892

Are you sure you want to change the base?

Speedup package:crypto (with a focus on md5) #892

Uh oh!

Conversation

jensjoha commented May 22, 2025

Uh oh!

github-actions bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Health

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lrhn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lrhn May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lrhn May 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented May 22, 2025 •

edited

Loading

lrhn May 23, 2025 •

edited

Loading