Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C# module benchmarks #1679

Merged
merged 6 commits into from
Sep 6, 2024
Merged

C# module benchmarks #1679

merged 6 commits into from
Sep 6, 2024

Conversation

RReverser
Copy link
Member

Description of Changes

These are ported from Rust benchmarks 1:1 (similarly to how sdk-test-cs is ported from sdk-test) by using Copilot + elbow grease for manual fixes.

When you set an environment variable STDB_BENCH_CS=1, this allows to run all the same module benchmarks as Rust currently does, except against an analogous C# implementation.

API and ABI breaking changes

If this is an API or ABI breaking change, please apply the
corresponding GitHub label.

Expected complexity level and risk

How complicated do you think these changes are? Grade on a scale from 1 to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex change.

This complexity rating applies not only to the complexity apparent in the diff,
but also to its interactions with existing and future code.

If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning ways.

Testing

Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!

  • Write a test you've completed here. Tried running C# benchmarks.
  • Write a test you want a reviewer to do here, so they can check it off when they're satisfied. Try running C# benchmarks.

@RReverser
Copy link
Member Author

Example run in Rust

special/stdb_module/large_arguments/64KiB
                        time:   [75.082 µs 75.849 µs 76.733 µs]
                        change: [-9.7556% -7.9926% -6.4687%] (p = 0.00 < 0.05)
                        Performance has improved.

vs C#

special/stdb_module/large_arguments/64KiB
                        time:   [1.2303 ms 1.2388 ms 1.2474 ms]
                        change: [+1500.6% +1523.1% +1543.4%] (p = 0.00 < 0.05)
                        Performance has regressed.

@RReverser
Copy link
Member Author

For the sake of posterity, here's EXPERIMENTAL_WASM_AOT vs regular C#:

special/stdb_module/large_arguments/64KiB
                        time:   [147.81 µs 149.06 µs 150.45 µs]
                        change: [-86.822% -86.542% -86.228%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)

only 2x slower than Rust (which is in line with my early microbenchmarks).

@cloutiertyler
Copy link
Contributor

This is great to have!

public static ulong MomentMilliseconds()
{
// Idk why Rust uses complicated math, but it seems like it should always return 1.
return 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was purely to copy the way was done and use the time methods, but is likely that this could be just a const

Copy link
Contributor

@mamcx mamcx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@RReverser RReverser enabled auto-merge September 6, 2024 17:54
@RReverser RReverser added this pull request to the merge queue Sep 6, 2024
Merged via the queue into master with commit 2cccac4 Sep 6, 2024
8 checks passed
@RReverser RReverser deleted the ingvar/csharp-bench-module branch September 6, 2024 22:22
kim added a commit that referenced this pull request Sep 27, 2024
Since #1679, a call to the publish endpoint that doesn't update but
create a database would drop errors and falsely return a success
response.

While the `ControlStateWriteAccess::publish` method can't currently
express the different error branches, we make it so creation errors are
returned as `Some(UpdateDatabaseResult::ErrorExecutingMigration)`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants