Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Computation Expression Benchmarks #16541

Merged
merged 4 commits into from
Jan 17, 2024
Merged

Conversation

dawedawe
Copy link
Contributor

@dawedawe dawedawe commented Jan 17, 2024

Description

While working on improving the compilation speed for CEs, it might be a good idea to have some benchmarks in the repo with different CE like with/without Custom Ops, different levels of nesting, different length.
I disabled CE100xnest10.fs for now, because it really runs for a long time...

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
Check100xNestingOf1 929.3 us 11.66 us 10.34 us 1.9531 - - 397.2 KB
Check100xNestingOf5 965.3 us 11.38 us 10.09 us 3.9063 - - 467.41 KB
Check200xNestingOf5 1,118.4 us 12.73 us 11.91 us 3.9063 1.9531 1.9531 544.12 KB
Check100xNestingOf10 2,776.8 us 57.31 us 165.35 us - - - 610.59 KB
CheckNesting1WithCustOps 1,037.0 us 14.78 us 13.83 us 3.9063 - - 486.02 KB
CheckNesting5WithCustOps 1,353.8 us 26.11 us 31.08 us 7.8125 5.8594 3.9063 703.79 KB
Compile100xNestingOf1 611,959.0 us 7,753.45 us 6,474.48 us 1000.0000 - - 249178.91 KB
Compile100xNestingOf5 6,232,674.9 us 40,350.93 us 37,744.28 us 18000.0000 4000.0000 2000.0000 2211470.08 KB
Compile200xNestingOf5 6,220,882.7 us 39,973.98 us 35,435.89 us 19000.0000 4000.0000 3000.0000 2208148.9 KB
Compile100xNestingOf10 113,578,524.0 us 2,248,102.35 us 6,154,143.90 us 415000.0000 37000.0000 23000.0000 53534013.59 KB
CompileNesting1WithCustOps 1,088.9 us 20.22 us 39.91 us 3.9063 - - 484.16 KB
CompileNesting5WithCustOps 1,352.9 us 25.09 us 24.65 us 7.8125 3.9063 3.9063 703.77 KB

Checklist

  • Test cases added

  • Performance benchmarks added in case of performance changes

  • Release notes entry updated:

    Please make sure to add an entry with short succinct description of the change as well as link to this pull request to the respective release notes file, if applicable.

    Release notes files:

    • If anything under src/Compiler has been changed, please make sure to make an entry in docs/release-notes/.FSharp.Compiler.Service/<version>.md, where <version> is usually "highest" one, e.g. 42.8.200
    • If language feature was added (i.e. LanguageFeatures.fsi was changed), please add it to docs/releae-notes/.Language/preview.md
    • If a change to FSharp.Core was made, please make sure to edit docs/release-notes/.FSharp.Core/<version>.md where version is "highest" one, e.g. 8.0.200.

    Information about the release notes entries format can be found in the documentation.
    Example:

    If you believe that release notes are not necessary for this PR, please add NO_RELEASE_NOTES label to the pull request.

@dawedawe dawedawe requested a review from a team as a code owner January 17, 2024 11:33
Copy link
Contributor

✅ No release notes required

Copy link
Member

@psfinaki psfinaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff - thanks!

I am currently working on doing some CI checks for benchmarks, as of now there is at least a build of the bench solution, we will probably also have some smoke test execution of the tests to be more sure the code stays up to date.

@psfinaki
Copy link
Member

/azp run

Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@psfinaki psfinaki merged commit 7e91ae6 into dotnet:main Jan 17, 2024
28 checks passed
@dawedawe dawedawe deleted the ce_benchmarks branch January 17, 2024 14:29
psfinaki added a commit that referenced this pull request Jan 25, 2024
* Name resolution: keep type vars in subsequent checks (#16456)

* Keep typars produced in name resolution

* Better debug errors

* Unwrap measure type vars

* Undo check declarations change

* Fix reported range

* Undo occurrence change

* Skip path typars

* Add test

* More freshen typar APIs properly

* Fantomas

* Cleanup

* Add release notes

* 123

---------

Co-authored-by: Vlad Zarytovskii <vzaritovsky@hotmail.com>

* Build benchmarks in CI (#16518)

* Remove profiling startpoint project

* Add bench build job

* Up

* up

* up

---------

Co-authored-by: Kevin Ransom (msft) <codecutter@hotmail.com>

* More ValueOption in compiler: part 1 (#16323)

* More ValueOption in compiler: part 1

* release notes

* Update CheckComputationExpressions.fs

* release notes

* `[Experimental]` `[WIP]` Transparent Compiler (#15179)

* Track CheckDeclarations.CheckModuleSignature activity. (#16534)

* Add Computation Expression Benchmarks (#16541)

* add benchmarks for various usages of CEs

* refactor

* move CE source files to dedicated ce folder

* Update Roslyn to a version which uses Immutable v7 (#16545)

* revert #16326 (addition of XliffTasks reference) (#16548)

* updated devcontainer image (#16551)

* Add higher-order-function-based API for working with untyped AST (#16462)

* Add module-based API for working with untyped AST

* Fantomas

* tryPickUntil → tryPickDownTo

* Don't need that

* Thread path while walking

* Update comment

* Simplify

* Expose `Ast.fold` and `Ast.tryPick`.
* Expose `SyntaxNode.(|Attributes|)`.
* Ensure a few more syntax node cases get hit.

* Update FCS release notes

* Update surface area

* Add back `foldWhile`; add `exists`, `tryNode`

* Put `Ast.foldWhile` back in.

* Add `Ast.exists`.

* Add `Ast.tryNode`.

* `SyntaxTraversal.Traverse` → `Ast.tryPick`…

* Replace uses of `SyntaxTraversal.Traverse` in `FSharpParseFileResults`
  with the appropriate function from the `Ast` module: `exists`,
  `tryPick`, `tryNode`.

* Update surface area

* Need that

* Just to be safe

* Add `Ast.tryPickLast`

* Handle multiple args mid-pipeline

* Before, no signature help was offered in a case like this:

  ```fsharp
  [1..10]
  |> List.fold (fun acc _ -> acc) ‸
  |> List.filter (fun x -> x > 3)
  ```

  The service will now offer help for the `state` parameter when the
  cursor ‸ is in that location.

* `*` instead of error

* `FSharpParseFileResults.TryRangeOfFunctionOrMethodBeingApplied` was
  previously returning the range of the (zero-width)
  `SynExpr.ArbitraryAfterError`. It now returns the range of the `*`
  (`op_Multiply`) instead.

* Update surface area

* Fmt

* Missed in merge

* Add VS release notes entry

* # → ###

* Add 	ryPick tests

* Add a few more tests

* \n

* Bump release notes

* Fmt

* `Ast` → `ParsedInput`

* Use `ParsedInput` as the main AST type.

* Move the `position` parameter rightward.

* Update surface area

* Less `function`

* Update untyped AST docs

* Add basic examples for `ParsedInput` module functions.

* Merge the existing `SyntaxVisitorBase` docs into the new file.

* Clean up doc comments

---------

Co-authored-by: Vlad Zarytovskii <vzaritovsky@hotmail.com>

* Move paren entries to appropriate releases (#16561)

* [main] Update dependencies from dotnet/source-build-reference-packages (#16532)

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240115.2

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24059.3 -> To Version 9.0.0-alpha.1.24065.2

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240116.1

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24059.3 -> To Version 9.0.0-alpha.1.24066.1

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240117.1

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24059.3 -> To Version 9.0.0-alpha.1.24067.1

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240117.1

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24059.3 -> To Version 9.0.0-alpha.1.24067.1

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240117.1

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24059.3 -> To Version 9.0.0-alpha.1.24067.1

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240117.1

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24059.3 -> To Version 9.0.0-alpha.1.24067.1

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Vlad Zarytovskii <vzaritovsky@hotmail.com>

* Attempt to make links from single identifier module names. (#16550)

* Add scenarios where parentheses are around module name.

* Address problem tighter to nameof usage.

* Restore missing commit and inline nameof ident check.

* Add release note entry.

* rewrite SizeOfValueInfo in Optimizer.fs to be tail-recursive (#16559)

* rewrite SizeOfValueInfo in Optimizer.fs to be tail-recursive

* use Brians rewrite into one local function

* stringbuilder is not threadsafe (#16557)

* Array postfix notation in fsharp core api (#16564)

* changed array types to postfix form in all signatures
* changed array types to postfix form in the implementation files

* Revert 16348 (#16536)

* Improve AsyncMemoize tests

* relax test condition

* Revert "Cancellable: set token from node/async in features code (#16348)"

This reverts commit d4e3b26.

* remove UsingToken

* remove UsingToken

* test improvement

* relax test condition

* use thread-safe collections when collecting events from AsyncMemoize

* fix flaky test

* release note

* Small code reshuffle for diff minimization (#16569)

* Moving code around

* Small code reshuffle for diff minimization

* wat

* Refactor parens API (#16461)

* Refactor parens API

* Remove `UnnecessaryParentheses.getUnnecessaryParentheses`.

* Expose `SynExpr.shouldBeParenthesizedInContext`.

* Expose `SynPat.shouldBeParenthesizedInContext`.

* Expose `SyntaxTraversal.TraverseAll`.

* Fantomas

* Use `ParsedInput.fold`

* Tests

* Update surface area

* Clean up sigs & comments

* Update release notes

* Remove redundant async

* Remove stubs (no longer needed)

* Preserve original stacktrace in state machines if available (#16568)

* Preserve original stacktrace in state machines if available

* Update release notes

* Automated command ran: fantomas

  Co-authored-by: vzarytovskii <1260985+vzarytovskii@users.noreply.github.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* check reportErrors and feature support at top level (#16549)

* Align DU case augmentation with previous behavior in EraseUnions (#16571)

* Align DU case augment with previous behavior in EraseUnions

* Update 8.0.300.md

* modify tests

* Refresh debug surface area (#16573)

* Remove superfluous rec keywords and untangle some functions (#16544)

* remove some superfluous rec keywords and untangle two functions that aren't mutually recursive.

* Don't throw on invalid input in Graph construction (#16575)

* More ValueOption in compiler: part 2 (#16567)

* More ValueOption in complier: part 2

* Update release notes

* extra optimization

* extra optimization 2

* fantomas

* Update dependencies from https://github.com/dotnet/arcade build 20240123.2 (#16579)

Microsoft.DotNet.Arcade.Sdk
 From Version 8.0.0-beta.24060.4 -> To Version 8.0.0-beta.24073.2

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>

* [main] Update dependencies from dotnet/source-build-reference-packages (#16574)

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240122.5

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24067.1 -> To Version 9.0.0-alpha.1.24072.5

* Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20240123.1

Microsoft.SourceBuild.Intermediate.source-build-reference-packages
 From Version 9.0.0-alpha.1.24067.1 -> To Version 9.0.0-alpha.1.24073.1

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Tomas Grosup <tomasgrosup@microsoft.com>

* Improve AsyncMemoize tests (#16580)

---------

Co-authored-by: Eugene Auduchinok <eugene.auduchinok@gmail.com>
Co-authored-by: Vlad Zarytovskii <vzaritovsky@hotmail.com>
Co-authored-by: Petr <psfinaki@users.noreply.github.com>
Co-authored-by: Kevin Ransom (msft) <codecutter@hotmail.com>
Co-authored-by: Petr Pokorny <petrpokorny@microsoft.com>
Co-authored-by: Florian Verdonck <florian.verdonck@outlook.com>
Co-authored-by: dawe <dawedawe@posteo.de>
Co-authored-by: Tomas Grosup <tomasgrosup@microsoft.com>
Co-authored-by: Martin <29605222+Martin521@users.noreply.github.com>
Co-authored-by: Brian Rourke Boll <brianrourkeboll@users.noreply.github.com>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Jakub Majocha <1760221+majocha@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants