Support call tracking within Q# code #1791

swernli · 2024-07-25T15:24:18Z

This change adds callable tracking APIs to the Diagnostics namespace that allow Q# code to register for tracking of specific callables and getting the count of times that callable was invoked. This is useful in katas and auto-grading scenarios to verify that exercise restrictions are being appropriately followed.

Fixes #1154

This change adds callable tracking APIs to the Diagnostics namespace that allow Q# code to register for tracking of specific callables and getting the count of times that callable was invoked. This is useful in katas and auto-grading scenarios to verify that exercise restrictions are being appropriately followed. Fixes #1154

github-actions · 2024-07-25T15:25:24Z

Change in memory usage detected by benchmark.

Memory Report for `b9a5957`

Test	This Branch	On Main	Difference
compile core + standard lib	17980128 bytes	17962616 bytes	17512 bytes

library/qs_source/src/std/diagnostics.qs

github-actions · 2024-07-25T15:36:38Z

Benchmark for `b9a5957`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	338.7±2.94µs	339.6±7.24µs	+0.27%
Array literal evaluation	192.7±3.61µs	187.2±3.24µs	-2.85%
Array update evaluation	417.3±4.37µs	416.9±3.58µs	-0.10%
Core + Standard library compilation	23.1±1.21ms	23.4±1.31ms	+1.30%
Deutsch-Jozsa evaluation	4.9±0.06ms	4.9±0.05ms	0.00%
Large file parity evaluation	34.3±1.77ms	34.7±0.59ms	+1.17%
Large input file compilation	13.3±0.67ms	13.5±0.61ms	+1.50%
Large input file compilation (interpreter)	51.7±3.83ms	54.9±2.67ms	+6.19%
Large nested iteration	32.6±0.37ms	33.0±1.14ms	+1.23%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1564.6±53.66µs	1587.0±88.18µs	+1.43%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.7±0.10ms	7.9±0.14ms	+2.60%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1420.4±40.01µs	1454.5±129.60µs	+2.40%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.9±1.04ms	28.2±0.84ms	+1.08%
Teleport evaluation	92.0±7.32µs	94.0±4.01µs	+2.17%

github-actions · 2024-07-25T16:48:47Z

Change in memory usage detected by benchmark.

Memory Report for `acda466`

Test	This Branch	On Main	Difference
compile core + standard lib	17980128 bytes	17962616 bytes	17512 bytes

github-actions · 2024-07-25T17:00:05Z

Benchmark for `acda466`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	335.0±2.40µs	341.8±2.80µs	+2.03%
Array literal evaluation	173.1±2.79µs	174.1±6.21µs	+0.58%
Array update evaluation	412.5±1.56µs	422.2±3.04µs	+2.35%
Core + Standard library compilation	22.9±1.29ms	21.7±0.51ms	-5.24%
Deutsch-Jozsa evaluation	4.9±0.06ms	5.0±0.06ms	+2.04%
Large file parity evaluation	34.0±0.13ms	34.7±0.55ms	+2.06%
Large input file compilation	13.1±0.28ms	13.2±0.43ms	+0.76%
Large input file compilation (interpreter)	52.0±2.13ms	51.8±1.64ms	-0.38%
Large nested iteration	32.2±0.53ms	33.2±0.18ms	+3.11%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1599.2±57.74µs	1561.7±35.87µs	-2.34%
Perform Runtime Capabilities Analysis (RCA) on large file sample	8.0±0.21ms	7.7±0.09ms	-3.75%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1455.6±94.46µs	1430.8±56.14µs	-1.70%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	28.4±0.26ms	27.6±0.25ms	-2.82%
Teleport evaluation	91.0±3.20µs	93.8±3.98µs	+3.08%

library/qs_source/src/std/diagnostics.qs

compiler/qsc_eval/src/lib.rs

library/src/tests/diagnostics.rs

compiler/qsc_eval/src/lib.rs

github-actions · 2024-08-05T05:15:06Z

Change in memory usage detected by benchmark.

Memory Report for `eb7cad7`

Test	This Branch	On Main	Difference
compile core + standard lib	17985040 bytes	17963208 bytes	21832 bytes

github-actions · 2024-08-05T05:26:23Z

Benchmark for `eb7cad7`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	339.9±3.43µs	334.6±9.60µs	-1.56%
Array literal evaluation	186.4±4.19µs	171.9±1.05µs	-7.78%
Array update evaluation	417.7±2.19µs	414.8±8.02µs	-0.69%
Core + Standard library compilation	21.7±0.61ms	22.0±0.72ms	+1.38%
Deutsch-Jozsa evaluation	5.0±0.06ms	5.1±0.08ms	+2.00%
Large file parity evaluation	34.5±0.22ms	34.6±1.94ms	+0.29%
Large input file compilation	12.9±0.25ms	13.0±0.21ms	+0.78%
Large input file compilation (interpreter)	50.0±1.68ms	52.2±1.98ms	+4.40%
Large nested iteration	33.3±1.20ms	32.7±0.31ms	-1.80%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1567.4±32.93µs	1580.6±65.34µs	+0.84%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.8±0.07ms	7.9±0.11ms	+1.28%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1429.7±44.36µs	1438.1±99.22µs	+0.59%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.7±0.19ms	27.7±0.20ms	0.00%
Teleport evaluation	93.2±3.10µs	94.9±3.82µs	+1.82%

github-actions · 2024-08-05T23:05:13Z

Change in memory usage detected by benchmark.

Memory Report for `7695dea`

Test	This Branch	On Main	Difference
compile core + standard lib	17984728 bytes	17962896 bytes	21832 bytes

github-actions · 2024-08-05T23:15:55Z

Benchmark for `7695dea`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	333.4±5.37µs	334.3±1.88µs	+0.27%
Array literal evaluation	186.4±1.06µs	190.5±2.77µs	+2.20%
Array update evaluation	417.0±5.24µs	416.5±4.10µs	-0.12%
Core + Standard library compilation	21.6±0.66ms	21.5±0.46ms	-0.46%
Deutsch-Jozsa evaluation	5.0±0.06ms	5.2±0.08ms	+4.00%
Large file parity evaluation	34.5±0.39ms	34.6±0.82ms	+0.29%
Large input file compilation	13.0±0.33ms	13.1±0.33ms	+0.77%
Large input file compilation (interpreter)	50.9±2.11ms	54.1±3.50ms	+6.29%
Large nested iteration	33.1±0.29ms	33.0±0.32ms	-0.30%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1562.5±42.47µs	1573.6±28.30µs	+0.71%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.7±0.08ms	7.9±0.30ms	+2.60%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1425.1±41.70µs	1441.1±41.28µs	+1.12%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.7±0.17ms	27.7±0.37ms	0.00%
Teleport evaluation	92.3±3.77µs	96.1±3.43µs	+4.12%

library/std/src/diagnostics.qs

library/src/tests/diagnostics.rs

github-actions · 2024-08-15T04:02:42Z

Change in memory usage detected by benchmark.

Memory Report for `dfb8b6d`

Test	This Branch	On Main	Difference
compile core + standard lib	17991976 bytes	17966344 bytes	25632 bytes

github-actions · 2024-08-15T04:13:54Z

Benchmark for `dfb8b6d`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	350.6±4.35µs	340.6±7.85µs	-2.85%
Array literal evaluation	192.2±0.83µs	190.3±1.76µs	-0.99%
Array update evaluation	429.0±2.57µs	413.0±18.23µs	-3.73%
Core + Standard library compilation	22.1±0.30ms	23.0±0.79ms	+4.07%
Deutsch-Jozsa evaluation	5.0±0.05ms	4.9±0.18ms	-2.00%
Large file parity evaluation	34.9±0.27ms	34.6±0.14ms	-0.86%
Large input file compilation	14.1±0.30ms	13.9±0.36ms	-1.42%
Large input file compilation (interpreter)	53.1±1.89ms	52.8±2.12ms	-0.56%
Large nested iteration	34.5±0.14ms	32.7±0.18ms	-5.22%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1564.5±43.28µs	1560.1±31.55µs	-0.28%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.8±0.11ms	8.0±0.31ms	+2.56%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1430.4±46.64µs	1429.0±52.36µs	-0.10%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.5±0.11ms	27.8±0.42ms	+1.09%
Teleport evaluation	93.8±3.22µs	95.3±3.55µs	+1.60%

github-actions · 2024-08-15T05:08:41Z

Change in memory usage detected by benchmark.

Memory Report for `dd8fcf4`

Test	This Branch	On Main	Difference
compile core + standard lib	17991976 bytes	17966344 bytes	25632 bytes

github-actions · 2024-08-15T05:19:31Z

Benchmark for `dd8fcf4`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	333.5±2.49µs	326.1±3.16µs	-2.22%
Array literal evaluation	194.0±3.99µs	172.5±3.13µs	-11.08%
Array update evaluation	415.8±1.91µs	408.0±1.69µs	-1.88%
Core + Standard library compilation	22.8±0.44ms	22.8±0.34ms	0.00%
Deutsch-Jozsa evaluation	5.0±0.07ms	4.9±0.06ms	-2.00%
Large file parity evaluation	34.9±0.47ms	34.5±0.65ms	-1.15%
Large input file compilation	14.2±0.40ms	14.3±0.42ms	+0.70%
Large input file compilation (interpreter)	54.5±1.70ms	54.7±2.06ms	+0.37%
Large nested iteration	33.0±0.24ms	32.6±0.18ms	-1.21%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1562.9±43.76µs	1570.4±60.42µs	+0.48%
Perform Runtime Capabilities Analysis (RCA) on large file sample	8.0±0.12ms	8.0±0.08ms	0.00%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1426.5±43.96µs	1435.4±55.53µs	+0.62%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.8±0.28ms	27.9±0.22ms	+0.36%
Teleport evaluation	93.1±3.88µs	94.2±3.73µs	+1.18%

github-actions · 2024-08-15T05:57:01Z

Change in memory usage detected by benchmark.

Memory Report for `78971ae`

Test	This Branch	On Main	Difference
compile core + standard lib	17991952 bytes	17966344 bytes	25608 bytes

github-actions · 2024-08-15T06:08:14Z

Benchmark for `78971ae`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	335.2±2.87µs	326.3±1.78µs	-2.66%
Array literal evaluation	197.7±4.63µs	171.6±0.94µs	-13.20%
Array update evaluation	416.7±2.79µs	406.4±1.56µs	-2.47%
Core + Standard library compilation	22.9±0.68ms	23.0±0.98ms	+0.44%
Deutsch-Jozsa evaluation	5.0±0.07ms	4.9±0.05ms	-2.00%
Large file parity evaluation	34.6±0.21ms	34.3±0.21ms	-0.87%
Large input file compilation	14.2±0.47ms	14.2±0.65ms	0.00%
Large input file compilation (interpreter)	54.3±2.25ms	53.5±2.21ms	-1.47%
Large nested iteration	32.9±0.19ms	33.0±0.89ms	+0.30%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1569.7±58.77µs	1567.3±74.79µs	-0.15%
Perform Runtime Capabilities Analysis (RCA) on large file sample	8.0±0.25ms	7.9±0.07ms	-1.25%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1426.1±33.51µs	1427.8±47.58µs	+0.12%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	28.0±0.31ms	27.8±0.53ms	-0.71%
Teleport evaluation	92.9±3.69µs	93.7±3.59µs	+0.86%

swernli requested review from DmitryVasilevsky, cesarzc, idavis and minestarks as code owners July 25, 2024 15:24

swernli commented Jul 25, 2024

View reviewed changes

library/qs_source/src/std/diagnostics.qs Show resolved Hide resolved

swernli added 2 commits July 25, 2024 09:32

Fix test case

81aee08

Merge remote-tracking branch 'origin/main' into swernli/issue1154

9929c4a

billti reviewed Jul 26, 2024

View reviewed changes

library/qs_source/src/std/diagnostics.qs Show resolved Hide resolved

billti reviewed Jul 26, 2024

View reviewed changes

compiler/qsc_eval/src/lib.rs Show resolved Hide resolved

billti reviewed Jul 26, 2024

View reviewed changes

library/src/tests/diagnostics.rs Outdated Show resolved Hide resolved

billti approved these changes Jul 26, 2024

View reviewed changes

cesarzc reviewed Aug 3, 2024

View reviewed changes

compiler/qsc_eval/src/lib.rs Show resolved Hide resolved

cesarzc approved these changes Aug 3, 2024

View reviewed changes

swernli requested a review from tcNickolas August 5, 2024 05:00

swernli added 2 commits August 4, 2024 22:03

Merge remote-tracking branch 'origin/main' into swernli/issue1154

f0aa41a

Add doc comments

d9a3c83

Merge remote-tracking branch 'origin/main' into swernli/issue1154

d9dd390

tcNickolas reviewed Aug 14, 2024

View reviewed changes

swernli added 2 commits August 14, 2024 14:04

Take PR suggestions

dcb84db

Updating error pattern with tests, adding examples to docs

abb97b0

Fix tests

be08d6e

Fix doc comment description

2479923

tcNickolas approved these changes Aug 15, 2024

View reviewed changes

swernli added this pull request to the merge queue Aug 15, 2024

Merged via the queue into main with commit 22c80c9 Aug 15, 2024

swernli deleted the swernli/issue1154 branch August 15, 2024 21:15

Support call tracking within Q# code #1791

Support call tracking within Q# code #1791

Uh oh!

Conversation

swernli commented Jul 25, 2024

Uh oh!

github-actions bot commented Jul 25, 2024

Memory Report for b9a5957

Uh oh!

Uh oh!

github-actions bot commented Jul 25, 2024

Benchmark for b9a5957

Uh oh!

github-actions bot commented Jul 25, 2024

Memory Report for acda466

Uh oh!

github-actions bot commented Jul 25, 2024

Benchmark for acda466

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 5, 2024

Memory Report for eb7cad7

Uh oh!

github-actions bot commented Aug 5, 2024

Benchmark for eb7cad7

Uh oh!

github-actions bot commented Aug 5, 2024

Memory Report for 7695dea

Uh oh!

github-actions bot commented Aug 5, 2024

Benchmark for 7695dea

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 15, 2024

Memory Report for dfb8b6d

Uh oh!

github-actions bot commented Aug 15, 2024

Benchmark for dfb8b6d

Uh oh!

github-actions bot commented Aug 15, 2024

Memory Report for dd8fcf4

Uh oh!

github-actions bot commented Aug 15, 2024

Benchmark for dd8fcf4

Uh oh!

github-actions bot commented Aug 15, 2024

Memory Report for 78971ae

Uh oh!

github-actions bot commented Aug 15, 2024

Benchmark for 78971ae

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Memory Report for `b9a5957`

Benchmark for `b9a5957`

Memory Report for `acda466`

Benchmark for `acda466`

Memory Report for `eb7cad7`

Benchmark for `eb7cad7`

Memory Report for `7695dea`

Benchmark for `7695dea`

Memory Report for `dfb8b6d`

Benchmark for `dfb8b6d`

Memory Report for `dd8fcf4`

Benchmark for `dd8fcf4`

Memory Report for `78971ae`

Benchmark for `78971ae`