Skip to content

Commit 361d14f

Browse files
robbytrobbyt
authored
harmonize test structure, improve coverage (#21)
* add and adjust tests * remove extra t.Parallel() calls * consolidate response_test code * use the mocks package for polyscript_test * fix up evaluator_test * more test cleanup * update tests for extism/compiler * more test cleanup * clean up starlark converter tests * remove extra t.Parallel() calls * readd t.Parallel in some places * minor test cleanup * harmonize several tests harmonize several tests * add testing guidelines for machines * add benchmark results
1 parent 8f35742 commit 361d14f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+6285
-4761
lines changed

‎benchmarks/results/benchmark_2025-04-10_14-26-23.json

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{"Time":"2025-04-10T14:26:34.196527-04:00","Action":"start","Package":"github.com/robbyt/go-polyscript/engine"}
2+
{"Time":"2025-04-10T14:26:34.212069-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Output":"goos: darwin\n"}
3+
{"Time":"2025-04-10T14:26:34.212168-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Output":"goarch: amd64\n"}
4+
{"Time":"2025-04-10T14:26:34.212178-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Output":"pkg: github.com/robbyt/go-polyscript/engine\n"}
5+
{"Time":"2025-04-10T14:26:34.212189-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Output":"cpu: Intel(R) Xeon(R) W-3275M CPU @ 2.50GHz\n"}
6+
{"Time":"2025-04-10T14:26:34.2122-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns"}
7+
{"Time":"2025-04-10T14:26:34.21221-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns","Output":"=== RUN BenchmarkEvaluationPatterns\n"}
8+
{"Time":"2025-04-10T14:26:34.212225-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns","Output":"BenchmarkEvaluationPatterns\n"}
9+
{"Time":"2025-04-10T14:26:34.212525-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/SingleExecution"}
10+
{"Time":"2025-04-10T14:26:34.212555-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/SingleExecution","Output":"=== RUN BenchmarkEvaluationPatterns/SingleExecution\n"}
11+
{"Time":"2025-04-10T14:26:34.212572-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/SingleExecution","Output":"BenchmarkEvaluationPatterns/SingleExecution\n"}
12+
{"Time":"2025-04-10T14:26:35.417933-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/SingleExecution","Output":"BenchmarkEvaluationPatterns/SingleExecution-56 \t 4477\t 262703 ns/op\t 460987 B/op\t 1173 allocs/op\n"}
13+
{"Time":"2025-04-10T14:26:35.418013-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/CompileOnceRunMany"}
14+
{"Time":"2025-04-10T14:26:35.41803-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/CompileOnceRunMany","Output":"=== RUN BenchmarkEvaluationPatterns/CompileOnceRunMany\n"}
15+
{"Time":"2025-04-10T14:26:35.418048-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/CompileOnceRunMany","Output":"BenchmarkEvaluationPatterns/CompileOnceRunMany\n"}
16+
{"Time":"2025-04-10T14:26:36.661812-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkEvaluationPatterns/CompileOnceRunMany","Output":"BenchmarkEvaluationPatterns/CompileOnceRunMany-56 \t 7039\t 173882 ns/op\t 373306 B/op\t 436 allocs/op\n"}
17+
{"Time":"2025-04-10T14:26:36.673916-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders"}
18+
{"Time":"2025-04-10T14:26:36.673943-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders","Output":"=== RUN BenchmarkDataProviders\n"}
19+
{"Time":"2025-04-10T14:26:36.673957-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders","Output":"BenchmarkDataProviders\n"}
20+
{"Time":"2025-04-10T14:26:36.673967-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/StaticProvider"}
21+
{"Time":"2025-04-10T14:26:36.673987-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/StaticProvider","Output":"=== RUN BenchmarkDataProviders/StaticProvider\n"}
22+
{"Time":"2025-04-10T14:26:36.674002-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/StaticProvider","Output":"BenchmarkDataProviders/StaticProvider\n"}
23+
{"Time":"2025-04-10T14:26:37.930088-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/StaticProvider","Output":"BenchmarkDataProviders/StaticProvider-56 \t 6999\t 178169 ns/op\t 373314 B/op\t 436 allocs/op\n"}
24+
{"Time":"2025-04-10T14:26:37.93016-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/ContextProvider"}
25+
{"Time":"2025-04-10T14:26:37.930176-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/ContextProvider","Output":"=== RUN BenchmarkDataProviders/ContextProvider\n"}
26+
{"Time":"2025-04-10T14:26:37.930183-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/ContextProvider","Output":"BenchmarkDataProviders/ContextProvider\n"}
27+
{"Time":"2025-04-10T14:26:39.124363-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/ContextProvider","Output":"BenchmarkDataProviders/ContextProvider-56 \t 6654\t 176240 ns/op\t 372992 B/op\t 435 allocs/op\n"}
28+
{"Time":"2025-04-10T14:26:39.124432-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/CompositeProvider"}
29+
{"Time":"2025-04-10T14:26:39.124441-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/CompositeProvider","Output":"=== RUN BenchmarkDataProviders/CompositeProvider\n"}
30+
{"Time":"2025-04-10T14:26:39.12445-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/CompositeProvider","Output":"BenchmarkDataProviders/CompositeProvider\n"}
31+
{"Time":"2025-04-10T14:26:40.470563-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkDataProviders/CompositeProvider","Output":"BenchmarkDataProviders/CompositeProvider-56 \t 7592\t 174850 ns/op\t 374157 B/op\t 445 allocs/op\n"}
32+
{"Time":"2025-04-10T14:26:40.470657-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison"}
33+
{"Time":"2025-04-10T14:26:40.470669-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison","Output":"=== RUN BenchmarkVMComparison\n"}
34+
{"Time":"2025-04-10T14:26:40.470676-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison","Output":"BenchmarkVMComparison\n"}
35+
{"Time":"2025-04-10T14:26:40.471566-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/RisorVM"}
36+
{"Time":"2025-04-10T14:26:40.471582-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/RisorVM","Output":"=== RUN BenchmarkVMComparison/RisorVM\n"}
37+
{"Time":"2025-04-10T14:26:40.471589-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/RisorVM","Output":"BenchmarkVMComparison/RisorVM\n"}
38+
{"Time":"2025-04-10T14:26:41.745863-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/RisorVM","Output":"BenchmarkVMComparison/RisorVM-56 \t 7122\t 176101 ns/op\t 373331 B/op\t 436 allocs/op\n"}
39+
{"Time":"2025-04-10T14:26:41.74593-04:00","Action":"run","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/StarlarkVM"}
40+
{"Time":"2025-04-10T14:26:41.745944-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/StarlarkVM","Output":"=== RUN BenchmarkVMComparison/StarlarkVM\n"}
41+
{"Time":"2025-04-10T14:26:41.74596-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/StarlarkVM","Output":"BenchmarkVMComparison/StarlarkVM\n"}
42+
{"Time":"2025-04-10T14:26:43.21085-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Test":"BenchmarkVMComparison/StarlarkVM","Output":"BenchmarkVMComparison/StarlarkVM-56 \t 114759\t 11738 ns/op\t 7049 B/op\t 65 allocs/op\n"}
43+
{"Time":"2025-04-10T14:26:43.210933-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Output":"PASS\n"}
44+
{"Time":"2025-04-10T14:26:43.238187-04:00","Action":"output","Package":"github.com/robbyt/go-polyscript/engine","Output":"ok \tgithub.com/robbyt/go-polyscript/engine\t9.042s\n"}
45+
{"Time":"2025-04-10T14:26:43.238259-04:00","Action":"pass","Package":"github.com/robbyt/go-polyscript/engine","Elapsed":9.042}

‎benchmarks/results/benchmark_2025-04-10_14-26-23.txt

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
goos: darwin
2+
goarch: amd64
3+
pkg: github.com/robbyt/go-polyscript/engine
4+
cpu: Intel(R) Xeon(R) W-3275M CPU @ 2.50GHz
5+
BenchmarkEvaluationPatterns/SingleExecution-56 4154 285805 ns/op 461073 B/op 1173 allocs/op
6+
BenchmarkEvaluationPatterns/CompileOnceRunMany-56 6836 177584 ns/op 373308 B/op 436 allocs/op
7+
BenchmarkDataProviders/StaticProvider-56 6912 179514 ns/op 373329 B/op 436 allocs/op
8+
BenchmarkDataProviders/ContextProvider-56 7078 180442 ns/op 372985 B/op 435 allocs/op
9+
BenchmarkDataProviders/CompositeProvider-56 6172 178387 ns/op 374177 B/op 445 allocs/op
10+
BenchmarkVMComparison/RisorVM-56 6769 175350 ns/op 373351 B/op 436 allocs/op
11+
BenchmarkVMComparison/StarlarkVM-56 121674 11599 ns/op 7046 B/op 65 allocs/op
12+
PASS
13+
ok github.com/robbyt/go-polyscript/engine 8.915s

‎benchmarks/results/comparison.txt

+24-24
Original file line numberDiff line numberDiff line change
@@ -4,39 +4,39 @@ pkg: github.com/robbyt/go-polyscript/engine
44
cpu: Intel(R) Xeon(R) W-3275M CPU @ 2.50GHz
55
│ previous │ current │
66
│ sec/op │ sec/op vs base │
7-
EvaluationPatterns/SingleExecution-56 306.8µ ± ∞ ¹ 315.2µ ± ∞ ¹ ~ (p=1.000 n=1) ²
8-
EvaluationPatterns/CompileOnceRunMany-56 183.6µ ± ∞ ¹ 189.5µ ± ∞ ¹ ~ (p=1.000 n=1) ²
9-
DataProviders/StaticProvider-56 185.8µ ± ∞ ¹ 201.0µ ± ∞ ¹ ~ (p=1.000 n=1) ²
10-
DataProviders/ContextProvider-56 186.4µ ± ∞ ¹ 193.0µ ± ∞ ¹ ~ (p=1.000 n=1) ²
11-
DataProviders/CompositeProvider-56 185.6µ ± ∞ ¹ 191.7µ ± ∞ ¹ ~ (p=1.000 n=1) ²
12-
VMComparison/RisorVM-56 183.8µ ± ∞ ¹ 187.7µ ± ∞ ¹ ~ (p=1.000 n=1) ²
13-
VMComparison/StarlarkVM-56 11.72µ ± ∞ ¹ 12.52µ ± ∞ ¹ ~ (p=1.000 n=1) ²
14-
geomean 134.1µ 139.8µ +4.26%
7+
EvaluationPatterns/SingleExecution-56 315.2µ ± ∞ ¹ 285.8µ ± ∞ ¹ ~ (p=1.000 n=1) ²
8+
EvaluationPatterns/CompileOnceRunMany-56 189.5µ ± ∞ ¹ 177.6µ ± ∞ ¹ ~ (p=1.000 n=1) ²
9+
DataProviders/StaticProvider-56 201.0µ ± ∞ ¹ 179.5µ ± ∞ ¹ ~ (p=1.000 n=1) ²
10+
DataProviders/ContextProvider-56 193.0µ ± ∞ ¹ 180.4µ ± ∞ ¹ ~ (p=1.000 n=1) ²
11+
DataProviders/CompositeProvider-56 191.7µ ± ∞ ¹ 178.4µ ± ∞ ¹ ~ (p=1.000 n=1) ²
12+
VMComparison/RisorVM-56 187.7µ ± ∞ ¹ 175.4µ ± ∞ ¹ ~ (p=1.000 n=1) ²
13+
VMComparison/StarlarkVM-56 12.52µ ± ∞ ¹ 11.60µ ± ∞ ¹ ~ (p=1.000 n=1) ²
14+
geomean 139.8µ 129.1µ -7.69%
1515
¹ need >= 6 samples for confidence interval at level 0.95
1616
² need >= 4 samples to detect a difference at alpha level 0.05
1717

1818
│ previous │ current │
1919
│ B/op │ B/op vs base │
20-
EvaluationPatterns/SingleExecution-56 450.0Ki ± ∞ ¹ 450.1Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
21-
EvaluationPatterns/CompileOnceRunMany-56 364.3Ki ± ∞ ¹ 364.4Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
22-
DataProviders/StaticProvider-56 364.4Ki ± ∞ ¹ 364.4Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
23-
DataProviders/ContextProvider-56 364.1Ki ± ∞ ¹ 364.1Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
24-
DataProviders/CompositeProvider-56 364.8Ki ± ∞ ¹ 364.9Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
25-
VMComparison/RisorVM-56 364.4Ki ± ∞ ¹ 364.5Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
26-
VMComparison/StarlarkVM-56 6.875Ki ± ∞ ¹ 6.666Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
27-
geomean 213.0Ki 212.1Ki -0.43%
20+
EvaluationPatterns/SingleExecution-56 450.1Ki ± ∞ ¹ 450.3Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
21+
EvaluationPatterns/CompileOnceRunMany-56 364.4Ki ± ∞ ¹ 364.6Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
22+
DataProviders/StaticProvider-56 364.4Ki ± ∞ ¹ 364.6Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
23+
DataProviders/ContextProvider-56 364.1Ki ± ∞ ¹ 364.2Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
24+
DataProviders/CompositeProvider-56 364.9Ki ± ∞ ¹ 365.4Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
25+
VMComparison/RisorVM-56 364.5Ki ± ∞ ¹ 364.6Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
26+
VMComparison/StarlarkVM-56 6.666Ki ± ∞ ¹ 6.881Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
27+
geomean 212.1Ki 213.1Ki +0.50%
2828
¹ need >= 6 samples for confidence interval at level 0.95
2929
² need >= 4 samples to detect a difference at alpha level 0.05
3030

3131
│ previous │ current │
3232
│ allocs/op │ allocs/op vs base │
33-
EvaluationPatterns/SingleExecution-56 1.166k ± ∞ ¹ 1.168k ± ∞ ¹ ~ (p=1.000 n=1) ²
34-
EvaluationPatterns/CompileOnceRunMany-56 432.0 ± ∞ ¹ 434.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
35-
DataProviders/StaticProvider-56 432.0 ± ∞ ¹ 434.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
36-
DataProviders/ContextProvider-56 431.0 ± ∞ ¹ 433.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
37-
DataProviders/CompositeProvider-56 438.0 ± ∞ ¹ 440.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
38-
VMComparison/RisorVM-56 432.0 ± ∞ ¹ 434.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
39-
VMComparison/StarlarkVM-56 65.00 ± ∞ ¹ 60.00 ± ∞ ¹ ~ (p=1.000 n=1) ²
40-
geomean 380.4 377.4 -0.79%
33+
EvaluationPatterns/SingleExecution-56 1.168k ± ∞ ¹ 1.173k ± ∞ ¹ ~ (p=1.000 n=1) ²
34+
EvaluationPatterns/CompileOnceRunMany-56 434.0 ± ∞ ¹ 436.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
35+
DataProviders/StaticProvider-56 434.0 ± ∞ ¹ 436.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
36+
DataProviders/ContextProvider-56 433.0 ± ∞ ¹ 435.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
37+
DataProviders/CompositeProvider-56 440.0 ± ∞ ¹ 445.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
38+
VMComparison/RisorVM-56 434.0 ± ∞ ¹ 436.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
39+
VMComparison/StarlarkVM-56 60.00 ± ∞ ¹ 65.00 ± ∞ ¹ ~ (p=1.000 n=1) ²
40+
geomean 377.4 383.6 +1.64%
4141
¹ need >= 6 samples for confidence interval at level 0.95
4242
² need >= 4 samples to detect a difference at alpha level 0.05

‎benchmarks/results/latest.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
benchmark_2025-03-27_19-39-40.txt
1+
benchmark_2025-04-10_14-26-23.txt

0 commit comments

Comments
 (0)