Don't consider panic fixes as "new failures"

[Razican]

Previously, if a panic was fixed and it was now failing, it would be considered as a "new failure". This considers it a panic fix.

[github-actions]

Test262 conformance changes:

Test result	master count	PR count	difference
Total	78,873	78,897	+24
Passed	26,667	26,666	-1
Ignored	15,604	15,628	+24
Failed	36,602	36,603	+1
Panics	14	15	+1
Conformance	33.81%	33.80%	-0.01%

New panics:

test/built-ins/Map/prototype/keys/returns-iterator.js (previously Passed)

[Razican]

I also took the opportunity to improve a bit the code and update the Test262 suite.

[github-actions]

Benchmark for fb48172

Click to view benchmark

Test	PR Benchmark	Master Benchmark	%
Arithmetic operations (Execution)	338.4±17.41ns	342.2±13.55ns	-1.11%
Arithmetic operations (Full)	252.5±13.34µs	258.5±9.70µs	-2.32%
Array access (Execution)	5.8±0.38µs	5.9±0.29µs	-1.69%
Array access (Full)	272.8±13.28µs	279.1±14.44µs	-2.26%
Array creation (Execution)	2.7±0.13ms	2.7±0.11ms	0.00%
Array creation (Full)	3.0±0.13ms	3.0±0.12ms	0.00%
Array pop (Execution)	854.0±45.53µs	840.3±42.40µs	+1.63%
Array pop (Full)	1332.8±59.65µs	1411.6±35.02µs	-5.58%
Boolean Object Access (Execution)	4.7±0.28µs	5.2±0.10µs	-9.62%
Boolean Object Access (Full)	264.4±13.93µs	279.4±7.49µs	-5.37%
Clean js (Execution)	562.8±26.78µs	591.0±16.59µs	-4.77%
Clean js (Full)	869.8±43.31µs	914.0±24.91µs	-4.84%
Clean js (Parser)	37.1±2.10µs	38.7±1.39µs	-4.13%
Create Realm	402.8±17.73ns	402.5±20.24ns	+0.07%
Dynamic Object Property Access (Execution)	4.6±0.31µs	5.0±0.18µs	-8.00%
Dynamic Object Property Access (Full)	273.6±8.93µs	277.7±11.23µs	-1.48%
Expression (Parser)	7.2±0.41µs	6.9±0.28µs	+4.35%
Fibonacci (Execution)	752.7±37.51µs	775.9±41.92µs	-2.99%
Fibonacci (Full)	1046.4±35.87µs	1087.2±34.79µs	-3.75%
For loop (Execution)	20.5±0.73µs	21.3±0.75µs	-3.76%
For loop (Full)	275.0±12.42µs	304.3±4.88µs	-9.63%
For loop (Parser)	18.8±0.71µs	19.2±0.70µs	-2.08%
Goal Symbols (Parser)	13.3±0.47µs	13.6±0.41µs	-2.21%
Hello World (Parser)	3.7±0.23µs	3.9±0.09µs	-5.13%
Long file (Parser)	717.3±44.07ns	769.0±18.50ns	-6.72%
Mini js (Execution)	483.8±30.41µs	536.7±17.20µs	-9.86%
Mini js (Full)	802.6±41.22µs	822.8±32.87µs	-2.46%
Mini js (Parser)	32.4±1.73µs	33.9±1.60µs	-4.42%
Number Object Access (Execution)	3.6±0.26µs	3.8±0.15µs	-5.26%
Number Object Access (Full)	271.0±10.90µs	261.5±12.86µs	+3.63%
Object Creation (Execution)	4.3±0.19µs	4.1±0.23µs	+4.88%
Object Creation (Full)	260.7±12.07µs	277.7±8.68µs	-6.12%
RegExp (Execution)	10.9±0.20µs	10.5±0.39µs	+3.81%
RegExp (Full)	270.5±16.68µs	278.2±13.35µs	-2.77%
RegExp Literal (Execution)	11.1±0.18µs	10.6±0.50µs	+4.72%
RegExp Literal (Full)	276.6±11.40µs	285.5±8.48µs	-3.12%
RegExp Literal Creation (Execution)	8.6±0.54µs	9.1±0.45µs	-5.49%
RegExp Literal Creation (Full)	272.7±8.58µs	273.1±12.26µs	-0.15%
Static Object Property Access (Execution)	4.3±0.28µs	4.6±0.16µs	-6.52%
Static Object Property Access (Full)	277.0±9.85µs	264.8±10.59µs	+4.61%
String Object Access (Execution)	6.7±0.37µs	6.9±0.17µs	-2.90%
String Object Access (Full)	269.7±11.79µs	290.0±5.87µs	-7.00%
String comparison (Execution)	6.2±0.37µs	6.4±0.13µs	-3.13%
String comparison (Full)	259.8±15.40µs	273.1±9.49µs	-4.87%
String concatenation (Execution)	5.0±0.29µs	5.0±0.23µs	0.00%
String concatenation (Full)	274.2±8.92µs	271.9±7.90µs	+0.85%
String copy (Execution)	3.6±0.19µs	3.9±0.10µs	-7.69%
String copy (Full)	258.2±12.63µs	272.1±7.69µs	-5.11%
Symbols (Execution)	3.1±0.18µs	3.2±0.18µs	-3.13%
Symbols (Full)	254.5±9.61µs	261.0±9.39µs	-2.49%

[github-actions]

Benchmark for 59a6743

Click to view benchmark

Test	PR Benchmark	Master Benchmark	%
Arithmetic operations (Execution)	350.0±10.55ns	345.5±12.53ns	+1.30%
Arithmetic operations (Full)	257.5±8.84µs	256.3±7.44µs	+0.47%
Array access (Execution)	6.1±0.20µs	6.2±0.16µs	-1.61%
Array access (Full)	284.9±7.38µs	282.4±7.80µs	+0.89%
Array creation (Execution)	2.8±0.08ms	2.8±0.06ms	0.00%
Array creation (Full)	3.1±0.09ms	3.2±0.09ms	-3.13%
Array pop (Execution)	902.3±20.66µs	887.7±24.09µs	+1.64%
Array pop (Full)	1381.1±48.65µs	1400.0±33.65µs	-1.35%
Boolean Object Access (Execution)	5.0±0.16µs	5.0±0.18µs	0.00%
Boolean Object Access (Full)	274.3±8.23µs	278.1±7.98µs	-1.37%
Clean js (Execution)	588.0±21.82µs	600.7±17.47µs	-2.11%
Clean js (Full)	915.4±16.48µs	925.7±27.84µs	-1.11%
Clean js (Parser)	39.4±0.89µs	39.1±1.19µs	+0.77%
Create Realm	418.2±10.56ns	423.0±8.53ns	-1.13%
Dynamic Object Property Access (Execution)	5.0±0.14µs	5.1±0.15µs	-1.96%
Dynamic Object Property Access (Full)	283.0±6.20µs	285.8±7.42µs	-0.98%
Expression (Parser)	7.2±0.13µs	7.2±0.20µs	0.00%
Fibonacci (Execution)	780.5±18.11µs	768.4±26.19µs	+1.57%
Fibonacci (Full)	1064.6±30.92µs	1096.8±33.13µs	-2.94%
For loop (Execution)	21.0±0.48µs	21.1±0.67µs	-0.47%
For loop (Full)	294.9±9.90µs	299.0±7.22µs	-1.37%
For loop (Parser)	19.2±0.49µs	18.9±0.56µs	+1.59%
Goal Symbols (Parser)	13.9±0.25µs	13.6±0.30µs	+2.21%
Hello World (Parser)	3.8±0.12µs	3.8±0.10µs	0.00%
Long file (Parser)	754.8±24.72ns	770.7±10.15ns	-2.06%
Mini js (Execution)	548.1±8.81µs	528.9±17.04µs	+3.63%
Mini js (Full)	846.8±20.37µs	860.2±24.17µs	-1.56%
Mini js (Parser)	35.1±0.56µs	34.4±0.84µs	+2.03%
Number Object Access (Execution)	4.0±0.09µs	3.9±0.12µs	+2.56%
Number Object Access (Full)	269.1±8.70µs	269.8±6.80µs	-0.26%
Object Creation (Execution)	4.3±0.17µs	4.2±0.13µs	+2.38%
Object Creation (Full)	279.0±8.70µs	281.3±3.44µs	-0.82%
RegExp (Execution)	10.8±0.32µs	10.9±0.18µs	-0.92%
RegExp (Full)	285.4±8.91µs	284.5±9.87µs	+0.32%
RegExp Literal (Execution)	10.7±0.29µs	10.7±0.38µs	0.00%
RegExp Literal (Full)	284.8±8.07µs	288.9±6.28µs	-1.42%
RegExp Literal Creation (Execution)	9.3±0.27µs	9.4±0.18µs	-1.06%
RegExp Literal Creation (Full)	278.6±8.84µs	278.2±7.73µs	+0.14%
Static Object Property Access (Execution)	4.5±0.13µs	4.5±0.11µs	0.00%
Static Object Property Access (Full)	285.9±4.35µs	273.3±8.02µs	+4.61%
String Object Access (Execution)	7.0±0.18µs	7.0±0.20µs	0.00%
String Object Access (Full)	280.8±10.76µs	281.7±6.35µs	-0.32%
String comparison (Execution)	6.4±0.17µs	6.3±0.16µs	+1.59%
String comparison (Full)	279.6±7.85µs	279.9±6.73µs	-0.11%
String concatenation (Execution)	5.1±0.12µs	5.0±0.16µs	+2.00%
String concatenation (Full)	275.8±6.89µs	266.9±7.90µs	+3.33%
String copy (Execution)	3.9±0.08µs	3.8±0.12µs	+2.63%
String copy (Full)	260.2±8.35µs	270.2±4.66µs	-3.70%
Symbols (Execution)	3.3±0.08µs	3.3±0.09µs	0.00%
Symbols (Full)	256.3±8.31µs	262.0±7.73µs	-2.18%

[Razican]

It seems that the update of the Test262 submodule is breaking this, probably due to tc39/test262@56ca8ad

[0x7D2B]

Can you make it so that there's a way to tell apart tests that were panicking but now pass and tests that were panicking but now fail?

[Razican]

Can you make it so that there's a way to tell apart tests that were panicking but now pass and tests that were panicking but now fail?

Something that can be done is to consider a Panic -> Passed a normal "fixed" test. In the list, it will show that it was panicking before.

[0x7D2B]

I think we still want to know if a test that previously panicked now fails, since that's an improvement. Though if we get to zero panics and make sure to not merge anything that adds panics, we might not see this very often.

[Razican]

I think we still want to know if a test that previously panicked now fails, since that's an improvement. Though if we get to zero panics and make sure to not merge anything that adds panics, we might not see this very often.

Yes, the idea would be that if a "Panic" goes to "Pass", it will be considered a "new passing test", but if it goes from "panic" to "Fail", then it would be considered a "panic fix".

[github-actions]

Benchmark for ec781a0

Click to view benchmark

Test	PR Benchmark	Master Benchmark	%
Arithmetic operations (Execution)	350.2±6.27ns	367.6±2.57ns	-4.73%
Arithmetic operations (Full)	271.1±3.32µs	271.8±2.01µs	-0.26%
Array access (Execution)	6.2±0.11µs	6.3±0.07µs	-1.59%
Array access (Full)	295.6±2.94µs	306.9±11.61µs	-3.68%
Array creation (Execution)	2.9±0.11ms	2.9±0.03ms	0.00%
Array creation (Full)	3.3±0.01ms	3.3±0.01ms	0.00%
Array pop (Execution)	897.1±12.89µs	911.8±9.78µs	-1.61%
Array pop (Full)	1449.5±4.85µs	1439.7±11.26µs	+0.68%
Boolean Object Access (Execution)	5.1±0.16µs	5.2±0.03µs	-1.92%
Boolean Object Access (Full)	290.9±3.10µs	287.8±1.86µs	+1.08%
Clean js (Execution)	600.7±11.81µs	608.0±6.76µs	-1.20%
Clean js (Full)	931.0±11.28µs	937.8±7.89µs	-0.73%
Clean js (Parser)	40.4±0.46µs	41.0±0.09µs	-1.46%
Create Realm	430.5±11.33ns	439.8±11.44ns	-2.11%
Dynamic Object Property Access (Execution)	5.2±0.10µs	5.1±0.09µs	+1.96%
Dynamic Object Property Access (Full)	302.5±13.57µs	288.6±3.82µs	+4.82%
Expression (Parser)	7.5±0.20µs	7.4±0.05µs	+1.35%
Fibonacci (Execution)	794.7±9.40µs	801.8±7.44µs	-0.89%
Fibonacci (Full)	1122.7±27.84µs	1125.8±10.39µs	-0.28%
For loop (Execution)	22.1±0.76µs	22.2±0.69µs	-0.45%
For loop (Full)	305.1±4.78µs	304.4±2.77µs	+0.23%
For loop (Parser)	20.3±0.86µs	20.0±0.66µs	+1.50%
Goal Symbols (Parser)	13.9±0.35µs	14.7±0.46µs	-5.44%
Hello World (Parser)	3.9±0.06µs	4.1±0.20µs	-4.88%
Long file (Parser)	773.4±10.96ns	787.0±6.47ns	-1.73%
Mini js (Execution)	547.2±6.50µs	566.6±41.52µs	-3.42%
Mini js (Full)	874.9±6.77µs	872.5±7.98µs	+0.28%
Mini js (Parser)	35.1±0.53µs	35.7±0.24µs	-1.68%
Number Object Access (Execution)	3.9±0.08µs	4.1±0.02µs	-4.88%
Number Object Access (Full)	282.7±3.72µs	297.0±7.89µs	-4.81%
Object Creation (Execution)	4.3±0.08µs	4.4±0.08µs	-2.27%
Object Creation (Full)	288.5±3.80µs	285.7±2.77µs	+0.98%
RegExp (Execution)	10.8±0.27µs	11.6±0.69µs	-6.90%
RegExp (Full)	296.5±4.97µs	300.1±1.85µs	-1.20%
RegExp Literal (Execution)	10.8±0.25µs	11.2±0.13µs	-3.57%
RegExp Literal (Full)	300.2±1.32µs	299.9±2.40µs	+0.10%
RegExp Literal Creation (Execution)	9.7±0.29µs	9.8±0.05µs	-1.02%
RegExp Literal Creation (Full)	294.3±4.13µs	300.6±11.69µs	-2.10%
Static Object Property Access (Execution)	4.6±0.09µs	4.6±0.08µs	0.00%
Static Object Property Access (Full)	287.1±2.98µs	301.7±11.59µs	-4.84%
String Object Access (Execution)	7.4±0.19µs	7.2±0.06µs	+2.78%
String Object Access (Full)	292.3±3.55µs	292.5±1.43µs	-0.07%
String comparison (Execution)	6.4±0.12µs	6.9±0.22µs	-7.25%
String comparison (Full)	293.8±2.53µs	301.0±11.31µs	-2.39%
String concatenation (Execution)	5.2±0.08µs	5.3±0.05µs	-1.89%
String concatenation (Full)	283.4±2.20µs	282.7±2.14µs	+0.25%
String copy (Execution)	3.9±0.08µs	4.0±0.03µs	-2.50%
String copy (Full)	276.5±5.25µs	278.7±1.56µs	-0.79%
Symbols (Execution)	3.4±0.05µs	3.4±0.05µs	0.00%
Symbols (Full)	272.4±9.43µs	277.2±10.17µs	-1.73%

[github-actions]

Benchmark for 9297a37

Click to view benchmark

Test	PR Benchmark	Master Benchmark	%
Arithmetic operations (Execution)	339.0±13.69ns	334.8±18.31ns	+1.25%
Arithmetic operations (Full)	283.0±15.65µs	272.4±11.00µs	+3.89%
Array access (Execution)	6.9±0.32µs	6.8±0.28µs	+1.47%
Array access (Full)	326.9±13.41µs	308.8±23.40µs	+5.86%
Array creation (Execution)	2.7±0.10ms	2.8±0.12ms	-3.57%
Array creation (Full)	3.0±0.12ms	2.9±0.16ms	+3.45%
Array pop (Execution)	895.8±142.40µs	879.6±40.21µs	+1.84%
Array pop (Full)	1355.1±61.21µs	1337.4±87.63µs	+1.32%
Boolean Object Access (Execution)	5.4±0.40µs	5.3±0.41µs	+1.89%
Boolean Object Access (Full)	304.8±13.36µs	290.9±15.68µs	+4.78%
Clean js (Execution)	604.9±28.69µs	587.4±27.86µs	+2.98%
Clean js (Full)	926.8±45.83µs	908.1±40.90µs	+2.06%
Clean js (Parser)	42.7±4.62µs	42.8±2.35µs	-0.23%
Create Realm	397.5±19.42ns	411.7±25.39ns	-3.45%
Dynamic Object Property Access (Execution)	5.6±0.25µs	5.4±0.20µs	+3.70%
Dynamic Object Property Access (Full)	301.5±14.22µs	313.2±14.00µs	-3.74%
Expression (Parser)	7.2±1.08µs	7.1±0.40µs	+1.41%
Fibonacci (Execution)	894.1±43.27µs	888.3±36.81µs	+0.65%
Fibonacci (Full)	1179.5±52.55µs	1180.4±65.57µs	-0.08%
For loop (Execution)	22.4±1.14µs	22.3±1.01µs	+0.45%
For loop (Full)	313.9±10.92µs	319.7±26.97µs	-1.81%
For loop (Parser)	21.3±5.02µs	19.7±1.10µs	+8.12%
Goal Symbols (Parser)	13.9±0.66µs	13.7±0.74µs	+1.46%
Hello World (Parser)	3.8±0.17µs	3.9±0.23µs	-2.56%
Long file (Parser)	802.7±24.21ns	737.0±39.96ns	+8.91%
Mini js (Execution)	546.2±39.98µs	518.7±19.48µs	+5.30%
Mini js (Full)	863.7±49.50µs	870.7±36.82µs	-0.80%
Mini js (Parser)	37.0±1.88µs	35.8±1.46µs	+3.35%
Number Object Access (Execution)	4.2±0.18µs	4.1±0.21µs	+2.44%
Number Object Access (Full)	297.7±14.30µs	292.6±14.14µs	+1.74%
Object Creation (Execution)	4.9±0.28µs	4.7±0.15µs	+4.26%
Object Creation (Full)	306.4±15.66µs	290.2±9.80µs	+5.58%
RegExp (Execution)	11.9±0.53µs	12.0±6.00µs	-0.83%
RegExp (Full)	310.5±11.78µs	308.8±18.66µs	+0.55%
RegExp Literal (Execution)	11.7±0.58µs	11.3±0.44µs	+3.54%
RegExp Literal (Full)	311.1±22.41µs	307.9±14.40µs	+1.04%
RegExp Literal Creation (Execution)	10.3±0.69µs	10.0±0.51µs	+3.00%
RegExp Literal Creation (Full)	304.5±14.67µs	301.6±14.37µs	+0.96%
Static Object Property Access (Execution)	5.1±0.23µs	4.9±0.18µs	+4.08%
Static Object Property Access (Full)	295.9±12.95µs	305.0±16.40µs	-2.98%
String Object Access (Execution)	7.5±0.51µs	7.5±0.56µs	0.00%
String Object Access (Full)	308.0±19.16µs	297.8±17.51µs	+3.43%
String comparison (Execution)	7.4±0.33µs	6.7±0.48µs	+10.45%
String comparison (Full)	320.7±17.07µs	298.1±16.76µs	+7.58%
String concatenation (Execution)	5.7±0.34µs	5.4±0.27µs	+5.56%
String concatenation (Full)	297.7±17.06µs	288.2±17.88µs	+3.30%
String copy (Execution)	4.3±0.21µs	4.1±0.17µs	+4.88%
String copy (Full)	291.2±11.87µs	297.8±24.50µs	-2.22%
Symbols (Execution)	3.6±0.17µs	3.6±0.15µs	0.00%
Symbols (Full)	287.0±36.68µs	276.6±10.17µs	+3.76%

[github-actions]

Benchmark for 9d1243b

Click to view benchmark

Test	PR Benchmark	Master Benchmark	%
Arithmetic operations (Execution)	359.6±3.51ns	364.7±2.66ns	-1.40%
Arithmetic operations (Full)	268.8±2.70µs	273.9±7.74µs	-1.86%
Array access (Execution)	6.5±0.08µs	6.3±0.09µs	+3.17%
Array access (Full)	296.3±3.74µs	297.6±3.61µs	-0.44%
Array creation (Execution)	2.9±0.03ms	2.9±0.03ms	0.00%
Array creation (Full)	3.3±0.04ms	3.3±0.04ms	0.00%
Array pop (Execution)	921.4±12.12µs	926.0±9.46µs	-0.50%
Array pop (Full)	1422.5±15.22µs	1426.6±21.08µs	-0.29%
Boolean Object Access (Execution)	5.3±0.03µs	5.3±0.06µs	0.00%
Boolean Object Access (Full)	285.2±4.15µs	294.9±5.25µs	-3.29%
Clean js (Execution)	614.7±7.28µs	616.4±5.62µs	-0.28%
Clean js (Full)	942.5±7.22µs	927.8±12.19µs	+1.58%
Clean js (Parser)	40.6±0.59µs	40.6±0.53µs	0.00%
Create Realm	432.8±5.25ns	430.8±7.46ns	+0.46%
Dynamic Object Property Access (Execution)	5.2±0.06µs	5.3±0.07µs	-1.89%
Dynamic Object Property Access (Full)	288.2±4.23µs	291.1±3.62µs	-1.00%
Expression (Parser)	7.3±0.06µs	7.4±0.05µs	-1.35%
Fibonacci (Execution)	806.5±7.51µs	802.0±9.50µs	+0.56%
Fibonacci (Full)	1126.9±11.46µs	1121.5±14.61µs	+0.48%
For loop (Execution)	22.1±0.29µs	21.9±0.29µs	+0.91%
For loop (Full)	304.7±4.00µs	302.4±4.28µs	+0.76%
For loop (Parser)	19.7±0.23µs	19.9±0.16µs	-1.01%
Goal Symbols (Parser)	13.9±0.23µs	13.9±0.21µs	0.00%
Hello World (Parser)	4.0±0.03µs	4.0±0.03µs	0.00%
Long file (Parser)	792.5±10.21ns	794.3±12.25ns	-0.23%
Mini js (Execution)	546.5±7.46µs	554.5±4.22µs	-1.44%
Mini js (Full)	882.1±6.86µs	876.0±6.39µs	+0.70%
Mini js (Parser)	35.5±0.50µs	35.3±0.43µs	+0.57%
Number Object Access (Execution)	4.1±0.04µs	4.0±0.07µs	+2.50%
Number Object Access (Full)	282.5±3.33µs	288.0±3.52µs	-1.91%
Object Creation (Execution)	4.5±0.05µs	4.5±0.08µs	0.00%
Object Creation (Full)	289.4±3.64µs	285.4±3.30µs	+1.40%
RegExp (Execution)	11.1±0.16µs	11.2±0.12µs	-0.89%
RegExp (Full)	299.5±3.55µs	296.2±3.96µs	+1.11%
RegExp Literal (Execution)	11.1±0.15µs	11.1±0.16µs	0.00%
RegExp Literal (Full)	300.0±3.19µs	296.4±3.94µs	+1.21%
RegExp Literal Creation (Execution)	9.7±0.12µs	9.6±0.11µs	+1.04%
RegExp Literal Creation (Full)	288.7±3.68µs	291.1±3.69µs	-0.82%
Static Object Property Access (Execution)	4.8±0.04µs	4.7±0.06µs	+2.13%
Static Object Property Access (Full)	283.0±4.47µs	288.6±3.15µs	-1.94%
String Object Access (Execution)	7.3±0.06µs	7.2±0.06µs	+1.39%
String Object Access (Full)	292.0±3.59µs	290.9±3.36µs	+0.38%
String comparison (Execution)	6.6±0.09µs	6.5±0.10µs	+1.54%
String comparison (Full)	290.2±2.52µs	291.7±3.94µs	-0.51%
String concatenation (Execution)	5.4±0.04µs	5.3±0.08µs	+1.89%
String concatenation (Full)	282.7±5.66µs	282.3±3.49µs	+0.14%
String copy (Execution)	4.0±0.05µs	4.0±0.06µs	0.00%
String copy (Full)	279.6±2.93µs	276.3±4.33µs	+1.19%
Symbols (Execution)	3.4±0.05µs	3.4±0.05µs	0.00%
Symbols (Full)	272.5±3.69µs	273.4±2.51µs	-0.33%

[Razican]

This is ready for review. The extra panic seems this panic that happens only "sometimes". I'm unable to reproduce it locally.