Tests on OverflowExamples should fail when `Treat overflows as errors`

[alisevych]

Description

Tests generated by Symbolic Execution for OverflowExamples with Treat overflows as errors option on - are supposed to fail with ArithmeticException, but they are passing.

To Reproduce

1. Install one of the latest plugin build from main in IDEA
2. Run the 'UTBotJava' project in IntelliJ Idea
3. Set Overflow detection to Treat overflows as errors:

4. Use plugin to generate tests for the sample org.utbot.examples.math.OverflowExamples with Mocks off:

5. Find ///region SYMBOLIC EXECUTION: OVERFLOWS for method intOverflow(int, int) in the generated test

Expected behavior

Test is supposed to fail with ArithmeticException

Actual behavior

The test has comment that it produces AithmeticException, but it actually pass.

Visual proofs (screenshots, logs, images)

    ///region SYMBOLIC EXECUTION: OVERFLOWS for method intSubOverflow(int, int)

    /**
     * @utbot.classUnderTest {@link OverflowExamples}
     * @utbot.methodUnderTest {@link OverflowExamples#intSubOverflow(int, int)}
     * @utbot.throwsException {@link ArithmeticException} in: return x - y;
     */
    @Test
    @DisplayName("intSubOverflow: return x - y : True -> ThrowArithmeticException")
    public void testIntSubOverflow_ThrowArithmeticException() {
        OverflowExamples overflowExamples = new OverflowExamples();

        /* This test fails because method [org.utbot.examples.math.OverflowExamples.intSubOverflow] produces [java.lang.ArithmeticException: int  -  overflow] */
        overflowExamples.intSubOverflow(1426620925, -1610612736);
    }
    ///endregion

Environment

Windows 10 Pro
IntelliJ IDEA 2022.2.3

Additional context

There are 15 tests generated for OverflowExamples with the same behavior.

[CaelmBleidd]

Looks like the reason is an unexpected combination of settings: tests with exceptions = fail and treat overflows as errors. If we have an overflow during symbolic analysis, it produces a fake arithmetic exception, that will not be raised during concrete execution. Because of that, in the generated tests we have one that expects the exception, but don't get it, therefore, it fails. This how treat as errors works.

When we say that tests with exceptions should fail, it means we remove these throws annotations and everything like this. Seems like it removes it even for such arithmetic exceptions, which makes this test green.

I'm not sure, who should be assigned to this task, but seems like it should be someone from code generation. @EgorkaKulikov please, help me with assigning this task to the right person