Setting maxTestsPerMethod is ignored

[alisevych]

Description

With new functionality:

• Generated test output size should be customizable #1235

When maxTestsPerMethod=1 is set test generation for ExceptionExamples is made with default settings.
There are 2 tests generated in a region, 3 tests generated per method.

To Reproduce

1. In ~/.utbot/settings.properties: set maxTestsPerMethod=1
2. Install latest build from main into IDEA 2022.2.3
3. Open UTBotJava project
4. Generate tests with 95% Symbolic execution for ExceptionExamples#initAnArray(int)

Expected behavior

There should be only 1 test per method generated.

Actual behavior

There are up to 3 tests per method generated, including 2 in the same region.

Visual proofs (screenshots, logs, images)

    ///region Test suites for executable org.utbot.examples.exceptions.ExceptionExamples.initAnArray

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method initAnArray(int)
    /// Actual number of generated tests (3) exceeds per-method limit (1)
    /// The limit can be configured in '{HOME_DIR}/.utbot/settings.properties' with 'maxTestsPerMethod' property

    /**
     * @utbot.classUnderTest {@link ExceptionExamples}
     * @utbot.methodUnderTest {@link ExceptionExamples#initAnArray(int)}
     */
    @Test
    @DisplayName("initAnArray: n = 0 -> return -3")
    public void testInitAnArrayWithCornerCase() {
        ExceptionExamples exceptionExamples = new ExceptionExamples();

        int actual = exceptionExamples.initAnArray(0);

        assertEquals(-3, actual);
    }
    ///endregion

    ///region FUZZER: EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS for method initAnArray(int)

    /**
     * @utbot.classUnderTest {@link ExceptionExamples}
     * @utbot.methodUnderTest {@link ExceptionExamples#initAnArray(int)}
     */
    @Test
    @DisplayName("initAnArray: n = Int.MAX_VALUE -> throw OutOfMemoryError")
    public void testInitAnArrayThrowsOOMEWithCornerCase() {
        ExceptionExamples exceptionExamples = new ExceptionExamples();

        assertThrows(OutOfMemoryError.class, () -> exceptionExamples.initAnArray(Integer.MAX_VALUE));
    }
    ///endregion

    ///endregion

    ///region Test suites for executable org.utbot.examples.exceptions.ExceptionExamples.nestedExceptions

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method nestedExceptions(int)
    /// Actual number of generated tests (2) exceeds per-method limit (1)
    /// The limit can be configured in '{HOME_DIR}/.utbot/settings.properties' with 'maxTestsPerMethod' property

    /**
     * @utbot.classUnderTest {@link ExceptionExamples}
     * @utbot.methodUnderTest {@link ExceptionExamples#nestedExceptions(int)}
     */
    @Test
    @DisplayName("nestedExceptions: i > 0 -> return 100")
    public void testNestedExceptionsReturns100() {
        ExceptionExamples exceptionExamples = new ExceptionExamples();

        int actual = exceptionExamples.nestedExceptions(1);

        assertEquals(100, actual);
    }
    ///endregion

    ///endregion

Environment

Windows 10 Pro
IntelliJ IDEA 2022.2.3

Related to #1235

[alisevych]

@Vassiliy-Kudryashov
Reproducing

Steps to reproduce

1. Set maxTestsPerMethod=1 and maxTestsPerMethod=2
2. Installed the latest main in IDEA 2022.2.3
3. Generated tests with default setting on the following code:

public class CheckStatic {
    public int connect(Socket socket) throws IOException {
        socket.connect(new InetSocketAddress("0.0.0.0", 22));
        return 0;
    }

Actual result

The following tests were generated:

    ///region Test suites for executable CheckStatic.connect

    ///region FUZZER: SECURITY for method connect(java.net.Socket)

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int, boolean)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("jkmqh", 568849899, true);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "jkmqh" "resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect1() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("hlcoj", 1525745488);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "hlcoj" "resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int, boolean)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect2() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("", 22, true);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:22" "connect,resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int, boolean)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect3() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("jkmqh", 22, false);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "jkmqh" "resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int, boolean)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect4() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("", 0, true);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:0" "connect,resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect5() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("0.0.0.0", 22);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "0.0.0.0:22" "connect,resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect6() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("noimn", 427785292);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "noimn" "resolve")] */
    }

    /**
     * @utbot.classUnderTest {@link .CheckStatic}
     * @utbot.methodUnderTest {@link .CheckStatic#connect(Socket)}
     */
    @Test
    @DisplayName("connect: socket = Socket(String, int)")
    @Disabled(value = "Disabled due to sandbox")
    public void testConnect7() throws IOException {
        CheckStatic checkStatic = new CheckStatic();
        Socket socket = new Socket("", 0);

        /* This test fails because method [CheckStatic.connect] produces [java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:0" "connect,resolve")] */
    }
    ///endregion

    ///region Errors report for connect

    public void testConnect_errors() {
        // Couldn't generate some tests. List of errors:
        // 
        // 70 occurrences of:
        // Default concrete execution failed

    }
    ///endregion

    ///endregion

[alisevych]

One more scenario:

1. Set maxTestsPerMethod=2 in settings.properties
2. Installed the latest main in IDEA 2022.2.3
3. Generated tests with default setting on the following code:
   org.utbot.examples.controlflow.Cycles.loopInsideLoop

Actual result

There are 4 tests generated in region
///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method loopInsideLoop(int)

[Vassiliy-Kudryashov]

The key was renamed to maxTestsPerMethodInRegion within 32deee9