Enhance fuzzing for arrays #738 #755

volivan239 · 2022-08-22T11:09:09Z

Description

Now, ArrayModelProvider generates more interesting tests:

each element of array is generated recursively by fuzzing synthetic method.
arrays with "interesting" small sizes are added ("interesting" means that this number is obtained from source code)

Fixes #738

Type of Change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Manual Scenario

Generate tests for methods from ArrayExample class (using only fuzzer):

class Point {
    int x, y;

    public Point(int x, int y) {
        this.x = x;
        this.y = y;
    }

    boolean equals(Point other) {
        return (other.x == this.x && other.y == this.y);
    }
}

public class ArrayExample {
    boolean checkAllSame(Integer[] a) {    // Fuzzer should find parameters giving true as well as parameters giving false
        Set<Integer> s = new HashSet<>(Arrays.asList(a));
        return (s.size() <= 1);
    }

    public boolean checkAllSamePoints(Point[] a) { // Also works for classes
        if (a.length == 4) {
            return false; // Test with array of size 4 should be generated by fuzzer
        }
        for (int i = 1; i < a.length; i++) {
            if (!a[i].equals(a[i - 1]))
                return false;
        }
        return true;
    }

    public boolean checkRowsWithAllSame2D(int[][] a, int y) {
        int cntSame = 0;
        for (int i = 0; i < a.length; i++) {
            boolean same = true;
            for (int j = 1; j < a[i].length; j++) {
                if (a[i][j] != a[i][j - 1]) {
                    same = false;
                    break;
                }
            }
            if (same)
                cntSame++;
        }
        return (cntSame == y && y > 0 && y < a.length);
    }
}

For each of them, meaningful tests are generated that cover both true/false outputs.

Checklist (remove irrelevant options):

This is the author self-check list

The change followed the style guidelines of the UTBot project
Self-review of the code is passed
The change contains enough commentaries, particularly in hard-to-understand areas
New documentation is provided or existed one is altered
No new warnings
New tests have been added
All tests pass locally with my changes

Markoutte · 2022-08-23T12:02:47Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/ObjectModelProvider.kt

-            }
+    private val limit: Int =
+        when(recursion) {
+            1 -> Int.MAX_VALUE


The logic looks different: before when recursion is 1, 2, 3 and so on then limit is 1. Now it is harder to understand what's happening. The idea behind this was that the limit is articulated as: when we find constructors that accepts another objects then we construct them, but use no more than one available constructor. If recursion is 0, than we pass null to every object parameter in constructor. But usually, we look for constructors with primitive values.

Markoutte · 2022-08-23T12:07:43Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/RecursiveModelProvider.kt

+    protected val recursion: Int
+): ModelProvider {
+    // TODO: currently it is var due to tests, maybe we can make it private val?
+    var modelProviderForRecursiveCalls: ModelProvider =


Looks like we have lost an ability to change model providers for ObjectModelProvider, because before the changed model provider is passed into recursive call. Now, it uses only default providers.

Markoutte · 2022-08-23T12:18:18Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/ArrayModelProvider.kt

                })
            }
    }
+
+    private fun generateArrayLengths(description: FuzzedMethodDescription): Set<Int> {


Looks like this can be simplified. It is too much work to generate some values in interval from 0 up to 10

Markoutte · 2022-08-23T12:21:11Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/ObjectModelProvider.kt


-    var modelProvider: ModelProvider = objectModelProviders(idGenerator)
+    // TODO: can we make it private val (maybe depending on recursion)?


It is manual option and a part of public API

Markoutte · 2022-08-23T12:22:11Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/RecursiveModelProvider.kt

+    protected val idGenerator: IdentityPreservingIdGenerator<Int>,
+    protected val recursion: Int
+): ModelProvider {
+    // TODO: currently it is var due to tests, maybe we can make it private val?


It was a part of public API to change a set of model providers. Currently, it looks broken

Markoutte

Please, look at comments

Markoutte · 2022-08-29T05:55:40Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/RecursiveModelProvider.kt

+
+data class ModelConstructor(
+    val neededTypes: List<ClassId>,
+    val createModel: (List<FuzzedValue>) -> FuzzedValue


You can use notation like this:

val createModel: (inputs: List<FuzzedValue>) -> FuzzedValue

to clarify name for parameter. Also, doc will be very useful here

Added name for parameters. Also, added docs here and in other places where needed

Markoutte · 2022-08-29T06:38:46Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/RecursiveModelProvider.kt

+            modelProviderForRecursiveCalls.map {
+                if (it is RecursiveModelProvider)
+                    it.copy(idGenerator, recursionDepthLeft - 1).apply {
+                        modelProviderForRecursiveCalls = this@RecursiveModelProvider.modelProviderForRecursiveCalls


Why don't we copy all these values in the copy method? It looks better as we want to get a copy of this provider. Here we can leave values that should be changed like totalLimit (but in copy it should be copied, I think)

Renamed to createNewInstance for more clarity

Markoutte · 2022-08-29T06:44:10Z

utbot-fuzzers/src/test/kotlin/org/utbot/framework/plugin/api/ModelProviderTest.kt

@@ -412,6 +413,7 @@ class ModelProviderTest {
        }
    }

+    @Disabled


Test checks that Outer class is created using only one constructor from the inner and it is the simplest one. I think we should keep this behavior for ObjectModelProvider

Returned this behavior back

Markoutte · 2022-08-29T06:44:25Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/providers/RecursiveModelProvider.kt

+    val createModel: (List<FuzzedValue>) -> FuzzedValue
+)
+
+abstract class RecursiveModelProvider(


Please, add docs

Markoutte · 2022-08-29T07:27:29Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/Fuzzer.kt

+ */
+fun defaultModelProviders(idGenerator: IdentityPreservingIdGenerator<Int>, recursionDepth: Int = 1): ModelProvider =
+    if (recursionDepth >= 0)
+        nonRecursiveProviders(idGenerator).with(recursiveModelProviders(idGenerator, recursionDepth))


Note, that the other method called 'defaultModelProviders' without parameter with recursion has another set of model providers. Therefore. I'd recommend to change name for this method to avoid ambiguous calls.

This method should have replaced the method without recursionDepth parameter with no changes in behavior when this parameter is not specified. Seems like I accidently got duplication after rebase. Deleted method without parameter now. Could you please check that it didn't change the behavior?

It changes because PrimitivesModelProvider isn't used anymore. By default I want to keep an ability to set different providers by default, because I usually want to use shallower value set for it. Right now this change is only about PrimitivesModelProvider, but in the future I'd want to add some providers that are used for generating flat parameters and aren't used for recursive.

Markoutte

Please, see comments

Markoutte

Looks good but let's discuss changes about defaultModelProviders

* Add flattening to Combined * Change default value for RecursiveModelProvider.branchingLimit

volivan239 marked this pull request as ready for review August 22, 2022 13:09

volivan239 requested a review from Markoutte August 22, 2022 13:09

volivan239 force-pushed the volivan239/enhance_fuzzing_for_arrays branch 2 times, most recently from 16ea1b7 to 7ab9dc6 Compare August 23, 2022 09:52

Markoutte reviewed Aug 23, 2022

View reviewed changes

Markoutte requested changes Aug 23, 2022

View reviewed changes

volivan239 force-pushed the volivan239/enhance_fuzzing_for_arrays branch from 7ab9dc6 to f218d6b Compare August 26, 2022 13:34

Markoutte reviewed Aug 29, 2022

View reviewed changes

Markoutte requested changes Aug 29, 2022

View reviewed changes

volivan239 requested a review from Markoutte August 29, 2022 16:11

volivan239 added 4 commits August 29, 2022 19:54

Sketch for ArrayModelProvider

0dd9ca9

Add RecursiveModelProvider abstraction

1a730a9

Moved common logic from ArrayMP and ObjectMP to RecursiveMP

570809e

Minor changes & documentation added

36c749e

volivan239 force-pushed the volivan239/enhance_fuzzing_for_arrays branch from f4405fa to 36c749e Compare August 29, 2022 17:06

Markoutte requested changes Aug 30, 2022

View reviewed changes

Return behavior of defaultModelProviders and minor fixes:

d6d09f2

* Add flattening to Combined * Change default value for RecursiveModelProvider.branchingLimit

Markoutte approved these changes Aug 30, 2022

View reviewed changes

volivan239 enabled auto-merge (squash) August 30, 2022 11:41

volivan239 merged commit 9431dd8 into main Aug 30, 2022

volivan239 deleted the volivan239/enhance_fuzzing_for_arrays branch August 30, 2022 12:01

Markoutte mentioned this pull request Sep 30, 2022

Improve fuzzer recursive model providers #1039

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance fuzzing for arrays #738 #755

Enhance fuzzing for arrays #738 #755

volivan239 commented Aug 22, 2022 •

edited

Loading

Markoutte Aug 23, 2022

Markoutte Aug 23, 2022

Markoutte Aug 23, 2022

Markoutte Aug 23, 2022

Markoutte Aug 23, 2022

Markoutte left a comment

Markoutte Aug 29, 2022

volivan239 Aug 29, 2022

Markoutte Aug 29, 2022

volivan239 Aug 29, 2022

Markoutte Aug 29, 2022

volivan239 Aug 29, 2022

Markoutte Aug 29, 2022

Markoutte Aug 29, 2022

volivan239 Aug 29, 2022

Markoutte Aug 30, 2022

Markoutte left a comment

Markoutte left a comment


		var modelProvider: ModelProvider = objectModelProviders(idGenerator)
		// TODO: can we make it private val (maybe depending on recursion)?

Enhance fuzzing for arrays #738 #755

Enhance fuzzing for arrays #738 #755

Conversation

volivan239 commented Aug 22, 2022 • edited Loading

Description

Type of Change

How Has This Been Tested?

Manual Scenario

Checklist (remove irrelevant options):

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Markoutte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Markoutte left a comment

Choose a reason for hiding this comment

Markoutte left a comment

Choose a reason for hiding this comment

volivan239 commented Aug 22, 2022 •

edited

Loading