New fuzzing platform #1457

Markoutte · 2022-12-06T05:45:27Z

Description

Adds new fuzzing platform that simplifies developing fuzzing for new languages. Old implementation is still in codebase but will be deprecated and removed soon. Only base Java implementation migrated to the new fuzzing platform at the moment.

Please, see Fuzzing Platform Design doc for understanding of basic concepts. In nutshell now any work with fuzzing requires some 'seeds' or 'tasks' which can be 4 types:

Simple seed is just a value with optional mutator
Known [structure] seed is something that has common structure for different types, e.g. integers (byte, short, int, long) which can be represented as a bit vector. This type can be used instead of 1 to faster developing of defaults such as integers, floats, strings, etc.
Recursive seed can represent some recursive structures that require to fuzz some parameters before constructing itself.
Collection seed represents collections such as arrays, lists, sets, maps and others.

There are some demos that clarify these concepts.

This platform is fully implemented for Java and plugin can be tested with these samples.

Fixes #1448

Also note, that after short discussion this implementation doesn't use mocks because it looks wrong concepts in terms of fuzzing. Fuzzing doesn't analyze the code therefore it cannot supply correct mock calls for tests. Therefore #747 is discontinued starting this PR.

Type of Change

Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Regression and integration tests

Test samples for manual testing were added. Integration test for fuzzing is being under developing.

Automated Testing

All tests from the utbot-fuzzer and utbot-fuzzing modules must pass.

Manual Scenario

Samples for manual scenario.

Checklist:

The change followed the style guidelines of the UTBot project
Self-review of the code is passed
The change contains enough commentaries, particularly in hard-to-understand areas
New documentation is provided or existed one is altered
No new warnings
New tests have been added
All tests pass locally with my changes

docs/Fuzzing Platform.md

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt

denis-fokin · 2022-12-07T14:02:43Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt

+        )
+    }
+
+    private fun shouldPass(type: ClassId): Boolean {


What "should pass" means in this context?

Changed to isIgnored(type: ClassId)

denis-fokin · 2022-12-07T14:03:58Z

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt

+    val getter: Method?
+)
+
+internal fun findSuitableFields(classId: ClassId, packageName: String?): List<FieldDescription>  {


I would add a piece of documentation here. Suitable for what?

Changed to findAccessibleModifableFields(...)

denis-fokin · 2022-12-08T08:18:25Z

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Api.kt

+     * @param description contains user-defined information about current run. Can be used as a state of the run.
+     * @param values current values to run.
+     */
+    suspend fun run(description: DESCRIPTION, values: List<RESULT>): FEEDBACK


Please, find a better naming for the method. 'Run' makes me think that the method is supposed to be called only ones. I would suggest something like: 'hanlde', 'valueUdated' or something similar

SBOne-Kenobi

Amazing! LGTM :)

SBOne-Kenobi · 2022-12-08T16:01:52Z

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Api.kt

+        builder = PassRoutine("Main Routine"),
+        state = State(1, typeCache),
+    )
+    val dynamicallyGenerated = mutableListOf<Node<T, R>>()


As I understand you use it as a queue, can you replace it with any instance of queue such as LinkedList or Deque.

…case

volivan239

Looks monumental!

volivan239 · 2022-12-09T15:04:48Z

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/demo/AbcFuzzing.kt

+fun String.findMaxSubstring(s: String) : Int {
+    if (s.isEmpty()) return -1
+    for (i in s.indices) {
+        if (s[i] != this[i]) return i - 1
+    }
+    return s.length
+}


I think the name is misleading, because the function looks like a strange variation of CharSequence.commonPrefixWith()

Also, it may throw StringIndexOutOfBoundsException if this is a strict prefix of s. If this is intentional, please, add a comment describing how fuzzing handle exceptions in this example.

volivan239 · 2022-12-12T04:24:37Z

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Configuration.kt

+    /**
+     * Probability of creating shifted array values instead of generating new values for modification.
+     */
+    var probCollectionMutationInsteadCreateNew: Int = 50,


Here and below: imo, if probability is not in [0, 1] it is worth mentioning that it is in percents

volivan239 · 2022-12-12T04:54:55Z

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/utils/Functions.kt

+fun main() {
+    val endian = Endian.BE
+    println(255.toUByte().toBinaryString(endian))
+    println(2.toBinaryString(endian))
+    println(BitVectorValue.fromInt(2).toBinaryString(endian))
+    print(8.75f.toBinaryString(endian))
+    print(8.75.toBinaryString(endian))
+}


I think this should be either moved to tests or deleted. Same for IEEE754Value.kt

volivan239 · 2022-12-12T05:29:51Z

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt

-            TrieBasedFuzzerStatistics(coveredInstructionValues), methodUnderTestDescription, *defaultModelMutators().toTypedArray()
-        )
-        fuzzedValues.forEach { values ->
+        val names = graph.body.method.tags.filterIsInstance<ParamNamesTag>().firstOrNull()?.names ?: emptyList()


This names initialization looks weird. Can't we just use methodUnderTest.parameters.map { it.name } here? If not, please, add comment reasoning such an initialization

Markoutte requested review from denis-fokin, SBOne-Kenobi, tamarinvs19, Vassiliy-Kudryashov, volivan239 and amandelpie December 6, 2022 05:47

denis-fokin reviewed Dec 6, 2022

View reviewed changes

docs/Fuzzing Platform.md Outdated Show resolved Hide resolved

Markoutte requested a review from denis-fokin December 7, 2022 08:13

amandelpie approved these changes Dec 7, 2022

View reviewed changes

denis-fokin reviewed Dec 7, 2022

View reviewed changes

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Outdated Show resolved Hide resolved

denis-fokin reviewed Dec 7, 2022

View reviewed changes

Markoutte force-pushed the pelevin/fuzzing_platform branch from 6e39690 to 97d8336 Compare December 8, 2022 05:12

Markoutte requested a review from denis-fokin December 8, 2022 05:20

denis-fokin reviewed Dec 8, 2022

View reviewed changes

Markoutte requested a review from denis-fokin December 8, 2022 12:00

SBOne-Kenobi approved these changes Dec 8, 2022

View reviewed changes

Markoutte added 8 commits December 12, 2022 07:19

New fuzzing platform

a809321

Fixes for summarization

fddb530

Fix fuzzing when collection with generics is this object

5423727

Fix for summarization when list of parameters was the same for every …

6f9fa08

…case

Changes to fuzzer doc

cc89dc5

Clarify some ObjectValueProvider entities

3897c5e

Fuzzer can pass one object into several arguments of same type

47a4c3c

Fix double values mutations

aa239b8

volivan239 approved these changes Dec 12, 2022

View reviewed changes

Fix summarization display name

f759f20

Markoutte force-pushed the pelevin/fuzzing_platform branch from b70b8af to f759f20 Compare December 12, 2022 05:40

Markoutte merged commit 3542afb into main Dec 12, 2022

Markoutte deleted the pelevin/fuzzing_platform branch December 12, 2022 06:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New fuzzing platform #1457

New fuzzing platform #1457

Markoutte commented Dec 6, 2022 •

edited

Loading

denis-fokin Dec 7, 2022

Markoutte Dec 8, 2022

denis-fokin Dec 7, 2022

Markoutte Dec 8, 2022

denis-fokin Dec 8, 2022

SBOne-Kenobi left a comment

SBOne-Kenobi Dec 8, 2022

volivan239 left a comment

volivan239 Dec 9, 2022

volivan239 Dec 12, 2022

volivan239 Dec 12, 2022

volivan239 Dec 12, 2022 •

edited

Loading

New fuzzing platform #1457

New fuzzing platform #1457

Conversation

Markoutte commented Dec 6, 2022 • edited Loading

Description

Type of Change

How Has This Been Tested?

Regression and integration tests

Automated Testing

Manual Scenario

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SBOne-Kenobi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

volivan239 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

volivan239 Dec 12, 2022 • edited Loading

Choose a reason for hiding this comment

Markoutte commented Dec 6, 2022 •

edited

Loading

volivan239 Dec 12, 2022 •

edited

Loading