Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add output collectors to support output described in the draft 2020-12 #128

Merged
merged 35 commits into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ac49f87
Add information about schema uri for reference
OptimumCode May 31, 2024
f7d6db5
Migrate to output collectors
OptimumCode Jun 5, 2024
c06d246
Optimize error transformation
OptimumCode Jun 8, 2024
222e69a
Rewrite relative method for json pointer
OptimumCode Jun 8, 2024
66d52f7
Add methods to use output collectors. Make sure keyword location is u…
OptimumCode Jun 9, 2024
2e518e2
Add tests for output collectors
OptimumCode Jun 9, 2024
61f1906
Avoid index parsing when insertLast is used
OptimumCode Jun 9, 2024
7f8714e
Delay allocation of inner collection in collectors until it is really…
OptimumCode Jun 9, 2024
9036eec
Minimize publich api
OptimumCode Jun 9, 2024
02a956d
Generate API dump
OptimumCode Jun 9, 2024
5d87e75
Add benchmarks for new output collectors
OptimumCode Jun 9, 2024
b99c0a6
Do not create child collectors for flag output once the failid result…
OptimumCode Jun 9, 2024
1acc755
Increase nested block depth limit
OptimumCode Jun 9, 2024
0dd2c71
Return value instead of using blackhole object
OptimumCode Jun 9, 2024
9488dd1
Reduce allocations in outputc collectors. Cache hashcode to avoid dee…
OptimumCode Jun 9, 2024
9694a8c
Correct output classes to match the output schema from draft 2020-12
OptimumCode Jun 9, 2024
8d90abc
Hide output property from public API
OptimumCode Jun 9, 2024
67ea7f3
Add serizalization for absolute location
OptimumCode Jun 10, 2024
3a3ac20
Add seriailzation to json pointer
OptimumCode Jun 10, 2024
284ac26
Add serialization to output results
OptimumCode Jun 10, 2024
9df8015
Make custom serizalizers internal
OptimumCode Jun 10, 2024
26c8283
Update api dump
OptimumCode Jun 10, 2024
18f8e27
Remove unused index prop during pointer parsing
OptimumCode Jun 10, 2024
e6a69ee
Add additional test case for relative pointer
OptimumCode Jun 10, 2024
cc0a02d
Add an alternative strategy for joining pointers with big length
OptimumCode Jun 11, 2024
f0db8c8
Add documentation to new API
OptimumCode Jun 11, 2024
189aa5b
Fix reporting behavior for anyOf and oneOf
OptimumCode Jun 11, 2024
e843137
Suppress detekt error for detailed collector
OptimumCode Jun 11, 2024
4bbfa14
Add testcase for json pointer with length longer than max length for …
OptimumCode Jun 12, 2024
35f52e4
Correct doc. Simplify some methods
OptimumCode Jun 12, 2024
2d84366
Update readme
OptimumCode Jun 12, 2024
1152bef
Remove redundant nullability from error transformer
OptimumCode Jun 12, 2024
c479aeb
Fix bug in detailed output. Add tests
OptimumCode Jun 12, 2024
4acf6bc
Hide collector behind a provider to prevent possible sharing between …
OptimumCode Jun 12, 2024
ba87c05
Correct long line in comments
OptimumCode Jun 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,25 @@ val elementToValidate: JsonElement = loadJsonToValidate()
val valid = schema.validate(elementToValidate, errors::add)
```

You can also use predefined `ValidationOutput`s to collect the results.
Output formats are defined in [draft 2020-12](https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#section-12.4).
The most performance can be achieved by using either `flag` or `basic` collectors.
The `detailed` and `verbose` provide more structured information but this adds additional cost to the validation process
(because they collect hierarchical output).

```kotlin
import io.github.optimumcode.json.schema.JsonSchema
import io.github.optimumcode.json.schema.OutputCollector
import io.github.optimumcode.json.schema.ValidationOutput.Flag
import io.github.optimumcode.json.schema.ValidationOutput.Basic
import io.github.optimumcode.json.schema.ValidationOutput.OutputUnit

val flag: Flag = schema.validate(elementToValidate, OutputCollector.flag())
val basic: Basic = schema.validate(elementToValidate, OutputCollector.basic())
val detailed: OutputUnit = schema.validate(elementToValidate, OutputCollector.detailed())
val verbose: OutputUnit = schema.validate(elementToValidate, OutputCollector.verbose())
```

If you need to use more than one schema, and they have references to other schemas you should use `JsonSchemaLoader` class.

```kotlin
Expand Down
155 changes: 149 additions & 6 deletions api/json-schema-validator.api

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package io.github.optimumcode.json.schema.benchmark

import io.github.optimumcode.json.schema.ErrorCollector
import io.github.optimumcode.json.schema.JsonSchema
import io.github.optimumcode.json.schema.OutputCollector
import kotlinx.benchmark.Benchmark
import kotlinx.benchmark.Blackhole
import kotlinx.benchmark.Setup
Expand Down Expand Up @@ -49,4 +50,24 @@ abstract class AbstractCommonBenchmark {
fun validate(bh: Blackhole) {
bh.consume(schema.validate(document, ErrorCollector.EMPTY))
}

@Benchmark
fun validateFlag(bh: Blackhole) {
bh.consume(schema.validate(document, OutputCollector.flag()))
}

@Benchmark
fun validateBasic(bh: Blackhole) {
bh.consume(schema.validate(document, OutputCollector.basic()))
}

@Benchmark
fun validateDetailed(bh: Blackhole) {
bh.consume(schema.validate(document, OutputCollector.detailed()))
}

@Benchmark
fun validateVerbose(bh: Blackhole) {
bh.consume(schema.validate(document, OutputCollector.verbose()))
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,12 @@ import com.networknt.schema.JsonSchemaFactory
import com.networknt.schema.OutputFormat
import com.networknt.schema.SchemaValidatorsConfig
import com.networknt.schema.SpecVersion.VersionFlag.V7
import com.networknt.schema.output.OutputFlag
import com.networknt.schema.output.OutputUnit
import io.github.optimumcode.json.schema.ErrorCollector
import io.github.optimumcode.json.schema.OutputCollector
import io.github.optimumcode.json.schema.ValidationError
import io.github.optimumcode.json.schema.ValidationOutput
import io.github.optimumcode.json.schema.fromStream
import io.openapiprocessor.jackson.JacksonConverter
import io.openapiprocessor.jsonschema.reader.UriReader
Expand All @@ -18,8 +22,8 @@ import io.openapiprocessor.jsonschema.schema.Output.FLAG
import io.openapiprocessor.jsonschema.schema.SchemaStore
import io.openapiprocessor.jsonschema.validator.Validator
import io.openapiprocessor.jsonschema.validator.ValidatorSettings
import io.openapiprocessor.jsonschema.validator.steps.ValidationStep
import kotlinx.benchmark.Benchmark
import kotlinx.benchmark.Blackhole
import kotlinx.benchmark.Setup
import kotlinx.serialization.ExperimentalSerializationApi
import kotlinx.serialization.json.Json
Expand Down Expand Up @@ -95,29 +99,49 @@ abstract class AbstractComparisonBenchmark {
}

@Benchmark
fun validateOpenApi(bh: Blackhole) {
bh.consume(openapiValidator.validate(openapiSchema, openapiDocument))
fun validateOpenApi(): ValidationStep {
return openapiValidator.validate(openapiSchema, openapiDocument)
}

@Benchmark
fun validateNetworkntFlag(bh: Blackhole) {
bh.consume(networkntSchema.validate(networkntDocument, OutputFormat.FLAG))
fun validateNetworkntFlag(): OutputFlag? {
return networkntSchema.validate(networkntDocument, OutputFormat.FLAG)
}

@Benchmark
fun validateKmpFlag(bh: Blackhole) {
bh.consume(schema.validate(document, ErrorCollector.EMPTY))
fun validateNetworkntDetailed(): OutputUnit {
return networkntSchema.validate(networkntDocument, OutputFormat.LIST)
}

@Benchmark
fun validateNetworkntCollectErrors(bh: Blackhole) {
bh.consume(networkntSchema.validate(networkntDocument, OutputFormat.LIST))
fun validateNetworkntVerbose(): OutputUnit {
return networkntSchema.validate(networkntDocument, OutputFormat.HIERARCHICAL)
}

@Benchmark
fun validateKmpCollectErrors(bh: Blackhole) {
fun validateKmpEmptyCollector(): Boolean {
return schema.validate(document, ErrorCollector.EMPTY)
}

@Benchmark
fun validateKmpCollectErrors(): List<ValidationError> {
val errors = arrayListOf<ValidationError>()
schema.validate(document, errors::add)
bh.consume(errors)
return errors
}

@Benchmark
fun validateKmpFlag(): ValidationOutput.Flag {
return schema.validate(document, OutputCollector.flag())
}

@Benchmark
fun validateKmpDetailed(): ValidationOutput.OutputUnit {
return schema.validate(document, OutputCollector.detailed())
}

@Benchmark
fun validateKmpVerbose(): ValidationOutput.OutputUnit {
return schema.validate(document, OutputCollector.verbose())
}
}
2 changes: 1 addition & 1 deletion build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ kotlin {

dependencies {
api(libs.kotlin.serialization.json)
implementation(libs.uri)
api(libs.uri)
// When using approach like above you won't be able to add because block
implementation(libs.kotlin.codepoints.get().toString()) {
because("simplifies work with unicode codepoints")
Expand Down
2 changes: 1 addition & 1 deletion config/detekt/detekt.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ complexity:
ignoreArgumentsMatchingNames: false
NestedBlockDepth:
active: true
threshold: 4
threshold: 5
NestedScopeFunctions:
active: false
threshold: 1
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package io.github.optimumcode.json.pointer

import kotlinx.serialization.Serializable
import kotlin.jvm.JvmField
import kotlin.jvm.JvmStatic

Expand All @@ -13,6 +14,7 @@ public fun JsonPointer(path: String): JsonPointer = JsonPointer.compile(path)
* Implementation of a JSON pointer described in the specification
* [RFC6901](https://datatracker.ietf.org/doc/html/rfc6901).
*/
@Serializable(JsonPointerSerializer::class)
public sealed class JsonPointer(
internal open val next: JsonPointer? = null,
) {
Expand All @@ -29,7 +31,13 @@ public sealed class JsonPointer(
*/
public fun atIndex(index: Int): JsonPointer {
require(index >= 0) { "negative index: $index" }
return atProperty(index.toString())
return insertLast(
SegmentPointer(
propertyName = index.toString(),
depth = 1,
index = index,
),
)
}

/**
Expand All @@ -40,7 +48,10 @@ public sealed class JsonPointer(
* val pointer = JsonPointer.ROOT.atProperty("prop1").atProperty("prop2") // "/prop1/prop2"
* ```
*/
public fun atProperty(property: String): JsonPointer = insertLast(SegmentPointer(property))
public fun atProperty(property: String): JsonPointer =
insertLast(
SegmentPointer(depth = 1, propertyName = property),
)

override fun toString(): String {
val str = asString
Expand All @@ -66,19 +77,59 @@ public sealed class JsonPointer(
if (this !is SegmentPointer) {
return last
}
var parent: PointerParent? = null
var node: JsonPointer = this
while (node is SegmentPointer) {
parent =
PointerParent(
parent,
node.propertyName,
if (depth < MAX_POINTER_DEPTH_FOR_RECURSIVE_INSERT) {
return insertLastDeepCopy(this, last)
}
// avoid recursion when pointer depth is greater than a specified limit
// this should help with avoiding stack-overflow error
// when this method called for a pointer that has too many segments
//
// Using queue is less efficient than recursion (around 10%) but saves us from crash
val queue = ArrayDeque<SegmentPointer>(depth)
var cur: JsonPointer = this
while (cur is SegmentPointer) {
queue.add(cur)
cur = cur.next
}
val additionalDepth = last.depth
var result = last
while (queue.isNotEmpty()) {
val segment = queue.removeLast()
result =
SegmentPointer(
propertyName = segment.propertyName,
depth = segment.depth + additionalDepth,
index = segment.index,
next = result,
)
node = node.next
}
return buildPath(last, parent)
return result
}

// there might be an issue with stack in case this function is called deep on the stack
private fun insertLastDeepCopy(
pointer: SegmentPointer,
last: SegmentPointer,
): JsonPointer =
with(pointer) {
val additionalDepth = last.depth
if (next is SegmentPointer) {
SegmentPointer(
propertyName = propertyName,
depth = depth + additionalDepth,
index = index,
next = insertLastDeepCopy(next, last),
)
} else {
SegmentPointer(
propertyName = propertyName,
depth = depth + additionalDepth,
index = index,
next = last,
)
}
}

private fun escapeJsonPointer(propertyName: String): String {
if (propertyName.contains(SEPARATOR) || propertyName.contains(QUOTATION)) {
return buildString(capacity = propertyName.length + 1) {
Expand Down Expand Up @@ -132,6 +183,7 @@ public sealed class JsonPointer(
}

public companion object {
private const val MAX_POINTER_DEPTH_FOR_RECURSIVE_INSERT = 20
internal const val SEPARATOR: Char = '/'
internal const val QUOTATION: Char = '~'
internal const val QUOTATION_ESCAPE: Char = '0'
Expand Down Expand Up @@ -174,13 +226,15 @@ public sealed class JsonPointer(
lastSegment: SegmentPointer,
parent: PointerParent?,
): JsonPointer {
var depth = lastSegment.depth
var curr = lastSegment
var parentValue = parent
while (parentValue != null) {
curr =
parentValue.run {
SegmentPointer(
segment,
++depth,
curr,
)
}
Expand Down Expand Up @@ -269,12 +323,11 @@ private fun StringBuilder.appendEscaped(ch: Char) {
internal object EmptyPointer : JsonPointer()

internal class SegmentPointer(
segment: String,
val propertyName: String,
val depth: Int = 1,
override val next: JsonPointer = EmptyPointer,
val index: Int = parseIndex(propertyName),
) : JsonPointer(next) {
val propertyName: String = segment
val index: Int = parseIndex(segment)

companion object {
private const val NO_INDEX: Int = -1
private const val LONG_LENGTH_THRESHOLD = 10
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package io.github.optimumcode.json.pointer

import kotlinx.serialization.KSerializer
import kotlinx.serialization.descriptors.PrimitiveKind
import kotlinx.serialization.descriptors.PrimitiveSerialDescriptor
import kotlinx.serialization.descriptors.SerialDescriptor
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder

internal object JsonPointerSerializer : KSerializer<JsonPointer> {
override val descriptor: SerialDescriptor =
PrimitiveSerialDescriptor(
"io.github.optimumcode.json.pointer.JsonPointer",
PrimitiveKind.STRING,
)

override fun deserialize(decoder: Decoder): JsonPointer = JsonPointer(decoder.decodeString())

override fun serialize(
encoder: Encoder,
value: JsonPointer,
) {
encoder.encodeString(value.toString())
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -72,17 +72,23 @@ public operator fun JsonPointer.plus(otherPointer: JsonPointer): JsonPointer {
* @throws IllegalArgumentException when [other] is an empty pointer
*/
public fun JsonPointer.relative(other: JsonPointer): JsonPointer {
if (this is EmptyPointer) {
if (this !is SegmentPointer) {
return other
}
require(other !is EmptyPointer) { "empty pointer is not relative to any" }
val currentValue = this.toString()
val otherValue = other.toString()
val relative = otherValue.substringAfter(currentValue)
return if (relative == otherValue) {
other
require(other is SegmentPointer) { "empty pointer is not relative to any" }
var currentValue: JsonPointer = this
var otherValue: JsonPointer = other
while (currentValue is SegmentPointer && otherValue is SegmentPointer) {
if (currentValue.propertyName != otherValue.propertyName) {
return other
}
currentValue = currentValue.next
otherValue = otherValue.next
}
return if (currentValue is EmptyPointer) {
otherValue
} else {
JsonPointer(relative)
other
}
}

Expand Down
Loading
Loading