Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A trimmed-down API #136

Merged
merged 83 commits into from
Jun 27, 2023
Merged
Show file tree
Hide file tree
Changes from 75 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
492c601
Remove ByteString-related API
fzhinkin May 25, 2023
87a44b2
First round of the refactoring
fzhinkin May 26, 2023
ee9c395
Enable Dokka
fzhinkin May 26, 2023
8847c34
Enable Kover for core-io
fzhinkin May 30, 2023
597bb41
Suppressed/fixed warnings
fzhinkin May 30, 2023
fb7b533
Update API dump
fzhinkin May 30, 2023
5efafef
Extract utf8-related methods from Sink/Source into ext functions
fzhinkin May 30, 2023
800bccb
Updated documentation, added additional tests
fzhinkin May 31, 2023
78bc776
Add more JVM-specific tests
fzhinkin Jun 1, 2023
372c3c9
Remove some unused and commented out code
fzhinkin Jun 1, 2023
1cc5e14
Update documentation
fzhinkin Jun 2, 2023
2f25f19
Added module description
fzhinkin Jun 2, 2023
863b359
Move internal utf8-related code to -Utf8.kt
fzhinkin Jun 5, 2023
8b00289
Enabled toString test, fixed implementation.
fzhinkin Jun 5, 2023
28cda06
Cleaned up the code.
fzhinkin Jun 5, 2023
198b140
Improve test coverage
fzhinkin Jun 5, 2023
34510ff
Add more tests on UTF-8
fzhinkin Jun 5, 2023
efc3a27
Fix FS-related tests on Windows
fzhinkin Jun 5, 2023
45da73f
Updated documentation
fzhinkin Jun 5, 2023
cd368b4
Updated usage example
fzhinkin Jun 6, 2023
90c9ba3
Enable explicit API mode
fzhinkin Jun 6, 2023
1568f29
Treat all warnings as errors for native compilation tasks
fzhinkin Jun 6, 2023
9b94d6b
Get rid of unneeded annotations
fzhinkin Jun 6, 2023
e6cf5c9
Cleanup imports
fzhinkin Jun 6, 2023
36e779a
Remove more Throws annotations
fzhinkin Jun 6, 2023
93c37af
Extract some JVM-specific member functions to extension functions
fzhinkin Jun 6, 2023
04ab7f7
Move Source.readByteArray to extensions
fzhinkin Jun 7, 2023
18c8637
Move readFully([B) to extensions
fzhinkin Jun 7, 2023
ee0dd50
Remove 'cancel' method
fzhinkin Jun 8, 2023
fadd590
Accept Byte and Short in corresponding write methods, support unsigne…
fzhinkin Jun 9, 2023
0fc1fad
Don't implement channels on JVM, return proxy objects instead
fzhinkin Jun 9, 2023
336da54
Mark buffer getters and emitCompleteSegments as delicate API requirin…
fzhinkin Jun 9, 2023
33545dc
Source::readFully should accept RawSink instead of Buffer
fzhinkin Jun 9, 2023
f349aa8
Fixed line reading tests
fzhinkin Jun 9, 2023
f48baab
Cleanup
fzhinkin Jun 9, 2023
ddfcd11
Annotate methods using delicate API with corresponding OptIn
fzhinkin Jun 12, 2023
dc35855
Cleanup comments and docs
fzhinkin Jun 12, 2023
b82fb82
Improve implementations
fzhinkin Jun 12, 2023
863cf29
Rename tests
fzhinkin Jun 12, 2023
08e7cd2
Cleanup
fzhinkin Jun 12, 2023
a9d113e
Fixed imports
fzhinkin Jun 12, 2023
e55e9b1
Don't override equals/hashCode for Buffer
fzhinkin Jun 12, 2023
d55e78a
Enable source /w options test
fzhinkin Jun 15, 2023
d3b9179
Throw IllegalArgumentException instead of IOOBE
fzhinkin Jun 15, 2023
2f73fd5
Don't return a receiver from write methods
fzhinkin Jun 16, 2023
79c927c
Enabled missing targets
fzhinkin Jun 16, 2023
008cc82
Get rid of SharedImmutable
fzhinkin Jun 16, 2023
b492ae5
Get rid of some platform-specific implementations
fzhinkin Jun 16, 2023
cfe3f40
Implement AutoClosable from stdlib instead of custom Closable interface
fzhinkin Jun 16, 2023
b404e72
Simplified receiver decl for Sink's extensions
fzhinkin Jun 16, 2023
c6a9395
Make Source and Sink regular interfaces, not expects
fzhinkin Jun 16, 2023
29845d6
Update Buffer::toString doc
fzhinkin Jun 16, 2023
ef49ac5
Make public API intended for internal use less accessible
fzhinkin Jun 16, 2023
d0cff32
Fixed pathSourceWithOptions test on Windows
fzhinkin Jun 16, 2023
a6a0f64
Apply new naming conventions
fzhinkin Jun 19, 2023
c3720f1
Use start + end indices instead of offset + byteCount
fzhinkin Jun 19, 2023
b29ea3c
Change readByteArray signature to avoid too large arrays allocation
fzhinkin Jun 19, 2023
76fc0ec
Improve exception messages by adding more context to it
fzhinkin Jun 19, 2023
281f9c7
Fix readDecimalLong and readHexadecimalUnsignedLong behavior
fzhinkin Jun 19, 2023
2de3209
Cleanup
fzhinkin Jun 19, 2023
0e7646e
Document naming convention and methods behavior
fzhinkin Jun 19, 2023
5d3e19c
Fixed typo
fzhinkin Jun 19, 2023
115e8ea
Updated API dump
fzhinkin Jun 19, 2023
31821a7
Fix arguments handling and closed-state checks
fzhinkin Jun 19, 2023
0041f9f
Improve test coverage
fzhinkin Jun 19, 2023
6a7a03f
Fixed Buffer::toString
fzhinkin Jun 19, 2023
e6785c1
Rewrite hex- and decimal-long reading extensions w/o peek
fzhinkin Jun 19, 2023
ada8927
Support array's subrange in Source::readTo
fzhinkin Jun 20, 2023
873661e
Renamed Buffer::writeTo(OutputStream) to readTo
fzhinkin Jun 20, 2023
535a8f7
Updated sample in module description
fzhinkin Jun 20, 2023
dc7483a
Renamed unsigned int write methods to follow the common naming conven…
fzhinkin Jun 20, 2023
afcfd08
Removed JVM-specific unnecessary extensions
fzhinkin Jun 22, 2023
7c49413
Update dependencies
fzhinkin Jun 23, 2023
e5fec12
Improved exception messages, fixed few typos, updated docs
fzhinkin Jun 23, 2023
eb97548
Add -Xjvm-default=all to compiler args
fzhinkin Jun 23, 2023
32f207f
Renamed methods returning wrappers for Sink/Source and vice versa
fzhinkin Jun 23, 2023
2b7eb60
Change Buffer::toString format
fzhinkin Jun 23, 2023
5171a28
Clarify copyTo behavior in the documentation
fzhinkin Jun 23, 2023
a96726f
Make parameter names in public API more verbose
fzhinkin Jun 26, 2023
c5f8d0e
Rename BlackholeSink to DiscardingSink
fzhinkin Jun 26, 2023
a579e33
Explicitly specify what does the 'buffer bounds' mean
fzhinkin Jun 26, 2023
3a506b8
Removed unused code
fzhinkin Jun 26, 2023
47d2964
Hide utf8Size and code-point related functions
fzhinkin Jun 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 24 additions & 14 deletions benchmarks/src/commonMain/kotlin/BufferOps.kt
Original file line number Diff line number Diff line change
Expand Up @@ -97,18 +97,21 @@ open class DecimalLongBenchmark: BufferRWBenchmarkBase() {
var value = 0L

override fun padding(): ByteArray {
val tmpBuffer = Buffer()
while (tmpBuffer.size < minGap) {
// use space as a delimiter between consecutive decimal values
tmpBuffer.writeDecimalLong(value).writeByte(' '.code)
return with (Buffer()) {
while (size < minGap) {
writeDecimalLong(value)
// use space as a delimiter between consecutive decimal values
writeByte(' '.code.toByte())
}
readByteArray()
}
return tmpBuffer.readByteArray()
}

@Benchmark
fun benchmark(): Long {
// use space as a delimiter between consecutive decimal values
buffer.writeDecimalLong(value).writeByte(' '.code)
buffer.writeDecimalLong(value)
buffer.writeByte(' '.code.toByte())
val l = buffer.readDecimalLong()
buffer.readByte() // consume the delimiter
return l
Expand All @@ -120,16 +123,19 @@ open class HexadecimalLongBenchmark: BufferRWBenchmarkBase() {
var value = 0L

override fun padding(): ByteArray {
val tmpBuffer = Buffer()
while (tmpBuffer.size < minGap) {
tmpBuffer.writeHexadecimalUnsignedLong(value).writeByte(' '.code)
return with(Buffer()) {
while (size < minGap) {
writeHexadecimalUnsignedLong(value)
writeByte(' '.code.toByte())
}
readByteArray()
}
return tmpBuffer.readByteArray()
}

@Benchmark
fun benchmark(): Long {
buffer.writeHexadecimalUnsignedLong(value).writeByte(' '.code)
buffer.writeHexadecimalUnsignedLong(value)
buffer.writeByte(' '.code.toByte())
val l = buffer.readHexadecimalUnsignedLong()
buffer.readByte()
return l
Expand Down Expand Up @@ -317,7 +323,11 @@ open class IndexOfBenchmark {
if (valueOffset >= 0) array[valueOffset] = VALUE_TO_FIND

val padding = ByteArray(paddingSize)
buffer.write(padding).write(array).skip(paddingSize.toLong())
with(buffer) {
write(padding)
write(array)
skip(paddingSize.toLong())
}
}

@Benchmark
Expand Down Expand Up @@ -358,7 +368,7 @@ open class BufferReadWriteByteArray: BufferRWBenchmarkBase() {
@Benchmark
fun benchmark(blackhole: Blackhole) {
buffer.write(inputArray)
buffer.readFully(outputArray)
buffer.readTo(outputArray)
blackhole.consume(outputArray)
}
}
Expand All @@ -377,6 +387,6 @@ open class BufferReadNewByteArray: BufferRWBenchmarkBase() {
@Benchmark
fun benchmark(): ByteArray {
buffer.write(inputArray)
return buffer.readByteArray(size.toLong())
return buffer.readByteArray(size)
}
}
9 changes: 6 additions & 3 deletions benchmarks/src/jvmMain/kotlin/ByteBufferBenchmarks.kt
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@

package kotlinx.io.benchmarks

import kotlinx.benchmark.Benchmark
import kotlinx.benchmark.Param
import kotlinx.benchmark.Setup
import kotlinx.io.readAtMostTo
import kotlinx.io.write
import java.nio.ByteBuffer

import kotlinx.benchmark.*

open class ByteBufferReadWrite: BufferRWBenchmarkBase() {
private var inputBuffer = ByteBuffer.allocate(0)
private var outputBuffer = ByteBuffer.allocate(0)
Expand All @@ -28,7 +31,7 @@ open class ByteBufferReadWrite: BufferRWBenchmarkBase() {
inputBuffer.rewind()
outputBuffer.clear()
buffer.write(inputBuffer)
while (buffer.read(outputBuffer) > 0) {
while (buffer.readAtMostTo(outputBuffer) > 0) {
// do nothing
}
return outputBuffer
Expand Down
5 changes: 3 additions & 2 deletions benchmarks/src/jvmMain/kotlin/SegmentPoolBenchmarkMT.kt
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,16 @@

package kotlinx.io.benchmarks

import kotlinx.io.*
import kotlinx.benchmark.*
import kotlinx.io.*
import org.openjdk.jmh.annotations.Group
import org.openjdk.jmh.annotations.GroupThreads

@State(Scope.Benchmark)
open class SegmentPoolBenchmarkMT {
private fun testCycle(): Buffer {
val buffer = Buffer().writeByte(0)
val buffer = Buffer()
buffer.writeByte(0)
buffer.clear()
return buffer
}
Expand Down
10 changes: 8 additions & 2 deletions benchmarks/src/jvmMain/kotlin/StreamBenchmarks.kt
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,13 @@

package kotlinx.io.benchmarks

import kotlinx.benchmark.*
import kotlinx.benchmark.Benchmark
import kotlinx.benchmark.Blackhole
import kotlinx.benchmark.Param
import kotlinx.benchmark.Setup
import kotlinx.io.inputStream
import kotlinx.io.outputStream
import kotlinx.io.readTo

open class InputStreamByteRead: BufferRWBenchmarkBase() {
private val stream = buffer.inputStream()
Expand Down Expand Up @@ -61,7 +67,7 @@ open class OutputStreamByteArrayWrite: StreamByteArrayBenchmarkBase() {
@Benchmark
fun benchmark(blackhole: Blackhole) {
stream.write(outputArray)
buffer.readFully(inputArray)
buffer.readTo(inputArray)
blackhole.consume(inputArray)
}
}
17 changes: 15 additions & 2 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@
* Use of this source code is governed by the Apache 2.0 license that can be found in the LICENSE file.
*/

import org.jetbrains.kotlin.gradle.tasks.*
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
import org.jetbrains.kotlin.gradle.tasks.KotlinNativeCompile

plugins {
id("org.jetbrains.kotlinx.binary-compatibility-validator") version "0.11.1"
id("org.jetbrains.kotlinx.binary-compatibility-validator") version "0.13.2"
id("org.jetbrains.dokka") version "1.8.20"
`maven-publish`
signing
}
Expand All @@ -33,6 +35,10 @@ apply(plugin = "maven-publish")
apply(plugin = "signing")

subprojects {
if (name.contains("benchmark")) {
return@subprojects
}

repositories {
mavenCentral()
}
Expand All @@ -58,6 +64,13 @@ subprojects {
tasks.withType<KotlinCompile>().configureEach {
kotlinOptions {
jvmTarget = JavaVersion.VERSION_1_8.toString()
allWarningsAsErrors = true
freeCompilerArgs += "-Xjvm-default=all"
}
}
tasks.withType<KotlinNativeCompile>().configureEach {
kotlinOptions {
allWarningsAsErrors = true
}
}
}
Expand Down
23 changes: 18 additions & 5 deletions buildSrc/src/main/kotlin/Platforms.kt
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,14 @@ fun KotlinMultiplatformExtension.configureNativePlatforms() {
tvosSimulatorArm64()
watchosArm32()
watchosArm64()
watchosX86()
watchosX64()
watchosSimulatorArm64()
watchosDeviceArm64()
linuxArm64()
androidNativeArm32()
androidNativeArm64()
androidNativeX64()
androidNativeX86()
// Required to generate tests tasks: https://youtrack.jetbrains.com/issue/KT-26547
linuxX64()
macosX64()
Expand All @@ -38,20 +43,28 @@ private val appleTargets = listOf(
"tvosSimulatorArm64",
"watchosArm32",
"watchosArm64",
"watchosX86",
"watchosX64",
"watchosSimulatorArm64"
"watchosSimulatorArm64",
"watchosDeviceArm64"
)

private val mingwTargets = listOf(
"mingwX64"
)

private val linuxTargets = listOf(
"linuxX64"
"linuxX64",
"linuxArm64"
)

val nativeTargets = appleTargets + linuxTargets + mingwTargets
private val androidTargets = listOf(
"androidNativeArm32",
"androidNativeArm64",
"androidNativeX64",
"androidNativeX86"
)

val nativeTargets = appleTargets + linuxTargets + mingwTargets + androidTargets

/**
* Creates a source set for a directory that isn't already a built-in platform. Use this to create
Expand Down
84 changes: 84 additions & 0 deletions core/Module.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Module kotlinx-io-core

The module provides core multiplatform IO primitives and integrates it with platform-specific APIs.

`kotlinx-io core` aims to provide a concise but powerful API along with efficient implementation.

The main interfaces for the IO interaction are [kotlinx.io.Source] and [kotlinx.io.Sink] providing buffered read and
write operations for integer types, byte arrays, and other sources and sinks. There are also extension functions
bringing support for strings and other types.
Implementations of these interfaces are built on top of [kotlinx.io.Buffer], [kotlinx.io.RawSource],
and [kotlinx.io.RawSink].

A central part of the library, [kotlinx.io.Buffer], is a container optimized to reduce memory allocations and to avoid
data copying when possible.

[kotlinx.io.RawSource] and [kotlinx.io.RawSink] are interfaces aimed for integration with anything that can provide
or receive data: network interfaces, files, etc. The module provides integration with some platform-specific IO APIs,
but if something not yet supported by the library needs to be integrated, then these interfaces are exactly what should
be implemented for that.

Example below shows how to manually serialize an object to [BSON](https://bsonspec.org/spec.html)
and then back to an object using `kotlinx.io`. Please note that the example aimed to show `kotlinx-io` API in action,
rather than to provide a robust BSON-serialization.
```kotlin
data class Message(val timestamp: Long, val text: String) {
companion object
}

fun Message.toBson(sink: Sink) {
val buffer = Buffer()
with (buffer) {
writeByte(0x9) // UTC-timestamp field
writeUtf8("timestamp") // field name
writeByte(0)
writeLongLe(timestamp) // field value
writeByte(0x2) // string field
writeUtf8("text") // field name
writeByte(0)
writeIntLe(text.utf8Size().toInt() + 1) // field value: length followed by the string
writeUtf8(text)
writeByte(0)
writeByte(0) // end of BSON document
}

// Write document length and then its body
sink.writeIntLe(buffer.size.toInt() + 4)
buffer.transferTo(sink)
sink.flush()
}

fun Message.Companion.fromBson(source: Source): Message {
source.require(4) // check if the source contains length
val length = source.readIntLe() - 4L
source.require(length) // check if the source contains the whole message

fun readFieldName(source: Source): String {
val delimiterOffset = source.indexOf(0) // find offset of the 0-byte terminating the name
check(delimiterOffset >= 0) // indexOf return -1 if value not found
val fieldName = source.readUtf8(delimiterOffset) // read the string until terminator
source.skip(1) // skip the terminator
return fieldName
}

// for simplicity, let's assume that the order of fields matches serialization order
var tag = source.readByte().toInt() // read the field type
check(tag == 0x9 && readFieldName(source) == "timestamp")
val timestamp = source.readLongLe() // read long value
tag = source.readByte().toInt()
check(tag == 0x2 && readFieldName(source) == "text")
val textLen = source.readIntLe() - 1L // read string length (it includes the terminator)
val text = source.readUtf8(textLen) // read value
source.skip(1) // skip terminator
source.skip(1) // skip end of the document
return Message(timestamp, text)
}
```

# Package kotlinx.io

Core IO primitives.

# Package kotlinx.io.files

Basic API for working with files.
Loading