Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Json configuration flag to allow comments #2592

Merged
merged 8 commits into from
Mar 26, 2024
Merged

Add Json configuration flag to allow comments #2592

merged 8 commits into from
Mar 26, 2024

Conversation

sandwwraith
Copy link
Member

in C/Java style for both string and stream parser.

This flag, together with allowTrailingCommas and isLenient, will help to cover most use cases for Json5, such as configuration files.

Fixes #2221
Fixes #797

@sandwwraith
Add Json configuration flag to allow comments
fa75b84 
in C/Java style for both string and stream parser.

This flag together with allowTrailingCommas and isLenient will help
to cover most use-cases for Json5, for example, configuration files.

Fixes #2221
Fixes #797
@qwwdfsad qwwdfsad requested review from fzhinkin and removed request for qwwdfsad March 5, 2024 17:41
fzhinkin
fzhinkin reviewed Mar 13, 2024
View reviewed changes
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/AbstractJsonLexer.kt Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/CommentLexers.kt Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/Json.kt Outdated Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/StringJsonLexer.kt Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/ReaderJsonLexer.kt Outdated Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/ReaderJsonLexer.kt Outdated Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/ReaderJsonLexer.kt Outdated Show resolved Hide resolved
sandwwraith and others added 3 commits March 14, 2024 17:13
@sandwwraith @fzhinkin
Apply suggestions from code review
5754583 
Co-authored-by: Filipp Zhinkin <filipp.zhinkin@gmail.com>
@sandwwraith
~minor fixups
1475f95
@sandwwraith
~extract isWhitespace
926fd40
shanshin
shanshin reviewed Mar 15, 2024
View reviewed changes
formats/json-tests/commonTest/src/kotlinx/serialization/features/JsonCommentsTest.kt
import kotlinx.serialization.json.*
import kotlin.test.*

class JsonCommentsTest: JsonTestBase() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need special tests for StringJsonLexer and ReaderJsonLexer. We now have two decoding branches that can be modified independently of each other.

Since JSON without comments is actually a subset of JSON with comments, it is necessary to test it for both cases.

We can do this by running all the JSON tests for allowComments = true, however, this will be very redundant.
It is easier to identify the main decoding cases and write a limited set of tests for them, which works the same regardless of the source (String or Reader) and the settings (with or without comments)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid we do not have particular 'main decoding cases' tests. The closes place I can think of is Json spec tests:

kotlinx.serialization/formats/json-tests/jvmTest/src/kotlinx/serialization/json/SpecConformanceTest.kt

Line 27 in 5e8ccad

class SpecConformanceTest {

Do you think it is enough to add allowComments = true there?

Also, 'JsonCommentsTest' may be already enough because it has arrays, nested objects, and strings, and there's really not much anything else in the json spec.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think SpecConformanceTest will be enough.
After all, this flag only changes parsing, but not the serialization logic.

formats/json-tests/commonTest/src/kotlinx/serialization/features/JsonCommentsTest.kt Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/CommentLexers.kt Show resolved Hide resolved
formats/json-tests/commonTest/src/kotlinx/serialization/features/JsonCommentsTest.kt Outdated Show resolved Hide resolved
formats/json/commonMain/src/kotlinx/serialization/json/internal/lexer/CommentLexers.kt Show resolved Hide resolved
@shanshin
Copy link
Contributor

Could you also please show the benchmark results?
I am interested in the results before the changes made, after them, as well as after the changes, but if you set allowComments = true for all available benchmark tests (without changing of JSON strings).

@sandwwraith
Copy link
Member Author

after the changes, but if you set allowComments = true for all available benchmark tests

I believe TwitterFeedCommentsBenchmark.decodeMicroTwitterCommentSupport does exactly this, as it decodes data without comments in allowComments = true mode. Although I can re-check with every test if you insist.

@sandwwraith
Copy link
Member Author

For other benchmarks

Before (dev branch):

Benchmark                                                    Mode  Cnt     Score    Error  Units
TwitterFeedBenchmark.decodeMicroTwitter                     thrpt   30  1036.555 ±  5.558  ops/s
TwitterFeedBenchmark.decodeMicroTwitterNoAltNames           thrpt   30  1097.467 ±  5.286  ops/s
TwitterFeedBenchmark.decodeMicroTwitterWithNamingStrategy   thrpt   30   971.212 ± 12.652  ops/s
TwitterFeedStreamBenchmark.decodeMicroTwitterJacksonStream  thrpt   30   605.712 ±  6.738  ops/s
TwitterFeedStreamBenchmark.decodeMicroTwitterReadText       thrpt   30   484.507 ±  5.646  ops/s
TwitterFeedStreamBenchmark.decodeMicroTwitterStream         thrpt   30   496.026 ±  1.806  ops/s

After:

Benchmark                                                             Mode  Cnt     Score    Error  Units
TwitterFeedBenchmark.decodeMicroTwitter                              thrpt   30  1016.684 ±  3.544  ops/s
TwitterFeedBenchmark.decodeMicroTwitterNoAltNames                    thrpt   30  1097.850 ±  4.275  ops/s
TwitterFeedBenchmark.decodeMicroTwitterWithNamingStrategy            thrpt   30   996.990 ± 22.545  ops/s
TwitterFeedCommentsBenchmark.decodeMicroTwitter                      thrpt   30  1057.715 ±  5.114  ops/s
TwitterFeedCommentsBenchmark.decodeMicroTwitterCommentInData         thrpt   30   860.290 ±  9.344  ops/s
TwitterFeedCommentsBenchmark.decodeMicroTwitterCommentInDataStream   thrpt   30   550.861 ± 14.833  ops/s
TwitterFeedCommentsBenchmark.decodeMicroTwitterCommentSupport        thrpt   30   908.675 ± 15.589  ops/s
TwitterFeedCommentsBenchmark.decodeMicroTwitterCommentSupportStream  thrpt   30   649.064 ±  5.674  ops/s
TwitterFeedStreamBenchmark.decodeMicroTwitterJacksonStream           thrpt   30   602.225 ±  1.965  ops/s
TwitterFeedStreamBenchmark.decodeMicroTwitterReadText                thrpt   30   465.921 ±  1.146  ops/s
TwitterFeedStreamBenchmark.decodeMicroTwitterStream                  thrpt   30   545.131 ± 19.263  ops/s

shanshin
shanshin approved these changes Mar 22, 2024
View reviewed changes
Copy link
Contributor

@shanshin shanshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice :)

@sandwwraith
Copy link
Member Author

@shanshin PTAL at changes at SpecConformanceTest

@sandwwraith sandwwraith merged commit 1ffdbaf into dev Mar 26, 2024
4 checks passed
This was referenced Apr 5, 2024
Closed
Merged
@shanshin shanshin deleted the json-comments-f branch August 2, 2024 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fzhinkin fzhinkin fzhinkin approved these changes

@shanshin shanshin shanshin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sandwwraith @shanshin @fzhinkin