-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
',' is invalid after a single JSON value. Expected end of data. #30405
Comments
I have tried setting |
@ericwj can you share your input JSON payload that is causing the issue and also the .NET object model you are deserializing to? Also, what version of dotnet sdk are you using (share the output of Please try the nightly SDK/runtime with the fix to verify it works (https://github.com/dotnet/core-sdk#installers-and-binaries), or wait till preview 9 ships. Alternatively, you could reference the latest S.T.Json NuGet package. For example: <ItemGroup>
<PackageReference Include="System.Text.Json" Version="4.6.0-preview9.19413.13" />
<PackageReference Include="System.Text.Encodings.Web" Version="4.6.0-preview9.19413.13" />
</ItemGroup> If your scenarios is the following (and you want to just read the first json object), then that is not supported today. We only support reading a single JSON object: This is also how the underlying string json = "{},"
byte[] input = Encoding.UTF8.GetBytes(jsonString);
var reader = new Utf8JsonReader(input,
new JsonReaderOptions() { AllowTrailingCommas = true });
while (reader.Read())
;
// After EndObject throws System.Text.Json.JsonReaderException :
',' is invalid after a single JSON value. Expected end of data. LineNumber: 0 | BytePositionInLine: 2 |
Yes I am trying to read in a streaming fashion, so there is JSON following other JSON. The files can become fairly big. I think I might be able to use a Although it is quite a lot of code overhead, working with |
What would you expect TryDeserialize to do/return in your scenario where you have multiple json payloads within the file and it fails to deserialize? The deserialize call today throws, so I don't see returning false in the Try* API would help all that much. cc @steveharter
Yep, that's true. Let me think about the scenario you presented and get back to you (I want to see how much effort is required if a motivated dev wants to support this case via the low-level reader). If you already have a pipe/utf8jsonreader-based implementation that works, please share. |
A quick and dirty implementation that works as long as the input is okay is here.
I would like it to return class JsonSerializer {
public static bool TryDeserialize<T>(
ref Utf8JsonReader reader,
out T result,
JsonSerializerOptions options);
} The pipe reader code will search for I don't immediately see the need for I seriously don't think using the |
Given this is currently by-design, and requires some thinking/design work to support this feature, moving to 5.0. |
I ran into this issue myself and ended up working around it by making a custom |
Actually if it were only being used by the enumerate-array-elements method it wouldn't need to support receiving a pushback character, as the character in question would always be either |
In case it is not obvious, specifically, if the value being deserialized is a string, an object or an array, then the character that tells you you are at the end of the JSON value is a part of the value itself and no further character needs to be read. But, if it is a number, you only know that you're past the end of the number once you read the first character that isn't part of the number. Technically speaking, when reading |
@logiclrd - You're making things hard on yourself. Assuming you've got some sort of delegating stream which is buffering the contents, you don't bother to rewind the stream, you just don't move the offending characters into the buffer (or rather, don't mark them as part of the buffer).
This unfortunately iterates over the stream an extra time (but only once), but is simple to implement and doesn't require messing around with states of readers. |
|
I have had a similar issue. I solved it by deleting the file before writing serialized JSON in it. If not do the deletion, the I hope my experience could solve the issue for someone else. |
It's good to point this out in case people with corrupt data end up on this issue by searching for the error message, but I want to make it clear that this issue isn't about corrupt data, but about a design limitation in System.Text.Json that makes it impossible to intentionally decode a JSON value from the middle of a stream. Note that there is another approach you can use: call |
I ran into a similar issue with my json: I can't figure out what is the problem |
Duplicate of #33030, specifically the discussion about |
Wouldn't a |
At the very least the other one is the duplicate - it is three quarters of a year newer than this issue. Looks like about the same although I think |
I should point out that me closing this or the other issue is not assigning credit -- we just need to keep a focused backlog. It just happens that the other issue was triaged ahead of time and as such contains more recent information. I would recommend upvoting the other issue or contributing to the conversation there.
Why do you think it's misleading? The default behavior is to fail on trailing data and we're not changing that. We can discuss adding an optional flag for disabling this, just like Json.NET is doing. |
Thats fine.
I think the name |
Sure, this is not an API proposal, it's merely pointing out the name of the feature in Json.NET |
I still feel like these are fundamentally not the same issue. The other issue is about allowing parsing in situations where the source data is not syntactically valid JSON, because it is smushing multiple documents together. This issue is about making the parsing of JSON arrays piecewise -- it's still a single, valid document, I just don't want to parse the whole thing in one call. But, I want the parsing state preserved so that I can go back to it and get the next piece. These are not the same functionality. EDIT: I thought I was commenting on a related issue that I had encountered. Rereading the top of this thread, @ericwj is not coming at this from quite the same place I am, and my issue was a comment further down the thread. |
Just looking through how this works, I also think this issue should be reopened, over this request: Although neither the In the former case, with small objects, in spans, or single read results, @logiclrd is also helped using the pair |
@logiclrd is this presumably related to #30405 (comment)? I'm not sure I entirely understand the first part of the post, so I would recommend creating a new issue containing a reproduction of the behavior so we can make a recommendation. Regarding the second part, .NET 6 does ship with a |
@ericwj a related request was discussed in #54557 (comment). Per design guidelines |
Mm, |
This too is not an API proposal. I'd be fine with error codes, preferably the most reasonable ones defined in an enum while there is a whole series of errors for which its fine to still throw in this case, too - like out of memory, etc. but also say invalid characters for the current JavaScript decoder. Other than that, for the use case at the start of the article, all that would be needed is The benefit of the former variant would be that much higher levels of customizations can be achieved at very high performance. Would still need to instruct it to succeed without error if all that is wrong is that data follows the JSON, whatever how long the valid part is. |
Given the scenario where you just want a single value, I'd probably take the approach of passing in a JsonPath (e.g. "$.MyObject.MyPropertyToReturn" to a helper like: static async Task<byte[]> GetJsonFromPath(Stream utf8Json, string jsonPath, JsonSerializerOptions options);
// Return the raw bytes to be deserialized in any way (serializer, node, element) or a helper that directly deserializes those bytes using the serializer\node\element which will avoid an extra byte[] allocation: static async Task<TValue> GetValueFromPath(Stream utf8Json, string jsonPath, JsonSerializerOptions options);
// Return the value from the serializer. Similar JsonPath functionality was proposed with #31068 and also in #55827 where I said I'll provide a sample to do this with V6 code, however Stream+SingleValue wasn't mentioned (which complicates things). Some options for Stream:
All options would have minimal processing of the unneeded values -- e.g. values that are not the target JsonPath will not be deserialized. I can work on the sample now if there's interest, but would like to know if "drain" is acceptable or not. |
No thank you @steveharter, repeatedly parsing is unacceptable for the problems described in this issue and in #33030. |
OK I'll work on a prototype; since it involves async+Stream the prototype may need to change internal STJ code instead of just an add-on sample. |
I am reading a single complex object from a
FileStream
usingJsonSerializer.ReadAsync<T>(stream, options, token)
and I seriously just want the one value, but I am getting this exception.The JSON in the file is preceded by bytes which I have skipped or already processed and it is followed by more bytes which I can skip or process separately. However I get stuck by this exception and don't get the value that is presumably correctly read.
I have no idea how long the value is. Pre-parsing the JSON at a lower level to just get the byte length is a bad solution.
Wouldn't it be better to not throw in this situation and let the consumer of the API choose to check for end of stream?
Similar to #29947.
The text was updated successfully, but these errors were encountered: