No way to parse JSON from the middle of a stream

[logiclrd]

Description

If I have a string that has a properly-formed JSON object in the middle, there doesn't appear to be a way to parse it with System.Text.Json, because System.Text.Json requires that when it reaches the end of the object, it is also at the hard end of the underlying stream.

A practical case where this has come up is when trying to transmit an IAsyncEnumerable<> over a JSON boundary. System.Text.Json doesn't support IAsyncEnumerable<>, but the structure of an array is pretty simple to layer on top of a series of serializater operations.

When serializing, it is easy enough to emit a [, and then loop through the elements (possibly using await) and serialize each one in turn, putting a , before each element after the first one, and finishing with a ] character.

But, when deserializing, if the code recognizes the [, and then tries to deserialize the first element to return it (again, possibly in an async context), System.Text.Json throws an error:

System.Text.Json.JsonException: '',' is invalid after a single JSON value. Expected end of data.

I can't find any way to permit this situation, and for the time being have worked around it by creating a specialized Stream type that pretends it is at the end of the stream every time it reaches the end of a JSON value. This analysis requires a moderately complex state machine, and while I have created a suite of tests covering every arrangement that came to mind, I do not have 100% confidence that it can never break.

It would be ideal if there were a way directly to support this scenario. It is supported (at least for elements that are all JSON objects and arrays) by Newtonsoft.Json. It occurs to me also, though, that if System.Text.Json had a specialized "read IAsyncEnumerable<>" method that did this mechanic but with internal support, yielding the objects as they are read, it would obviate the issue, not in general, but in this specific case. This specific case is one which I've had come up repeatedly in the context of web services.

Configuration

(all versions with System.Text.Json that I've tried)

[BreyerW]

Basically dupe of either #1570 or #45188

[logiclrd]

Well, the specific suggestion that IAsyncEnumerable be supported innately is, but if that isn't a specific approach being taken, then the bug reported here isn't captured or addressed anywhere.

[svick]

I think the specific request is also a duplicate of #30405.

[logiclrd]

Okay, yep. I'll add a comment there with my IAsyncEnumerable idea.

[layomia]

Triage: merging this with #30405.