Pass dictionary to FileHandler

[DamianMehers]

In my case (and perhaps this is out of the ordinary) I need a header for a FilePart that is not one of the ones extracted from the parameters dictionary passed into ParseFilePart in StreamingMultipartFormDataParser.ParseFilePart

// Read the parameters
parameters.TryGetValue("name", out string name);
parameters.TryGetValue("filename", out string filename);
parameters.TryGetValue("content-type", out string contentType);
parameters.TryGetValue("content-disposition", out string contentDisposition);

... and subsequently passed to FileHandler:

FileHandler(name, filename, contentType, contentDisposition, fullBuffer, endPos - bufferNewlineLength, partNumber++);

Consequently I've modified (in my own copy) FileHandler to just take the Dictionary<string, string> parameters as a parameter and not pass in name, filename, contentType, contentDisposition:

FileHandler(parameters, fullBuffer, endPos - bufferNewlineLength, partNumber++);

This would be a breaking change, so I am not sure what to do ... are other people finding that they need a parameter that is not one of the ones that are passed?

I'm happy to submit a pull request if this change makes sense.

[Jericho]

Can you provide an example to help me understand?

[DamianMehers]

As I said, this may not be a common scenario. The Amazon Alexa voice service streams multipart data which just has a content type and content id.

Here I added logging to the ParseSection method in StreamingMultipartFormDataParser:

ParseSection Parsing line: Content-ID: <Terminal_amzn1.ask.skill.48a656db-56f8-46e7-aa1e-4a5d856878c7_3bb1d60a-cb56-4e23-bb4b-86cac587c576_CloudInitiated_664250913>
ParseSection Adding 'content-id' : '<Terminal_amzn1.ask.skill.48a656db-56f8-46e7-aa1e-4a5d856878c7_3bb1d60a-cb56-4e23-bb4b-86cac587c576_CloudInitiated_664250913>'
ParseSection Reading a line
ParseSection Parsing line: Content-Type: application/octet-stream
ParseSection Adding 'content-type' : 'application/octet-stream'
ParseSection Reading a line
ParseSection Parsing line: 
ParseSection Parsing file parts ...

I've had to make other changes since the logic that detected "Files" didn't work in my scenario - I have massive non-file binary data parts that stream. I have it working, but I wasn't sure if others might need the kind of changes I made.

[Jericho]

What would be helpful is an example of data sent by Alexa. I’m trying to understand what data you are trying to access that is not already provided to the FileHandler. Also, it would allow me to write a quick unit test to reproduce your situation.

[Jericho]

Also, I suggest you raise a separate issue to discuss the logic to determine if a section contains a file or not.

[DamianMehers]

Re a unit test, fundamentally the stream will have at least two parts:
The first with the header:

• Content-Type: application/json
  and json content

...and the second with headers:

• Content-ID: <some string>
• Content-Type: application/octet-stream
  and binary content of very large indeterminate length (streaming) which I need to get access to as it arrives (it is audio data that I am sending to the speaker).

Additional sections may then come down over the stream (I'm reading over an HTTP/2 stream where content can be pushed).

I don't think the library handles large binary streaming non-file parts (I could have missed something).

A couple of years ago I made local changes to this library to support this scenario, since the existing library assumed that files were large and everything else was small ... not the case in my situation. I just came back to the library and saw you were making updates (thanks!) so I thought I'd see if my situation was an outlier.

[Jericho]

If you give me the payload you receive, I'll take care of the unit test (feel free to truncate the binary data to make it more manageable).

Based on my understanding so far, I think it should look something like this:

-----------------------------41952539122868
Content-Disposition: form-data; name="myJson"

{
    jsonProperty1 : value1,
    jsonProperty2 : value2
}
-----------------------------41952539122868
Content-Disposition: form-data; name="alexa"; filename="alexa.wav"
Content-ID: <some string>
Content-Type: application/octet-stream

<... binary data ...>
-----------------------------41952539122868--

[DamianMehers]

Will try to capture it and send it today. One correction: there is no content-disposition header - no name and no filename for the binary data - it is just another part.

-----------------------------41952539122868
Content-Type: application/json

{
    jsonProperty1 : value1,
    jsonProperty2 : value2
}
-----------------------------41952539122868
Content-ID: <some string>
Content-Type: application/octet-stream

<... binary data ...>
-----------------------------41952539122868--

[Jericho]

A quick observation: up until release 4.2.0, the fact that name and filename are omitted was problematic. The fact that filename is not present would cause the section to be treated as a regular parameter (as opposed to a file) but the name is required in order to identify regular parameters. Therefore you would have gotten a "Key not found" exception. See #83 for more details.

This logic was changed in release 4.3.0 so that sections without name and filename are now treated as a file (files do not require the name and filename). This change was made to accommodate mjpeg streams where each JPEG frame is sent as a file but the name and filename are omitted.

It just occurred to me that this change also benefits you because the audio stream you receive from the Alexa service does not have name and filename either and by treating it as a file it causes the content to be streamed.

[DamianMehers]

Hi Jericho,

I've decided to continue using the older version of this library from a couple of years ago that I modified extensively - the new one takes dependencies on Microsoft.IO.RecyclableMemoryStream and System.Buffers, and while they would probably be OK, because I use this library in an app that runs on many platforms (iOS, WatchOS, Android, WearOS, Mac, Windows, xBox, and Tizen) I'm hesitant in case there are platform issues. WatchOS especially has some weirdness because of the AOT compilation to BitCode.

I really appreciate your effort with this library, and hope to return and use the latest version.

If by any chance you use the Amazon Alexa and have a smartwatch, let me know - I use Http-Multipart-Data-Parser in my "Voice in a Can" app.

[Jericho]

The two nuget packages you mentioned were added to the project in an effort to reduce memory allocation. See #77 for benchmark and before/after comparison. Since you are targeting platforms such as watches and wearables were presumably memory is quite limited, I would think you would be interested in the improved memory utilization.

By the way, I worked on drafting a version of the HttpMultipartData parser that can capture additional parameters when processing files. It's based on my understanding of your scenario so far and I was hopping to get a sample payload from you to validate that my understanding is correct. If you have some time to check it out, I published it on my MyGet feed: https://www.myget.org/feed/jericho/package/nuget/HttpMultipartParser

[Jericho]

@DamianMehers I am planning the next release of the HttpMultipartDataParser which will contain a few breaking changes and I figured it would make sense to include the functionality you and I were discussing since it would cause a breaking change to the FileHandler delegate. Would you be available to collaborate with me and test this new feature?

By the way, did you review the memory utilization benchmark data I previously mentioned? If so, did it convince you to use the latest version of HttpMultipartDataParser or are you still using your custom built version?

[DamianMehers]

@DamianMehers I am planning the next release of the HttpMultipartDataParser which will contain a few breaking changes and I figured it would make sense to include the functionality you and I were discussing since it would cause a breaking change to the FileHandler delegate. Would you be available to collaborate with me and test this new feature?

By the way, did you review the memory utilization benchmark data I previously mentioned? If so, did it convince you to use the latest version of HttpMultipartDataParser or are you still using your custom built version?

I'm afraid not. I'm moving to Swift from .NET for my Apple Watch apps since Microsoft have said they won't be supporting the Apple Watch for .NET 6... so all my focus is on Swift and SwiftUI at the moment.

Sorry,

Damian