Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix reading CSV from non seekable network stream #1472

Merged

Conversation

Lenne231
Copy link
Contributor

@Lenne231 Lenne231 commented Jan 15, 2023

When using the FSharp.Data.Csv.Core package to read massive CSV data from an AWS S3 bucket we had some issues because sometimes many rows got lost.

The problem seems to be related to the StreamReader.Peek() method that is used. When StreamReader.Peek() returns -1 the whole reading stops with the current implementation. The documentation states that -1 is not only returned in case of the end of the stream, but also if the stream is not seekable and the stream doesn't read all data that was requested.

An integer representing the next character to be read, or -1 if there are no characters to be read or if the stream does not support seeking.

This can be fixed by using StreamReader.Read() instead, because this method is blocking. I have also tried to add a unit test to simulate the problem with a non seekable stream that always reads just 1 byte when calling Stream.Read(buffer, offset, count).

@cartermp
Copy link
Collaborator

@Lenne231 looks like you need to run the code formatter.

@Lenne231
Copy link
Contributor Author

@cartermp thanks! Done!

Copy link
Collaborator

@cartermp cartermp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@cartermp cartermp merged commit 1e4cb26 into fsprojects:main Jan 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants