Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSV reading isn't fully iterative #33

Closed
dracos opened this issue May 12, 2017 · 3 comments
Closed

CSV reading isn't fully iterative #33

dracos opened this issue May 12, 2017 · 3 comments

Comments

@dracos
Copy link

dracos commented May 12, 2017

If you call get_data with a file-type object and a CSV file type, if I've understood the code correctly, it nearly can be read in iteratively without loading the entire file into memory, apart from the fact that _load_from_stream does a full read() at

content = self._file_stream.read()
in order to look for separators. If that could be made optional (if you know you don't have separators for example), then the process would be fully iterative and only read from the file as you looped through it, which would be useful on extremely large files.

@chfw
Copy link
Member

chfw commented May 12, 2017

I will look at how to iterate content when it is in memory.

@chfw chfw closed this as completed in d82eb39 May 13, 2017
chfw added a commit that referenced this issue May 13, 2017
chfw added a commit that referenced this issue May 13, 2017
@chfw
Copy link
Member

chfw commented May 13, 2017

please evaluate the fixes and let me know how it goes.

@dracos
Copy link
Author

dracos commented May 19, 2017

Thank you for this, making the multiple streams optional is great :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants