Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read only the necessary bytes when parsing the header #36

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

DavidBruant
Copy link
Contributor

I use this library in a project i'm currently working on. It works beautifully, thank you very much!

The files i use it on are around 1.5GB. The streaming of the body is super cool, however, i noticed that to parse the header, the entire file was being read in memory

This is due to the use of fs.readFile in src/header.js
In this PR, the code is a bit lower-level to be able to read only the necessary bytes. The idea is:

  1. read on the file only the start field of the header,
  2. this information is used to know how many bytes the header is composed of, so only this number of bytes is read

I'm happy to discuss the change further if what i did here is unclear or if it doesn't adhere to the project standard

@DavidBruant
Copy link
Contributor Author

I can confirm that this change had an amazing perf impact on the header reading in my 1.5GB files. It's pretty much instantaneous now and i can run my script in parallel of my web browser without running out of memory 👌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant