-
Notifications
You must be signed in to change notification settings - Fork 183
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigTIFF Support/Directory Parsing #132
Comments
P.S I am happy to make PR's for either or both of these :) |
Ha! Aweseome! To explain: when I wrote both 64bit support and the automatic parsing of all file directories I always knew that the approach was not ideal, but there simply was no way to test the feasibility. So I'm grateful that the issue has finally come up and that people are actually using it on really big TIFFs. Regarding the 64bit integer parsing: I did not see the Mozilla page you mentioned, I tried to figure it out myself and obviously failed. I would love to see a PR, if you are willing to do so. Which route to take is up to you. I'm not sure what the performance of Regarding the parsing of all file directories: this may be a little more tricky. The issue is the internal TIFF structure (i.e: essentially a linked list of IFDs) and my laziness when I started to work on the lib. I don't think that it is too hard to do, it is basically just a deferred parsing of the IFDs up to the point what page was requested, and continue later on from there. One drawback of that is that some operations that seem lightweight (like getting the number of pages) can take an unexpected amount of work. If you want, you can prepare a PR for that aswell, but I would advise to keep it separate from the other one. Thanks for raising these issues and for the interest in geotiff.js! |
Yes, no problem! That sounds like a good plan regarding the separate PR's. I will probably not use As for the IFD's, I had not thought of that. I will look into this problem as well, but I have a less clear idea of the solution space. Looking forward to resolving this! |
You can have a look at it. If you don't want the hassle just tell me here and I'll try to fix it myself. Thanks for your contribution! |
I have put some thought into grabbing the
For example 2097448 = ((1024 ** 2) * 2) + 296 which is the tile size (1024) squared times 2 (each pixel is two bytes) for the single-tile image plus 296 bytes of metadata/pad (I assume - I don't know too much about this and the number appears to vary at different pyramid levels, but will look into it more), at the lowest resolution. 4194632 bytes represents the next lowest resolution, where there are only two tiles per image. Does this sound like a direction that makes sense? Perhaps a class inheritance would be better, where you override a function? I could look into this as well but I don't know much about the codebase so would be concerned about overriding something that is important. |
Unfortunately, that assumption would not work for all cases. The IFDs would not necessarily be in ascending order. Currently, there is no other way than to traverse the linked list to reliably get to the image metadata/data. |
Could you clarify a bit as to why this means a |
I poked around a bit allowing for a |
I'll add a PR today that will address the directory parsing. I'll do it in a fashion, that one can parse IFDs from arbitrary offsets in the file (at own risk ;-) ). Does this suit your requirements? |
Yes, that would be perfect! |
Hi @constantinius, thanks for getting the PR started. How is it going? I would be willing to help out on it if you are busy with other things. Let me know! |
@ilan-gold Is this issue now fixed? |
#136 looks like the solution, yes! |
Hello, I have a use case of large BigTIFF files (hundreds of image pages) and I am having trouble parsing them as things stand. I believe the issue centers around how 64 bit integers are handled for this type https://github.com/geotiffjs/geotiff.js/blob/master/src/dataslice.js#L97. Currently, the operation for combining the two 32 bit integers is
but on the Mozilla page, one of two is recommended:
or
The first approach is not guaranteed to be precise past 2 ** 53, though, hence the warning. I have done some testing and not had problems with this approach but I could see things going wrong (although 2 ** 53 is a lot, so not sure we'd actually ever run into that if all this method is for is byte offsets; 2 ** 53 bytes is nine million gigabytes, I think).
The behavior I was seeing originally was similar to #63 and #68 where I would get negative numbers. I see in #68 that you added a
BigInt
support but I am not sure that is necessary (I also tried it using the above approach and did not have success, and tried your approach but it nearly crashed my browser).Additionally, the current state of
parseFileDirectories
seems problematic for me because it attempts to parse all the file directories, which is unnecessary in our use case. Could we add a flag to stop that? Our use case only needs the first image pane to get some metadata and can infer what is in the rest. If you think there is a good reason for parsing everything every time, I'd love to discuss - I'm new to the tiff ecosystem so could certainly be missing a crucial use-case.Thanks! This package is quite the workhorse, great stuff!!!! :) :)
The text was updated successfully, but these errors were encountered: