Problem loading bigTIFF larger than 2^32 bytes

[sroboubi]

We've run into issues loading bigTIFF files that are larger than 2^32 bytes. The issue happens regardless of compression but results in different types of errors. For DEFLATE data we get a 406 HTTP error because the library is requesting an invalid (negative) range. For uncompressed data we get:
RangeError: Offset is outside the bounds of the DataView
or
RangeError: Invalid field type: 615

These might all have the same root cause, which might be related to how 64 bit addresses are handled as 2 x 32 bit values. Do you have any bigTIFF test data that is larger than 2^32 bytes that you can use to reproduce the problem? If not, is there a good way we could send you our test data?

Thanks.

[constantinius]

Hi @sroboubi

What is strange, is that obviously some indices are actually read correctly.
I'd like to investigate this, do you have the test file for this issue?

[sroboubi]

I do have the test data. How can I send it? Google drive/dropbox or do you have a better method?

[constantinius]

@sroboubi Yes, a link to the file would be ideal!

[sroboubi]

@constantinius
Here are the files:

https://s3-us-west-2.amazonaws.com/bigtiff-example-0c3d5b35-37ec-416c-9a64-4b8f9f1ba9c9/14SEP04080617-M2AS-BigTiff_TestPattern_DEFLATE.TIF

Getting 416 (Range Not Satisfiable): Requested Range: bytes=-1145289660--1145224125

https://s3-us-west-2.amazonaws.com/bigtiff-example-0c3d5b35-37ec-416c-9a64-4b8f9f1ba9c9/14SEP04080617-M2AS-BigTiff_TestPattern_noCompression.TIF

Getting RangeError: Offset is outside the bounds of the DataView

Let me know if you have any issues downloading them. Thanks.

[constantinius]

@sroboubi
Okay, the issue is two-fold:
First off, the parsing of 64bit integers was off which is easy to correct.

Second (and here is the real issue): the current method to read 64bit integers is inherently flawed, as JavaScript only deals with 64 bit float values and treating them as integers simply does not cut it.
For this problem there is the so-called BigInt type, which is not yet fully ratified (Stage 3 currently AFAIK). So, in modern browsers we can fix this issue, but I don't think that this will work on node (just checked on my installation (v9.11.1) and it did not).

I'll open a branch for the BigInt solution for you to test. For the future: hopefully we'll find a solution that works on both worlds.

[sroboubi]

How can I get and test this fix? I typically just use "npm i geotiff" to get the latest from master. I tried manually downloading the branched code from https://github.com/geotiffjs/geotiff.js/tree/bigtiff-bigint and building using "npm run build" but that didn't generate a dist folder. Thanks.

[constantinius]

@sroboubi hm, that is odd, when I try to run npm run build I can see the result, even when I delete the folder first. Is there any command output?

[constantinius]

@sroboubi Additionally, you can use the npm link functionality (which I use a lot in development):

It works like this:

• Check out the geotiff.js repo, navigate there and do npm link, which globally registers the directory
• Now in your project (where you would usually do npm install geotiff) do npm link geotiff

Now you should have linked the repository as your geotiff package source

[sroboubi]

@constantinius
I don't have a node project so I dont use npm to build/install. I just use it to get the geotiff.bundle.js which I include manually (or through require.js). Note that I am using mingw on windows so it looks like the npm bundle:prod was giving errors because it couldn't set NODE_ENV.
I managed to build the project by running the following in order:
npm install
export NODE_ENV=production
npm run build:clean
npm run bundle:dev
npm run bundle:prod
npm run build:babel

Note that I updated the package.json to remove NODE_ENV=production from the "bundle:prod" script.

This only produced geotiff.bundle.min.js, but not geotiff.browserify.js. Using the minified code I managed to load normal geotiffs but when loading a bigtiff the browser would hang (after making 3 successful partial GET requests) and I can't debug this because I only have the minified code.

[sroboubi]

@constantinius
By the way, it looks like the latest version of node has bigint support as well? Does that make it easier to test?
https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V10.md#2018-10-10-version-10120-current-targos