Ingest: Add geotiff support

[raprasad]

For GeoTIFF files added to Dataverse:

1. Identify the file as a GeoTIFF and make an appropriate thumbnail.
   • http://gis.stackexchange.com/questions/18672/how-to-determine-if-a-tiff-is-georeferenced-or-not
2. Consider adding WorldMap support/visualization. (WorldMap already supports GeoTIFF

[eunices]

Hi, would like to ask if there are any updates on this issue?

geotiff file thumbnails are broken. This issue only applies to some geotiff but not all (see https://researchdata.ntu.edu.sg/dataverse/ASE?q=tiff&types=dataverses%3Adatasets%3Afiles&sort=score&order=desc&page=1).

---

A colleague replicated it on the demo.dataverse.org instance.
Two geotiff files with this issue were uploaded and thumbnails associated with the file are broken.

Dataset uploaded on: https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/TV3OWZ
Files involved: middle_lower_Amazon_bathymetry_v2.tiff, Table2_Sediment.tif

The thumbnail is 0 bytes according to Jim Myers on https://groups.google.com/g/dataverse-community/c/K9eUCw8erKc.

Both files are geotiff, verified with gdal

(base) D:_dev\dataverse>gdalinfo data-issue-20210830/middle_lower_Amazon_bathymetry_v2.tiff
Driver: GTiff/GeoTIFF
Files: data-issue-20210830/middle_lower_Amazon_bathymetry_v2.tiff
Size is 7316, 2005
Coordinate System is:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-59.898000000000003,-1.742771843488000)
Pixel Size = (0.000903557453000,-0.000903557453000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( -59.8980000, -1.7427718) ( 59d53'52.80"W, 1d44'33.98"S)
Lower Left ( -59.8980000, -3.5544045) ( 59d53'52.80"W, 3d33'15.86"S)
Upper Right ( -53.2875737, -1.7427718) ( 53d17'15.27"W, 1d44'33.98"S)
Lower Right ( -53.2875737, -3.5544045) ( 53d17'15.27"W, 3d33'15.86"S)
Center ( -56.5927868, -2.6485882) ( 56d35'34.03"W, 2d38'54.92"S)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
Min=1.764 Max=10.351
Minimum=1.764, Maximum=10.351, Mean=5.436, StdDev=1.919
NoData Value=-3.4028234663852886e+38
Metadata:
RepresentationType=ATHEMATIC
STATISTICS_COVARIANCES=3.684419636731698
STATISTICS_MAXIMUM=10.35097694397
STATISTICS_MEAN=5.4357336787474
STATISTICS_MINIMUM=1.7639776468277
STATISTICS_SKIPFACTORX=1
STATISTICS_SKIPFACTORY=1
STATISTICS_STDDEV=1.9194842111181

(base) D:_dev\dataverse>gdalinfo data-issue-20210830/Table2_Sediment.tif
Driver: GTiff/GeoTIFF
Files: data-issue-20210830/Table2_Sediment.tif
Size is 274, 170
Coordinate System is:
PROJCRS["CGCS2000_3_Degree_GK_CM_114E",
BASEGEOGCRS["China Geodetic Coordinate System 2000",
DATUM["China 2000",
ELLIPSOID["CGCS2000",6378137,298.257222101004,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4490]],
CONVERSION["Transverse Mercator",
METHOD["Transverse Mercator",
ID["EPSG",9807]],
PARAMETER["Latitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",114,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Scale factor at natural origin",1,
SCALEUNIT["unity",1],
ID["EPSG",8805]],
PARAMETER["False easting",500000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["easting",east,
ORDER[1],
LENGTHUNIT["metre",1]],
AXIS["northing",north,
ORDER[2],
LENGTHUNIT["metre",1]],
ID["EPSG",4547]]
Data axis to CRS axis mapping: 1,2
Origin = (474514.743200000375509,2527758.696948537603021)
Pixel Size = (250.000000000000000,-250.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 474514.743, 2527758.697) (113d45' 6.10"E, 22d50'53.86"N)
Lower Left ( 474514.743, 2485258.697) (113d45' 8.58"E, 22d27'52.25"N)
Upper Right ( 543014.743, 2527758.697) (114d25' 8.74"E, 22d50'52.57"N)
Lower Right ( 543014.743, 2485258.697) (114d25' 4.55"E, 22d27'50.98"N)
Center ( 508764.743, 2506508.697) (114d 5' 7.00"E, 22d39'23.67"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
Min=0.000 Max=688.500
Minimum=0.000, Maximum=688.500, Mean=26.463, StdDev=32.489
NoData Value=-3.4028234663852886e+38
Metadata:
RepresentationType=ATHEMATIC
STATISTICS_MAXIMUM=688.5
STATISTICS_MEAN=26.463243982434
STATISTICS_MINIMUM=0
STATISTICS_SKIPFACTORX=1
STATISTICS_SKIPFACTORY=1
STATISTICS_STDDEV=32.488883884369

[qqmyers]

There are a lot of things that could be done. At a minimum, Dataverse shouldn't send 0 byte thumbnails and should use the default icon instead. More usefully, I found that ImageMagick and GraphMagick (#8014), with the latter producing a clearer thumb in my quick test, can create a png thumb from these files, so using it instead of the jai-imageio library in these cases (when jai has an error/produces 0 bytes) could help. (FWIW: I also tried updating to the latest jai version - that didn't help).

[PaulBoon]

Hi, got the same problem on our dataverse.nl instance (v5.6), where a recently published dataset with geoTiff files destroys the layout of the root verse (home) page which is annoying.

When uploading one of this geoTiff files on a 5.10.1 version it seems to be fixed, but I can't find any info about that.

Anyways, the only thing that helped was overwriting the tumbs with another image. Only did that for the datasets (thumb48) which need a 48x48 PNG image which I took from a screendump.
Maybe others find this useful.

After upgrading to 5.10.1 is there a way to force regeneration of the thumbs via an API, or should I just remove them on disk @pdurbin ?

[pdurbin]

@PaulBoon a few things...

There are some APIs for managing thumbnails but they aren't documented. Please feel free to create an issue about this! For now, please see https://groups.google.com/g/dataverse-community/c/goFs0D61es4/m/B4pLV-_lAwAJ

There are a few related issues you might be interested in:

• Feature Request/Idea: API to delete file thumbnails #8167
• Feature Request/Idea: add superuser api to upload or generate thumbnails for image files of arbitrary size #8664

With regard to removing the thumbnails on disk, if it works, please go for it! Thumbnails are regenerated when they don't exist. You can even generate thumbnails of arbitrary size. See imageThumb=64 at https://guides.dataverse.org/en/5.11.1/api/dataaccess.html

[mreekie]

grooming:

• discussed in NetCDF meeting.
• not in the queue. Not adding tag.

[cmbz]

2024/08/26: @pdurbin could you please look at this issue and let us know if it should be closed or we should keep it open to help support other geospatial needs (e.g., CAFE).

[pdurbin]

@cmbz I'll close it.

Folks in the community, please feel free to open a fresh geotiff issue if you like. And please link to it from here so it can be found! Thanks. ❤️