Sentinel-1 datacube using force-cube

[henitsl]

I try to create force datacube from Sentinel-1 GRD data. I exported Sentinel-1 data from ESA SNAP in GeoTiff format and converted into Int16 with NoData value -9999 using GDAL. I used force-cube program and a predidefined datacube.prj file that was created by Sentinel-2 force level2 preprocessing.
I used the following command:
force-cube -o /opt/data/sentinel1/level2 -s 10 -n -9999 -t Int16 -b 20220308_LEVEL2_S1AID_SIG /opt/data/sentinel1/processing/20220308_LEVEL2_S1AID_SIG.tif
force-cube successfully created 30 km x 30 km Sentinel-1 image tiles, but lot of image contain only NoData.
Can force-cube automatically delete this file?

[davidfrantz]

Dear @henitsl,

I believe this should happen automatically. Please see the changelog in here:
https://force-eo.readthedocs.io/en/latest/history/v3/v340.html

Can you please check whether the values in your input images correspond to the -9999 nodata value?

Cheers,
David

[ernstste]

Thanks for clarifying. Is the gdalinfo output from the original image or from one of the images that is produced by force-cube?
force-cube checks the output of gdalinfo to determine if an image should be deleted. In the gdalinfo example you posted above it would not delete the image, since the band statistics indicate valid data.

[henitsl]

Yes the gdalinfo output is from original image. Here is a gdalinfo output of the force tile X0022_Y0004. It was created from the original image using force-cube. It contains only nodata.

Driver: GTiff/GeoTIFF
Files: /opt/data/sentinel/level2/X0022_Y0004/20220308_LEVEL2_S1AID_SIG.tif
Size is 3000, 3000
Coordinate System is:
PROJCRS["ETRS89-extended / LAEA Europe",
BASEGEOGCRS["ETRS89",
DATUM["European Terrestrial Reference System 1989",
ELLIPSOID["GRS 1980",6378137,298.257222101,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4258]],
CONVERSION["Europe Equal Area 2001",
METHOD["Lambert Azimuthal Equal Area",
ID["EPSG",9820]],
PARAMETER["Latitude of natural origin",52,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",10,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["False easting",4321000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",3210000,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["northing (Y)",north,
ORDER[1],
LENGTHUNIT["metre",1]],
AXIS["easting (X)",east,
ORDER[2],
LENGTHUNIT["metre",1]],
USAGE[
SCOPE["Statistical analysis."],
AREA["Europe - European Union (EU) countries and candidates. Europe - onshore and offshore: Albania; Andorra; Austria; Belgium; Bosnia and Herzegovina; Bulgaria; Croatia; Cyprus; Czechia; Denmark; Estonia; Faroe Islands; Finland; France; Germany; Gibraltar; Greece; Hungary; Iceland; Ireland; Italy; Kosovo; Latvia; Liechtenstein; Lithuania; Luxembourg; Malta; Monaco; Montenegro; Netherlands; North Macedonia; Norway including Svalbard and Jan Mayen; Poland; Portugal including Madeira and Azores; Romania; San Marino; Serbia; Slovakia; Slovenia; Spain including Canary Islands; Sweden; Switzerland; Turkey; United Kingdom (UK) including Channel Islands and Isle of Man; Vatican City State."],
BBOX[24.6,-35.58,84.17,44.83]],
ID["EPSG",3035]]
Data axis to CRS axis mapping: 2,1
Origin = (5318445.731224000453949,2831323.297503000125289)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 5318445.731, 2831323.298) ( 23d23' 6.84"E, 47d46'46.40"N)
Lower Left ( 5318445.731, 2801323.298) ( 23d18'55.94"E, 47d30'52.55"N)
Upper Right ( 5348445.731, 2831323.298) ( 23d46'46.99"E, 47d43'48.57"N)
Lower Right ( 5348445.731, 2801323.298) ( 23d42'29.13"E, 47d27'55.77"N)
Center ( 5333445.731, 2816323.298) ( 23d32'49.61"E, 47d37'21.50"N)
Band 1 Block=3000x300 Type=Int16, ColorInterp=Gray
ERROR 1: /opt/data/sentinel/level2/X0022_Y0004/20220308_LEVEL2_S1AID_SIG.tif, band 1: Failed to compute statistics, no valid pixels found in sampling.
NoData Value=-9999
Metadata:
STATISTICS_VALID_PERCENT=0
Band 2 Block=3000x300 Type=Int16, ColorInterp=Undefined
ERROR 1: /opt/data/sentinel/level2/X0022_Y0004/20220308_LEVEL2_S1AID_SIG.tif, band 2: Failed to compute statistics, no valid pixels found in sampling.
NoData Value=-9999
Metadata:
STATISTICS_VALID_PERCENT=0

[ernstste]

Okay I see now, thanks.
@davidfrantz I'm not sure what the rationale behind checking against the max value statistic being below 1 was.
If it's just about removing images that only consist of nodata values (this is what the changelog says), this should do the trick (line 155 onwards):

VALID=$($RASTER_INFO_EXE -stats "$FOUT" | grep STATISTICS_VALID_PERCENT | head -n 1 | sed 's/ //g; s/[=,]/ /g' | cut -d ' ' -f2 | sed 's/\..*//')
 rm "$FOUT.aux.xml"
 debug "max: $MAX"

    if [ $VALID -eq 0 ]; then

Thinking about it: I wonder if there would ever be a case where the first band may contain only nodata, why other bands contain valid data.
A more robust way would be to sum up the values of STATISTICS_VALID_PERCENT:

VALID=$($RASTER_INFO_EXE -stats "$FOUT" | grep STATISTICS_VALID_PERCENT | sed 's/ //g; s/[=,]/ /g' | cut -d ' ' -f2 | awk '{sum +=$1} END {print sum != 0}'
 rm "$FOUT.aux.xml"
 debug "max: $MAX"

    if [ $VALID -eq 0 ]; then

[davidfrantz]

hmm, this worked in the past...

Can you please post the GDAL version that is being used?
gdal-config --version

And could you please upload /opt/data/sentinel/level2/X0022_Y0004/20220308_LEVEL2_S1AID_SIG.tif, then I can directly debug on that image.

[henitsl]

My GDAL version: 3.2.1.
FORCE version: 3.7.6.
I uploaded 20220308_LEVEL2_S1AID_SIG.tif and datacube-definition.prj file to this google drive link.
https://drive.google.com/file/d/1wK_hf_7ggL97c-zZbNjq7VxJTCxHf8Lt/view?usp=sharing

[davidfrantz]

Hi both,

I fixed this now in the develop branch. The previous check was only active for vector-to-cube.
It is now also implemented for raster-to-cube.

I also redirected error messages (incl. GDAL ERROR 1) to /dev/null, which should give a clean output when nodata-only images are encountered.

@ernstste. although the previous check would have worked too, I went for your solution as it is much cleaner. Thanks.

Cheers,
David

[ernstste]

That was quick, thanks David.

[henitsl]

Thanks @davidfrantz and @ernstste!
I tested on develop branch and it worked correctly. No more NoData images. :) Minor comment that new code created lot of empty folder but this is not a big problem for me.
Thanks for the quick answers and fix.