force-level2 - Transformation failed. Computing tile origin in dst_srs failed. Starting datacube failed.

[thielfab]

Hi David,

I wanted to ingest new data and started force-level2 with the new version 3.7.4.
In the log files I get the error
/home/thielf54/edc/level1/landsat/LE07_L1TP_190022_20030327_20200916_02_T1.tar: Transformation failed. Computing tile origin in dst_srs failed. Starting datacube failed.
for every file.
Rolling back to 3.7.3 with same queue and parameter file seems to work just fine.
Did something change inside the docker image on the gdal or PROJ side of things?

FORCE 3.7.4 (dockerhub)
Ubuntu Server 21.10

PROJECTION string from parameter file:

PROJCS["ETRS89-extended / LAEA Europe",GEOGCS["ETRS89",DATUM["European_Terrestrial_Reference_System_1989",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6258"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4258"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["latitude_of_center",52],PARAMETER["longitude_of_center",10],PARAMETER["false_easting",4321000],PARAMETER["false_northing",3210000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3035"]]

passes gdalsrsinfo -V in both containers (3.7.3 and 3.7.4)

[davidfrantz]

Hi Fabian,

no, not really, I just quickly checked the diff between those releases.
There was only one change in the lower level module.
But this is unrelated and won't even be compiled unless using the debug Docker image.

I rebuilt the base image between the versions, though...
https://github.com/davidfrantz/base_image/commits/main

Probably, there is a difference in the GDAL version?

[thielfab]

Guess I could have checked that earlier, GDAL and proj versions are the same 🙈
Well, I will report when I figure out what is/was going on. If you say that level2 module or the datacubing wasn't touched and gdal versions are the same, it should work (in theory).
Lets see... running 3.7.3 is fine for now 👍

$ singularity exec ~/force_3.7.4.sif gdalinfo --version             
GDAL 3.0.4, released 2020/01/28                                                                        
$ singularity exec ~/force_3.7.3.sif gdalinfo --version                 
GDAL 3.0.4, released 2020/01/28                                                                        
$ singularity exec ~/force_3.7.4.sif apt list --installed | grep libproj                                                                                                      
libproj-dev/focal,now 6.3.1-1 amd64 [installed,automatic]                                              
libproj15/focal,now 6.3.1-1 amd64 [installed,automatic]                                                
$ singularity exec ~/force_3.7.3.sif apt list --installed | grep libproj                                                          
libproj-dev/focal,now 6.3.1-1 amd64 [installed,automatic]                                              
libproj15/focal,now 6.3.1-1 amd64 [installed,automatic]                                                

[davidfrantz]

hmm, if the GDAL/OGR versions are the same, I wonder what causes this...

You can compare releases if you click on the release on GitHub.
Probably, you can spot a difference there that I missed.

[thielfab]

I don't know if it is related to the issue, but I get the following errors using gdalsrsinfo and gdaltransform. In the latter case it also happens using the WKT strings instead of EPSG codes.
Can you reproduce this in the davidfrantz/force:3.7.4 docker image?

$ gdalsrsinfo -e EPSG:3035

ERROR 1: PROJ: proj_create_from_database: SQLite error on SELECT name, coordinate_system_auth_name, coordinate_system_code, geodetic_crs_auth_name, geodetic_crs_code, conversion_auth_name, conversion_code, area_of_use_auth_name, area_of_use_code, text_definition, deprecated FROM projected_crs WHERE auth_name = ? AND code = ?: no such column: area_of_use_auth_name ERROR 1: ERROR - failed to load SRS definition from EPSG:3035

similar with
$ gdaltransform -s_srs EPSG:4326 -t_srs EPSG:3035

ERROR 1: PROJ: proj_create_from_database: SQLite error on SELECT name, type, coordinate_system_auth_name, coordinate_system_code, datum_auth_name, datum_code, area_of_use_auth_name, area_of_use_code, text_definition, deprecated FROM geodetic_crs WHERE auth_name = ? AND code = ?: no such column: area_of_use_auth_name ERROR 1: Translating source or target SRS failed: EPSG:4326

I am currently processing a new study area, and here the switch back to version 3.7.3 did not do the trick... I get a similar error (compared to inital post)
Transformation failed.computing upper left in dst_srs failed. Starting datacube failed.Starting datacube(s) failed.

[davidfrantz]

Hi @thielfab,

looks good here...

frantz@ahsoka:~$ dforce force -v                                                                                                                                         
FORCE version: 3.7.4                                                                                                                                                     
frantz@ahsoka:~$ dforce gdalsrsinfo -e EPSG:3035                                                                                                                         
                                                                                                                                                                         
EPSG:3035                                                                                                                                                                
                                                                                                                                                                         
PROJ.4 : +proj=laea +lat_0=52 +lon_0=10 +x_0=4321000 +y_0=3210000 +ellps=GRS80 +units=m +no_defs                                                                         
                                                                                                                                                                         
OGC WKT2:2018 :                                                                                                                                                          
PROJCRS["ETRS89-extended / LAEA Europe",                                                                                                                                 
    BASEGEOGCRS["ETRS89",                                                                                                                                                
        DATUM["European Terrestrial Reference System 1989",                                                                                                              
            ELLIPSOID["GRS 1980",6378137,298.257222101,                                                                                                                  
                LENGTHUNIT["metre",1]]],                                                                                                                                 
        PRIMEM["Greenwich",0,                                                                                                                                            
            ANGLEUNIT["degree",0.0174532925199433]],                                                                                                                     
        ID["EPSG",4258]],                                                                                                                                                
    CONVERSION["Europe Equal Area 2001",                                                                                                                                 
        METHOD["Lambert Azimuthal Equal Area",                                                                                                                           
            ID["EPSG",9820]],                                                                                                                                            
        PARAMETER["Latitude of natural origin",52,                                                                                                                       
            ANGLEUNIT["degree",0.0174532925199433],                                                                                                                      
            ID["EPSG",8801]],                                                                                                                                            
        PARAMETER["Longitude of natural origin",10,                                                                                                                      
            ANGLEUNIT["degree",0.0174532925199433],                                                                                                                      
            ID["EPSG",8802]],                                                                                                                                            
        PARAMETER["False easting",4321000,                                                                                                                               
            LENGTHUNIT["metre",1],                                                                                                                                       
            ID["EPSG",8806]],                                                                                                                                            
        PARAMETER["False northing",3210000,                                                                                                                              
            LENGTHUNIT["metre",1],                                                                                                                                       
            ID["EPSG",8807]]],                                                                                                                                           
    CS[Cartesian,2],                                                                                                                                                     
        AXIS["northing (Y)",north,                                                                                                                                       
            ORDER[1],                                                                                                                                                    
            LENGTHUNIT["metre",1]],                                                                                                                                      
        AXIS["easting (X)",east,                                                                                                                                         
            ORDER[2],                                                                                                                                                    
            LENGTHUNIT["metre",1]],                                                                                                                                      
    USAGE[                                                                                                                                                               
        SCOPE["unknown"],                                                                                                                                                
        AREA["Europe - LCC & LAEA"],                                                                                                                                     
        BBOX[24.6,-35.58,84.17,44.83]],                                                                                                                                  
    ID["EPSG",3035]]              

[thielfab]

Thanks for checking David! Also no issues with the force image on my local WSL docker. I also deleted the image + cache and re-pulled it.
Confusing... so maybe there is an issue with the container engine or the kernel on the host system. But that does not make much sense, does it. Aren't containers supposed to circumvent such compatibility issues 😅

[thielfab]

Well, turns out the problem was that I ran the commands from a terminal where my conda gdal environment was loaded (I recently started using VSCode with Remote SSH and I use the gdal env as my default python interpreter, which also loads the conda env in the Terminal). This caused gdal/proj to look for the proj.db from the conda environment instead of the one the container comes with 🤯.

[davidfrantz]

The charming conda again :)

Great that you got to the bottom of it!