Replace Layer for raster creates a new dataset intead of replace the previous one

[mattiagiupponi]

I'm trying to use the GeoServer importer to replace an existing raster layer.
GeoServer UI:

The importer session is correctly created:

{
  "task": {
    "id": 0,
    "href": "http://localhost:8080/geoserver/rest/imports/96/tasks/0",
    "state": "READY",
    "updateMode": "REPLACE",
    "data": {
      "type": "file",
      "format": "GeoTIFF",
      "file": "AC04078710.tif"
    },
    "target": {
      "href": "http://localhost:8080/geoserver/rest/imports/96/tasks/0/target",
      "coverageStore": {
        "name": "AC04078710",
        "type": "GeoTIFF"
      }
    },
    "progress": "http://localhost:8080/geoserver/rest/imports/96/tasks/0/progress",
    "layer": {
      "name": "AC04078710",
      "href": "http://localhost:8080/geoserver/rest/imports/96/tasks/0/layer"
    },
    "transformChain": {
      "type": "raster",
      "transforms": [
        
      ]
    }
  }
}

But when the import is completed, I'm expecting to have my current layer replaced, instead a new one is added:
Note the store and the layer name
Geoserver UI:

importer:

{
  "task": {
    "id": 0,
    "href": "http://localhost:8080/geoserver/rest/imports/96/tasks/0",
    "state": "COMPLETE",
    "updateMode": "REPLACE",
    "data": {
      "type": "file",
      "format": "GeoTIFF",
      "file": "AC04078710.tif"
    },
    "target": {
      "href": "http://localhost:8080/geoserver/rest/imports/96/tasks/0/target",
      "coverageStore": {
        "name": "AC0407871013",
        "type": "GeoTIFF"
      }
    },
    "progress": "http://localhost:8080/geoserver/rest/imports/96/tasks/0/progress",
    "layer": {
      "name": "AC040787100",
      "href": "http://localhost:8080/geoserver/rest/imports/96/tasks/0/layer"
    },
    "transformChain": {
      "type": "raster",
      "transforms": [
        
      ]
    }
  }
}

The only information that I have from the logs is an error related to the SecuredLayerInfo

2022-03-10 16:10:49,517 ERROR [geoserver.rest] - class org.geoserver.security.decorators.SecuredLayerInfo cannot be cast to class org.geoserver.catalog.impl.LayerInfoImpl (org.geoserver.security.decorators.SecuredLayerInfo and org.geoserver.catalog.impl.LayerInfoImpl are in unnamed module of loader org.apache.catalina.loader.ParallelWebappClassLoader @7e242b4d)
java.lang.ClassCastException: class org.geoserver.security.decorators.SecuredLayerInfo cannot be cast to class org.geoserver.catalog.impl.LayerInfoImpl (org.geoserver.security.decorators.SecuredLayerInfo and org.geoserver.catalog.impl.LayerInfoImpl are in unnamed module of loader org.apache.catalina.loader.ParallelWebappClassLoader @7e242b4d)
        at org.geoserver.importer.rest.ImportTaskController.updateLayer(ImportTaskController.java:523)
        at org.geoserver.importer.rest.ImportTaskController.handleTaskPut(ImportTaskController.java:433)
        at org.geoserver.importer.rest.ImportTaskController.lambda$taskPut$4(ImportTaskController.java:208)
        at org.geoserver.importer.rest.converters.ImportWrapperMessageConverter.writeJSON(ImportWrapperMessageConverter.java:84)
        at org.geoserver.importer.rest.converters.ImportWrapperMessageConverter.writeInternal(ImportWrapperMessageConverter.java:66)
        at org.geoserver.importer.rest.converters.ImportWrapperMessageConverter.writeInternal(ImportWrapperMessageConverter.java:26)
        at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227)
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:284)

NOTE: Instead, the same approach works smoothly with vector datasets

cc: @afabiani @giohappy

[giohappy]

@aaime is gs_importer's REPLACE method supposed to work the same for feature types and coverages? The documentation doesn't tell anything in this regard.
If it's supposed to work we need to do an investigation to understand what's going wrong here.

[mattiagiupponi]

I just want to add that this could be useful.
Without providing a target_store to the importer it creates a new one. If provided GeoServer raise the errors in attach
error.log
It looks like that try to add the geotiff to a mosaic

[aaime]

As a summary:

• the importer has no idea what UpdateMode = REPLACE is, for the raster import path
• there is no general API to replace a raster (like we have it for vectors), it's new ground, needs coding
• we can try to do something similar to the CoverageFileController#coverageStorePut which uploads the file, and then replaces the file the store points to, to the new file. I don't see anything related to cleaning up the old file though (which might be dangerous).

I'm guessing it's somewhere between 1 and 2 days of work.

[giohappy]

@aaime given the estimate I would do it. Could you open an issue for the implementation?
Do you think the missing cleanup you were mentioning could be included?

[aaime]

@giohappy whoever implements the code will open a Jira ticket for it.

About the cleanup, it should be optional. A few thoughts about it:

• We don't know if the original file was also uploaded, and if it should be kept or not. So a parameter "purge=true" or something like that should be added to control removal of the file.
• The endpoint controls all sorts of file-based rasters (e.g., not mosaics), there is no API to remove their contents... I'm guessing we could use some heuristics to fine the eventual sidecar files (world, projection, masks)... or we create a new API for it, which would be better, but would also cause backport issues. Also, some readers can point to blobstore files, not sure we can do anything about them (again, no deletion in the API).

[giohappy]

ok @aaime thx.
@nmco when do you think this work could be scheduled?

[nmco]

Hi @giohappy, at this stage I would ask @taba90 to have a look Monday and if the estimate still stands, we should be able to have it ready by the end of next week (18th of March 20212), beginning the week after the latest.

[giohappy]

fine @nmco thanks

[giohappy]

@nmco @taba90 news?

[taba90]

My apologizes for the delay on this @giohappy.
Here the final implementation plan and estimate.
Functional aspects:

• we are going to add support for UPDATE mode REPLACE. The update will replace the file the store points to with the imported one.
• a parameter purge true should be supported in order to remove the old files.

Implementation aspect:

• this part will be refactored to a separate method called loadIntoCoverageStore.
• if the prune flag was specified the old files needs to be deleted. We will retrieve all the coverageInfo trough the reader, and pickup the files using the getInfo method of the AbstractGridCoverage2DReader.
• The loading process will replace the file url in the CoverageInfo object as it is done here and will configure the various CoverageInfo.
• If everything went fine and the prune option was set we can then delete the old files.

I'm not that match expert neither of raster data neither of the importer so If I've to do the task I would need around 2,5 days to be on safe side.

[giohappy]

ok @taba90 let's do it. Esitmated delivery?

[nmco]

@dromagnoli will work on this, ETA 1st of APRIL, wors case 6th of April.

We are targeting GeoServer 2.20.x, branch we need to check where this would land.

[dromagnoli]

Hi @mattiagiupponi, @giohappy
I'm going to check this issue next week and proceed with the suggested implementaion.

Could you please provide me some steps (and eventually a few samples) to replicate the issue?
I.e. :

• initial coveragestore example (is it a mosaic?) and sample to be re-imported
• REST calls to be performed to involve the interesting part.

So I can start full speed on Monday.
(I'm sorry but I didn't use importer very often... probably only used once the past year for another debug)

Thanks

[mattiagiupponi]

Hi @dromagnoli
I mainly interface with GeoServer and the importer through GeoNode (the replace is the development in progress), so I don't have some API calls to forward to you. I don't know if @giohappy has some Postman collection ready.

[dromagnoli]

Hi @giohappy. any comment on this?

[giohappy]

@dromagnoli as we spoke on the chat the way GeoNode executes the import is:

• create the import (initial POST)
• execute a multipart POST to send the files
• configure the task with either CREATE, APPEND or REPLACE
• trigger the execution by calling the import endpoint with the ?sync=true&exec=true params

[dromagnoli]

Hi @giohappy. Thanks
And the purge parameter would be an additional param of the request, right?

i.e.
?sync=true&exec=true&purge=true

[giohappy]

@dromagnoli I suppose yes. This was the proposal from @aaime and @taba90

[dromagnoli]

Hi @giohappy, after some sync up with Andrea it turned out that the javadoc for class UpdateMode.REPLACE says:

The target DataStore will be removed and replaced with the specified input.

And REST documentation states that again:

Changing it to “REPLACE” instead will preserve the layer, but remove the old contents and replace them with the newly uploaded ones

So we were wondering if we actually need the purge parameter or we can just remove the old content for consistency.

Also note that the nativeName of a layer is based on the file name itself so in order to have same names for the layer we also need to have same file in input. When replacing a file in a store, we are going to send the same file name of the original store right? (eventually with a different content... i.e. the previous file has been badly preprocessed and we want just to update the content)

[giohappy]

@dromagnoli I agree that purge is redundant.

Regarding the file name let's hear @mattiagiupponi, but I think it's ok to assume the same name.

[mattiagiupponi]

Regarding the file name let's hear @mattiagiupponi, but I think it's ok to assume the same name.

It's ok, already from UI we force the user to use the same filename
https://github.com/GeoNode/geonode/blob/070bdbe31ad312dd569e1afaa2ea310cd931796f/geonode/layers/utils.py#L1158

[dromagnoli]

Hi guys.
quick question that came to my mind when updating the documentation:

reading the documentation:
Other possible values are “REPLACE”, that is, delete the existing features in the target layer and replace them with the task source ones. For vectorial entities I think that it makes sense: so you provide a new shapefile (containing new features) and you want to remove the old ones in the layer and replace them with the new features imported.

However, I think that for rasters, when dealing with the Structured GridCoverageReader (i.e. ImageMosaic) the behaviour should keep using the harvesting path. So that when sending a new file (even using a REPLACE) only that file will affect the ImageMosaic update. I think that it could be very dangerous and unexpected if a REPLACE of a data file into an ImageMosaic layer would do the remove of everything else but the new data.

Do you agree?

[giohappy]

Definetely @dromagnoli. The full replace is meant only for single coverages.

[dromagnoli]

@giohappy, @mattiagiupponi while waiting for the PR review, here, a testing jar based on GS 2.19.x to early testing of the functionality

[giohappy]

@dromagnoli as agreed with @simboss we want to backport this to 2.18.x.

[dromagnoli]

The PR has been merged to main.
Here, a version based on 2.19.x. I'm going to backport the change to 2.20.x, 2.19.x and 2.18.x as soon as you manage to test and confirm it works
gs-importer-core-2.19-SNAPSHOT.jar.zip

[mattiagiupponi]

@dromagnoli @giohappy
I made some tests, but looks like the "replace" for raster data is not working.

The importer session is the following:

{
  "task": {
    "id": 0,
    "href": "http://localhost:8080/geoserver/rest/imports/379/tasks/0",
    "state": "ERROR",
    "updateMode": "REPLACE",
    "data": {
      "type": "file",
      "format": "GeoTIFF",
      "file": "csdi_r1_p1.tif"
    },
    "target": {
      "href": "http://localhost:8080/geoserver/rest/imports/379/tasks/0/target",
      "coverageStore": {
        "name": "csdi_r1_p1",
        "type": "GeoTIFF"
      }
    },
    "progress": "http://localhost:8080/geoserver/rest/imports/379/tasks/0/progress",
    "layer": {
      "name": "csdi_r1_p1",
      "href": "http://localhost:8080/geoserver/rest/imports/379/tasks/0/layer"
    },
    "errorMessage": "java.io.FileNotFoundException: /opt/data/geoserver_data/geonode/importer_data/tmp6134601418283653642/csdi_r1_p1.tif (No such file or directory)\n/opt/data/geoserver_data/geonode/importer_data/tmp6134601418283653642/csdi_r1_p1.tif (No such file or directory)",
    "transformChain": {
      "type": "raster",
      "transforms": [
     
      ]
    }
  }
}

Attach the geoserver.log with the error raised.

This is the XML provided by the WMS service when clicking on the Layer Preview:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE ServiceExceptionReport SYSTEM "http://localhost:8080/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1" >   
    <ServiceException>
      java.lang.IllegalArgumentException: java.io.FileNotFoundException: /opt/data/geoserver_data/geonode/importer_data/tmp6134601418283653642/csdi_r1_p1.tif (No such file or directory)
java.io.FileNotFoundException: /opt/data/geoserver_data/geonode/importer_data/tmp6134601418283653642/csdi_r1_p1.tif (No such file or directory)
/opt/data/geoserver_data/geonode/importer_data/tmp6134601418283653642/csdi_r1_p1.tif (No such file or directory)
    </ServiceException>
</ServiceExceptionReport>

[mattiagiupponi]

I made another test, it looks like that GeoServer keeps the file path as a new source for the Tiff image, if deleted it raises an error.

Test:
Step 1 - Upload from GeoNode
GeoServer Importer session:

{
  "task": {
    "id": 0,
    "href": "http://localhost:8080/geoserver/rest/imports/380/tasks/0",
    "state": "COMPLETE",
    "updateMode": "CREATE",
    "data": {
      "type": "file",
      "format": "GeoTIFF",
      "file": "cwd_r1_p1.tif"
    },
    "target": {
      "href": "http://localhost:8080/geoserver/rest/imports/380/tasks/0/target",
      "coverageStore": {
        "name": "cwd_r1_p1",
        "type": "GeoTIFF"
      }
    },
    "progress": "http://localhost:8080/geoserver/rest/imports/380/tasks/0/progress",
    "layer": {
      "name": "cwd_r1_p1",
      "href": "http://localhost:8080/geoserver/rest/imports/380/tasks/0/layer"
    },
    "transformChain": {
      "type": "raster",
      "transforms": [
        
      ]
    }
  }
}

On the GeoServer data dir, the tiff file is present:

root@DESKTOP-J0K8ERH:/opt/data/geoserver_data/geonode# find | grep tif
./importer_data/tmp10150666626376475118/cwd_r1_p1.tif

Step 2 - Replace Layer

GeoServer importer say that everything is fine

{
  "task": {
    "id": 0,
    "href": "http://localhost:8080/geoserver/rest/imports/381/tasks/0",
    "state": "COMPLETE",
    "updateMode": "REPLACE",
    "data": {
      "type": "file",
      "format": "GeoTIFF",
      "file": "cwd_r1_p1.tif"
    },
    "target": {
      "href": "http://localhost:8080/geoserver/rest/imports/381/tasks/0/target",
      "coverageStore": {
        "name": "cwd_r1_p1",
        "type": "GeoTIFF"
      }
    },
    "progress": "http://localhost:8080/geoserver/rest/imports/381/tasks/0/progress",
    "layer": {
      "name": "cwd_r1_p1",
      "href": "http://localhost:8080/geoserver/rest/imports/381/tasks/0/layer"
    },
    "transformChain": {
      "type": "raster",
      "transforms": [
        
      ]
    }
  }
}

The Image is deleted from the geoserver data_dir
root@DESKTOP-J0K8ERH:/opt/data/geoserver_data/geonode# find | grep tif
We don't persist the TIFF that we use for the replace, since are temporary files, so when we want to preview the layer, it raises a

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE ServiceExceptionReport SYSTEM "http://localhost:8080/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd"> <ServiceExceptionReport version="1.1.1" >   <ServiceException>
      java.lang.IllegalArgumentException: java.io.FileNotFoundException: /opt/data/geoserver_data/geonode/importer_data/tmp14320844651562588145/cwd_r1_p1.tif (No such file or directory)
java.io.FileNotFoundException: /opt/data/geoserver_data/geonode/importer_data/tmp14320844651562588145/cwd_r1_p1.tif (No such file or directory)
/opt/data/geoserver_data/geonode/importer_data/tmp14320844651562588145/cwd_r1_p1.tif (No such file or directory)
</ServiceException></ServiceExceptionReport>

[simboss]

@mattiagiupponi I asked @aaime to double check this as @dromagnoli is busy with a training.

@aaime might check with you to make sure the expectations are right.
@etj @giohappy keep an eye on this.

[mattiagiupponi]

@dromagnoli as requested follows the call made by GeoNode as post_multipart

url - >http://{geoserver_location}/geoserver/rest/imports/{import_id}/tasks?expand=3
example -> http://localhost:8080/geoserver/rest/imports/385/tasks?expand=3

Method: POST
Content-Type: 'multipart/form-data; boundary=----------ThIs_Is_tHe_bouNdaRY_$'

Payload in case of Tiff image:
------------ThIs_Is_tHe_bouNdaRY_$\r\nContent-Disposition: form-data; name="{absolute filepath in geonode}"; filename="{file_name with extension}"\r\nContent-Type: {image type}{open_imaged decoded in ISO-8859-1}------------ThIs_Is_tHe_bouNdaRY_$--\r\n'

Follows a python example to upload a file:

import requests

url = 'http://localhost:8080/geoserver/rest/imports/387/tasks?expand=3'
data = '------------ThIs_Is_tHe_bouNdaRY_$\r\nContent-Disposition: form-data; name="/opt/core/geonode/geonode/uploaded/tmp14_mbefq/test_grid.tif"; filename="test_grid.tif"\r\nContent-Type: image/tiff\r\n\r\nII*\x00\x08\x00\x00\x00\x10\x00\x00\x01\x03\x00\x01\x00\x00\x00\x05\x00\x00\x00\x01\x01\x03\x00\x01\x00\x00\x00\x07\x00\x00\x00\x02\x01\x03\x00\x01\x00\x00\x00@\x00\x00\x00\x03\x01\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00\x06\x01\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00\x11\x01\x04\x00\x01\x00\x00\x00t\x01\x00\x00\x15\x01\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00\x16\x01\x03\x00\x01\x00\x00\x00\x07\x00\x00\x00\x17\x01\x04\x00\x01\x00\x00\x00\x18\x01\x00\x00\x1c\x01\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00S\x01\x03\x00\x01\x00\x00\x00\x03\x00\x00\x00\x0e\x83\x0c\x00\x03\x00\x00\x00Î\x00\x00\x00\x82\x84\x0c\x00\x06\x00\x00\x00æ\x00\x00\x00¯\x87\x03\x00 \x00\x00\x00\x16\x01\x00\x00°\x87\x0c\x00\x02\x00\x00\x00V\x01\x00\x00±\x87\x02\x00\x0e\x00\x00\x00f\x01\x00\x00\x00\x00\x00\x00$Ó\x18â\x95z\x9f?$Ó\x18â\x95z\x9f?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00w¾\x9f\x1a/=X@L¾*¥Ô6\x15À\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x07\x00\x00\x04\x00\x00\x01\x00\x02\x00\x01\x04\x00\x00\x01\x00\x01\x00\x00\x08\x00\x00\x01\x00æ\x10\x01\x08±\x87\r\x00\x00\x00\x06\x08\x00\x00\x01\x00\x8e#\t\x08°\x87\x01\x00\x01\x00\x0b\x08°\x87\x01\x00\x00\x00\x88mt\x96\x1d¤r@\x00\x00\x00@¦TXAGCS_WGS_1984|\x00\x00\x00\x00àHhI@\x00\x00\x00@\x94>I@\x00\x00\x00\x00_=I@\x00\x00\x00ÀY\nI@\x00\x00\x00 £\x16I@\x00\x00\x00\x00\x00\x00\x08@\x00\x00\x00\x00\x80\x87ÃÀ\x00\x00\x00\x00\x80\x87ÃÀ\x00\x00\x00\x00\x80\x87ÃÀ\x00\x00\x00\x80\x8bzI@\x00\x00\x00 A\x10I@\x00\x00\x00@ùLIÀ\x00\x00\x00@¹EI@\x00\x00\x00\x80ÛsI@\x00\x00\x00 ö!I@\x00\x00\x00\x80étI@\x00\x00\x00\x00\x00\x00.À\x00\x00\x00 ÛRI@\x00\x00\x00\xa0RDI@\x00\x00\x00\xa0XvI@\x00\x00\x00 ñPI@\x00\x00\x00 \x08/I@\x00\x00\x00\x00\x9eEI@\x00\x00\x00\x00\x80\x87ÃÀ\x00\x00\x00\x80¤\x1cI@\x00\x00\x00\x00\x80\x87ÃÀ\x00\x00\x00@v~I@\x00\x00\x00@K\\I@\x00\x00\x00 ´iI@\x00\x00\x00\x00Ñ/I@\x00\x00\x00à¥#I@\x00\x00\x00\xa0Ô\x04I@\x00\x00\x00 áBI@\x00\x00\x00ÀF+I@\x00\x00\x00@3\x0bI@\r\n------------ThIs_Is_tHe_bouNdaRY_$--\r\n'
headers = {'Content-Type': 'multipart/form-data; boundary=----------ThIs_Is_tHe_bouNdaRY_$'}

response = requests.post(url, data=data, headers=headers, auth=('admin', 'geoserver'))
print(response.status_code)

FYI @aaime @etj @giohappy

[dromagnoli]

@mattiagiupponi , @giohappy

Tentative fix doing a copy of the data file so that the replacing file is internally stored in the dedicated GeoServer datadir/importer uploads folder.

If it's still not working as you expect, please make sure to provide sample calls to be sent via POST against GeoServer to exactly simulate the rest calls made by GeoNode / python scripts. Calls including import creation, data upload and data replace.

gs-importer-core-2.19-SNAPSHOT.jar.zip

[mattiagiupponi]

@dromagnoli @giohappy @etj
Good news and bad news:

• Good news: the "replace" is working as expected, the file is uploaded in the directory and is not relying anymore on the original file.
• Bad news: when a "replace" is performed, a new file is uploaded, but the original file is not deleted and is persisted in the upload dir.

Example:
The first file upload will create correctly the new image in the importer_data folder

tomcat@DESKTOP-J0K8ERH:/opt/data/geoserver_data$ find . | grep csdi_r1_p1.tif
./geonode/importer_data/tmp13450636897756784357/csdi_r1_p1.tif

When running the "Replace", a new file is uploaded and the one created during the upload phase is persisted:

tomcat@DESKTOP-J0K8ERH:/opt/data/geoserver_data$ find . | grep csdi_r1_p1.tif
./geonode/importer_data/tmp15207103194059995/csdi_r1_p1.tif
./geonode/importer_data/tmp9657244487043802753/csdi_r1_p1.tif

NB: if I run another replace, the file is correctly substituted

[dromagnoli]

After final internal sync up on Friday, it looks like it's working as expected. Starting backports

[giohappy]

@dromagnoli @mattiagiupponi what about the replace not removing the original file? Did you solve it somehow?

[mattiagiupponi]

@dromagnoli @mattiagiupponi what about the replace not removing the original file? Did you solve it somehow?

was an issue related to my local environment. I clean-up my local env, retry the replace, and worked as expected