Implement geo upload #47

mtiessen1175 · 2023-11-30T13:49:57Z

references #44

Implement the structure for the upload mechanism in the volume-edit view. Allow the upload of both - browsing layers and context layers - i.e. upload of geoTIFF and (iFDO) metadata, linkage of WMS.

Additional layers can be added in the volume edit view
An additional layer can be a georeferenced file (geoTIFF) or a link to a WMS source
- geoTIFF upload done (but should be tested more thoroughly by adding more example .tif files, e.g. a geoTIFF using the ModelTransformationTag. Corresponding code implemented here. Update: Tests fully cover this case; see commits cafb808 and bb6b2c6)
- WMS
Layers can be ordered in the volume edit view
Each layer can either be a "browsing layer" (Geospatial browsing visualisation #44) or a "context layer" or both
The filter-map shows all configured "browsing" layers
Option to upload the full image extent as metadata (also from iFDO, see Geospatial context fusion display #45)

associated PR's:

…rmat.

mzur · 2024-10-23T14:36:11Z

This is most certainly due to the transformation error in the proj4php package, as discussed #47 (comment).

Did you find the overlay? Where do I have to look? 😃

Yes, I created a normalization of the color-band (as geoTIFF's often contain more color-information that can be set/adjusted manually by the user). So the #47 (comment) solution for now was to normalize the color-band to 0 - 255 in case it differs from that range (see code).

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

mtiessen1175 · 2024-10-23T14:45:01Z

Did you find the overlay? Where do I have to look? 😃

Ahh, yeah it is a fairly small example-TIFF! The coordinates in EPSG:3857 are: -325165.2, 7779955.1 (Loch Davan close to Dinnet, Scotland)

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

I see, so I should check what goes wrong in the normalization.

mtiessen1175 · 2024-10-28T10:15:21Z

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

I see, so I should check what goes wrong in the normalization.

The formula used for image normalization should be working. It seems that the derived min- and max-values from libvips are not providing the correct min/max colour-band values as I expected them to be (i.e. they are different to what I see in the QGIS colour settings). Moreover, I tested with two different geoTIFF's and surprisingly the min/max of the two geoTIFF's were identical (min: -3.4028230607371E+38, max: 162.15870666504).
So I need to find a different approach to get the min/max colour-value information from the image.

mzur · 2024-10-31T08:56:12Z

You should try it with the Vips CLI too. If this does not show the expected values you can ask in the Vips repo. The maintainer is extremely responsive and helpful if you provide a good description of the problem and an example file.

mtiessen1175 · 2024-11-04T15:42:20Z

You should try it with the Vips CLI too. If this does not show the expected values you can ask in the Vips repo. The maintainer is extremely responsive and helpful if you provide a good description of the problem and an example file.

I tried with the CLI and get the correct max() value but an incorrect min(). This must be due to the NoData values of the geoTIFF images (the black frame around the geoTIFF when converting it to an image). So the min() always returns this value (e.g. -99999) instead of the desired min-value (e.g. -4123.31).

mzur · 2024-11-05T09:02:05Z

(please use @mzur, otherwise I don't get notified of activity here)

If this can be fixed by creating the geoTIFF in any other way then we could add an explanation to the manual and just expect geoTIFFs with the correct format. Then there is no need for special handling of edge cases (for now).

mzur · 2024-11-06T09:43:17Z

I tested the WMS now and it works fine.

I noticed an error on the geo map of a volume (not the map in the filter modal). When I open the label trees tab there, it says Error in event handler for "sidebar.toggle": "TypeError: this.map is undefined".

…nated single arithmetic operations.

mtiessen1175 · 2024-11-12T12:02:49Z

@mzur,

If this can be fixed by creating the geoTIFF in any other way then we could add an explanation to the manual and just expect geoTIFFs with the correct format. Then there is no need for special handling of edge cases (for now).

I've implemented the color normalization such that NoData values of -99999 will get excluded in order to calculate the true min (see conversation on libvips discussion). I therefore would add an excerpt to the manual stating that color-band normalization should either be done beforehand or the NoData values of the geoTIFF should be set to -99999 in order to be handled correctly by BIIGLE.

mzur · 2024-11-12T14:22:09Z

Sounds good 👍

mtiessen1175 · 2024-11-12T14:59:42Z

@mzur,

I noticed an error on the geo map of a volume (not the map in the filter modal). When I open the label trees tab there, it says Error in event handler for "sidebar.toggle": "TypeError: this.map is undefined".

That's fixed now.

When I upload a GeoTIFF, I get JS errors (see below). The upload seems to succeed, though.
14:31:59.876 [Vue warn]: $listeners is readonly.

found in

---> <Portal>
       <Tab>
         <Tabs>
           <Root> volumes.js:3924:15 

Regarding the latter, I tried to find where the warning is coming from by debugging the code, but I couldn't figure out what causes it. It is still unclear to me...

where the readonly is allegedly violated and
to what kind of object the attributes refer to

I looked into the whole upload process of the geoTIFF files, comprised of:

and

geoTiffOverlayController.php

commenting lines step by step to see where the warning stems from, but it's still not obvious to me. Do you have any idea or instinct what could cause such warning?

See: #47 (review)

mzur · 2024-12-03T14:30:22Z

The error came from the Portal component which is used by the uiv Tab. The portal imports Vue but since biigle/geo compiles it's own JS package, this was a separate Vue instance than the one of biigle/core, which caused the error. To fix this, biigle/geo now uses the Tab and Tabs components exported from biigle/core, which use the correct Vue instance.

The issue is described here: LinusBorg/portal-vue#201

mtiessen1175 · 2024-12-03T17:40:35Z

The error came from the Portal component which is used by the uiv Tab. The portal imports Vue but since biigle/geo compiles it's own JS package, this was a separate Vue instance than the one of biigle/core, which caused the error. To fix this, biigle/geo now uses the Tab and Tabs components exported from biigle/core, which use the correct Vue instance.

The issue is described here: LinusBorg/portal-vue#201

Thanks for the help and clarification on this issue! 👍

mzur · 2024-12-13T10:00:06Z

src/resources/assets/js/volumes/components/webmapOverlayForm.vue

@@ -46,7 +42,7 @@ export default {
        data.append('volumeId', this.volumeId);
        geoApi.saveWebMap({id: this.volumeId}, data)
          .then(this.handleSuccess, this.handleError)


I currently get a 500 server error if the WMS URL is invalid.

The 500 error should not occur. instead it should return a 422 validation error.

Even if a 500 occurs, an error message should be shown in the UI.

mzur · 2024-12-13T10:04:46Z

src/Jobs/TileSingleOverlay.php

@@ -104,7 +104,7 @@ protected function getVipsImage($path)
     */
    protected function imageNormalization($vipsImage, $min, $max)
    {
-        // band intensity normalization x' = (x - $min * (255 / ($max - $min))


I still get only black&white tiles.

BIIGLE:

QGIS:

Is this because of the "incorrect" preprocessing of the TIFF we talked about?

Also, the position is also not 100% correct but I guess this is the error we talked about and which can't be fixed.

mzur · 2024-12-13T10:20:56Z

tests/Jobs/TileSingleOverlayTest.php

This produces undesired console output from processing actual files. Maybe this can be mocked?

mzur · 2024-12-13T10:28:38Z

src/Database/migrations/2024_07_30_155058_add_geo_overlays_table.php

+            $table->foreign('volume_id')
+                  ->references('id')
+                  ->on('volumes')
+                  ->onDelete('cascade');


We need an event handler/observer that deletes the geo overlay files when the whole volume or project is deleted.

mzur · 2024-12-13T10:32:57Z

src/GeoServiceProvider.php

@@ -40,10 +40,14 @@ public function boot(Modules $modules, Router $router)
            'viewMixins' => [
                'imagesIndex',
                'manualTutorial',
+                'manualVolumes',


There is not manualVolumes mixin location on the manual index page. It can be added in biigle/core#725, though.

There is also the volume-map manual article. The description of the geo filter there is no longer correct.

mzur · 2024-12-17T11:11:16Z

src/routes.php

+    $router->get('geo-overlays/{id}/file', [
+        'uses' => 'GeoOverlayController@showFile',
+    ]);


Where is this used?

mzur · 2024-12-17T11:12:12Z

tests/GeoOverlayTest.php

+        $faker = \Faker\Factory::create();
+        $model = new GeoOverlay;
+        $model->name = $faker->company();
+        $model->volume_id = VolumeTest::create()->id;
+        $model->type = 'geotiff';
+        $model->browsing_layer = false;
+        $model->context_layer = false;
+        $model->layer_index = null;
+        $model->attrs = [
+            'top_left_lng' => $faker->randomFloat(),
+            'top_left_lat' => $faker->randomFloat(),
+            'bottom_right_lng' => $faker->randomFloat(),
+            'bottom_right_lat' => $faker->randomFloat(),
+            'width' => $faker->randomNumber(), 
+            'height' => $faker->randomNumber()
+        ];
+        $model->save();


Better define a factory for the model.

mzur · 2024-12-17T11:12:48Z

tests/GeoOverlayTest.php

+        $model = self::createGeotiffOverlay();
+        $model->volume->delete();
+        $this->assertNull($model->fresh());


Also test if the overlay files are removed.

mzur · 2024-12-17T12:45:10Z

tests/Http/Controllers/Api/GeoTiffOverlayControllerTest.php

These tests are quite slow because they process actual files. Maybe this can also be mocked (e.g. implement a method "read exif" that returns static data in the test without opening an actual file)?

mzur · 2024-12-17T12:46:00Z

tests/Http/Controllers/Api/WebMapOverlayControllerTest.php

Make sure that this test does not actually send HTTP requests!

mtiessen1175 added 20 commits November 22, 2023 14:17

include basic volumesEditRight mixin in geo

c0aa459

add basic geoOverlay model and controller

a71b958

Use old geo-overlay structure

0746ebe

fill geoOverlay model with functions

444cbd1

configure storage-disk for geo-overlays

d676299

fix uiv Tab(s) import

d4bcc71

rename geoOverlay to GeoOverlay

a073726

Add routes and controller structure for volume-geo-overlay

34c92bf

added some missing imports

2b4ebcd

add simple tabs for file upload

947f673

Merge branch 'browsing_vis' into geo_edit

2638389

commit production ready files

95706ba

create new geoTIFF component

d686aa2

include production files

ee70bfa

edit text and names of overlay upload-tabs

6a73c5e

Modified upload function.

fb78c78

Include geotiff.js for testing

89af5cf

added transform-coordinates package to transform any CRS to WGS 84 fo…

df0ad5c

…rmat.

Merge branch 'browsing_vis' into geo_upload

0e772f3

add production files

5761319

Base automatically changed from browsing_vis to master December 8, 2023 14:43

mtiessen1175 added 9 commits December 11, 2023 15:20

remove JS geotiff packages and code

eb718c8

pass "raw" geotiff-data to backend

dda9101

Add existing code for geo-overlay retrieval and deletion + test

9df6bec

delete the removal of geo-overlays-table

351c681

add test for VolumeGeoOverlayController

fdc0e79

Linting and fixing files

7004442

Test native php exif function

c2d9cfd

test native php function for identifying geotiff-tagnames

38f1aaf

include php exif dependency

52823b0

mtiessen1175 added 2 commits October 23, 2024 16:30

Fix table appearance.

99181d2

remove unnecessary success-messages and api route.

ddf61fe

Fix not existing "loading" variable

b7a13ea

re-write normalization formula

6bd7301

Commented color-band-normalization for testing

b1b0de9

reapply color-band normalization

296f2a9

mtiessen1175 added 2 commits November 12, 2024 12:33

Exclude NoData values from geoTIFF's for calculating correct min

bd3ee4e

Improve color-normalization by using vips_linear() instead of concate…

23c2ae1

…nated single arithmetic operations.

Fixed "this.map is undefined" error on sidebar-toggle

7a48b90

Remove unnecessary variable

8693f2d

mtiessen1175 and others added 2 commits November 13, 2024 19:05

Add color-band normalization note to user-manual.

19d2251

Fix issue with Vue warning

64ec0a0

See: #47 (review)

add test for tile-directory

56d76f9

mtiessen1175 added 3 commits December 3, 2024 18:49

fixed new testcase

c745a5f

remove tests (creation of tiles-dir is not covered in tests)

3b5c206

fixed image normalisation.

b40a81f

mtiessen1175 requested a review from mzur December 4, 2024 12:24

mzur requested changes Dec 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement geo upload #47

Implement geo upload #47

mtiessen1175 commented Nov 30, 2023 •

edited by mzur

Loading

mzur commented Oct 23, 2024

mtiessen1175 commented Oct 23, 2024

mtiessen1175 commented Oct 28, 2024

mzur commented Oct 31, 2024

mtiessen1175 commented Nov 4, 2024

mzur commented Nov 5, 2024

mzur commented Nov 6, 2024

mtiessen1175 commented Nov 12, 2024

mzur commented Nov 12, 2024

mtiessen1175 commented Nov 12, 2024

mzur commented Dec 3, 2024

mtiessen1175 commented Dec 3, 2024

mzur Dec 13, 2024

mzur Dec 13, 2024

mzur Dec 13, 2024

mzur Dec 13, 2024

mzur Dec 13, 2024

mzur Dec 13, 2024

mzur Dec 17, 2024

mzur Dec 17, 2024

mzur Dec 17, 2024

mzur Dec 17, 2024

mzur Dec 17, 2024

Implement geo upload #47

Are you sure you want to change the base?

Implement geo upload #47

Conversation

mtiessen1175 commented Nov 30, 2023 • edited by mzur Loading

mzur commented Oct 23, 2024

mtiessen1175 commented Oct 23, 2024

mtiessen1175 commented Oct 28, 2024

mzur commented Oct 31, 2024

mtiessen1175 commented Nov 4, 2024

mzur commented Nov 5, 2024

mzur commented Nov 6, 2024

mtiessen1175 commented Nov 12, 2024

mzur commented Nov 12, 2024

mtiessen1175 commented Nov 12, 2024

mzur commented Dec 3, 2024

mtiessen1175 commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtiessen1175 commented Nov 30, 2023 •

edited by mzur

Loading