2024-10-30-fix-loosing-coordinates-interlis-import #484

sjib · 2024-10-30T16:39:38Z

Add function geometry3D_convert

Reach_point
Cover
Wastewater_node

sjib · 2024-10-30T16:40:41Z

Should solve #475

for more information, see https://pre-commit.ci

… is ok Call with 'Nan' does not work yet

…ttps://github.com/teksi/wastewater into 2024-10-30-fix-loosing-coordinates-interlis-import

for more information, see https://pre-commit.ci

sjib · 2024-10-30T20:01:28Z

Conversion with 'Nan' not yet as expected - result is None

2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000008930444100000000a96331410000000000d49d40.
2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000009d30444100000000ae6331410000000000da9d40.
2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000009330444100000000a96331410000000000d49d40.
2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000808130444100000000a96331410000000000d49d40.
2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000007030444100000000956331410000000000d09d40.
2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a00808000000000000e03344410000000068723141cdcccccccc32a140.
2024-10-30T20:47:07        teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a0080800005eba49ccde334441d122dbf981723141000000000034a140.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01325A- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01325E- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01326A- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01326E- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01327A- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01327E- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01328A- situation3d_geometry z-value set to "Nan" instead: None.
2024-10-30T20:47:07        teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01328E- situation3d_geometry z-value set to "Nan" instead: None.

sjib · 2024-10-30T20:05:20Z

error: cannot format plugin/teksi_wastewater/interlis/interlis_model_mapping/interlis_importer_to_intermediate_schema.py: Cannot parse: 434:91: geom = self.session_tww.scalar(ST_Force3D(coteattribute, COALESCE (NULL,'NaN'))::numeric)

sjib · 2024-10-30T20:07:04Z

@ponceta Any idea how to adapt that?

error: cannot format plugin/teksi_wastewater/interlis/interlis_model_mapping/interlis_importer_to_intermediate_schema.py: Cannot parse: 434:91: geom = self.session_tww.scalar(ST_Force3D(coteattribute, COALESCE (NULL,'NaN'))::numeric)

ponceta · 2024-10-31T07:25:24Z

...teksi_wastewater/interlis/interlis_model_mapping/interlis_importer_to_intermediate_schema.py

+            # situation3d_geometry=self.session_tww.scalar(ST_Force3D(row.lage, row.kote)),
+            #geom = self.session_tww.scalar(ST_Force3D(coteattribute, "Nan"))
+            # (COALESCE (NULL,'NaN'))::numeric)
+            geom = self.session_tww.scalar(ST_Force3D(coteattribute, COALESCE (NULL,'NaN'))::numeric)


Suggested change

geom = self.session_tww.scalar(ST_Force3D(coteattribute, COALESCE (NULL,'NaN'))::numeric)

geom = self.session_tww.scalar(ST_Force3D(coteattribute))

This is python geoalchemy stuff, the proposed syntax is for postgis pgsql syntax. They are similar but not identical.

this still creates this:

2024-10-31T08:56:44 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000009330444100000000a96331410000000000d49d40. 2024-10-31T08:56:44 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000808130444100000000a96331410000000000d49d40. 2024-10-31T08:56:44 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000007030444100000000956331410000000000d09d40. 2024-10-31T08:56:44 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a00808000000000000e03344410000000068723141cdcccccccc32a140. 2024-10-31T08:56:44 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a0080800005eba49ccde334441d122dbf981723141000000000034a140. 2024-10-31T08:56:44 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01325A- situation3d_geometry z-value set to "Nan" instead: None. 2024-10-31T08:56:44 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01325E- situation3d_geometry z-value set to "Nan" instead: None. 2024-10-31T08:56:44 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01326A- situation3d_geometry z-value set to "Nan" instead: None. 2024-10-31T08:56:44 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01326E- situation3d_geometry z-value set to "Nan" instead: None.

And if I provider 'NULL' I get a crash:
geom = self.session_tww.scalar(ST_Force3D(coteattribute), 'NULL')

Ein Fehler trat bei der Ausführung von Python-Code auf:

sqlalchemy.exc.ArgumentError: mapping or list expected for parameters
Traceback (most recent call last):
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\teksi_wastewater_plugin.py", line 650, in actionImportClicked
self.interlisImporterExporter.action_import()
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\gui\interlis_importer_exporter_gui.py", line 87, in action_import
raise exception
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\gui\interlis_importer_exporter_gui.py", line 68, in action_import
self.interlis_importer_exporter.interlis_import(
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_importer_exporter.py", line 152, in interlis_import
raise exception
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_importer_exporter.py", line 106, in interlis_import
tww_session = self._import_from_intermediate_schema(import_model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_importer_exporter.py", line 265, in _import_from_intermediate_schema
interlisImporterToIntermediateSchema.tww_import(skip_closing_tww_session=True)
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_importer_to_intermediate_schema.py", line 41, in tww_import
raise exception
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_importer_to_intermediate_schema.py", line 33, in tww_import
self._tww_import(skip_closing_tww_session)
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_importer_to_intermediate_schema.py", line 66, in _tww_import
self._import_sia405_abwasser()
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_importer_to_intermediate_schema.py", line 120, in _import_sia405_abwasser
self._import_haltungspunkt()
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_importer_to_intermediate_schema.py", line 1971, in _import_haltungspunkt
situation3d_geometry=self.geometry3D_convert(row.lage, row.kote, row.t_ili_tid),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_pre_202403/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_importer_to_intermediate_schema.py", line 435, in geometry3D_convert
geom = self.session_tww.scalar(ST_Force3D(coteattribute), 'NULL')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Stefan\AppData\Roaming\Python\Python312\site-packages\sqlalchemy\orm\session.py", line 2410, in scalar
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Stefan\AppData\Roaming\Python\Python312\site-packages\sqlalchemy\orm\session.py", line 2242, in _execute_internal
return conn.scalar(
^^^^^^^^^^^^
File "C:\Users\Stefan\AppData\Roaming\Python\Python312\site-packages\sqlalchemy\engine\base.py", line 1301, in scalar
distilled_parameters = _distill_params_20(parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "lib\sqlalchemy\cyextension\util.pyx", line 37, in sqlalchemy.cyextension.util._distill_params_20
sqlalchemy.exc.ArgumentError: mapping or list expected for parameters

My bug - sure that geom = self.session_tww.scalar(ST_Force3D(coteattribute)) fails - as it is not existing.

I have to take the geometryattribute only
geom = self.session_tww.scalar(ST_Force3D(geometryattribute))

@ponceta Does this look ok with the geometries?

2024-10-31T09:14:30 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01353E- situation3d_geometry z-value set to "Nan" instead: 01010000a008080000e17a14ae912d44410c022b872e8c31410000000000000000. 2024-10-31T09:14:30 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01354A- situation3d_geometry z-value set to "Nan" instead: 01010000a008080000e17a14ae912d44410c022b872e8c31410000000000000000. 2024-10-31T09:14:30 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01354E- situation3d_geometry z-value set to "Nan" instead: 01010000a0080800004e6210f8b32d44416f1283e04e8c31410000000000000000. 2024-10-31T09:14:30 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a008080000000000007030444100000000956331410000000000d09d40. 2024-10-31T09:14:30 teksi_wastewater.utils: debug: controling situation3d_geometry : 01010000a00808000000000000c12f4441cdcccc4c4c6231410000000000c09d40.

We do now not 'Nan' but just leave it away. How should the debug message look like?

How about:

2024-10-31T09:14:30 teksi_wastewater.utils:No reach_point.cote (Haltungpunkt.Kote) provided for object ch080qwzHP01353E- situation3d_geometry with no z-value: 01010000a008080000e17a14ae912d44410c022b872e8c31410000000000000000.

sjib · 2024-10-31T08:41:06Z

Do we need also treat the case when there is a level but no geometry? What should we do then?

…ewater_node also, rename coteattribute to levelattribute

sjib · 2024-10-31T09:02:40Z

Now also for cover:

and wastewater_node:

for more information, see https://pre-commit.ci

sjib · 2024-10-31T09:03:41Z

This now also improves network following:

Downstream works from the beginning

ponceta · 2024-10-31T09:12:38Z

Do we need also treat the case when there is a level but no geometry? What should we do then?

ST_FORCE3D(None) = None

So technically it is already supported, but we should encourage the user to fix geometries otherwise he will face issues when using TWW (network, ...).

ponceta · 2024-10-31T10:21:05Z

Thank you @sjib for providing this!

sjib · 2024-10-31T10:33:21Z

sjib · 2024-10-31T10:33:47Z

Logger Messages for all 4 cases, example cover:

2024-10-31T11:31:44        teksi_wastewater.utils: debug: situation3d_geometry created with geometry (x,y) and level (z): 01010000a008080000000000008827444100000000b2a031410000000000fcaa40.
2024-10-31T11:31:44        teksi_wastewater.utils:No cover.level (Deckel.Deckelkote) and geometry (Lage) provided for object ch080qwzDE001408 - situation3d_geometry cannot be defined! Object cannot be displayed in TEKSI TWW!
2024-10-31T11:31:44        teksi_wastewater.utils:No cover.level (Deckel.Deckelkote) and geometry (Lage) provided for object ch080qwzDE001409 - situation3d_geometry cannot be defined! Object cannot be displayed in TEKSI TWW!
2024-10-31T11:31:44        teksi_wastewater.utils:No cover.level (Deckel.Deckelkote) provided for object ch080qwzDE001410- situation3d_geometry with no z-value created: 01010000a008080000000000408827444100000080b2a031410000000000000000.
2024-10-31T11:31:44        teksi_wastewater.utils:cover.level (Deckel.Deckelkote) provided but no geometry (Lage) provided for object ch080qwzDE001411 - situation3d_geometry cannot be defined! Object cannot be displayed in TEKSI TWW!
2024-10-31T11:31:44        teksi_wastewater.utils: debug: situation3d_geometry created with geometry (x,y) and level (z): 01010000a008080000000000009227444100000000fca131416666666666fcaa40.
2024-10-31T11:31:44        teksi_wastewater.utils: debug: situation3d_geometry created with geometry (x,y) and level (z): 01010000a0080800007d3f35fe40294441e17a142edf9d31413333333333d8a840.
2024-10-31T11:31:44        teksi_wastewater.utils: debug: situation3d_geometry created with geometry (x,y) and level (z): 01010000a008080000000000804d2a444100000000bb9f31410000000000b4aa40.

sjib · 2024-10-31T10:41:20Z

When I import cases with no coordinates I get this network following error again: #476

sjib · 2024-10-31T10:42:10Z

When I import cases with no coordinates I get this network following error again: #476

So we have to finish this also: #457

sjib · 2024-11-01T08:33:07Z

Danke @tomep for this testing: Ich habe die angepassten Files ausgetauscht und 2 Erfolgreiche Imports machen können 😊 Das funktioniert also, da freue ich mich aber. Super Sache. Ich sehe nun auch wieder alle SAA-Objekte und die Netzverfolgung kann ausgeführt werden.

ponceta · 2024-11-01T12:55:37Z

@sjib are we ready to integrate this? Please undraft and merge if ready!

sjib · 2024-11-01T15:21:26Z

Should I change some of the messages to logger.debug?

for more information, see https://pre-commit.ci

sjib · 2024-11-01T16:27:39Z

I leave as is - logger levels seem reasonable to me.

Add function geometry3D_convert

1036403

sjib added the fix Fixing something not working label Oct 30, 2024

sjib added this to the TEKSI Wastewater 2024.0 milestone Oct 30, 2024

sjib self-assigned this Oct 30, 2024

sjib marked this pull request as draft October 30, 2024 16:40

pre-commit-ci bot and others added 5 commits October 30, 2024 16:41

[pre-commit.ci] auto fixes from pre-commit.com hooks

6914a9b

for more information, see https://pre-commit.ci

improve geometry3D_convert - add geom variable to check if conversion…

6a84f12

… is ok Call with 'Nan' does not work yet

Merge branch '2024-10-30-fix-loosing-coordinates-interlis-import' of h…

93b22ec

…ttps://github.com/teksi/wastewater into 2024-10-30-fix-loosing-coordinates-interlis-import

correct )

599ac84

[pre-commit.ci] auto fixes from pre-commit.com hooks

4bb7278

for more information, see https://pre-commit.ci

sjib added 2 commits October 30, 2024 21:02

add coalesce

1177b1e

Correct brackets

8e58081

ponceta reviewed Oct 31, 2024

View reviewed changes

correct ST_Force3D - take geometryattribute instead

160dd71

Improve geometry3D_convert - all 4 cases, integrate to cover and wast…

52bfd49

…ewater_node also, rename coteattribute to levelattribute

[pre-commit.ci] auto fixes from pre-commit.com hooks

86f305d

for more information, see https://pre-commit.ci

ponceta approved these changes Oct 31, 2024

View reviewed changes

sjib and others added 2 commits November 1, 2024 17:12

cleaning up code

4403972

[pre-commit.ci] auto fixes from pre-commit.com hooks

dc1bc9f

for more information, see https://pre-commit.ci

sjib marked this pull request as ready for review November 1, 2024 16:14

sjib merged commit 7679eb4 into main Nov 1, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2024-10-30-fix-loosing-coordinates-interlis-import #484

2024-10-30-fix-loosing-coordinates-interlis-import #484

sjib commented Oct 30, 2024 •

edited by ponceta

Loading

sjib commented Oct 30, 2024

sjib commented Oct 30, 2024

sjib commented Oct 30, 2024

sjib commented Oct 30, 2024

ponceta Oct 31, 2024

sjib Oct 31, 2024

sjib Oct 31, 2024

sjib Oct 31, 2024

sjib Oct 31, 2024

sjib Oct 31, 2024

sjib Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

ponceta commented Oct 31, 2024

ponceta commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Nov 1, 2024 via email

ponceta commented Nov 1, 2024

sjib commented Nov 1, 2024 via email

sjib commented Nov 1, 2024

	geom = self.session_tww.scalar(ST_Force3D(coteattribute, COALESCE (NULL,'NaN'))::numeric)
	geom = self.session_tww.scalar(ST_Force3D(coteattribute))

2024-10-30-fix-loosing-coordinates-interlis-import #484

2024-10-30-fix-loosing-coordinates-interlis-import #484

Conversation

sjib commented Oct 30, 2024 • edited by ponceta Loading

sjib commented Oct 30, 2024

sjib commented Oct 30, 2024

sjib commented Oct 30, 2024

sjib commented Oct 30, 2024

ponceta Oct 31, 2024

Choose a reason for hiding this comment

sjib Oct 31, 2024

Choose a reason for hiding this comment

sjib Oct 31, 2024

Choose a reason for hiding this comment

sjib Oct 31, 2024

Choose a reason for hiding this comment

sjib Oct 31, 2024

Choose a reason for hiding this comment

sjib Oct 31, 2024

Choose a reason for hiding this comment

sjib Oct 31, 2024

Choose a reason for hiding this comment

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

ponceta commented Oct 31, 2024

ponceta commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Oct 31, 2024

sjib commented Nov 1, 2024 via email

ponceta commented Nov 1, 2024

sjib commented Nov 1, 2024 via email

sjib commented Nov 1, 2024

sjib commented Oct 30, 2024 •

edited by ponceta

Loading