Example: use SceneKit in custom layer #154

astojilj · 2021-03-02T15:44:51Z

Port "antenna building" example from gl-js https://docs.mapbox.com/mapbox-gl-js/example/add-3d-model/

Screen.Recording.2021-03-17.at.14.34.49.mov

The example demonstrates direct port from https://docs.mapbox.com/mapbox-gl-js/example/add-3d-model/ using Three.js to SceneKit.

For that purpose, original GLTF model is converted (using Blender) to dae. Textures are embedded in attached dae file. When exporting in Blender, it is important to set with Y-up in export setting. Original model is available from https://nasa3d.arc.nasa.gov/detail/jpl-vtad-dsn34.

Code comments provide in depth explanation on matrices transformations required to geo-reference models on map.

Initial version used SceneKit SCNMatrix4 operations for positioning model on the map. That led to visible jittering of model relative to camera. Similar issue was observed using simd floating point calculations. The jittering looked like this:

Screen.Recording.2021-03-17.at.14.24.23.mov

To resolve that, it is necessary to use simd double precision calculation for following:

mapbox-maps-ios/Examples/Examples/All Examples/SceneKitExample.swift

Lines 177 to 187 in 9804c12

    
           let modelScale = makeScaleMatrix(xScale: meterInMercatorCoordinateUnits, yScale: -meterInMercatorCoordinateUnits, zScale: 1) 
        
           // Translate scaled model to model origin (in web mercator coordinates) and elevate to model origin's altitude (in meters). 
        
           let origin = try! Projection.project(for: modelOrigin, zoomScale: pow(2, parameters.zoom)) 
        
           var elevation = 0.0 
        
           if let elevationData = parameters.elevationData, let elevationValue = elevationData.getElevationFor(self.modelOrigin) { 
        
               elevation = elevationValue.doubleValue 
        
           } 
        
           let translateModel = makeTranslationMatrix(tx: origin.x, ty: origin.y, tz: elevation) 
        
           let transform = transformSimd * translateModel * modelScale

Issues:

There was a lag/delay when panning the map. projectionTransform property is animatable and it is need to call SCNTransaction.flush() to use it in current frame.

Follow up: After #155 , implement CPU side occlusion as an option.

CLAassistant · 2021-03-02T15:44:57Z

All committers have signed the CLA.

julianrex · 2021-03-02T16:05:22Z

Examples/Examples/All Examples/SceneKitExample.swift

+        transformSimd[3, 2] = m[14].doubleValue
+        transformSimd[3, 3] = m[15].doubleValue
+
+        let origin = try! Projection.project(for: modelOrigin, zoomScale: 1.0 / 512.0);


What's the 512 here?

Explained in code comment:

// Projection.project(for: modelOrigin, zoomScale: 1.0 / 512.0) corresponds to gl-js's // mapboxgl.MercatorCoordinate.fromLngLat(). origin is in spherical mercator normalized to // 0..1 for the width of the world. In other words, (x,y) E [0..1) is used to represent // coordinates in one world copy, values of x +/- 1 represent wrap. let origin = try! Projection.project(for: modelOrigin, zoomScale: 1.0 / 512.0)

julianrex · 2021-03-02T16:07:12Z

Examples/Examples/All Examples/SceneKitExample.swift

+    public func render(_ parameters: CustomLayerRenderParameters, mtlCommandBuffer: MTLCommandBuffer, mtlRenderPassDescriptor: MTLRenderPassDescriptor) {
+        let m = parameters.projectionMatrix;
+
+        // It is essential to use double precision for computation below: using simd instead


Can you say why - presumably because of jittering with floats?

Yes, mercator coordinate addresses whole world and there's just not enough precision if using 32-bit floating point computation.

Can you add that to the comment please?

julianrex · 2021-03-02T16:13:53Z

Examples/Examples/All Examples/SceneKitExample.swift

+        if #available(iOS 13.0, *) {
+            renderer.usesReverseZ = false
+        } else {
+            // Fallback on earlier versions, disable depth in render()


What happens in these cases? Should we disable this example completely if < iOS 13?

We should enable it, just there is a need to use CPU side occlusion (the same approach that could be used when geo-referencing UI elements to map). The idea is to disable depth buffer (now disabled in demo) and show or hide model with animated fade in / fade out (similar to behavior of markers in gl-js).

As we cannot have occlusion done yet, current behavior on iOS < 13 is like here (model always visible):

Screen.Recording.2021-03-03.at.19.28.43.mov

astojilj · 2021-03-03T17:43:24Z

There was a lag/delay when panning the map. projectionTransform property is animatable and it is need to call SCNTransaction.flush() to use it in current frame. Updated video shows improved behavior.

@julianrex PTAL

julianrex · 2021-03-04T19:33:35Z

Examples/Examples/All Examples/SceneKitExample.swift

+    override public func viewDidAppear(_ animated: Bool) {
+        super.viewDidAppear(animated)
+         // The below line is used for internal testing purposes only.
+        self.finish()


Let's move this to renderingWillEnd (and rebase on main once #158 lands). These examples get used as tests, and we want to ensure that we render the custom layer.

Can you make sure this example runs as a test (Cmd-U on the Examples target should be sufficient).

Done. Test passes. However, as it is the case with this and several other tests, I could see

Example timed out, was this intentional? Call finish() if possible. Test Case '-[ExamplesTests.TestableExampleTests testSceneKitExample]' passed (20.534 seconds).

julianrex · 2021-03-05T04:30:53Z

Examples/Examples/All Examples/SceneKitExample.swift

+        let insertHillshadeBelow = try! map.styleLayerExists(forLayerId: "water") ?
+            LayerPosition(above: nil, below: "water", at: nil) : try! map.styleLayerExists(forLayerId: "hillshade") ?
+            LayerPosition(above: nil, below: "hillshade", at: nil) : nil


This is a little hard to understand; can we split this into multiple lines.

Unnecessary complication - replaced the code with simple LayerPosition(above: nil, below: "water", at: nil) as it is present in style used.

julianrex · 2021-03-05T04:38:58Z

Examples/Examples/All Examples/SceneKitExample.swift

+        let modelScale = makeScaleMatrix(xScale: meterInMercatorCoordinateUnits, yScale: -meterInMercatorCoordinateUnits, zScale: meterInMercatorCoordinateUnits)
+
+        // mercator scale is specific to gl-native example because gl-js's customLayerMatrix computes this
+        // internaly: https://github.com/mapbox/mapbox-gl-js/blob/main/src/geo/transform.js#L1316


nit: internally.

julianrex · 2021-03-05T04:51:16Z

Examples/Examples/All Examples/SceneKitExample.swift

+        // mapboxgl.MercatorCoordinate.fromLngLat(). origin is in spherical mercator normalized to
+        // 0..1 for the width of the world. In other words, (x,y) E [0..1) is used to represent
+        // coordinates in one world copy, values of x +/- 1 represent wrap.
+        let origin = try! Projection.project(for: modelOrigin, zoomScale: 1.0 / 512.0)


I still don't understand what the 512 represents here and in the subsequent code; shouldn't the projection calculation here involve parameters.zoom?

This is a good point. I have apparently approached this from wrong angle; the idea was to complicate matrices and calculation, enhance the complexity with differences between gl-js and gl-native custom layer transformation matrix and Web mercator scaling internals, in order to present 1-1 mapping of gl-js example to approach here. It is wrong and if using parameters.zoom this is much simpler to follow (and less code).

1ec5

This is super exciting to see. The simd usage significantly improves upon the approach in mapbox/ios-sdk-examples#111 (comment).

It would be wonderful to see MapboxMaps eventually encapsulate SceneKit integration to a greater degree – perhaps even as a “SceneKitLayer” class. There are a lot of transformations and settings here that would be common to most usage of SceneKit. I think developers would likely copy-paste the whole example rather than take advantage of any flexibility gained by implementing it at the application level.

/ref mapbox/ios-sdk-examples#111

astojilj · 2021-03-11T16:47:27Z

@julianrex PTAL

julianrex

A few comments that would be good to address, but otherwise LGTM. Approved so you're not blocked.

julianrex · 2021-03-12T21:38:18Z

Examples/Examples/All Examples/SceneKitExample.swift

+
+        var demSource = RasterDemSource()
+        demSource.url = "mapbox://mapbox.mapbox-terrain-dem-v1"
+        demSource.tileSize = 514


Is 514 correct?

514 is used for padded (border ready) DEM tiles. I wanted to test it here and left the value.

Padded DEM tiles are performance improvement, especially for hillshade: borders are prepared on server side and there is no need to backfill borders from neighboring DEM tiles.

Anyway, it is unnecessary, additional information for this PR. I'll remove before merging.

julianrex · 2021-03-12T21:40:30Z

Examples/Examples/All Examples/SceneKitExample.swift

+    public func render(_ parameters: CustomLayerRenderParameters, mtlCommandBuffer: MTLCommandBuffer, mtlRenderPassDescriptor: MTLRenderPassDescriptor) {
+        let m = parameters.projectionMatrix;
+
+        // It is essential to use double precision for computation below: using simd instead


Can you add that to the comment please?

Examples/Examples/All Examples/SceneKitExample.swift

jmkiley

Thank you for the PR!

Port "antenna building" example from gl-js https://docs.mapbox.com/mapbox-gl-js/example/add-3d-model/

Cal-um · 2021-06-11T15:54:05Z

hey @astojilj thanks for this great example. Im looking at using something similar to this with models on a map. Do you have any idea how to respond to a touch on one of these models? I am trying out the hittest method on the renderer but having no luck unless you are directly above it and close.

NickLambert95 · 2021-06-14T20:53:02Z

@astojilj Is there an example for this on Android SDK?

astojilj · 2021-06-15T06:31:06Z

@Cal-um,
Please check mapbox/mapbox-gl-js#10595 (comment): there you'll find explanation for a solution in gl-js with custom layer in THREE.js. Following the same example, this would mean that we could try to use

- (NSArray<SCNHitTestResult *> *)hitTestWithSegmentFromPoint:(SCNVector3)pointA 
                                                     toPoint:(SCNVector3)pointB 
                                                     options:(NSDictionary<NSString *,id> *)options;

for hit testing. If you have time to experiment with it, please try, I will also allocate some time later this week to check it.
It could be useful to know a bit more about number, size and shape of models you use and if terrain is active.

astojilj · 2021-06-15T07:19:56Z

@astojilj Is there an example for this on Android SDK?

@NickLambert95 ,
I've answered in mapbox/mapbox-navigation-android#4502 (comment). Let's see if that helps.

NickLambert95 · 2021-06-16T00:49:25Z

@astojilj I just replied to your comment, it seems you closed the issue thread so I just wanted to let you know

Cal-um · 2021-06-16T08:00:02Z

thanks @astojilj Ill have a play around with it.

The use case I have is displaying multiple models (15+) on the map similar to the satellite dish in your example so they could be a range of shapes and sizes. I know this may complicate things but I'm getting an idea of what kind of challenge this would be to identify taps on all of them.

Cal-um · 2021-06-22T16:11:23Z

Hey @astojilj Sorry for pestering you on this but I am a complete noob at this. Perhaps I am approaching this wrong but I have tried taking the CGPoint of the tap and unprojecting it from the renderer method with a SCNVector3 of z 0 and 1. It seems to work when rendering a line with it but it's at completely the wrong point on the map.

On a different note would you need a new scene and layer for every distinct model rendered on the map?

Thanks in advance.

julianrex reviewed Mar 2, 2021

View reviewed changes

astojilj requested a review from julianrex March 3, 2021 17:42

astojilj self-assigned this Mar 3, 2021

This was referenced Mar 4, 2021

When it will support 3D model? mapbox/mapbox-gl-native#14494

Open

Add 3D model to iOS map at coordinates mapbox/mapbox-gl-native#15049

Open

julianrex mentioned this pull request Mar 4, 2021

Ensure Examples' finish is called only once. #158

Merged

7 tasks

julianrex reviewed Mar 5, 2021

View reviewed changes

1ec5 reviewed Mar 5, 2021

View reviewed changes

astojilj force-pushed the astojilj-scenekit branch from 73f8a45 to 5c204bc Compare March 11, 2021 15:53

astojilj force-pushed the astojilj-scenekit branch from 5c204bc to 85b813d Compare March 12, 2021 07:25

julianrex requested a review from jmkiley March 12, 2021 20:20

julianrex approved these changes Mar 12, 2021

View reviewed changes

julianrex requested a review from nishant-karajgikar March 12, 2021 21:46

jmkiley approved these changes Mar 16, 2021

View reviewed changes

astojilj added 4 commits March 17, 2021 13:41

SceneKit in custom layer

c67984f

Port "antenna building" example from gl-js https://docs.mapbox.com/mapbox-gl-js/example/add-3d-model/

Gesture lag and review fixes.

d26cf99

Simplification. Review comments fixes.

65c4284

Rebasing. Review comments.

8832550

astojilj force-pushed the astojilj-scenekit branch from 85b813d to 8832550 Compare March 17, 2021 12:17

astojilj merged commit 9804c12 into main Mar 17, 2021

astojilj deleted the astojilj-scenekit branch March 17, 2021 12:31

macdrevx added the skip changelog Add this label if this item does not need to be included in the changelog label Mar 29, 2021

NickLambert95 mentioned this pull request Jun 14, 2021

V10 import a 3D object onto Android SDK? mapbox/mapbox-navigation-android#4502

Closed

1ec5 mentioned this pull request Jun 29, 2021

Example for SceneKit content in the course view mapbox/mapbox-navigation-ios-examples#9

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example: use SceneKit in custom layer #154

Example: use SceneKit in custom layer #154

astojilj commented Mar 2, 2021 •

edited

Loading

CLAassistant commented Mar 2, 2021 •

edited

Loading

julianrex Mar 2, 2021

astojilj Mar 3, 2021

julianrex Mar 2, 2021

astojilj Mar 3, 2021

julianrex Mar 12, 2021

julianrex Mar 2, 2021

astojilj Mar 3, 2021

astojilj commented Mar 3, 2021

julianrex Mar 4, 2021

astojilj Mar 11, 2021

julianrex Mar 5, 2021

astojilj Mar 11, 2021

julianrex Mar 5, 2021

julianrex Mar 5, 2021

astojilj Mar 11, 2021

1ec5 left a comment •

edited

Loading

astojilj commented Mar 11, 2021

julianrex left a comment

julianrex Mar 12, 2021

astojilj Mar 15, 2021

julianrex Mar 12, 2021

jmkiley left a comment

Cal-um commented Jun 11, 2021

NickLambert95 commented Jun 14, 2021

astojilj commented Jun 15, 2021

astojilj commented Jun 15, 2021

NickLambert95 commented Jun 16, 2021

Cal-um commented Jun 16, 2021

Cal-um commented Jun 22, 2021

	let modelScale = makeScaleMatrix(xScale: meterInMercatorCoordinateUnits, yScale: -meterInMercatorCoordinateUnits, zScale: 1)

	// Translate scaled model to model origin (in web mercator coordinates) and elevate to model origin's altitude (in meters).
	let origin = try! Projection.project(for: modelOrigin, zoomScale: pow(2, parameters.zoom))
	var elevation = 0.0
	if let elevationData = parameters.elevationData, let elevationValue = elevationData.getElevationFor(self.modelOrigin) {
	elevation = elevationValue.doubleValue
	}
	let translateModel = makeTranslationMatrix(tx: origin.x, ty: origin.y, tz: elevation)

	let transform = transformSimd * translateModel * modelScale

Example: use SceneKit in custom layer #154

Example: use SceneKit in custom layer #154

Conversation

astojilj commented Mar 2, 2021 • edited Loading

CLAassistant commented Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astojilj commented Mar 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1ec5 left a comment • edited Loading

Choose a reason for hiding this comment

astojilj commented Mar 11, 2021

julianrex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmkiley left a comment

Choose a reason for hiding this comment

Cal-um commented Jun 11, 2021

NickLambert95 commented Jun 14, 2021

astojilj commented Jun 15, 2021

astojilj commented Jun 15, 2021

NickLambert95 commented Jun 16, 2021

Cal-um commented Jun 16, 2021

Cal-um commented Jun 22, 2021

astojilj commented Mar 2, 2021 •

edited

Loading

CLAassistant commented Mar 2, 2021 •

edited

Loading

1ec5 left a comment •

edited

Loading