Offline map support on public branch for testing

[brnkhy]

Hey everyone!
I have been working on offline map support (https://docs.mapbox.com/help/troubleshooting/mobile-offline/) for a while now and wanted to let you know it's on a public branch for testing at the moment.

You can find it here; https://github.com/mapbox/mapbox-unity-sdk/tree/offlineMaps

And usage goes something like this;

 var region = new OfflineRegion(
            MapName,
            _minLatLng,
            _maxLatLng,
            _minZoom,
            _maxZoom,
            elevationTilesetId,
            imageryTilesetId,
            vectorTilesetId);
var response = MapboxAccess.Instance.OfflineManager.CreateOfflineMap(region.Name, region);

This will download and cache all tiles in this region in given zoom range. I just pushed it today so there might be bugs and issues of course.
Just to be clear, this doesn't mean you can ship data with your application or anything. This is just a variation of regular ambient cache. Regular ambient cache time out, recycle out etc. Offline maps adds a flag to tiles so they won't time out like regular ambient cache tiles. So it ensures those tiles will be in the cache.

In example, let's say you're building a hiking app where people go off grid frequently to hike. You can let them predownload and cache their hiking area in advance, at their home when they have connectivity for example. SDK will ensure cached tiles will not be removed/recycled/deleted so when they go hiking and don't have connection, their tile data will still be in the cache.

This feature also comes with a lot of changes in Sqlite management so please let me know if you notice any bugs and/or performance issues regarding data in general.

[brnkhy]

for anyone trying this out, please let me know if you run into troubles, it's super beta so there will be bugs but I want to finish this asap so any fixes to this will be priority for me 🙏

[Markovicho]

first note:

OfflineMapDemoController.cs
remove using directive for:

using UnityEditor.Rendering.LookDev;

it seems like it's not used and will throw errors if you are not using the right pipeline:
Assets\Mapbox\Examples\9_OfflineMap\Scripts\OfflineMapDemoController.cs(7,29): error CS0234: The type or namespace name 'LookDev' does not exist in the namespace 'UnityEditor.Rendering' (are you missing an assembly reference?)

by the way thx for the afford. This feature is pretty important for a lot of showcases. Looking forward to check this out

Postscript:

Running on Unity 2019.4.5f LTS on hitting play will result in those errors. It's running on a german system which could possibly cause some formatting/encoding issues (?)

[brnkhy]

Hey @Markovicho ,
Removed the using directive, I have no idea it was there really. Thanks!

Second issue is happening because your existing old SQLite table doesn't work with the new offline maps code. If you use Mapbox -> Clear File Cache from the top menu, it'll delete the existing file and recreate it using new system. Then you shouldn't get the error.

[Markovicho]

Thx! clearing the file cache worked.

postscript:

Building for Android will run into another unused using directive:

CachingWebFileSource.cs ==> using Unity.UNetWeaver

using Unity.UNetWeaver;
causing:
Assets\Mapbox\Core\mapbox-sdk-cs\Platform\Cache\CachingWebFileSource.cs(1,13): error CS0234: The type or namespace name 'UNetWeaver' does not exist in the namespace 'Unity' (are you missing an assembly reference?)

Test on PC:

Configuration:

minZoom: 10
maxZoom: 18

Zoom while Downloading: 16

After the download completed i started to zoom out/in. But only zoomLevel 15+16 seems to work offline oO

I attached a videocapture. there is a little textfield which is showing the current zoomLevel for debug purposes

https://drive.google.com/file/d/1kGao-LXjOX_Sao7f_dcpvcrX1MgdVlgk/view?usp=sharing

[brnkhy]

@Markovicho I'm not sure what's up with those unused directives, great catch!

I'll check the zoom thing but I think it might be something about overzooming (16+ levels). I can't remember for sure now but I might intentionally or not might have capped it at z16. I'll check and update asap.
Thanks for the video by the way, it's great to see it's working well otherwise!

[brnkhy]

@Markovicho I think I fixed the zoom issue. The problem was with the demo scene, it used satellite imagery but sample code downloaded streets imagery so when you went offline, it wasn't able to find different zoom levels of satellite. Somehow, z15-16 was cached though, probably with regular ambient caching.

Also found and fixed a bug with cache expiration date. Thanks a lot!

[Markovicho]

This seems to be solved and is working now as expected :-)
(Tested on Windows and Android 11)

How is the feature behaving on entering the following situations:

• Display standby : ==> Unity-Application will go to sleep...
• Switching to another application ==> Unity-App will go to sleep as well here ...

This could be an interesting point regarding larger downloads or slow internet connections. I assume it would be a large task to outsource this functionality into an native plugin to avoid pausing the download (right ?)
Otherwise we need to find some way to keep the user in the foreground application while the download is running (UX-Design needs to be aware of that point)

[brnkhy]

@Markovicho awesome, thanks a lot with the help, I really appreciate it! As I said I want to get this done as soon as possible.

I'm afraid I have no idea about the issue you described though. I don't do mobile stuff much so I'm not sure how it works but I suspect OS freezes the app when it's in the background? Or doing tricky something in the background without app developers knowledge might not be preferable either. Either way I think it's safer to leave that to app developer. I'm not sure but as far as I know other Mapbox SDKs doesn't do this either right?

[Markovicho]

Like you said no matter if you're using Android or iOS, the application will freeze completely. The only way to keep functionality active in this case is to write an separate native plugin for both plattforms which will continue the tasks while the unity app is freezed. Maybe thats even more work than the current solution itself.
As long as the download will continue correctly after returning to the app thats not a big deal and more an question of the app design.
Anyways this feature is such a big improvement for the unity sdk. thx in advance !!

[raggnic]

Nice ! Is it ok with the mapbox data licence tough?

[brnkhy]

@raggnic it's the same offline map support other Mapbox SDKs provide. Why did you thought that might be an issue? Am I missing something? You got me worried :)

[raggnic]

Ok sorry I may have read the intro too fast. I was not aware of the statement you've made, about shipping an app with included data.
It may be super useful to us but we're still on a fork from 1.45.... so long way to go

[brnkhy]

@raggnic I highly suggest updating but if it'll be too much work, I'm hoping to release (raster) performance work public soon as well (1-2 weeks) so you might want to wait for that.

[JackMoljerc]

It works!! Is there any way to change the third-party tile Servers ,such as my own local geoserver?

[Markovicho]

How about the releaseplan ? Is it worth to use this branch or should we wait for an upcoming official release of the mapbox unity
sdk ?

[brnkhy]

@JackMoljerc that would be a different feature and I probably won't have a chance to look into that for a while. Sorry.

@Markovicho I think I'll wait on this for now and release it together with performance branch. Both works developed separately and they both customize sqlite database so to prevent more issues, I'll just try to update it all in once. I'm afraid it won't be trivial though so might take some time. I really want to finish all this before 2021 though, so this isn't official or anything but I want raster performance improvements + offline maps + ton of bug fixes in before 2021.

[brnkhy]

@Markovicho I started working on that merge I mentioned yesterday already but admittedly I found a few points would use some improvement. Even some possible bugs. So I think it would be wiser not to use it in final product for a little longer.

[Markovicho]

@brnkhy sure we will wait for the final release. The mentioned changes in your last post (raster performance etc) sounds pretty much like what we need to get a stable release ready version. Is there a public branch for testing the raster performance ? maybe we could help improve that process by some testing feedback :-)

[brnkhy]

@Markovicho I want to release it as soon as possible (for a while now) for exactly that reason. But it's such a big change that I want to release it at least in a decent state. I thought I was there really, I had a reliable version but then I realized merging that with offline maps won't be trivial and again cause a lot of changes so I decided to get that out of way first. I'll definitely let you all know when it's ready though, I can really use some help/feedback 🙏

[Kundankmr500]

@brnkhy I have downloaded this branch to use the offline map feature but when I open the OfflineMap scene there is OfflineMapListPrefab prefab reference is missing in OfflineMapControl Gameobject. please have a look at the screenshot. Can you tell me the OfflineMapListPrefab Prefab name and Path?

[LauraLaureus]

Hello @Kundankmr500 did you manage to get any advances on this line ?

[brnkhy]

Hey @Kundankmr500 @LauraLaureus
I'm sorry I just saw this. Turns out that prefabs was indeed missing 🤦‍♂️ I'm really sorry for that.
I pushed he prefab on same branch. I suspect you still might have issues as it was crated using 2019.4 and previous versions might not detect it properly. It's OfflineMapListItem under offline maps example folder.
It's used only for the demo and shouldn't effect the feature itself.

fix commit; 1039e40

[LauraLaureus]

Thank you @brnkhy ! Now it works :)
Can you guide me, or at least put me in the right track, on how to load the offline maps ?

[avgarint]

Can you guide me, or at least put me in the right track, on how to load the offline maps ?

Yes, that would be nice please 😉

[brnkhy]

@AlexandreAero
I know it's high but it works perferctly fine on mobile 🤔
well if you say so.... but please be careful with your usage numbers. That's a lot of tiles in a single moment and server requests might explode with a mistake or multiple users etc etc.

[avgarint]

Do you know if you will make any changes to modify the branch and improve other elements?

[Markovicho]

@brnkhy

Android build fails since pulling latest changeset from performance branch on android:

git changeset : 07a1b2c (5th of may)

[brnkhy]

@AlexandreAero there will be other improvements, especially on vector layer. But can't say much about the size&ETA for those yet.

@Markovicho that class was renamed, I think your git client didn't update it;

mapbox-unity-sdk/sdkproject/Assets/Mapbox/Unity/MapboxAccess.cs

Line 28 in 5c91bc1

public DataFetchingManager DataManager;

[Markovicho]

@brnkhy

@Markovicho that class was renamed, I think your git client didn't update it;

mapbox-unity-sdk/sdkproject/Assets/Mapbox/Unity/MapboxAccess.cs

Line 28 in 5c91bc1

public DataFetchingManager DataManager;

Indeed it is renamed but still incorrect in some preprocessor defines which only occur when building:

the fix would be changing MapboxAccess.cs Line 161

from
DataManager = new MapboxDataFetchingManager(_fileSource);
to
DataManager = new DataFetchingManager(_fileSource);

the second line which fails is this one:

fixed by changing MaboxAccess.cs Line 173

from
var fileCache = new FileCache(_fileSource);
to
var fileCache = new FileCache();

[brnkhy]

@Markovicho ah I see now, great catch!
I pushed the fix in 66a615e
Thanks a lot! 🙇

[Markovicho]

@brnkhy

Assets\Mapbox\Unity\CustomLayer\ImageFactoryBehaviour.cs(97,44): error CS1061: 'AbstractMapVisualizer' does not contain a definition for 'GetInactiveTiles' and no accessible extension method 'GetInactiveTiles' accepting a first argument of type 'AbstractMapVisualizer' could be found (are you missing a using directive or an assembly reference?)

which results out of this preprocessor statement not defining a method for the build:

i switched back to the commit of 7th june as well as leaving out the called preprocessor code. But anyways the current codebase is not correctly working on mobile for the zoomable map scene. The map is loaded initially to the correct zoomlevel + camera frustum. But zooming or panning is not working anymore as there are no new tiles loaded afterwards.

add:

last working commit is this one :

#66a615e0e (2th June)

[Markovicho]

@brnkhy is there any schedule when the performance branch will be working on mobile again ? We are currently developing a branch including the new mapbox tile loading for our project and can't process on that one until it's fixed (refer to last post with issue description).

[Markovicho]

next findings:

Adding Map Features : IndexOutOfBounds
Adding a map feature layer will result an editor script to throw an indexoutofbounds expection:

Renaming feature layer not persisting

Renaming the layer will not persist when entering play more. After playmode was active the layer is renamed back to "Untitled". Maybe this is caused by the same reason of the above indexoutofbounds exception (?)

Android Build broken : 'AbstractMapVisualizer' does not contain a definition for 'GetInactiveTiles'

refer to my last postings

@brnkhy should i continue with posting bug-reports related to performance-bnrach into this thread or outsource them into separate issues ?

[brnkhy]

Hey @Markovicho ,
Sorry I wasn't aware that's a breaking bug. But that line is just for debugging purposes and deleting it (with and follow ups using it) should be OK I believe. I'll check it.

Adding Map Features : IndexOutOfBounds & Renaming feature layer not persisting
I remember occasionally getting this in the past but haven't ran into it for a long time now. My recent changes might have broke something like serialization I guess. I'll see what I can do.

I think separate threads might be better, it would be easier to track.
Thanks a lot @Markovicho!

[Markovicho]

Sorry I wasn't aware that's a breaking bug. But that line is just for debugging purposes and deleting it (with and follow ups using it) should be OK I believe. I'll check it.

It's okay to generate the build. But the tile-loading isn't working properly anymore (refer to post ) since a specific commit.

I think separate threads might be better, it would be easier to track.

allright & thx

[brnkhy]

@Markovicho
It's okay to generate the build. But the tile-loading isn't working properly anymore (refer to post ) since a specific commit.
ah are you getting a warning/log like requests is already in queue etc etc? There was an issue with file paths on iOS and android we ran into recently, might be the same issue. Where map loads on first run, with empty cache but nothing works on second run.

edit: a6c79ae commit on performance branch should have fixed the issue I mentioned.

[Markovicho]

@brnkhy

mb_notLoading.mp4

This is what it looks like in a android mobile onDevice build. Like i said this behaviour has not changed since a specific commit in june (#66a615e0e 2th June). Playmode in Unity-Editor (Android plattform target) is working.

Android-Logcat is throwing a bunch of expections:

Maybe it would be an approach deploying on a real phone. Unity-Editor Playmode with a specific target-plattform is not conclusive enough.

I attached a logcat snippet which should help you finding the issue

com.mapbox.unitysdk-logcat.txt

[brnkhy]

@Markovicho I think I know the issue, I pushed a quick fix. Admittedly I couldn't test it myself yet but I think it should solve the problem. Sorry about this.
commit 4f4a327

[Markovicho]

Seems good so far. thank in advance :-)

[Markovicho]

@brnkhy

I was happy too soon. Refer to this:

#1798

and the little issue with the Preprocessor-Flag:

#1799

[brnkhy]

@Markovicho ah admittedly I forgot to look into the preprocessor flag thing but responded to the libcurl one, I'm afraid there's no fix for that.

[jaspervandenbarg]

@brnkhy

How do I import this to my Unity project? Do I need to remove my old mapbox from the assets folder?

Also I managed to make the TileCacher from issue #1312 ~ https://github.com/mapbox/mapbox-unity-sdk/tree/TilePrefetching work for different zoom levels aswell.

How does this offline map compare to the branch I used for downloading map tiles? I read you mentioned something about adding a flag to the cached tiles.

Also how would I be able to combine this with the performance branch?

Thanks in advance!

[avgarint]

How the branch is coming along? Is it stable yet? If no, when it will be?

[RomanAtBrief]

How to properly install Performance branch into existing project with MapBox 2.1.1?
Also what would be a good strategy to update it in future ? as new bug fixes come through

[DevNMY]

How to properly install Performance branch into existing project with MapBox 2.1.1? Also what would be a good strategy to update it in future ? as new bug fixes come through

You should check out this thread: #1813

[Markovicho]

outsourced comment to separate issue --> #1849

[useronym]

What is the status of this feature?

[Markovicho]

Maybe something interersting while implementing smoothquadtree branch @brnkhy @Entryist

#1868

[ajcalabro]

Hey I love the offline option and I got it working in my Unity project. My issue is how do I load the map from the cache once it’s made? I have my app downloading the files then initializing the abstract map but it always waits until the next load to use the cached data. Any idea how to use the cache right after it finished downloading and use that as the map?

[teechoi]

Updates on this branch? Will it be merged with develop for sdk access? I tried downloading and accessing the offlineMaps branch itself but am unable to get the examples running.