Offline maps

[ljbade]

It is a personal goal to have the ability to view maps without needing an Internet connection.

Very useful for activities such as bushwalking our mountain biking.

Plus we know there are others interested in this feature too.

[ljbade]

Some discussion I had with @kkaefer about what was needed for offline maps:

• native core does not yet support support offline maps - on roadmap
• The cache sqlite DB can be reused for storing tiles offline
  • need to add ability to swap out the cache database and set it to a read-only mode
• need a tool that builds an offline database that can be distributed with the app/downloaded
• could also download all individual tiles on first launch, however this is not good strategy
• could be useful to have a way to differentially update only the tiles that have been changed. Like some sort of patch DB, that has higher priority than main DB, and can eventually be somehow merged into the main DB.

First things to work on:

1. enable offline storage in cache DB
2. create a tool to export a map to a cache DB

[incanus]

Removing label & editing title — this isn't specific to Android.

We've accomplished a lot of this in past across various toolsets on both iOS and Android.

• MBTiles is a spec for raster or vector tiles as blobs in SQLite with a known format and metadata table. This is used for prebunding offline maps with an app. It's not designed for adding to later, but could be remedied in some way if this seems like a good idea.
  • https://github.com/mapbox/mapbox-ios-sdk/blob/release/MapView/Map/RMMBTilesSource.h
  • add MBTiles support mbxmapkit#3
  • https://github.com/mapbox/mapbox-android-sdk/blob/65a1c81f72b3419ae22bc74e1b28adb0a71fbb11/MapboxAndroidSDK/src/main/java/com/mapbox/mapboxsdk/tileprovider/tilesource/MBTilesLayer.java

• need to add ability to swap out the cache database set it to a read-only mode

  • Agreed.

• need a tool that builds an offline database that can be distributed with the app/downloaded

  • https://github.com/mapbox/mbutil

• could also download all individual tiles on first launch, however this is not good strategy

  • It can be a good strategy, depending on usage, and becomes much more viable with vector tiles than raster.
  • https://github.com/mapbox/mbxmapkit/blob/master/MBXMapKit/MBXOfflineMapDownloader.h
  • https://github.com/mapbox/mapbox-ios-sdk/blob/release/MapView/Map/RMTileCache.h#L159
  • https://github.com/mapbox/mapbox-android-sdk/issues/533

[ljbade]

MBTiles is a spec for raster or vector tiles as blobs in SQLite with a known format and metadata table. This is used for prebunding offline maps with an app. It's not designed for adding to later, but could be remedied in some way if this seems like a good idea.

Being able to push out small update datasets will be needed if people start creating maps of whole countries, e.g. topo maps for all of Australia or NZ cause it will be annoying to redownload hundreds of MBs of map data every time the dataset gets updated (NZ updates their topo data monthly)

[ljbade]

It can be a good strategy, depending on usage, and becomes much more viable with vector tiles than raster.

True keep forgetting that vector data is a lot smaller.

[ljbade]

Perhaps as a first step, we could add support for pre-caching a bbox + zoom level of data?

[ljbade]

Some ideas of minimum functionality:

• function to calculate number of tiles needed for given bounding box and zoom range
• function to start offline caching - will take bbox, zoom max & min, some sort of callback/interface to receive progress notifications
• notification that receives - total number of tiles, tiles downloaded so far, number of failures (and possibly list of the failures), overall finished & successful indication

[ljbade]

After a quick look through the code here is a strategy:

• build list of all Sources needed for each layer
• for each Source tell it to fetch all needed TileData
• ensure TileData downloads and saves data into SQLiteStore via FileSource/HTTPRequest
• disable cache expiry check in HTTPRequest
• ???

[ljbade]

We will also need to save in cache:

• Style JSON
• Tile source JSON
• PNGs for styles

Is there anything else needed to be downloaded to render a map?

[ljbade]

Hmm, going to take me a while to figure out how to do this tidily.

Suggestions?

[incanus]

Perhaps as a first step, we could add support for pre-caching a bbox + zoom level of data?

We have built this out pretty robustly on iOS, so that's a good starting spot.

iOS SDK

In RMTileCache:

Estimates

-tileCountForSouthWest:northEast:minZoom:maxZoom:

Start/stop

-beginBackgroundCacheForTileSource:southWest:northEast:minZoom:maxZoom:
-cancelBackgroundCache

Progress

-tileCache:didBeginBackgroundCacheWithCount:forTileSource:
-tileCache:didBackgroundCacheTile:withIndex:ofTotalTileCount:
-tileCacheDidFinishBackgroundCache:
-tileCacheDidCancelBackgroundCache:

Storage format is a SQLite-based common store keyed on tile source, per the convention of the upstream project that we built this SDK off of. This same cache is used for performance caching during online use as well as offline fallback.

MBXMapKit

In MBXOfflineMapDownloader:

Start/stop

-beginDownloadingMapID:mapRegion:minimumZ:maximumZ:
-cancel
-resume
-suspend

Estimates & Progress

-offlineMapDownloader:stateChangedTo:
-offlineMapDownloader:totalFilesExpectedToWrite:
-offlineMapDownloader:totalFilesWritten:totalFilesExpectedToWrite:
-offlineMapDownloader:didEncounterRecoverableError:
-offlineMapDownloader:didCompleteOfflineMapDatabase:withError:

Here we had a combination of a clean slate for how to do this along with integration with an existing Apple API. Storage format is again SQLite but offline maps are treated as atomic, immutable documents.

We don't necessarily want to mirror the APIs to Java for any sort of undue parity, but these are some ideas. I think the idea of bounding box + zoom range, start/pause/cancel, and progress callbacks are the general model we want.

However, we should also decide whether to do this at the C++ level in some core way.

In my eyes, despite this being eventually important, it's far lower priority than general Android parity with current iOS tools + buildout of client side data (#507) so that mobile apps can start deploying on GL in the wild.

[hallahan]

Whatever happened with this issue? Is there now MBTiles support for Android? Is this done natively? This feature is a requirement for me to use this library as an Android app.

[incanus]

There is not MBTiles support yet in this project. Offline support will be coming in a future release; pretty sure this will include MBTiles.

[hallahan]

Darn, is there a branch this is being tried with? Is this going to be native C++ for all builds? For me, this is the most important feature I need. I'm interested in helping.

[incanus]

There's no movement on it yet, AFAIK. Right now we are sprinting towards a series of iOS beta releases, after which we'll focus on Android and a broader feature set. I would expect it to be in C++, yes, with Java/ObjC bindings.

Happy to take work in a branch, but it just likely won't get many eyeballs during March due to priorities elsewhere.

[kristfal]

Hey,

I know this is a large feature that will be implemented later, but I'm very curious regd. the performance differences, so wondering:

Is there currently a more elegant way to load tiles bundled with an app besides doing this and then load individual .pbfs referenced through a tilejson?

[xwz]

I've made some progress in adding mbtiles source. It is very basic at the moment, it simply loads tiles from the mbtiles file using sqlite.

[ljbade]

@xwz Great, have you put your changes on GitHub?

[xwz]

@ljbade See master...xwz:mbtiles-source

[vespakoen]

Great stuff, @xwz what is the reason that there is no pull request for your changes?

[speechlogger]

Will referring the map's source to local on-device raster-tiles through the style json not work offline?
Does the SDK need to verify online the access-token before making the map work, even if it doesn't use Mapbox hosted maps?

[jfirebaugh]

An access token will be required only when downloading resources from Mapbox APIs.

If by "local on-device raster-tiles" you're referring to the use of asset:// URLs, no -- offline will not support asset:// URLs (asset:// URLs are already inherently "offline"). But if the tiles are served by a local server with localhost URLs, they will work with offline the same as other http URLs.

[jfirebaugh]

⇢ #3715, #3891, #3892

[nnhubbard]

Since this was closed, do we have a release date?

[JonasVautherin]

They said in March! ;-)

[twbell]

Tracking progress and feedback on Offline here: #4178

[rm3dbeckers]

We've recently tested it and although it looks like the mapView itself accepts styleURLs looking like "asset://style.json" (they load and the tiles are being requested correctly), but the class OfflineTilePyramidRegionDefinition only seems to accept styleUrls of the form "mapbox://styles/mapbox/streets-v9". styleURLs pointing into the assets seem to later cause an error like getHost() called on null reference (or something along that line) and then result in a termination from within the mapbox-gl.so

[zugaldia]

@rm3dbeckers could you open a separate ticket to track this issue? Thanks for the feedback.

[kkaefer]

@rm3dbeckers asset:// URLs are already offline, so they shouldn't be stored in the cache.

[rm3dbeckers]

The problem is that I don't have any tiles in the assets but just the style.json which I want to use to download tiles. But using asset:// urls for the styleURL seems to only work for the mapview itself, not for the OfflineTilePyramidRegionDefinition.

btw: I've created an extra ticket at #5271 as requested.

[Stophface]

Is there mbtiles for ios and/or android?

[nitrag]

Chrckout openmaptiles. They have an extension for this. Here, no.

…

On Fri, Sep 29, 2017 at 11:19 AM Christoph ***@***.***> wrote: Is there mbtiles for ios and/or android? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#584 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMBkuv48XHtccxWPYiY8PvZVBtylBmkks5snQp0gaJpZM4C5RQr> .

[Stophface]

@nitrag thanks for the fast reply. I am searching for ready-to-use mbtiles for iOS or android. Although for android a solution might be osmdroid.

[nitrag]

Pretty sure it's ready to use. You just have to pay them for their module.

…

On Fri, Sep 29, 2017 at 11:29 AM Christoph ***@***.***> wrote: @nitrag <https://github.com/nitrag> thanks for the fast reply. I am searching for *ready-to-use* mbtiles for iOS or android. Although for android a solution might be osmdroid. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#584 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMBkjJ1PBWDRc5IgEGlNWP6F5vowL75ks5snQzXgaJpZM4C5RQr> .

[songyuyang0918]

Everybody is good! How does the latest IOS SDK load an offline map?
I have my own mbTiles file and don't want to download it via MapBox.
If there is a plug-in, please give me an address, thank you very much!

[tobrun]

https://docs.mapbox.com/help/troubleshooting/mobile-offline/

[ssshake]

Sorry to be posting on an old issue but I'm not finding a clear answer on the state of this in 2021.

We have a react native app using mapbox-gl-js so I'm realizing now I'm in the wrong repo. We have a need to pre-cache tiles for offline use as we are producing an app to be used on ipad by workers out in remote areas. They need to cache their tiles then go out to the job site.

To complicate matters we also need this cache to be in an application group / shared storage. Which is an ios concept to allow 2 or more applications by the same vendor to access common data storage.

We have two applications, one using mapbox-gl-js and the other using the unity mapbox SDK. The react native application needs to download and cache the tiles in shared storage. The unity application need to be able to access the offline tiles from shared storage.

Any help indicating where to go would be very helpful as I'm finding the number of libraries/sdks for mapbox and the supported feature set between them to be quite unclear. Thanks!

[hjhimanshu01]

@ssshake, were you able to render maps in offline mode?

[ssshake]

Yes (removed an essay about my specific issues as it's off topic)

[hjhimanshu01]

@ssshake, thanks for explaining, seems like quite an intricate solution you have got there. What I'm trying to achieve is related to 1), just to cache the tile data in a DB and read it later when offline, though I'm trying to achieve it in iOS using swift. Is it possible for you to just share some pointers on how did you cached those tiles and read it back in mapbox?

Once again thanks for explaining it in so much detail, cheers!

[ssshake]

Ah I see, sorry I thought you were a maintainer of this package trying to fix my issue.

I'm using the react-native-mapbox-gl package but I would hope it has the same methods as the native mapbox GL. You need to create a "pack" and that is implicitly saved to the mapbox sqlite DB.

Something like this:

    MapboxGL.offlineManager.setTileCountLimit(100000)

    await MapboxGL.offlineManager.createPack(
        {
            name: packName,
            styleURL: mapStyleUrl,
            minZoom: 8,
            maxZoom: 16,
            bounds: [
                [furthest.east, furthest.north],
                [furthest.west, furthest.south]
            ]
        },
        onProgress,
        errorListener
    )

Where furthest is an object containing the lat/long bounds that you want. We use two arrays of longitudes and latitudes for all of the points on the map we care about. Then we just get the min and max to construct the bounds.

{
        north: Math.max(...latitudes),
        east: Math.max(...longitudes),
        south: Math.min(...latitudes),
        west: Math.min(...longitudes)
}

I'm redacting some info in my prior comment because it's not helpful to getting my solution and is too off topic.

[hjhimanshu01]

Perfect, thanks for the info!