Desert Atlas is a general purpose, fully self-hosted map app for the Sandstorm platform (see What is Sandstorm? below), based on OpenStreetMap. All map data is stored on your own server; you choose the regions you want. The goal is to provide the basic functionality of Google Maps while respecting privacy.
You can check out a live demo here or install it on your Sandstorm instance.
Desert Atlas allows you to plan trip destinations with friends, store important locations, etc, and export the result to a convenient phone application like OrganicMaps for navigation. Perhaps some day (with some work on the phone app side) there could be a more automatic sync option. If there are other bits of standard functionality you would like to see, please let me know!
Some other self-hosted map apps will still get tile data from external sources on-demand, leaking some amount of your usage patterns to the outside world, and making you dependent on their services. With Desert Atlas, all of the map data is fully self-hosted in your Sandstorm grain, in the same way that OrganicMaps fully holds regions of the map on your phone. The map data needs to come from somewhere of course, so as with OrganicMaps you need to download the regions you need (about the size of a US state) from a map data server, but you only need to do this once per grain (until you want to update the data). Perhaps some day the data could be shared between grains on the same server, reducing the amount of downloads further, increasing privacy. But that's very much a stretch goal.
There are other fully self-hosted OSM solutions, admittedly including at least one that uses docker and is thus probably easy to set up. But thanks to Sandstorm, and Desert Atlas aiming for the "simplest version of everything" in the OSM ecosystem, Desert Atlas is even easier. A "full stack" OSM setup generally includes something like postgres for generating tiles and elasticsearch for search, neither of which play nicely with Sandstorm. Desert Atlas by contrast uses protomaps for tiles, and (as of now) sqlite3 with fts5 for search (a home-grown solution, importing search data extracted from raw OSM data using python bindings for the common tool osmium). No shade on the other solutions, they have more sophisticated search and nicer map renders, but Desert Atlas is simple enough to be a consumer app.
This is now at a "minimum viable product" status. Search is basic but functional. UI could use some tweaks. But it's usable. Try it out! Let me know what you think. Chime in on the Sandstorm groups, Github discussions, or file a feature request or bug report on Github. The more feedback I get (positive or negative) the more I know it's worth spending time on this. (If you know something about OSM or UI and think you could make this better, you're likely right. See here!)
Sandstorm is a user friendly platform for self-hosted web applications. Installation of the Sandstorm platform itself is pretty easy, and it updates itself in the background automatically. Once the platform is installed, everything is administered in-browser. Installing individual apps is like using an app store on the phone or desktop. The headaches of maintaning individual web apps are gone.
From a developer's point of view it can be a double-edged sword. Sandstorm has usability and security features built in, taking care of a lot of the overhead, allowing devs to focus on the core features of the app. On the other hand, it comes with a lot of restrictions, which can be especially challenging when porting existing apps. Desert Atlas is a new app, but it reuses some existing components that took a little work to fit into Sandstorm's restrictions.
You can read about Sandstorm's model in some depth on their website, but we'll go over it briefly here.
Most self-hosted web applications come in a single installation to handle all of its users. A standard word processor web application, for example, might have two users, each owning three documents, all represented in one database.
With Sandstorm's model, a similar word processor app would be designed to be controlled by only one user and have only one document. Those same two users would be able to each have 3 documents by spinning up three different instances of the app on demand, each with its own database. These instances are called "grains".
Sandstorm apps run in a restrictive, isolated container environment. No grain has access to any other grain without explicit user permission (via a popup in the UI), whether of the same or different apps.
Sandstorm restricts the app's access to the kernel API, which adds security. The environment is also single-user (as in, Unix user), which makes running databases like Postgres very difficult. However sqlite tends to scale just fine on Sandstorm because each grain will rarely have more than a few users interacting with it. Sqlite also starts faster, which is good since grains start and stop often on Sandstorm. The standard "full OpenStreetMap stack" uses Postgres and Elasticsearch, so Desert Atlas uses some stripped-down alternatives.
All of the grain's data ends up in one directory, making grain backups a simple matter of clicking a button in the UI to download a zip file.
Each grain's url contains a randomly generated string, making it hard for attackers to guess. A user can share one of their grains with other users (or even non-users) via a share link that contains its own hard-to-guess string. Each share link is revokable, and has a permission level associated. While the permissions UI is built into Sandstorm, the app defines the permission levels and is responsible for implementing them. For instance Desert Atlas has separate permissions for downloading map data and editing bookmarks.
Inbound connections from the browser go through a proxy to handle authentication. It adds special headers to tell the app which Sandstorm user is making the request and what permissions they're supposed to have.
Outbound network connections from the backend require explicit user permission (again via a popup) to prevent a malicious app from "phoning home". Desert Atlas makes use of this as users request to download map data for a given region.
First, check out the git submodules:
git submodule init
git submodule update dependencies
Normal stuff; vagrant-spk vm up, vagrant-spk dev. Just check out the Sandstorm packaging tutorial if you're unfamiliar.
spk-only is a bit more advanced. It's only recommended if you have a reason not to use Vagrant (such as developing on QubesOS that isn't as friendly to VirtualBox).
WARNING this should be built on an environment you do not mind modifying and risking breaking, etc. This will require installing things on your system and maybe changing things. The main author uses QubesOS and thus has a dedicated VM for this project.
Assumes Debian 11 (Bullseye) and that your current directory is the repository root
# This is how vagrant-spk likes to work, and it's easier to just follow suit
sudo mkdir /opt/app
sudo mount --bind . /opt/app
sudo .sandstorm/setup.sh
.sandstorm/build.sh
cd .sandstorm
spk dev
At this point it should be possible to run spk pack (within the .sandstorm directory) / vagrant-spk pack without running it first. All of the files should be there.
Please do not take what you see as being up to my standards. I hacked this out to the point where I could have a minimum viable product. Once released I'd like to clean it up.
Right now the back end is Python and the frontend is JavaScript, Leaflet, some JQuery thrown in. I'd like to rewrite the backend in Go. Maybe the frontend as well (WebAssembly with TinyGo).
The data generation scripts are mostly bash and python. I'd be happy with moving everything to Python.
spk dev
Sometimes useful. I should add some instructions. But it's currently shaky, so not worth documenting since it'll probably change.
But it involves running ./server.py. This will start a web server on port 3857 and possibly open a browser.
For generating map tile and search data, see here.
The original code in this repo is licensed MIT. However, Desert Atlas strings together a lot of different bits of software and data with different licenses, most of which are bundled in the resulting app. With a couple exceptions noted below, these things aren't vendored in this repository. In any case, I am noting the licenses related to third party work here.
In summary, big thanks for:
- Map Data: OpenStreetMap Foundation and all OSM editors, GeoNames, Natural Earth, Lexman, and Open Knowledge Foundation
- Map Tiles: mkgmap, Tilemaker, Protomaps and Geofabrik,
- Search: SQLite and PyOsmium
- Front end code: Leaflet, Leaflet Search, and JQuery
- Sandstorm integation: Powerbox Proxy
This is data that is built into the app, and is thus available as soon as the user starts a new grain. It includes a "wireframe" world and U.S. map, and the ability to search for cities and large towns.
- US States https://datahub.io/core/geo-admin1-us - Open Data Commons Public Domain Dedication and License
- World https://datahub.io/core/geo-countries - Open Data Commons Public Domain Dedication and License
Note that this is a separate data source than OpenStreetMap (as are the "wireframes"). As I understand, its license makes it incompatible as a source for importing to OSM's database, but I think it's fine to use side by side as we do here. (Let me know if this is wrong, and I can derive this data from OSM data instead)
This section relates to the periodic process of generating the regions of the world map which users explicitly choose to download after the grain starts. I.e. they are not built into the app (though some form of it is checked into this repository, as seen below).
These attributions are for the data generation tools, not the libraries in the app that use the data.
All downloadable regions data is derived from planet.osm.pbf from OpenStreetMap®. I also include a much reduced planet-test.osm.pbf in this repository.
- https://planet.osm.org/ - Open Data Commons Open Database License (ODbL) by the OpenStreetMap Foundation (OSMF).
Note
I believe that these licenses only apply to the splitter, not the app or data itself.
Mkgmap splitter splits the planet into smaller osm.pbf chunks.
- https://www.mkgmap.org.uk/download/splitter.html - GPL3, LGPL3, Apache 2, XPP3 (apache-like license)
Note
I believe that these licenses only apply to the tilemaker, go-pmtiles, and Shortbread, not the app or data itself.
Tilemaker converts each region from osm.pbf to mbtiles, go-pmtiles converts from mbtiles to pmtiles. I started from the Shortbread schema from Geofabrik, edited it to make into Protomaps schema. See generate-data directory.
- https://github.com/systemed/tilemaker/ - FTWPL
- https://github.com/protomaps/go-pmtiles/ - BSD 3-Clause
- https://github.com/shortbread-tiles/shortbread-docs - CC0
Note
I believe that this license only applies to pyosmium, not the app or data itself.
Extracting search data from raw OSM data for each region into a csv. See generate-data directory.
- https://osmcode.org/pyosmium/ - BSD 2-Clause
- Leaflet - Map UI framework - BSD 2-Clause
- Leaflet Search - plugin for search UI - MIT
- Protomaps JS (with modifications) - Display pmtiles vector map files - BSD 3-Clause
- JQuery - MIT
- Powerbox Proxy - Facilitate outbound requests from Sandstorm - Apache 2
- In this repository:
assets/js/powerbox-helper.js, which is built using typescript from a file in the Powerbox repository. (I didn't want to require npm for building)
- In this repository:
Each downloaded map region contains a csv file that is imported into an SQLite database with the FTS5 plugin for full-text search.
- SQLite+FTS5 - included in Sandstorm application package via Debian, see below for licensing
The Desert Atlas icon (in this repository, licensed via BSD 2-Clause due to the below licenses) was put together from a few pieces of clip art:
Source images:
- https://openclipart.org/detail/336198/home-map-colour-remix - CC0
- https://openclipart.org/detail/306084/digital-landscape-illustration-2 - CC0
- https://github.com/Leaflet/Leaflet/blob/0042d0b0ddac8e9159ee4f64742bb25b518b9e0f/src/images/marker.svg - BSD 2-Clause
In this repository, possibly edited from original, which can be found here: https://github.com/Leaflet/Leaflet/blob/0042d0b0ddac8e9159ee4f64742bb25b518b9e0f/src/images/marker.svg
Various other things from Debian Bullseye (via QubesOS) when creating the Sandstorm package.