Ambient cache initialization speed improvement #1959

geolives-contact · 2023-12-14T14:03:52Z

In order to solve #1815

Here is the matching design proposal : #1877

This reverts commit 07f7fa4.

…Resource / putInternal and getTile / getResource which support that Fix for NOT NULL contraint violation for "compressed" field when inserting tile or resource Changed SQL query for ambient cache initialization

…iles table when resource / tile is needed in an offline region download

…s tables

…s tables (added to the upgrade script) + queries modified to manage LRU based on new tables

…n yet at the moment but it is working

… cleaner

…and c_str()

Fixed existing tests to comply with new database model (work in progress) + temporarily disabled some database merging tests which crashed + added new MigrateFromV6Schema test method

added v6.db in test/fixtures/offline_database

…deleted after deleting an offline region

…ew database model version (v7)

…w database model version (v7)

for more information, see https://pre-commit.ci

geolives-contact · 2023-12-14T14:08:59Z

The OfflineDatabase.CorruptDatabaseOnQuery still fails.

We would need to create a corrupted database in version 7 of the database model, which opens without any error, but where the first query triggers error of corruption

(with test database in version 6, simply instantiating the object triggers the corruption error when it tries to create the new tables for the model upgrade, which makes the test fail)

louwers · 2023-12-18T23:02:05Z

How did you make the databases?

platform/ios/app/MBXAppDelegate.m

geolives-contact · 2023-12-19T14:16:52Z

@louwers

For databases used for merging / sideloading tests (satellite_test.db, sideload_ambient.db, sideload_sat.db, and sideload_sat_multiple.db), I simply upgraded them with a SQLite database tool by running the upgrade script to model version 7.

For v6.db database (which is used to test the upgrade from version 6 to version 7), I made a copy of v5.db and upgraded it with a SQLite database tool by running the upgrade script to model version 6.

I still haven't managed to create the upgraded corrupt-delayed.db database (a corrupted database in version 7 of the database model, which opens without any error, but where the first query triggers error of corruption) to make the OfflineDatabase.CorruptDatabaseOnQuery test succeed :-(

github-actions · 2023-12-19T15:00:56Z

Bloaty Results (iOS) 🐋

Compared to main

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.1% +20.0Ki  +0.1% +16.0Ki    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results-ios/pr-1959-compared-to-main.txt

CorruptDatabaseOnQuery test temporarily disabled

for more information, see https://pre-commit.ci

geolives-contact · 2024-02-16T08:46:56Z

CorruptDatabaseOnQuery test has been temporarily disabled as discussed with @louwers
Could you please proceed with merge? :-)
Thanks a lot.

github-actions · 2024-02-16T08:48:02Z

Bloaty Results 🐋

Compared to main

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0% +51.8Ki  +0.1% +21.6Ki    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-1959-compared-to-main.txt

Compared to d387090 (legacy)

    FILE SIZE        VM SIZE    
 --------------  -------------- 
   +19% +22.2Mi  +401% +24.0Mi    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-1959-compared-to-legacy.txt

louwers · 2024-02-28T23:09:53Z

platform/default/src/mbgl/storage/offline_database.cpp

+            "SELECT length(data) FROM ambient_tiles WHERE url_template = ?1 AND pixel_ratio = ?2 AND x = ?3 AND y = ?4 "
+            "AND z = ?5");
+        if (selectAmbientTilesResult) {
+            // std::cout << "-------- HASTILE - FOUND IN AMBIENT_TILES\n";


There are commented out lines that print used for debugging throughout this file.

Please remove them or change them to (debug) logging.

louwers · 2024-02-29T00:16:33Z

I ran the offline_database.benchmark.cpp benchmarks:

$ python3 vendor/benchmark/tools/compare.py --no-color benchmarks ./mbgl-benchmark-runner-main ./mbgl-benchmark-runner --benchmark_filter='OfflineDatabase/.*'

Benchmark                                                     Time             CPU      Time Old      Time New       CPU Old       CPU New
------------------------------------------------------------------------------------------------------------------------------------------
OfflineDatabase/InvalidateRegion                           -0.0228         -0.0226        144618        141326        144231        140973
OfflineDatabase/DeleteRegion                               +0.0586         +0.0588      22971082      24316111      22911370      24259681
OfflineDatabase/InsertTileRegion                           -0.0590         -0.0587        236233        222301        235708        221865
OfflineDatabase/InvalidateAmbientCache                     -0.5156         -0.5156         81160         39313         81146         39308
OfflineDatabase/ClearAmbientCache                          +0.0617         +0.0620      23023497      24444877      22966607      24389660
OfflineDatabase/InsertTileCache                            -0.0130         -0.0131        218526        215695        218123        215269
OfflineDatabase/InsertBigTileCache                         -0.0167         -0.0166       2157706       2121650       2157331       2121517
OfflineDatabase/GetTile                                    +0.0092         +0.0092        138014        139284        138001        139266
OfflineDatabase/AddTilesToFullDatabase                     -0.0043         -0.0043        166905        166193        166889        166178
OfflineDatabase/AddTilesToDisabledDatabase                 +0.0309         +0.0309           932           961           932           961
OfflineDatabase/GetTileFromDisabledDatabase                +0.0375         +0.0376           744           772           744           772
OfflineDatabase/ResizeDatabase                             +0.0141         +0.0144            11            11            11            11
OVERALL_GEOMEAN                                            -0.0516         -0.0515             0             0             0             0

No significant difference interestingly.

Removed logs in comments

…ves/maplibre-native into fix/ambient-cache-slowness

geolives-contact · 2024-02-29T10:47:39Z

Hello @louwers

I have just removed the cout logs in comments.

About the performance test, the improvements are significant when there is a lot of offline data downloaded as offline regions (for example 1,5 GB). The startup of the app is much faster in that case :-)

for more information, see https://pre-commit.ci

platform/default/src/mbgl/storage/offline_database.cpp

louwers · 2024-03-04T14:27:53Z

platform/default/src/mbgl/storage/offline_database.cpp

-                                  const Response& response,
-                                  const std::string& data,
-                                  bool compressed) {
+bool OfflineDatabase::putResource(


There is too much code duplication in this function and the putTile function below.

Instead of adding a bool I would for example define an enum class that can be converted to a string to get the related database name using another utility function.

We agree with you about the code duplication issue, but we encountered difficulties in resolving it :-(

Initially, we attempted to minimize code duplication in methods like getTile(), getResource(), putTile(), putResource(), etc., by using lambdas and c_str() to concatenate the table name in the query. However, this approach led to memory issues (EXC_BAD_ACCESS) and erratic behavior, particularly errors during the insertion of tiles/resources, for example :
[DEBUG] {abort at 23 in [INSERT INTO ambient_tiles (url_template, pixel_ratio, x, y, z, modified, must_revalidate, etag, expires, accessed, data, compressed) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12)]: NOT NULL constraint failed: ambient_tiles.compressed}Database: %s

Consequently, we had to revert our changes, even though it resulted in increased code duplication.

You can find here an old version of the code here with our trials for getTile() and getResource() method: Geolives@cb8cf11

As our expertise in C++ is not as strong as it is in other higher-level languages, we welcome any assistance from the community to help resolve this issue of code duplication without causing adverse effects :-)

Thanks in advance.

louwers · 2024-03-04T21:10:58Z

I downloaded 4GB of offline data.

This is a build before this PR. It loads pretty quickly...?

https://youtube.com/shorts/CZqLHsIXg4I?si=abdOzSF47ZvxOeJ5

This is a build that includes this PR:

https://youtube.com/shorts/ocZza9EIdzk?si=9-9kGuG6XKy4xnXd

I guess that means that initAmbientCacheSize is not called in this instance.

louwers · 2024-03-05T13:50:31Z

platform/default/include/mbgl/storage/offline_schema.hpp

@@ -35,6 +35,35 @@ static constexpr const char* offlineDatabaseSchema =
    "  must_revalidate INTEGER NOT NULL DEFAULT 0,\n"
    "  UNIQUE (url_template, pixel_ratio, z, x, y)\n"
    ");\n"


This file is generated and should not be modified manually.

Could you try updating offline_schema.js and running that?

OK, I didn't know that the file was autogenerated, sorry, I will try to do that :-)

louwers · 2024-03-05T14:09:56Z

platform/default/src/mbgl/storage/offline_database.cpp

+    // First, try to find the tile in the 'tiles' table
+    std::optional<int64_t> selectTilesResult = extractTileDataSize(
+        tile,
+        "SELECT length(data) FROM tiles WHERE url_template = ?1 AND pixel_ratio = ?2 AND x = ?3 AND y = ?4 AND z = ?5");


This will always fail for users not using any offline functionality at all. I'm not comfortable proceeding without first ensuring the performance impact is minimal, otherwise we need to keep track of if the tiles table contains any tiles at all and skip this check.

We agree with you on this point, a boolean variable which would be initialized on startup (by counting tiles / resources, or even by checking if there is an offline_region in the database or not) and set when we add the first resource / tile is a good idea in our opinion, we will try to implement that.

louwers

Again apologies for taking so long for me to review this, but I wasn't at all familiar with this subsystem of the library.

It looks like the problematic initAmbientCacheSize method was only introduced so that cache size could be determined without taking into account offline data. mapbox/mapbox-gl-native#15622

It seems these changed can be reverted after the split?

By the way, I was not able to reproduce the performance problems. Are you calling setMaximumAmbientCacheSize on startup? It seems somehow I was able to store 3GB of data with OfflineActivity.kt, without callingsetMaximumAmbientCacheSize and without an evict being triggered (these are the only places that call initAmbientCacheSize).

geolives-contact · 2024-03-05T15:09:15Z

@louwers

I downloaded 4GB of offline data.

This is a build before this PR. It loads pretty quickly...?

https://youtube.com/shorts/CZqLHsIXg4I?si=abdOzSF47ZvxOeJ5

This is a build that includes this PR:

https://youtube.com/shorts/ocZza9EIdzk?si=9-9kGuG6XKy4xnXd

I guess that means that initAmbientCacheSize is not called in this instance.

In our application, the problem was much more reproducible on iOS than on Android (the reason for this is unclear).

In the iOS test application integrated into the maplibre-native project, the problem could be reproduced as explained here: #1815, and as you can see in this video: https://github.com/maplibre/maplibre-native/assets/13694294/a7c2bf01-54ff-43ed-86b1-675efb79e00a.

On Android, the performance issue was significantly less pronounced.

louwers · 2024-03-05T22:31:50Z

@Geolives Alright, I will try iOS as well.

What do you think about removing initAmbientCacheSize() and some of the other changes introduced alongside it? It does not serve a purpose anymore after the database is split does it?

geolives-contact · 2024-03-06T14:51:06Z

What do you think about removing initAmbientCacheSize() and some of the other changes introduced alongside it? It does not serve a purpose anymore after the database is split does it?

It requires further investigation because we still need to compute the ambient cache size to avoid exceeding the maximum chosen size. However, you are right that we need to understand how it was done before integrating mapbox/mapbox-gl-native#15622

louwers · 2024-11-15T14:13:56Z

@geolives-contact Do you still want to get this merged?

geolives-contact · 2024-11-19T13:15:28Z

@geolives-contact Do you still want to get this merged?

Yes, unfortunately at the moment we don't have enough time to work on the requested changes / investigation needed, sorry, we hope to be able to work on it again in the beginning of 2025. Thanks for your patience.

louwers · 2024-11-19T14:02:18Z

No problem and thanks for the update.

Christophe Brasseur and others added 18 commits October 31, 2023 16:38

Update MBXViewController.m

07f7fa4

Revert "Update MBXViewController.m"

a878d90

This reverts commit 07f7fa4.

New tables created for ambient cache + first version of putTile / put…

b7a4bcf

…Resource / putInternal and getTile / getResource which support that Fix for NOT NULL contraint violation for "compressed" field when inserting tile or resource Changed SQL query for ambient cache initialization

Moving data from ambient_resources/ambient_tiles table to resources/t…

35ba68d

…iles table when resource / tile is needed in an offline region download

Indexes added on accessed field in ambient_tiles and ambient_resource…

864837f

…s tables

Indexes added on accessed field in ambient_tiles and ambient_resource…

e0bc77f

…s tables (added to the upgrade script) + queries modified to manage LRU based on new tables

Fix for EXC_BAD_ACCESS crash when downloading maps - code is not clea…

5c75a7b

…n yet at the moment but it is working

Code refactoring for hasTile and hasResource methods to make the code…

2085fcb

… cleaner

Refactoring of getTile and getResouce methods to avoid using lambdas …

cb8cf11

…and c_str()

Update offline_database.test.cpp

7758093

Fixed existing tests to comply with new database model (work in progress) + temporarily disabled some database merging tests which crashed + added new MigrateFromV6Schema test method

Create v6.db

67767ac

added v6.db in test/fixtures/offline_database

Resources and tiles which are not used in any offline region are now …

4307dc9

…deleted after deleting an offline region

Merge branch 'maplibre:main' into fix/ambient-cache-slowness

9d0f1d9

Re-enabled tests which were temporarily disabled

d0e6330

Databases merging (sideloading) feature now supports databases with n…

2857360

…ew database model version (v7)

Test databases used for database merging (sideloading) upgraded to ne…

14716aa

…w database model version (v7)

Merge branch 'maplibre:main' into fix/ambient-cache-slowness

b3aaeae

[pre-commit.ci] auto fixes from pre-commit.com hooks

67faa0b

for more information, see https://pre-commit.ci

Removed unused variable

a21ee01

louwers reviewed Dec 18, 2023

View reviewed changes

platform/ios/app/MBXAppDelegate.m Outdated Show resolved Hide resolved

louwers added the enhancement New feature or request label Dec 18, 2023

louwers requested a review from JesseCrocker December 18, 2023 23:04

Removed debug log

d948995

Geolives and others added 3 commits February 16, 2024 09:13

Update offline_database.test.cpp

f4838dd

CorruptDatabaseOnQuery test temporarily disabled

[pre-commit.ci] auto fixes from pre-commit.com hooks

16f3390

for more information, see https://pre-commit.ci

Merge branch 'maplibre:main' into fix/ambient-cache-slowness

4a1a8fd

louwers reviewed Feb 28, 2024

View reviewed changes

Geolives and others added 3 commits February 29, 2024 11:41

Merge branch 'maplibre:main' into fix/ambient-cache-slowness

dcd4d00

Update offline_database.cpp

760cb5e

Removed logs in comments

Merge branch 'fix/ambient-cache-slowness' of https://github.com/geoli…

ba243f5

…ves/maplibre-native into fix/ambient-cache-slowness

[pre-commit.ci] auto fixes from pre-commit.com hooks

a9e4fd3

for more information, see https://pre-commit.ci

louwers reviewed Mar 2, 2024

View reviewed changes

platform/default/src/mbgl/storage/offline_database.cpp Show resolved Hide resolved

louwers reviewed Mar 4, 2024

View reviewed changes

louwers reviewed Mar 5, 2024

View reviewed changes

louwers requested changes Mar 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ambient cache initialization speed improvement #1959

Ambient cache initialization speed improvement #1959

geolives-contact commented Dec 14, 2023

geolives-contact commented Dec 14, 2023

louwers commented Dec 18, 2023

geolives-contact commented Dec 19, 2023 •

edited

Loading

github-actions bot commented Dec 19, 2023 •

edited

Loading

geolives-contact commented Feb 16, 2024

github-actions bot commented Feb 16, 2024 •

edited

Loading

louwers Feb 28, 2024

louwers commented Feb 29, 2024 •

edited

Loading

geolives-contact commented Feb 29, 2024

louwers Mar 4, 2024

geolives-contact Mar 5, 2024

louwers commented Mar 4, 2024 •

edited

Loading

louwers Mar 5, 2024

louwers Mar 5, 2024

geolives-contact Mar 5, 2024

louwers Mar 5, 2024

geolives-contact Mar 6, 2024

louwers left a comment •

edited

Loading

geolives-contact commented Mar 5, 2024

louwers commented Mar 5, 2024

geolives-contact commented Mar 6, 2024

louwers commented Nov 15, 2024

geolives-contact commented Nov 19, 2024

louwers commented Nov 19, 2024

Ambient cache initialization speed improvement #1959

Are you sure you want to change the base?

Ambient cache initialization speed improvement #1959

Conversation

geolives-contact commented Dec 14, 2023

geolives-contact commented Dec 14, 2023

louwers commented Dec 18, 2023

geolives-contact commented Dec 19, 2023 • edited Loading

github-actions bot commented Dec 19, 2023 • edited Loading

Bloaty Results (iOS) 🐋

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results-ios/pr-1959-compared-to-main.txt

geolives-contact commented Feb 16, 2024

github-actions bot commented Feb 16, 2024 • edited Loading

Bloaty Results 🐋

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-1959-compared-to-main.txt

Choose a reason for hiding this comment

louwers commented Feb 29, 2024 • edited Loading

geolives-contact commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

louwers commented Mar 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

louwers left a comment • edited Loading

Choose a reason for hiding this comment

geolives-contact commented Mar 5, 2024

louwers commented Mar 5, 2024

geolives-contact commented Mar 6, 2024

louwers commented Nov 15, 2024

geolives-contact commented Nov 19, 2024

louwers commented Nov 19, 2024

geolives-contact commented Dec 19, 2023 •

edited

Loading

github-actions bot commented Dec 19, 2023 •

edited

Loading

github-actions bot commented Feb 16, 2024 •

edited

Loading

louwers commented Feb 29, 2024 •

edited

Loading

louwers commented Mar 4, 2024 •

edited

Loading

louwers left a comment •

edited

Loading