Fatal Exception: mapbox::sqlite::Exception: disk I/O error

[MikeKasperlik]

We see a crash that happens frequently. I don't know how to trigger this. We also do nothing fancy with mapbox currently, only setting the camera to the users location + creating some annotations.

According to crashlytics this only happens when the app is not in foreground - maybe an issue with file protection?

Platform: iOS 9.2.1 - iOS 9.3.1
Mapbox SDK version: 3.2.1

Crashed: DefaultFileSource
0  PACEApp                        0x1005075ac CLSProcessRecordAllThreads + 4300240300
1  PACEApp                        0x1005075ac CLSProcessRecordAllThreads + 4300240300
2  PACEApp                        0x1005079cc CLSProcessRecordAllThreads + 4300241356
3  PACEApp                        0x1004f8680 CLSHandler + 4300179072
4  PACEApp                        0x100505cc4 __CLSExceptionRecord_block_invoke + 4300233924
5  libdispatch.dylib              0x1811fd47c _dispatch_client_callout + 16
6  libdispatch.dylib              0x181208728 _dispatch_barrier_sync_f_invoke + 100
7  PACEApp                        0x100505770 CLSExceptionRecord + 4300232560
8  PACEApp                        0x1005052f4 CLSTerminateHandler() + 4300231412
9  libc++abi.dylib                0x180e0af44 std::__terminate(void (*)()) + 16
10 libc++abi.dylib                0x180e0a85c __cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*) + 134
11 Mapbox                         0x10149fc04 mapbox::sqlite::Statement::run() + 156
12 Mapbox                         0x10149a50c long long mbgl::OfflineDatabase::getPragma<long long>(char const*) + 36
13 Mapbox                         0x101498f80 mbgl::OfflineDatabase::evict(unsigned long long) + 72
14 Mapbox                         0x101498e74 mbgl::OfflineDatabase::putInternal(mbgl::Resource const&, mbgl::Response const&, bool) + 280
15 Mapbox                         0x101489f40 mbgl::DefaultFileSource::Impl::Task::Task(mbgl::Resource, std::__1::function<void (mbgl::Response)>, mbgl::DefaultFileSource::Impl*)::'lambda'(mbgl::Response)::operator()(mbgl::Response) const + 40
16 Mapbox                         0x101489e84 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4mbgl17DefaultFileSource4Impl4TaskC1ENS3_8ResourceENS_8functionIFvNS3_8ResponseEEEEPS5_EUlS9_E_S9_EEEvDpOT_ + 40
17 Mapbox                         0x10148f368 mbgl::util::RunLoop::Invoker<auto std::__1::unique_ptr<mbgl::WorkRequest, std::__1::default_delete<std::__1::default_delete> > mbgl::util::RunLoop::invokeWithCallback<auto mbgl::util::Thread<mbgl::OnlineFileSource::Impl>::bind<void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>)>(void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>))::'lambda'(void (mbgl::OnlineFileSource::Impl::*&&)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>)), std::__1::function<void (mbgl::Response)>&, mbgl::OnlineFileSource::request(mbgl::Resource const&, std::__1::function<void (mbgl::Response)>)::OnlineFileRequest*, mbgl::Resource&>(void (mbgl::OnlineFileSource::Impl::*&&)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>), std::__1::function<void (mbgl::Response)>&&&, mbgl::OnlineFileSource::request(mbgl::Resource const&, std::__1::function<void (mbgl::Response)>)::OnlineFileRequest*&&, mbgl::Resource&&&)::'lambda'(auto mbgl::util::Thread<mbgl::OnlineFileSource::Impl>::bind<void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>)>(void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>))::'lambda'(void (mbgl::OnlineFileSource::Impl::*&&)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>)))::operator()<mbgl::Response>(auto mbgl::util::Thread<mbgl::OnlineFileSource::Impl>::bind<void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>)>(void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>))::'lambda'(void (mbgl::OnlineFileSource::Impl::*&&)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>))) const::'lambda'(auto mbgl::util::Thread<mbgl::OnlineFileSource::Impl>::bind<void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>)>(void (mbgl::OnlineFileSource::Impl::*)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>))::'lambda'(void (mbgl::OnlineFileSource::Impl::*&&)(mbgl::FileRequest*, mbgl::Resource, std::__1::function<void (mbgl::Response)>))), std::__1::tuple<mbgl::Response> >::operator()() + 108
18 Mapbox                         0x101482754 mbgl::util::RunLoop::process() + 320
19 Mapbox                         0x1014eac38 uv__async_event + 80
20 Mapbox                         0x1014eadf8 uv__async_io + 168
21 Mapbox                         0x1014f6780 uv__io_poll + 1248
22 Mapbox                         0x1014eb30c uv_run + 424
23 Mapbox                         0x1014879d0 void mbgl::util::Thread<mbgl::DefaultFileSource::Impl>::run<std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long long&>, 0ul, 1ul>(mbgl::util::ThreadContext, std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long long&>&&, std::__1::integer_sequence<unsigned long, 0ul, 1ul>) + 164
24 Mapbox                         0x1014878cc std::__1::__thread_proxy<std::__1::tuple<mbgl::util::Thread<mbgl::DefaultFileSource::Impl>::Thread<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long long&>(mbgl::util::ThreadContext const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, unsigned long long&&&)::'lambda'()> >(void*, void*) + 168
25 libsystem_pthread.dylib        0x181417b28 _pthread_body + 156
26 libsystem_pthread.dylib        0x181417a8c _pthread_body + 154
27 libsystem_pthread.dylib        0x181415028 thread_start + 4

Other threads:

OnlineFileSource
0  libsystem_kernel.dylib         0x181335528 kevent + 8
1  Mapbox                         0x1014f6538 uv__io_poll (kqueue.c:142)
2  Mapbox                         0x1014eb30c uv_run (core.c:341)
3  Mapbox                         0x10148fba4 void mbgl::util::Thread<mbgl::OnlineFileSource::Impl>::run<std::__1::tuple<int>, 0ul>(mbgl::util::ThreadContext, std::__1::tuple<int>&&, std::__1::integer_sequence<unsigned long, 0ul>) + 116
4  Mapbox                         0x10148fad0 std::__1::__thread_proxy<std::__1::tuple<mbgl::util::Thread<mbgl::OnlineFileSource::Impl>::Thread<int>(mbgl::util::ThreadContext const&, int&&)::'lambda'()> >(void*, void*) + 168
5  libsystem_pthread.dylib        0x181417b28 _pthread_body + 156
6  libsystem_pthread.dylib        0x181417a8c _pthread_body + 154
7  libsystem_pthread.dylib        0x181415028 thread_start + 4

AssetFileSource
0  libsystem_kernel.dylib         0x181335528 kevent + 8
1  Mapbox                         0x1014f6538 uv__io_poll (kqueue.c:142)
2  Mapbox                         0x1014eb30c uv_run (core.c:341)
3  Mapbox                         0x1014d69cc void mbgl::util::Thread<mbgl::AssetFileSource::Impl>::run<std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, 0ul>(mbgl::util::ThreadContext, std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&&, std::__1::integer_sequence<unsigned long, 0ul>) + 112
4  Mapbox                         0x1014d68fc std::__1::__thread_proxy<std::__1::tuple<mbgl::util::Thread<mbgl::AssetFileSource::Impl>::Thread<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(mbgl::util::ThreadContext const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&)::'lambda'()> >(void*, void*) + 168
5  libsystem_pthread.dylib        0x181417b28 _pthread_body + 156
6  libsystem_pthread.dylib        0x181417a8c _pthread_body + 154
7  libsystem_pthread.dylib        0x181415028 thread_start + 

Map
0  libsystem_kernel.dylib         0x18134bf24 __psynch_cvwait + 8
1  libsystem_pthread.dylib        0x181416ce8 _pthread_cond_wait + 648
2  libc++.1.dylib                 0x180da342c std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 56
3  Mapbox                         0x101400144 mbgl::MapContext::pause() + 112
4  Mapbox                         0x1013fc518 mbgl::util::RunLoop::Invoker<auto mbgl::util::Thread<mbgl::MapContext>::bind<void (mbgl::MapContext::*)()>(void (mbgl::MapContext::*)())::'lambda'(void (mbgl::MapContext::*&&)()), std::__1::tuple<> >::operator()() + 80
5  Mapbox                         0x101482754 mbgl::util::RunLoop::process() + 320
6  Mapbox                         0x1014eac38 uv__async_event + 80
7  Mapbox                         0x1014eadf8 uv__async_io + 168
8  Mapbox                         0x1014f6780 uv__io_poll + 1248
9  Mapbox                         0x1014eb30c uv_run + 424
10 Mapbox                         0x1013fb780 void mbgl::util::Thread<mbgl::MapContext>::run<std::__1::tuple<mbgl::View&, mbgl::FileSource&, mbgl::MapMode&, mbgl::GLContextMode&, float>, 0ul, 1ul, 2ul, 3ul, 4ul>(mbgl::util::ThreadContext, std::__1::tuple<mbgl::View&, mbgl::FileSource&, mbgl::MapMode&, mbgl::GLContextMode&, float>&&, std::__1::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul>) + 132
11 Mapbox                         0x1013fb69c std::__1::__thread_proxy<std::__1::tuple<mbgl::util::Thread<mbgl::MapContext>::Thread<mbgl::View&, mbgl::FileSource&, mbgl::MapMode&, mbgl::GLContextMode&, float>(mbgl::util::ThreadContext const&, mbgl::View&&&, mbgl::FileSource&&&, mbgl::MapMode&&&, mbgl::GLContextMode&&&, float&&)::'lambda'()> >(void*, void*) + 168
12 libsystem_pthread.dylib        0x181417b28 _pthread_body + 156
13 libsystem_pthread.dylib        0x181417a8c _pthread_body + 154
14 libsystem_pthread.dylib        0x181415028 thread_start + 4

Worker (4 instances with equal trace)
0  libsystem_kernel.dylib         0x181335528 kevent + 8
1  Mapbox                         0x1014f6538 uv__io_poll (kqueue.c:142)
2  Mapbox                         0x1014eb30c uv_run (core.c:341)
3  Mapbox                         0x101474510 void mbgl::util::Thread<mbgl::Worker::Impl>::run<std::__1::tuple<> >(mbgl::util::ThreadContext, std::__1::tuple<>&&, std::__1::integer_sequence<unsigned long>) + 96
4  Mapbox                         0x101474450 std::__1::__thread_proxy<std::__1::tuple<mbgl::util::Thread<mbgl::Worker::Impl>::Thread<>(mbgl::util::ThreadContext const&)::'lambda'()> >(void*, void*) + 168
5  libsystem_pthread.dylib        0x181417b28 _pthread_body + 156
6  libsystem_pthread.dylib        0x181417a8c _pthread_body + 154
7  libsystem_pthread.dylib        0x181415028 thread_start + 4

[1ec5]

Looks like an uncaught C++ exception while evicting resources from the cache.

/cc @jfirebaugh

[pnull]

Any news on this?

[tmpsantos]

Sounds like something to do withData Protection classes. I.e. flags for how we open the cache.
https://www.apple.com/business/docs/iOS_Security_Guide.pdf

Can you trigger it by browsing and immediately locking the phone?

[MikeKasperlik]

That seems to trigger it sometimes (i guess only when i'm fast enough).

[MikeKasperlik]

As we expiring these crashes frequently we need a solution. In which directory are these caches saved? I will try to mark these directory with no file protection to see if these errors disappear.

[1ec5]

In iOS SDK v3.2.0 and above, the cache is saved at ~/Library/Application Support/tld.app.bundle.id/cache.db, where tld.app.bundle.id is your application’s bundle identifier.

[jfirebaugh]

#5592 gives the hint that you can use SQLITE_OPEN_FILEPROTECTION_NONE to disable data protection on the sqlite files.

[friedbunny]

It may turn out to be unnecessary, but as @tmpsantos suggested, the system file protection key for cache.db may also need to be modified. NSFileProtectionCompleteUntilFirstUserAuthentication would be a good starting point, while completely disabling file protection via NSFileProtectionNone should be a last resort.

[jfirebaugh]

I think the right fix for this is #5236, so that no IO happens when an app is backgrounded.

[jfirebaugh]

Actually, I take that back -- you would want offline downloads to continue in the background. The pause/resume API should only be about automatic revalidation.

[MikeKasperlik]

Setting the cache.db and the containing directory to NSFileProtectionNone seemed to fix it - instead #5651 pops up now.

[1ec5]

Note that the cache.db file moved as a result of #5601, so you may need to set NSFileProtectionNone on different paths now.

[MikeKasperlik]

Seems to be fixed in 3.3.0.

[1ec5]

Thanks, @MikeKasperlik! Let us know if it crops back up.

[friedbunny]

This is reportedly still a problem in v3.3.4.

[boundsj]

We expect that this issue will be resolved by #8125 that is in the Mapbox iOS SDK as of 3.5.0 beta 2. I'll leave this issue open for tracking until we can collect more evidence that this particular sqlite error no longer occurs.

[jadar]

This is still a problem for me in 3.5.4. It seems to happen the first time the app launches after a fresh install.

[boundsj]

Thanks for the report @jadar and I'm sorry to hear that.

It seems to happen the first time the app launches after a fresh install.

That's interesting. If you have any other details that you can share (i.e. patterns of device / iOS version, crash logs, frequency, etc.) we would greatly appreciate it.

[jadar]

Thanks for the quick reply. Right now I'm testing on an iPhone 7 with iOS 10.3.1. The issue happens about every time when we're logging into our app and all the views are initializing (including a MGLMapView.)

After that first login, though, the app launches pretty reliably each subsequent time with only non-crashing exceptions being thrown & caught on. Maybe because the cache SQLite is fresh?

Here's the stack trace I'm seeing.

[jfirebaugh]

@jadar Are you actually getting an application crash, or is it just stopping in the debugger, e.g. because you have an exception breakpoint enabled? Exceptions are used for flow control in this specific instance, so the fact that one is being thrown isn't necessarily an indication of a bug.

[jadar]

Yes, you're right. I'm getting both and it's sometimes hard for me to tell which one I'm getting. It seems indeterminate if it's an exception or a crash, but the crashes are happening.

[jfirebaugh]

What are the symptoms that lead you to believe there's a crash, and not just a (handled) exception?

[jadar]

Just some more background, this app uses CoreData and it does some data processing in a background CoreData context. CoreData also uses SQLite to provide persistence.

Here are the call stacks for when an actual crash happens. It appears it happens while we're doing more background data processing, which would explain why it only happens during some launches.

[jfirebaugh]

Can you post the console output and the full stack trace from the "DefaultFileSource (13)" thread? (In the screenshot frames 1-13 have been collapsed.)

[jadar]

Crash log: https://pastebin.com/cJY5Km80
Console log: https://pastebin.com/vH82A6cQ

[jadar]

I might have just figured out my issue. I found a spot in our app where the entire Application Support directory was deleted to remove all caches. I changed it to only remove certain directories and it seems that the crash has stopped occurring.

I will admit that was our oversight. But there would be two things that would be extremely helpful for both us and other people to debug:

1. The Mapbox cache directory in the Application Support directory shouldn't be hidden with a dot prefix. No one will see it anyway and it just confuses us developers who aren't as familiar with the Framework.
2. There could be a more descriptive error thrown. Perhaps the error "Disk I/O error" should have tipped me off but it still feels pretty broad.