Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stats: hitting flushing while already flushing assertion #748

Closed
junr03 opened this issue Mar 13, 2020 · 1 comment
Closed

stats: hitting flushing while already flushing assertion #748

junr03 opened this issue Mar 13, 2020 · 1 comment

Comments

@junr03
Copy link
Member

junr03 commented Mar 13, 2020

SIGABRT: Abort program
        at tgkill(/system/lib64/libc.so:440272)
        at abort(/system/lib64/libc.so:122644)
        at Envoy::Stats::ThreadLocalStoreImpl::mergeHistograms(std::__ndk1::function<void ()>)(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:15809952)
        at Envoy::Server::InstanceImpl::flushStats()(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:14918788)
        at Envoy::Event::DispatcherImpl::runPostCallbacks()(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:15486116)
        at 0x7e65064ce4(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:18566372)
        at event_base_loop(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:18549540)
        at Envoy::Server::InstanceImpl::run()(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:14924404)
        at Envoy::MainCommonBase::run()(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:14104948)
        at Envoy::Engine::run(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >)(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:13849616)
        at void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, envoy_status_t (Envoy::Engine::*)(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >), Envoy::Engine*, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >(void*)(/data/app/me.lyft.android.beta-Kvj80d62eAicoIFuNGT_HA==/lib/arm64/libenvoy_jni.so:13923388)
        at 0x7e9d042150(/system/lib64/libc.so:426320)
        at 0x7e9cff9668(/system/lib64/libc.so:128616)
        at 0x0(unknown)

Due to this being a sigabort it is clear to see that we are hitting the following assertion: https://github.com/envoyproxy/envoy/blob/609458bbd178100f956ee8587c1092abfb9a99f2/source/common/stats/thread_local_store.cc#L167

This started happening since we enabled #717. It makes sense as the app might have the opportunity to trigger a subsequent flush that causes merging before the previous merge finishes.

@junr03 junr03 added this to the v0.4 "Contorno" milestone Mar 13, 2020
junr03 added a commit that referenced this issue Mar 13, 2020
Description: disable until #748 is fixed.
Risk Level: low given stats are being flushed every 5 secs.

Signed-off-by: Jose Nino <jnino@lyft.com>
junr03 added a commit that referenced this issue Mar 17, 2020
Description: tag with two updates:
- #749. Disable lifecycle event-base stat flushing until #748 is resolved.
- #751 to get additional data of the number of retries attempted by envoy mobile

Signed-off-by: Jose Nino <jnino@lyft.com>
@junr03
Copy link
Member Author

junr03 commented Mar 17, 2020

closing this in favor of #754, as the crash itself won't happen now that the lifecycle flushing is disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant