Skip to content

Conversation

@masaori335
Copy link
Contributor

I have been running my own eBPF based mutex profiling tool. Recently a mutex lock shows up with the latest mater branch that I haven't seen before.

Report from profiler

        mutex [unknown] ::: wait time 228741.80us ::: hold time 57441.78us ::: enter count 6754 ::: try-lock failure count 0
                std::__1::mutex::lock()+0x6 [libc++.so.1.0] (7f31fb7347f6)
                Cache::open_read(Continuation*, ats::CryptoHash const*, HTTPHdr*, OverridableHttpConfigParams const*, CacheFragType, char const*, int)+0x42 [traffic_server] (55dd98786a32)
                CacheProcessor::open_read(Continuation*, HttpCacheKey const*, HTTPHdr*, OverridableHttpConfigParams const*, long, CacheFragType)+0x27 [traffic_server] (55dd9876b5c7)
                HttpCacheSM::open_read(HttpCacheKey const*, URL*, HTTPHdr*, OverridableHttpConfigParams const*, long)+0x9d [traffic_server] (55dd985f9cad)
                HttpSM::do_cache_lookup_and_read()+0x169 [traffic_server] (55dd985ec0d9)

It looks like the std::__1::mutex::lock is called from ReplaceablePtr<CacheHostTable>::ScopedReader introduced by #9046.

Vol *
Cache::key_to_vol(const CacheKey *key, const char *hostname, int host_len)
{
ReplaceablePtr<CacheHostTable>::ScopedReader hosttable(&this->hosttable);

It loosk like replacing std::shared_lock with ts::shared_mutex mitigates the issue. At least the profiler stoped complaining.

@masaori335 masaori335 added this to the 10.0.0 milestone Nov 18, 2022
@masaori335 masaori335 self-assigned this Nov 18, 2022
@SolidWallOfCode
Copy link
Member

I wonder if it is because that workload is write heavy.

@masaori335
Copy link
Contributor Author

Sorry, it looks like I tested with my messed working branch and misunderstood. If I look carefully, the call stack of the report doesn't match with the Cache::key_to_vol.

#9040 seems fine.

@masaori335 masaori335 closed this Nov 18, 2022
@zwoop zwoop removed this from the 10.0.0 milestone Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants