Skip to content

Track latency of per item eviction/promotion between memory tiers #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 31, 2023
Merged

Track latency of per item eviction/promotion between memory tiers #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
08d8f33
edit dockerfile
byrnedj Jul 24, 2023
316133c
these submodules work
byrnedj Jul 25, 2023
8d2c390
Track latency of per item eviction/promotion between memory tiers
guptask Jul 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions cachelib/allocator/Cache.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,6 +478,10 @@ void CacheBase::updateGlobalCacheStats(const std::string& statPrefix) const {

visitEstimates(uploadStatsNanoToMicro, stats.allocateLatencyNs,
statPrefix + "allocate.latency_us");
visitEstimates(uploadStatsNanoToMicro, stats.bgEvictLatencyNs,
statPrefix + "background.eviction.latency_us");
visitEstimates(uploadStatsNanoToMicro, stats.bgPromoteLatencyNs,
statPrefix + "background.promotion.latency_us");
visitEstimates(uploadStatsNanoToMicro, stats.moveChainedLatencyNs,
statPrefix + "move.chained.latency_us");
visitEstimates(uploadStatsNanoToMicro, stats.moveRegularLatencyNs,
Expand Down
7 changes: 3 additions & 4 deletions cachelib/allocator/CacheAllocator-inl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,17 +426,16 @@ CacheAllocator<CacheTrait>::allocateInternalTier(TierId tid,
uint32_t expiryTime,
bool fromBgThread,
bool evict) {
util::LatencyTracker tracker{stats().allocateLatency_};

util::LatencyTracker tracker{stats().allocateLatency_, static_cast<size_t>(!fromBgThread)};
SCOPE_FAIL { stats_.invalidAllocs.inc(); };

// number of bytes required for this item
const auto requiredSize = Item::getRequiredSize(key, size);

// the allocation class in our memory allocator.
const auto cid = allocator_[tid]->getAllocationClassId(pid, requiredSize);
util::RollingLatencyTracker rollTracker{
(*stats_.classAllocLatency)[tid][pid][cid]};

util::RollingLatencyTracker rollTracker{(*stats_.classAllocLatency)[tid][pid][cid]};
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this will still track latency for both user and bg allocations? If yes, it would be nice to put some comment regarding that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're not tracking rolling average latency for background threads separately. This feature is not in use anywhere. It was meant to provide a low overhead tracker for profile-driven batch evictions. Profile-driven approach has not been explored since Daniel Obiri worked on it a year back and found it didn't provide any significant improvement. I'm not sure Meta will accept the rolling stats upstream PR - it has not been reviewed yet.


(*stats_.allocAttempts)[tid][pid][cid].inc();

Expand Down
4 changes: 3 additions & 1 deletion cachelib/allocator/CacheAllocator.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2016,7 +2016,8 @@ class CacheAllocator : public CacheBase {
// exposed for the background evictor to iterate through the memory and evict
// in batch. This should improve insertion path for tiered memory config
size_t traverseAndEvictItems(unsigned int tid, unsigned int pid, unsigned int cid, size_t batch) {
auto& mmContainer = getMMContainer(tid, pid, cid);
util::LatencyTracker tracker{stats().bgEvictLatency_, batch};
auto& mmContainer = getMMContainer(tid, pid, cid);
size_t evictions = 0;
size_t evictionCandidates = 0;
std::vector<Item*> candidates;
Expand Down Expand Up @@ -2089,6 +2090,7 @@ auto& mmContainer = getMMContainer(tid, pid, cid);
}

size_t traverseAndPromoteItems(unsigned int tid, unsigned int pid, unsigned int cid, size_t batch) {
util::LatencyTracker tracker{stats().bgPromoteLatency_, batch};
auto& mmContainer = getMMContainer(tid, pid, cid);
size_t promotions = 0;
std::vector<Item*> candidates;
Expand Down
2 changes: 2 additions & 0 deletions cachelib/allocator/CacheStats.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ void Stats::populateGlobalCacheStats(GlobalCacheStats& ret) const {
ret.numNvmItemDestructorAllocErrors = numNvmItemDestructorAllocErrors.get();

ret.allocateLatencyNs = this->allocateLatency_.estimate();
ret.bgEvictLatencyNs = this->bgEvictLatency_.estimate();
ret.bgPromoteLatencyNs = this->bgPromoteLatency_.estimate();
ret.moveChainedLatencyNs = this->moveChainedLatency_.estimate();
ret.moveRegularLatencyNs = this->moveRegularLatency_.estimate();
ret.nvmLookupLatencyNs = this->nvmLookupLatency_.estimate();
Expand Down
2 changes: 2 additions & 0 deletions cachelib/allocator/CacheStats.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,8 @@ struct GlobalCacheStats {

// latency and percentile stats of various cachelib operations
util::PercentileStats::Estimates allocateLatencyNs{};
util::PercentileStats::Estimates bgEvictLatencyNs{};
util::PercentileStats::Estimates bgPromoteLatencyNs{};
util::PercentileStats::Estimates moveChainedLatencyNs{};
util::PercentileStats::Estimates moveRegularLatencyNs{};
util::PercentileStats::Estimates nvmLookupLatencyNs{};
Expand Down
2 changes: 2 additions & 0 deletions cachelib/allocator/CacheStatsInternal.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,8 @@ struct Stats {

// latency stats of various cachelib operations
mutable util::PercentileStats allocateLatency_;
mutable util::PercentileStats bgEvictLatency_;
mutable util::PercentileStats bgPromoteLatency_;
mutable util::PercentileStats moveChainedLatency_;
mutable util::PercentileStats moveRegularLatency_;
mutable util::PercentileStats nvmLookupLatency_;
Expand Down
2 changes: 2 additions & 0 deletions cachelib/cachebench/cache/Cache-inl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -743,6 +743,8 @@ Stats Cache<Allocator>::getStats() const {
static_cast<int64_t>(itemRecords_.count()) - totalDestructor_;

ret.cacheAllocateLatencyNs = cacheStats.allocateLatencyNs;
ret.cacheBgEvictLatencyNs = cacheStats.bgEvictLatencyNs;
ret.cacheBgPromoteLatencyNs = cacheStats.bgPromoteLatencyNs;
ret.cacheFindLatencyNs = cacheFindLatency_.estimate();

// Populate counters.
Expand Down
6 changes: 6 additions & 0 deletions cachelib/cachebench/cache/CacheStats.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ struct Stats {
uint64_t numNvmItemRemovedSetSize{0};

util::PercentileStats::Estimates cacheAllocateLatencyNs;
util::PercentileStats::Estimates cacheBgEvictLatencyNs;
util::PercentileStats::Estimates cacheBgPromoteLatencyNs;
util::PercentileStats::Estimates cacheFindLatencyNs;

double nvmReadLatencyMicrosP50{0};
Expand Down Expand Up @@ -282,6 +284,8 @@ struct Stats {

printLatencies("Cache Find API latency", cacheFindLatencyNs);
printLatencies("Cache Allocate API latency", cacheAllocateLatencyNs);
printLatencies("Cache Background Eviction API latency", cacheBgEvictLatencyNs);
printLatencies("Cache Background Promotion API latency", cacheBgPromoteLatencyNs);
}
}

Expand Down Expand Up @@ -510,6 +514,8 @@ struct Stats {

counters["find_latency_p99"] = cacheFindLatencyNs.p99;
counters["alloc_latency_p99"] = cacheAllocateLatencyNs.p99;
counters["bg_evict_latency_p99"] = cacheBgEvictLatencyNs.p99;
counters["bg_promote_latency_p99"] = cacheBgPromoteLatencyNs.p99;

counters["ram_hit_rate"] = calcInvertPctFn(numCacheGetMiss, numCacheGets);
counters["nvm_hit_rate"] = calcInvertPctFn(numCacheGetMiss, numCacheGets);
Expand Down
11 changes: 6 additions & 5 deletions cachelib/common/PercentileStats.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,24 +107,24 @@ class PercentileStats {

class LatencyTracker {
public:
explicit LatencyTracker(PercentileStats& stats)
: stats_(&stats), begin_(std::chrono::steady_clock::now()) {}
explicit LatencyTracker(PercentileStats& stats, size_t nSamples = 1)
: stats_(&stats), nSamples_(nSamples), begin_(std::chrono::steady_clock::now()) {}
LatencyTracker() {}
~LatencyTracker() {
if (stats_) {
if (nSamples_ > 0 && stats_) {
auto tp = std::chrono::steady_clock::now();
auto diffNanos =
std::chrono::duration_cast<std::chrono::nanoseconds>(tp - begin_)
.count();
stats_->trackValue(static_cast<double>(diffNanos), tp);
stats_->trackValue(static_cast<double>(diffNanos/nSamples_), tp);
}
}

LatencyTracker(const LatencyTracker&) = delete;
LatencyTracker& operator=(const LatencyTracker&) = delete;

LatencyTracker(LatencyTracker&& rhs) noexcept
: stats_(rhs.stats_), begin_(rhs.begin_) {
: stats_(rhs.stats_), nSamples_(rhs.nSamples_), begin_(rhs.begin_) {
rhs.stats_ = nullptr;
}

Expand All @@ -138,6 +138,7 @@ class LatencyTracker {

private:
PercentileStats* stats_{nullptr};
size_t nSamples_{1};
std::chrono::time_point<std::chrono::steady_clock> begin_;
};
} // namespace util
Expand Down
2 changes: 1 addition & 1 deletion cachelib/external/fbthrift
Open in desktop
Submodule fbthrift updated 4984 files
2 changes: 1 addition & 1 deletion cachelib/external/fizz
Open in desktop
Submodule fizz updated 130 files
2 changes: 1 addition & 1 deletion cachelib/external/folly
Open in desktop
Submodule folly updated 393 files
2 changes: 1 addition & 1 deletion cachelib/external/wangle
Open in desktop
Submodule wangle updated 50 files
+1 −1 build/deps/github_hashes/facebook/folly-rev.txt
+1 −1 build/deps/github_hashes/facebookincubator/fizz-rev.txt
+0 −1 build/fbcode_builder/CMake/FBThriftCppLibrary.cmake
+0 −1 build/fbcode_builder/CMake/FindGflags.cmake
+10 −8 build/fbcode_builder/getdeps.py
+3 −0 build/fbcode_builder/getdeps/cargo.py
+3 −3 build/fbcode_builder/getdeps/dyndeps.py
+1 −0 build/fbcode_builder/getdeps/manifest.py
+0 −1 build/fbcode_builder/getdeps/platform.py
+17 −0 build/fbcode_builder/manifests/OpenNSA
+0 −13 build/fbcode_builder/manifests/benchmark
+0 −11 build/fbcode_builder/manifests/blake3
+1 −0 build/fbcode_builder/manifests/boost
+0 −10 build/fbcode_builder/manifests/date
+4 −1 build/fbcode_builder/manifests/eden
+0 −1 build/fbcode_builder/manifests/edencommon
+1 −0 build/fbcode_builder/manifests/fboss
+3 −3 build/fbcode_builder/manifests/fmt
+1 −1 build/fbcode_builder/manifests/libsodium
+2 −2 build/fbcode_builder/manifests/lz4
+68 −0 build/fbcode_builder/manifests/sapling
+39 −0 build/fbcode_builder/manifests/traffixr
+0 −33 build/fbcode_builder/manifests/zstrong
+0 −97 build/fbcode_builder/patches/blake3_CMakeLists_txt.patch
+4 −4 wangle/acceptor/AcceptObserver.h
+23 −86 wangle/acceptor/Acceptor.cpp
+5 −18 wangle/acceptor/Acceptor.h
+0 −5 wangle/acceptor/AcceptorHandshakeManager.cpp
+0 −7 wangle/acceptor/AcceptorHandshakeManager.h
+27 −30 wangle/acceptor/ConnectionManager.cpp
+13 −19 wangle/acceptor/ConnectionManager.h
+10 −25 wangle/acceptor/FizzAcceptorHandshakeHelper.cpp
+5 −29 wangle/acceptor/FizzAcceptorHandshakeHelper.h
+0 −3 wangle/acceptor/FizzConfig.h
+1 −3 wangle/acceptor/ManagedConnection.cpp
+0 −13 wangle/acceptor/ManagedConnection.h
+11 −3 wangle/acceptor/SSLContextSelectionMisc.h
+1 −16 wangle/acceptor/ServerSocketConfig.h
+1 −15 wangle/acceptor/SharedSSLContextManager.h
+2 −2 wangle/acceptor/TransportInfo.cpp
+23 −38 wangle/acceptor/test/AcceptorTest.cpp
+34 −124 wangle/acceptor/test/ConnectionManagerTest.cpp
+1 −27 wangle/bootstrap/ServerBootstrap-inl.h
+3 −7 wangle/bootstrap/ServerBootstrap.h
+0 −6 wangle/client/ssl/test/ThreadSafeSSLSessionCacheTest.cpp
+3 −0 wangle/ssl/ClientHelloExtStats.h
+0 −31 wangle/ssl/SNIConfig.h
+124 −142 wangle/ssl/SSLContextManager.cpp
+7 −67 wangle/ssl/SSLContextManager.h
+184 −355 wangle/ssl/test/SSLContextManagerTest.cpp
2 changes: 2 additions & 0 deletions docker/images/centos-8streams.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ json-c-devel \
perf \
numactl

# updated to fix compile errors and better symbol
# resolving in VTune
RUN dnf -y install gcc-toolset-12
RUN echo "source /opt/rh/gcc-toolset-12/enable" >> /etc/bashrc
SHELL ["/bin/bash", "--login", "-c"]
Expand Down