[Upstreaming] Added file-based SHM segment option in SHM module #74
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guptask
force-pushed
the
shm_segments
branch
10 times, most recently
from
0f84af5 to
2eb9eac
Compare
guptask
force-pushed
the
shm_segments
branch
11 times, most recently
from
e3c00af to
4275baf
Compare
Summary: GitHub commits: facebook@b17378e facebook/fb303@f46db3d facebook/fbthrift@9f520d1 facebookarchive/fbzmq@6a983b2 facebook/folly@415c7cd facebook/proxygen@0011dec facebook/wangle@25a90a6 facebook/watchman@0dda4e4 facebookexperimental/rust-shed@3896699 facebookincubator/fizz@51eff02 facebookincubator/katran@92cad23 facebook/mvfst@3d0780a facebookincubator/velox@79361bf Reviewed By: jurajh-fb fbshipit-source-id: f2c827f8775edef84c98434afe7306d247724ea0
Summary: GitHub commits: facebook@96fb269 facebook/fb303@4bd5cc9 facebook/fbthrift@05eee0a facebookarchive/fbzmq@251dcc1 facebook/proxygen@173c3ba facebook/wangle@1539594 facebook/watchman@b61b18c facebookexperimental/rust-shed@3e88384 facebookincubator/fizz@c515905 facebookincubator/katran@f476862 facebook/mvfst@908d06b facebookincubator/velox@b0bfc83 Reviewed By: jurajh-fb fbshipit-source-id: 395e763435bf40f790b121627ca547f95776d01d
Summary: GitHub commits: facebook@ffa3eec facebook/fb303@dd8a452 facebook/fbthrift@ade147f facebookarchive/fbzmq@1c635ed facebook/proxygen@3758c79 facebook/wangle@7aea6bd facebook/watchman@54269b0 facebookexperimental/rust-shed@935b600 facebookincubator/katran@674f224 facebook/mvfst@a768ce8 Reviewed By: jurajh-fb fbshipit-source-id: e823f8fee735769c3d18a8c5eedd5d301c2012c2
Summary: GitHub commits: facebook@6c4389f facebook/fb303@721357a facebook/fbthrift@2d6736f facebookarchive/fbzmq@968ab9f facebook/proxygen@3f1f2a8 facebook/watchman@97f3cde facebookexperimental/rust-shed@c34067d Reviewed By: jurajh-fb fbshipit-source-id: 2166f1bcd83f04271545e6d7bcd4277e77f18764
Summary: GitHub commits: facebook/fbthrift@80469bd facebook/litho@09210b4 facebook/watchman@6c29e8a facebookincubator/velox@fb7b62b Reviewed By: jurajh-fb fbshipit-source-id: 234fd86e822cef60781048bce907b9dae25187a0
Summary: Clean up `ItemHandle` in CacheAllocator Reviewed By: therealgymmy Differential Revision: D34334392 fbshipit-source-id: 0ba983dc0e4f2614b8074ad70585f91d1af3e5e9
Summary: GitHub commits: facebook@c8dc407 facebook/fbthrift@8900b1e Reviewed By: jurajh-fb fbshipit-source-id: f470e19dcba367551a58d23a5588a3bfa0e3ebfb
Summary: GitHub commits: facebook@67d39b5 facebook/fb303@ebeb87a facebook/fbthrift@ed12bea facebookarchive/fbzmq@a2463ec facebook/rocksdb@94e245a facebook/watchman@be1dc56 facebookexperimental/rust-shed@727a63c facebookincubator/velox@29143ef pytorch/FBGEMM@7fcc520 Reviewed By: jurajh-fb fbshipit-source-id: 53a7c4c1a292928ba2a4d4647d78cf205c2939fa
Summary: GitHub commits: facebook@ecd0d6d facebook/fb303@24d7a2f facebook/fbthrift@3d2149b facebookarchive/fbzmq@99e839b facebook/folly@91b554e facebook/proxygen@d3dd9e2 facebook/watchman@735ac45 Reviewed By: wittgenst fbshipit-source-id: 62dc2b07406ae4b435877c7684208a2a8c00ae51
Summary: GitHub commits: facebook@314a53e facebook/fb303@dd4e5e5 facebook/fbthrift@ad08e0c facebookarchive/fbzmq@b48f0ab facebook/proxygen@d3344c1 facebook/wangle@ffcc603 facebook/watchman@94c23b3 facebookexperimental/rust-shed@302157d facebookincubator/fizz@9820e01 facebookincubator/katran@205dd7f facebook/mvfst@5527687 facebookincubator/velox@5e34657 Reviewed By: wittgenst fbshipit-source-id: 914a2dcb5e06dcf72ade5707f1e034e6321afffc
Summary: GitHub commits: facebook@bdfda90 facebook/fb303@a81c3a8 facebook/fbthrift@faad955 facebookarchive/fbzmq@c72c353 facebook/folly@73e4b30 facebook/proxygen@37f8b7f facebook/wangle@85700f0 facebook/watchman@2e67cf0 facebookexperimental/rust-shed@3eb15e9 facebookincubator/katran@b70a445 facebook/mvfst@78015d4 Reviewed By: wittgenst fbshipit-source-id: 3eb69cbfac83ce8b1845ed1432c72dab5d34e471
Summary: GitHub commits: facebook@fcfa349 facebook/fb303@5d13aac facebook/fbthrift@66e0364 facebookarchive/fbzmq@629c277 facebook/proxygen@cc0fa5d facebook/wangle@4ac3e2b facebook/watchman@60fd139 facebookexperimental/rust-shed@83c7a9b facebookincubator/fizz@2f68e3c facebookincubator/katran@288cb00 facebook/mvfst@7d956c8 facebookincubator/velox@0bd3977 Reviewed By: wittgenst fbshipit-source-id: 4a40057be153a14100a3ec52c612027437b12157
Summary: GitHub commits: facebook@a98c18f facebook/fb303@ebacfce facebook/fbthrift@a9abfc8 facebookarchive/fbzmq@cf43732 facebook/proxygen@6c5a71f facebook/wangle@20a7de5 facebook/watchman@653cf66 facebookexperimental/rust-shed@00a0c03 facebookincubator/katran@bba0c7c facebook/mvfst@6225f75 Reviewed By: wittgenst fbshipit-source-id: ed594ec3fe3d7ec41c1ff7f477116155a8204137
Summary: GitHub commits: facebook@26f59c7 facebook/fb303@de4c64f facebook/fbthrift@355713e facebookarchive/fbzmq@73629c1 facebook/proxygen@b2cf97e facebook/watchman@966dd45 facebookexperimental/rust-shed@a411120 Reviewed By: wittgenst fbshipit-source-id: 635d1133584898b6a55ee3870d4d78b0ad17cfe2
Summary: GitHub commits: facebook/folly@847a391 Reviewed By: wittgenst fbshipit-source-id: 6fb2934c88a00aa3a346543952b21660cf019959
Summary: GitHub commits: facebook@06f51dc facebook/fb303@3262f2b facebook/fbthrift@23976d2 facebookarchive/fbzmq@656d062 facebook/proxygen@e389635 facebook/wangle@28a0e15 facebook/watchman@86321a0 facebookincubator/fizz@ce2089f facebookincubator/katran@9cc9d5d facebook/mvfst@42f284c facebookincubator/velox@3ea40c5 Reviewed By: wittgenst fbshipit-source-id: 5e7a2aad8404671fdb76c8a0ddfca7db271b99d1
Summary: GitHub commits: facebook@7928362 facebook/fb303@4123eb5 facebook/fbthrift@7fe5796 facebookarchive/fbzmq@e46c497 facebook/proxygen@c641d76 facebook/wangle@c5fd3fa facebook/watchman@a6fb749 facebookexperimental/rust-shed@a88ba78 facebookincubator/katran@9627abd facebook/mvfst@99af3b0 Reviewed By: wittgenst fbshipit-source-id: 283b5395042962576df234dde9cdde196bb5b73e
Summary: GitHub commits: facebook@315fb65 facebook/fb303@004f8e5 facebook/fbthrift@a1c1e6c facebookarchive/fbzmq@1edad5c facebook/proxygen@19e3f54 facebook/watchman@34c6d03 facebookexperimental/rust-shed@aac1fe7 Reviewed By: wittgenst fbshipit-source-id: ccd26682041d2956e21108dcd154c753a7056d2b
Summary: GitHub commits: facebook/fb303@7ac6c35 facebook/fbthrift@67729ba facebookarchive/fbzmq@d42e9b6 facebook/folly@053748a facebook/proxygen@161098c facebook/wangle@cd8a816 facebook/watchman@fb72d47 facebookexperimental/rust-shed@769b564 facebookincubator/fizz@939e803 facebookincubator/katran@63d5bc7 facebook/mvfst@9ca86a5 facebookarchive/profilo@d55c0c7 Reviewed By: wittgenst fbshipit-source-id: 3ed03ebda88048dbaf7ea98d661f2f529fda5453
Summary: GitHub commits: facebook@7ff9925 facebook/fb303@8f3ec79 facebook/fbthrift@37d9ca2 facebookarchive/fbzmq@a10645c facebook/proxygen@6113a32 facebook/wangle@bfce313 facebook/watchman@677830e facebookexperimental/rust-shed@f43332c facebookincubator/fizz@78392a6 facebookincubator/katran@3ffa1b9 facebook/mvfst@7215859 facebookincubator/velox@ec3b7a9 Reviewed By: wittgenst fbshipit-source-id: c689d4bf27be6ddcc0dc7fde3d06511a8f09fc06
Summary: GitHub commits: facebook@f56a17c facebook/fb303@373e44f facebook/fbthrift@66fdb75 facebookarchive/fbzmq@faa1227 facebook/proxygen@aed0372 facebook/wangle@d125926 facebook/watchman@84b621c facebookexperimental/rust-shed@bcc1958 facebookincubator/katran@fb9c186 facebook/mvfst@b4ca720 Reviewed By: wittgenst fbshipit-source-id: 78659911866ad8c76864632351e77f3d56c2b9ca
Summary: GitHub commits: facebook@39a9143 facebook/fb303@5eb0355 facebook/fbthrift@c28868e facebookarchive/fbzmq@7bbf4b0 facebook/proxygen@0d08a03 facebook/watchman@e2feeac facebookexperimental/rust-shed@b234fb9 Reviewed By: wittgenst fbshipit-source-id: bce160b7ccfd80d8bac28db04ac7a513469c47bc
Summary: GitHub commits: facebook@0c94a3b facebook/fb303@d279f99 facebook/fbthrift@bfa4da7 facebookarchive/fbzmq@ca78b2a facebook/watchman@3ff867e facebookexperimental/rust-shed@fbda957 Reviewed By: wittgenst fbshipit-source-id: ff3998d8932c3fc348ec5d3ecbdc7e13f92a1fa5
Summary: GitHub commits: facebook/fbthrift@9a8e286 Reviewed By: wittgenst fbshipit-source-id: 548ede7f73eef04044608c8c4ff08cde6ac0b1c6
Summary: GitHub commits: facebook/folly@1d196d6 Reviewed By: wittgenst fbshipit-source-id: 85b66175cbcac8d719708d4cc92533a96408ce84
Summary: GitHub commits: facebook@b8866d3 facebook/fb303@0ac8829 facebook/fbthrift@bd5896e facebookarchive/fbzmq@879617b facebook/proxygen@3e3fa72 facebook/wangle@b7e0c72 facebook/watchman@21a65c1 facebookincubator/fizz@73e93b6 facebookincubator/katran@df45d0b facebook/mvfst@d5f145f facebookincubator/velox@c69405f Reviewed By: wittgenst fbshipit-source-id: 7c71a7bc0e92bd815589d6748b4a214309bda972
Summary: GitHub commits: facebook@7ee65e1 facebook/fb303@d90de35 facebook/fbthrift@cd5d077 facebookarchive/fbzmq@592b59e facebook/proxygen@734c0ce facebook/wangle@6097be3 facebook/watchman@78cf46b facebookexperimental/rust-shed@1f168a0 facebookincubator/katran@b8a6f30 facebook/mvfst@129c720 Reviewed By: wittgenst fbshipit-source-id: e5bd83520711163e5b85c3e57192e65d941016cb
Summary: GitHub commits: facebook@1b1359b facebook/fb303@9da566f facebook/fbthrift@5a4e9d2 facebookarchive/fbzmq@1481ef6 facebook/proxygen@c85c232 facebook/watchman@ba5f786 facebookexperimental/rust-shed@05d9670 Reviewed By: wittgenst fbshipit-source-id: b00e12913cde2dd0d8a9805ad722ca8a6ef102ad
Summary: GitHub commits: facebook/fbthrift@85148bc Reviewed By: wittgenst fbshipit-source-id: d5e0a075a19357ddcc2f8cee208e257c9c5f4b4e
It's implementation is mostly based on PosixShmSegment. Also, extend ShmManager and ShmSegmentOpts to support this new segment type.
After introducing file segment type, nameToKey_ does not provide enough information to recover/remove segments on restart. This commit fixes that by replacing nameToKey_ with nameToOpts_. Previously, the Key from nameToKey_ map was only used in a single DCHECK().
It wrongly assumed that the only possible segment type is PosixSysV segment.
guptask
added a commit
to guptask/CacheLib
that referenced
this pull request
Mar 30, 2023
Update DML version in docker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added file-based SHM segment option in SHM module and CacheAllocator. This is first half of the changes needed to enable multi-tiering in CacheLib. The multi-tiering config APIs are in a separate PR (facebook#138). These config APIs are needed to enable multi-tiering. Once that PR is merged, the second half of the multi-tier changes can be upstreamed via a separate PR.
This change is