feat: Use Velox fs for ssd cache checkpoint file #11783
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
✅ Deploy Preview for meta-velox canceled.
|
zacw7
force-pushed
the
export-D66892136
branch
from
8a169ff to
ddda3ab
Compare
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 7, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
xiaoxmeng
reviewed
Dec 7, 2024
| @@ -383,6 +383,8 @@ void LocalWriteFile::truncate(int64_t newSize) { | |||
| 0, | |||
| "ftruncate failed in LocalWriteFile::truncate: {}.", | |||
| folly::errnoStr(errno)); | |||
| // Reposition the file offset to the end of the file for append(). | |||
There was a problem hiding this comment.
Underline LocalWriteFile->append, it's ::write:
velox/velox/common/file/File.cpp
Line 323 in ddda3ab
The writing takes place at an inexplicit offset (https://man7.org/linux/man-pages/man2/write.2.html), which won't be reset automatically after truncate.
velox/common/caching/SsdFile.cpp
Outdated
| state.write(checkpointVersion().data(), sizeof(int32_t)); | ||
| state.write(asChar(&maxRegions_), sizeof(maxRegions_)); | ||
| state.write(asChar(&numRegions_), sizeof(numRegions_)); | ||
| checkpointWriteFile_->append(checkpointVersion()); |
There was a problem hiding this comment.
You can allocate 1MB buffer from cache pool for checkpointing buffer.
velox/common/caching/SsdFile.cpp
Outdated
| } | ||
| } | ||
| checkpointWriteFile_->append(folly::IOBuf::copyBuffer( |
There was a problem hiding this comment.
We just need to sync data file right before write out kCheckpointEndMarker?
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 10, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
zacw7
force-pushed
the
export-D66892136
branch
from
ddda3ab to
01a2da0
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
zacw7
force-pushed
the
export-D66892136
branch
from
01a2da0 to
9afe3ad
Compare
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 10, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 11, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
zacw7
force-pushed
the
export-D66892136
branch
from
9afe3ad to
72262ca
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
xiaoxmeng
reviewed
Dec 11, 2024
velox/common/caching/SsdFile.cpp
Outdated
| } | ||
|
|
||
| std::string readString(common::FileInputStream* stream, int32_t length) { | ||
| std::string data; |
There was a problem hiding this comment.
std::string data(length);
There was a problem hiding this comment.
This requires a second arg. I'll put it as std::string data(length, '\0')
xiaoxmeng
reviewed
Dec 11, 2024
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 11, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
zacw7
force-pushed
the
export-D66892136
branch
from
72262ca to
dcc1f05
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 11, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
zacw7
force-pushed
the
export-D66892136
branch
from
dcc1f05 to
bb61e4e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
zacw7
added a commit
to zacw7/velox
that referenced
this pull request
Dec 12, 2024
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Differential Revision: D66892136
zacw7
force-pushed
the
export-D66892136
branch
from
bb61e4e to
8268555
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
…1783) Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Reviewed By: xiaoxmeng Differential Revision: D66892136
zacw7
force-pushed
the
export-D66892136
branch
from
8268555 to
fd46b9b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66892136 |
|
This pull request has been merged in ac13440. |
athmaja-n
pushed a commit
to athmaja-n/velox
that referenced
this pull request
Jan 10, 2025
…1783) Summary: Pull Request resolved: facebookincubator#11783 Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections. Reviewed By: xiaoxmeng Differential Revision: D66892136 fbshipit-source-id: d4da2df1f5da976b0a71c4a1a087a2c0ba569328
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Switch the ssd cache checkpoint file to use Velox filesystem for file r/w operations, so that more advanced testing can be built by leveraging features like fault injections.
Differential Revision: D66892136