download job related fix #4754
Merged
download job related fix #4754
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
critical27
added
ready-for-testing
critical27
requested review from
panda-sheep,
SuperYoko,
pengweisong and
shanlai
SuperYoko
reviewed
Oct 20, 2022
| @@ -49,6 +50,7 @@ Status HdfsCommandHelper::copyToLocal(const std::string& hdfsHost, | |||
| folly::Subprocess proc(std::vector<std::string>({"/bin/bash", "-c", command})); | |||
| auto result = proc.wait(); | |||
| if (!result.exited()) { | |||
| LOG(INFO) << "Failed to download: " << result.str(); | |||
There was a problem hiding this comment.
Here download error will failed this job too? Will them generate empty directories?
There was a problem hiding this comment.
Yeah, if download occurs error we should fail it. (Exchange will not generate empty dir, if it indeed generate empty dir, ingest will handle it)
Sophie-Xie
added
ready-for-testing
Sophie-Xie
pushed a commit
that referenced
this pull request
Oct 20, 2022
Sophie-Xie
added a commit
that referenced
this pull request
Oct 20, 2022
* fix dep of loop in go planner (#4736) * fix inappropriate error code from raft (#4737) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix variable types collected and graph crash (#4724) * Fix variable types collected and graph crash add test cases small fix * unskip some test cases related to multiple query parts * small delete * fmt * Fix ldbc BI R5 implementation Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * stats handle the flag use_vertex_key (#4738) * JobManager related fix (#4742) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * download job related fix (#4754) * fixed case-when error (#4744) * fixed case-when error * fix tck * fix tck * fix tck Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Refine go planner (#4750) * refine go planner * update * fix ctest Co-authored-by: jie.wang <38901892+jievince@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: kyle.cao <kyle.cao@vesoft.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: canon <87342612+caton-hpg@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What problem(s) does this PR solve?
Issue(s) number:
close #4753
close #4752
Description:
When exchange generate SST files, some part may do not have data, so there won't exist corresponding part directory. After we change the way to execute hdfs command, this will be an issue. Because there will be an error like
dir not exists, so when storage really start download, firstls, if the directory exists then copy.And we need to handle ingest no files into rocksdb.
How do you solve it?
Special notes for your reviewer, ex. impact of this fix, design document, etc:
Checklist:
Tests:
Affects:
Release notes:
Please confirm whether to be reflected in release notes and how to describe: