Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix invalid job path #227

Merged
merged 2 commits into from
Jul 19, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion doc/usage_cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,14 @@ scp -r my_training_data_dir/ user@tunnel-server:/mnt/hdfs_mulan/idl/idl-dl/mydir
paddlecloud命令集成了上传数据的功能,目前仅针对存储系统是CephFS的环境。如果希望上传,执行:

```bash
paddlecloud file put /path/to/dir
paddlecloud file src dest
```
- `src` 必须是当前目录的子目录,`../`是不允许的。
- `src` 如果以'/'结尾,则表示上传`src`目录下的文件,不会在`dest`下创建新的目录。
- `src` 如果没有以`/`结尾,则表示上传`src`目录,会在`dest`下创建一个新的目录。
- `dest` 必须包含`/pfs/{datacenter}/user/{username}`目录。



### 使用公共数据集

Expand Down
13 changes: 11 additions & 2 deletions go/paddlecloud/simplefile.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,13 +100,21 @@ func putFiles(src string, dest string) error {
if err != nil {
return err
}
if strings.HasPrefix(src, "..") {
return errors.New("src path should be inside your submiting path")
}
switch mode := f.Mode(); {
case mode.IsDir():
if err := filepath.Walk(src, func(path string, info os.FileInfo, err error) error {
if info.Mode().IsRegular() {
srcs := strings.Split(filepath.Clean(src), string(os.PathSeparator))
paths := strings.Split(path, string(os.PathSeparator))
destFile := filepath.Join(dest, strings.Join(paths[len(srcs)-1:len(paths)], string(os.PathSeparator)))
var destFile string
if strings.HasSuffix(src, "/") {
destFile = filepath.Join(dest, strings.Join(paths[len(srcs):len(paths)], string(os.PathSeparator)))
} else {
destFile = filepath.Join(dest, strings.Join(paths[len(srcs)-1:len(paths)], string(os.PathSeparator)))
}
putFile(path, destFile)
}
return nil
Expand All @@ -115,7 +123,8 @@ func putFiles(src string, dest string) error {
}

case mode.IsRegular():
return putFile(src, dest)
_, f := filepath.Split(src)
return putFile(src, filepath.Join(dest, f))
}
return nil
}
Expand Down
9 changes: 8 additions & 1 deletion go/paddlecloud/submit.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"fmt"
"os"
"path"
"strings"

"github.com/PaddlePaddle/cloud/go/utils/config"
"github.com/PaddlePaddle/cloud/go/utils/restclient"
Expand Down Expand Up @@ -108,7 +109,13 @@ func (s *Submitter) Submit(jobPackage string, jobName string) error {
_, pkgerr := os.Stat(jobPackage)
if pkgerr == nil {
dest := path.Join("/pfs", Config.ActiveConfig.Name, "home", Config.ActiveConfig.Username, "jobs", jobName)
return putFiles(jobPackage, dest)
if !strings.HasSuffix(jobPackage, "/") {
jobPackage = jobPackage + "/"
}
err := putFiles(jobPackage, dest)
if err != nil {
return err
}
} else if os.IsNotExist(pkgerr) {
glog.Warning("jobpackage not a local dir, skip upload.")
}
Expand Down