add pfsserver dockerfile #148

gongweibao · 2017-06-13T09:59:52Z

No description provided.

Yancey1989 · 2017-06-14T02:02:34Z

go/README_CN.md

@@ -0,0 +1,20 @@
+1. 如何创建pfsserver的DockerImage


pfsserver => PFSServer ?

Yancey1989 · 2017-06-14T02:04:48Z

go/README_CN.md

@@ -0,0 +1,20 @@
+1. 如何创建pfsserver的DockerImage
+  - 创建pfsserver的编译环境


创建pfsserver的编译环境
==>
构建PFSServer的开发环境镜像

Yancey1989 · 2017-06-14T02:05:15Z

go/README_CN.md

+  docker run  --rm -v  $(pwd):/root/gopath/src/github.com/PaddlePaddle/cloud/go  pfsserver:dev
+  ```
+
+  - 创建pfsserver的DockerImage


创建pfsserver的DockerImage
=>
构建PFSServer的生产环境镜像

Yancey1989 · 2017-06-14T02:08:05Z

go/Dockerfile

@@ -0,0 +1,5 @@
+FROM ubuntu:16.04


This Dockerfile is used for PFSServer, so we can move this to ./cmd/pfsserver ?

目前还不行，依赖了./tools/gen_config.sh，用来生成配置文件。后续解耦合以后可以放到./cmd/pfsserver下。
我在这里创建了一个ISSUE:#157

Yancey1989 · 2017-06-14T02:10:20Z

go/docker/build.sh

@@ -0,0 +1,17 @@
+cat > ./Dockerfile << EOF


I think we can use golang:1.8 as the base Docker image so that it will reduce the Image size.

golang:1.8最小的257M，这个只有125M。

Yep, but you should wget golang.tar.gz if you use ubuntu as the base image. and we also need VPN so that we can download the tar file from storage.googleapis.com.

编译环境和运行环境是分开的。运行环境不需要wget任何东西。
编译环境很长时间才需要构建一次。

typhoonzero · 2017-06-26T13:09:41Z

k8s/pfs_ingress.yaml

@@ -0,0 +1,19 @@
+apiVersion: extensions/v1beta1


Ingress 和python django部分现在可以完成分发了么？

…docker

typhoonzero · 2017-07-24T02:35:23Z

go/.tools/gen_config.sh

+cat > ~/.paddle/config << EOF
+datacenters:
+- name: datacenter1
+  username: user@baidu.com


user@baidu.com should not exist, use your-user-name instead.

typhoonzero · 2017-07-24T02:35:43Z

go/.tools/gen_config.sh

+datacenters:
+- name: datacenter1
+  username: user@baidu.com
+  password: T123


T123 have no meaning, use your-secret instead.

typhoonzero · 2017-07-24T02:37:50Z

go/README_CN.md

@@ -0,0 +1,61 @@
+1. 如何构建PFSServer的DockerImage


Just a mark, we need to put pfs and pcloud in the same image later.

typhoonzero · 2017-07-24T02:39:54Z

go/docker/build.sh

@@ -0,0 +1,19 @@
+cat > ./Dockerfile << EOF


Having both go/docker/ and docker/ seems confusing, move to the same direcotry or use docker/build_docker.sh.

typhoonzero · 2017-07-24T02:43:00Z

go/filemanager/pfsmodules/command.go

-	if len(a) < 3 {
+func checkUser(pathStr string, user string) error {
+	pathStr = path.Clean(strings.TrimSpace(pathStr))
+	a := strings.Split(pathStr, "/")


Should use os.PathSeparator instead of "/" to support Windows, or use filepath.Split()

这个地方的路径是server端路径，格式是固定的。

typhoonzero · 2017-07-24T02:47:02Z

go/.tools/gen_config.sh

+  username: user@baidu.com
+  password: T123
+  endpoint: http://127.0.0.1:8080
+  pfsendpoint: http://127.0.0.1:8080


It's not simple enough for users to configure if we have two "endpoints", just use static paths like http://127.0.0.1:8080/api/v1/jobs and http://127.0.0.1:8080/api/v1/pfs, so users only need to configure endpoint to http://127.0.0.1:8080

typhoonzero · 2017-07-25T12:15:40Z

k8s/cloud_deployment.yaml

@@ -39,7 +39,7 @@ spec:
      containers:
      - name: paddle-cloud
        imagePullPolicy: Always
-        image: paddlepaddle/cloud
+        image: gongweibao/pcloud


gongweibao/pcloud should not appear.

typhoonzero · 2017-07-25T12:16:00Z

k8s/cloud_ingress.yaml

@@ -4,10 +4,15 @@ metadata:
  name: paddle-cloud-ingress
 spec:
  rules:
-  - host: cloud.paddlepaddle.org
+  - host: cloud1.paddlepaddle.org


cloud1.paddlepaddle.org is confusing.

typhoonzero · 2017-07-25T12:16:18Z

k8s/pfs_deployment.yaml

+      containers:
+      - name: pfs
+        imagePullPolicy: Always
+        image: gongweibao/pfsserver:latest


gongweibao/pfsserver:latest should not appear.

gongweibao

改的次数太多，忘记把测试环境的改过来了。汗。

typhoonzero

LGTM++
except for one small comment

typhoonzero · 2017-07-26T04:54:36Z

k8s/pfs_deployment.yaml

+        image: paddlepaddle/pfsserver:latest
+        env:
+          - name: CURRENT_DATACENTER
+            value: "meiyan"


"meiyan" should be a template too.

我单独搞一个PR来处理template的问题

gongweibao added 2 commits June 13, 2017 17:57

add dockerfile

765d0d8

modify README_CN.md

e760269

gongweibao requested review from Yancey1989 and typhoonzero June 13, 2017 10:00

fix README_CN.md

b0de563

Yancey1989 reviewed Jun 14, 2017

View reviewed changes

gongweibao added 10 commits June 20, 2017 10:11

modify readme

eac9ffc

fix pfsserver bugs

22d0a8d

add cloud_pfsserver.yaml

122ee25

modify yaml

a520967

modify docker file

612f53c

add pfs config

03f244d

modify app

91b4426

modify pfs_app.yaml

64fbb84

modify travis

d896844

add cephfs support

79862db

typhoonzero reviewed Jun 26, 2017

View reviewed changes

gongweibao added 8 commits July 19, 2017 09:48

merge

a61462d

Merge branch 'develop' of https://github.com/PaddlePaddle/cloud into …

9655d39

…docker

mofidy

473f7ab

modify

2f6e9aa

modify

a7db6e7

Merge branch 'docker' of https://github.com/gongweibao/cloud into docker

de757f8

local test ok

b88dfe5

rm not need

bef6072

typhoonzero requested changes Jul 24, 2017

View reviewed changes

gongweibao added 3 commits July 25, 2017 10:35

fix

2633ad4

fix comments

e121c53

rm not need

1a9d15e

typhoonzero reviewed Jul 25, 2017

View reviewed changes

gongweibao added 2 commits July 25, 2017 20:27

fix

d4ce648

fix

1ccc748

gongweibao commented Jul 25, 2017

View reviewed changes

typhoonzero approved these changes Jul 26, 2017

View reviewed changes

gongweibao merged commit 0148f0f into PaddlePaddle:develop Jul 26, 2017

gongweibao deleted the docker branch July 26, 2017 06:43

wangkuiyi mentioned this pull request Feb 4, 2018

.travis.yml refers to nonexisting bash script #586

Closed

		@@ -0,0 +1,20 @@
		1. 如何创建pfsserver的DockerImage
		- 创建pfsserver的编译环境

add pfsserver dockerfile #148

add pfsserver dockerfile #148

Conversation

gongweibao commented Jun 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao Jun 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao Jul 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao left a comment

Choose a reason for hiding this comment

typhoonzero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao Jun 20, 2017 •

edited

Loading

gongweibao Jul 24, 2017 •

edited

Loading