Add python lint check to travis and fix python lint issues #91

jlewi · 2017-10-25T23:56:07Z

Add a python lint check
- Partial fix to Run lint (Python, Go) as a presubmit test #53
Capture stdout/stderr of the test and upload to GCS so we have a proper build log that outlives the pod lifetime.
- Fixes runner.py needs to create build-log.txt with stdout/stderr of test #82
runner.py should reuse the run commands in py/util.py rather than duplicating code.

Add lint to travis.

pylint on Travis complains about missing config file.

jlewi · 2017-10-26T17:47:48Z

/test all

jlewi · 2017-10-26T19:59:38Z

@krzyzacy The prow job is reported as failed but the data in Gubernator indicates success. How does Prow determine success? Is it determined by the exit code of the prow job?

I will try fixing #82 so that we get actual logs. My guess is there's a problem with running the lint check I added in this PR.

krzyzacy · 2017-10-26T20:05:44Z

/retest

krzyzacy · 2017-10-26T20:06:22Z

the build log is missing though, that might be the failure

krzyzacy · 2017-10-26T20:24:28Z

2017/10/26 20:17:51 Step 'helm install --set image=gcr.io/mlkube-testing/tf_operator:v20171026-d1ec26d /go/src/github.com/tensorflow/k8s/tf-job-operator-chart --namespace default --name dnz --wait' finished in 6.647918167s
2017/10/26 20:17:51 Running: helm test dnz
2017/10/26 20:20:46 RUNNING: dnz-tfjob-test-risxyt
PASSED: dnz-tfjob-test-risxyt
2017/10/26 20:20:46 Step 'helm test dnz' finished in 2m55.071202959s
2017/10/26 20:20:46 Running: helm delete dnz --purge
2017/10/26 20:20:49 release "dnz" deleted
2017/10/26 20:20:49 Step 'helm delete dnz --purge' finished in 3.672653442s
2017/10/26 20:20:49 Saved XML output to /tmp/tmpTfCrdTestkPhSuN/junit_01.xml.
INFO:root:Running: lint.sh
INFO:root:Uploading file /tmp/tmpTfCrdTestkPhSuN/build_info.yaml.
INFO:root:Uploading file /tmp/tmpTfCrdTestkPhSuN/junit_01.xml.
INFO:googleapiclient.discovery:URL being requested: DELETE https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/clusters/v20171026-d5bb?alt=json
INFO:root:Response {u'status': u'RUNNING', u'name': u'operation-1509049250665-298dbe8b', u'zone': u'us-central1-f', u'startTime': u'2017-10-26T20:20:50.66589927Z', u'targetLink': u'https://container.googleapis.com/v1/projects/323197036272/zones/us-central1-f/clusters/v20171026-d5bb', u'operationType': u'DELETE_CLUSTER', u'selfLink': u'https://container.googleapis.com/v1/projects/323197036272/zones/us-central1-f/operations/operation-1509049250665-298dbe8b'}
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:googleapiclient.discovery:URL being requested: GET https://container.googleapis.com/v1/projects/mlkube-testing/zones/us-central1-f/operations/operation-1509049250665-298dbe8b?alt=json
INFO:root:Cluster deletion done.
 {u'status': u'DONE', u'name': u'operation-1509049250665-298dbe8b', u'zone': u'us-central1-f', u'startTime': u'2017-10-26T20:20:50.66589927Z', u'targetLink': u'https://container.googleapis.com/v1/projects/323197036272/zones/us-central1-f/clusters/v20171026-d5bb', u'operationType': u'DELETE_CLUSTER', u'endTime': u'2017-10-26T20:22:12.864525794Z', u'selfLink': u'https://container.googleapis.com/v1/projects/323197036272/zones/us-central1-f/operations/operation-1509049250665-298dbe8b'}
Traceback (most recent call last):
  File "/go/src/github.com/tensorflow/k8s/test-infra/runner.py", line 540, in <module>
    main()
  File "/go/src/github.com/tensorflow/k8s/test-infra/runner.py", line 528, in main
    lint_success = run_lint(args.src_dir)
  File "/go/src/github.com/tensorflow/k8s/test-infra/runner.py", line 424, in run_lint
    run(["lint.sh"], cwd=src_dir)
  File "/go/src/github.com/tensorflow/k8s/test-infra/runner.py", line 63, in run
    subprocess.check_call(command, cwd=cwd)
  File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/workspace/bootstrap.py", line 145, in <module>
d1ec26d7ec4bad894f1b6a87e8ac8a9ec165c9a5

    main()
  File "/workspace/bootstrap.py", line 140, in main
    run(["python", runner, "--src_dir=" + src_dir, "--sha=" + sha])
  File "/workspace/bootstrap.py", line 39, in run
    subprocess.check_call(command, cwd=cwd)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '/go/src/github.com/tensorflow/k8s/test-infra/runner.py', '--src_dir=/go/src/github.com/tensorflow/k8s', u'--sha=d1ec26d7ec4bad894f1b6a87e8ac8a9ec165c9a5\n']' returned non-zero exit status 1

jlewi · 2017-10-26T20:35:58Z

@krzyzacy Thanks.

* runner.py should reuse py/util rather than redefining run.

jlewi · 2017-10-26T21:52:26Z

/test tf-k8s-presubmit

jlewi · 2017-10-26T21:58:15Z

/test all

jlewi · 2017-10-26T22:09:54Z

/test all

jlewi · 2017-10-26T22:14:10Z

/test all

jlewi · 2017-10-26T22:14:55Z

/test all

jlewi · 2017-10-26T22:20:50Z

/test all

jlewi · 2017-10-26T22:23:08Z

/test all

jlewi · 2017-10-26T22:26:14Z

/test all

which will have different libraries then the machine running the lint check.

…trc.

jlewi · 2017-10-26T23:28:39Z

/test all

jlewi added 4 commits October 25, 2017 16:55

Trivial change to test travis on pull requests.

d62f616

Run autopep8 on all py files.

388eac1

Fix lint issues in py/

92a536c

Fix python lint issues

a12418c

Add lint to travis.

jlewi changed the title ~~Fix travis for presubmits.~~ Add python lint check to travis and fix python lint issues Oct 26, 2017

jlewi added 4 commits October 25, 2017 20:16

Travis needs to install lint.

7ca32b3

Try installing pylint via apt.

ac1021b

Try explicitly setting location of .pylintrc file because otherwise

192f1f4

pylint on Travis complains about missing config file.

Run lint as part of our E2E prow tests.

d1ec26d

jlewi added 3 commits October 26, 2017 13:59

Create a log of the build and copy it to GCS.

cb13787

Fix command for invoking lint.sh.

3e5d62d

* Ensure we pipe subprocess output to logging module

1b4d963

* runner.py should reuse py/util rather than redefining run.

Fix directory from which we invoke runner.py

7dbfb44

jlewi added 3 commits October 26, 2017 15:29

Fix bug in run command.

8597548

Disable import-error message since some py files run inside containers

8c03697

which will have different libraries then the machine running the lint check.

Remove some disable messages for lint since we disabled them in pylin…

84cb1e1

…trc.

jlewi merged commit 16fc3d0 into kubeflow:master Oct 27, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add python lint check to travis and fix python lint issues #91

Add python lint check to travis and fix python lint issues #91

jlewi commented Oct 25, 2017 •

edited

Loading

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

krzyzacy commented Oct 26, 2017

krzyzacy commented Oct 26, 2017

krzyzacy commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

Add python lint check to travis and fix python lint issues #91

Add python lint check to travis and fix python lint issues #91

Conversation

jlewi commented Oct 25, 2017 • edited Loading

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

krzyzacy commented Oct 26, 2017

krzyzacy commented Oct 26, 2017

krzyzacy commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 26, 2017

jlewi commented Oct 25, 2017 •

edited

Loading