make coverage.html; make coverage-report fails #334

manu-chroma · 2017-03-14T13:19:10Z

make coverage.html 
make coverage-report fails

these commands fail due to the following error:

Finished processing dependencies for cwltool==1.0.20170309164828
export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; \
       cd ; ./run_test.sh RUNNER=cwltool
/bin/sh: 2: ./run_test.sh: not found
Makefile:110: recipe for target '.coverage' failed
make: *** [.coverage] Error 127

related line of code in makefile: https://github.com/common-workflow-language/cwltool/blob/master/Makefile#L111

manu-chroma · 2017-03-14T13:21:59Z

I'm interested in knowing if there is any way to assess the test coverage of the current codebase.

manu-chroma · 2017-03-15T19:56:53Z

@mr-c thoughts ?

mr-c · 2017-03-22T10:36:00Z

@manu-chroma run-test.sh is from https://github.com/common-workflow-language/common-workflow-language/ -- the Makefile should be clarified -- sorry!

manu-chroma · 2017-03-22T12:53:13Z

@mr-c thanks. I will try this. 👍

Last night only did I realize that we can run conformance tests locally. That they are already present in cwltool/cwltool/schemas directory.

I believe that this is something that should be briefly mentioned in the main README. I was under the impression that they were only setup on the jenkins server. I'd be happy to make a PR for the same.

mr-c · 2017-03-22T13:11:40Z

You've touched on an organizational problem we have, I've started to think about it out loud at common-workflow-language/common-workflow-language#413

In summary, please don't use the conformance tests inside cwltool/cwltool/schemas as they are out of date :-)

mr-c · 2017-03-22T13:12:03Z

Yes, we should explain how to run the conformance tests in the README -- PR gladly accepted

manu-chroma · 2017-03-22T13:29:25Z

In summary, please don't use the conformance tests inside cwltool/cwltool/schemas as they are out of date :-)

@mr-c thanks for pointing this out. I was using them yesterday while debugging my PR: #337

Yes, we should explain how to run the conformance tests in the README -- PR gladly accepted

I'm thinking of adding short description on running tox locally. And adding essential contents from https://github.com/common-workflow-language/cwltool/blob/master/cwltool/schemas/CONFORMANCE_TESTS.md

aa320ec Merge pull request common-workflow-language#342 from common-workflow-language/fix-mystery-package-errors 3c6eaff Update for rename of argparse2cwl to argparse2tool b14404b remove errant jsonldPredicate 7f116da Merge pull request common-workflow-language#339 from common-workflow-language/dockerOutputDirectory 4f65d1f Add test for dockerOutputDirectory option of DockerRequirement. cb1f928 Merge pull request common-workflow-language#335 from common-workflow-language/output-literals d8b6c8f Add tests for file and directory literals in output as produced by expression tool. ee3bb1c Merge pull request common-workflow-language#324 from common-workflow-language/secondaryfiles-array 5e6c2d4 Add NLeSC's Xenon cbe1c8d Merge pull request common-workflow-language#334 from StarvingMarvin/master 8300e43 output ids must start with '#' in draft-2 f87d218 draft-3 test fixes: explicit output for args 8b10319 Fixes for draft-2 tests: adding explicit args output 83ed4b4 Merge remote-tracking branch 'rabix-cwl/master' e0a5e7c Merge pull request common-workflow-language#328 from StarvingMarvin/master d906c4a dir4 test fix 21bdcaf Merge remote-tracking branch 'upstream/master' dee0f4e Disabled checksum check for listing input dir fc4cb0f Added args output and empty.json for null jobs a4950f4 Update tools list 8e1d67c Add Rabix Bunny CI badge 2a5b95e fixing mentions to draft3/4 1356c45 Add test for secondaryFiles with arrays b60125b Single instead of double quotes in doc 179835b Merge pull request common-workflow-language#319 from common-workflow-language/test-cwl-out2 ed40737 Add additional test for cwl.output.json behavior. a217bb6 Merge pull request common-workflow-language#317 from StarvingMarvin/master 8dd69f2 fixing test in v1.0 and v1.1.0-dev1 e831c20 Merge pull request common-workflow-language#314 from common-workflow-language/test-requirements-on-steps 8cad6f1 Add test for requirements/hints on workflow steps. 7679adb Merge pull request common-workflow-language#308 from common-workflow-language/embedded-subworkflow-test 1582677 Tighten up formatting 779c2d4 Add test for embedded subworkflow e22a9c2 Merge pull request common-workflow-language#307 from alaindomissy/patch-3 125cb5f Update UserGuide.yml 99f4f9f Merge pull request common-workflow-language#303 from common-workflow-language/include-stdin-in-docs 5324dbb put stdin shortcut in correct place git-subtree-dir: cwltool/schemas git-subtree-split: aa320ec

manu-chroma · 2017-06-18T18:01:39Z

I was able to get combined code coverage from unit tests and conformance tests. Here it is. Conformance Tests cover roughly 30% of the codebase which is a subset of the codebase covered by unit tests. So combining the code coverage did not increase the coverage by more than few lines of code.

Explicit 'args' output for three tests in draft-2, draft-3 and v1.0

mr-c · 2017-08-09T08:50:47Z

make a simple shell script: coverage run cwltool $@ and pass that path in as the RUNNER to gather test coverage from the conformance tests

manu-chroma · 2017-08-12T11:50:38Z

@mr-c little-confused here. I've created coverage.sh as: coverage run cwltool $@

How to integrate it here: https://github.com/common-workflow-language/cwltool/blob/master/Makefile#L117
I forgot what trick you told to make the RUNNER resolve correctly.

mr-c · 2017-08-12T12:26:10Z

@manu-chroma Did you try RUNNER=path/to/coverage.sh ?

manu-chroma · 2017-08-12T12:41:32Z

Yes. It's giving error in run_test.sh:

Bottom of the error message:

export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; \
       cd common-workflow-language; ./run_test.sh RUNNER="/home/manu/github/cwltool/coverage.sh"
run_test.sh: Run common workflow tool description language conformance tests.

Syntax:
        run_test.sh [RUNNER=/path/to/cwl-runner] [DRAFT=cwl-draft-version]

Options:
  -nT   Run a specific test.
  -l    List tests

runner '/home/manu/github/cwltool/coverage.sh' not found
Makefile:116: recipe for target '.coverage' failed
make: *** [.coverage] Error 1

Going into this loop:

if ! runner="$(which $RUNNER)" ; then
    echo >&2 "$helpmessage"
    echo >&2
    echo >&2 "runner '$RUNNER' not found"
    exit 1
fi

mr-c · 2017-08-12T13:05:21Z

Ah, it expects it to be on the PATH. Then try

export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; export PATH=$PATH:$PWD; \
       cd common-workflow-language; ./run_test.sh RUNNER=coverage.sh

manu-chroma · 2017-08-12T13:13:34Z

Updating PATH is breaking run_test.sh. Not sure why

Using /home/manu/github/cwltool/venv3/lib/python3.5/site-packages
Finished processing dependencies for cwltool==1.0.20170811195303
export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; export PATH=ATH:WD; \
       cd common-workflow-language; ./run_test.sh RUNNER=coverage.sh
./run_test.sh: line 3: basename: command not found
./run_test.sh: line 3: basename: command not found
./run_test.sh: line 18: uname: command not found
./run_test.sh: line 47: cut: command not found
./run_test.sh: line 47: cut: command not found
./run_test.sh: line 47: =: command not found
./run_test.sh: line 52: which: command not found
: Run common workflow tool description language conformance tests.

Syntax:
         [RUNNER=/path/to/cwl-runner] [DRAFT=cwl-draft-version]

Options:
  -nT   Run a specific test.
  -l    List tests

runner 'cwl-runner' not found
Makefile:116: recipe for target '.coverage' failed
make: *** [.coverage] Error 1
(venv3) manu@hp:~/github/cwltool$ which cwl-runer
(venv3) manu@hp:~/github/cwltool$ which cwl-runner 
/home/manu/github/cwltool/venv3/bin/cwl-runner

mr-c · 2017-08-12T13:25:40Z

Ah, if you're in a Makefile the $ have to be escaped: export PATH=$$PATH:$$PWD;

manu-chroma · 2017-08-12T13:35:42Z

ok, PATH behavior is fine now. But it shows coverage.sh as invalid runner

 COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; export PATH=$PATH:$PWD; \
       cd common-workflow-language; ./run_test.sh RUNNER=coverage.sh
run_test.sh: Run common workflow tool description language conformance tests.

Syntax:
        run_test.sh [RUNNER=/path/to/cwl-runner] [DRAFT=cwl-draft-version]

Options:
  -nT   Run a specific test.
  -l    List tests

runner 'coverage.sh' not found
Makefile:116: recipe for target '.coverage' failed
make: *** [.coverage] Error 1

mr-c · 2017-08-12T15:55:58Z

Is it marked executable? `chmod a+x script.sh`

manu-chroma · 2017-08-12T17:01:28Z

ok, added that. Output now:

Using /home/manu/github/cwltool/venv3/lib/python3.5/site-packages
Finished processing dependencies for cwltool==1.0.20170811195303
export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; export PATH=$PATH:$PWD; \
       cd common-workflow-language; ./run_test.sh RUNNER=coverage.sh
--- Running conformance test v1.0 on /home/manu/github/cwltool/coverage.sh ---
No file to run: 'cwltool'

mr-c · 2017-08-12T17:35:33Z

I would debug this by using coverage.sh by hand as if it were cwltool

mr-c · 2017-08-14T07:11:45Z

@manu-chroma What happens when you run a single conformance test directly by hand with coverage.sh (without using run_test.sh or cwltest)?

manu-chroma · 2017-08-14T11:44:44Z

(venv3) manu@hp:~/.../common-workflow-language/v1.0$ coverage.sh --debug --outdir=/tmp/tmprcwzi_kf v1.0/scatter-wf1.cwl v1.0/scatter-job1.json

This runs successfully. Trying to debug for the whole suite.

Also, I had modified coverage.sh to:

coverage run /home/manu/github/cwltool/venv3/bin/cwltool $@

Should use readlink cwltool here I suppose

manu-chroma · 2017-08-14T12:13:59Z

@mr-c Working now. Need to remove hardcoded paths though

(venv) manu@hp:~/.../cwltool/common-workflow-language$ coverage report --omit="*/site-packages/*","*/schema_salad/*"
Name                                                         Stmts   Miss  Cover
--------------------------------------------------------------------------------
/home/manu/github/cwltool/cwltool.py                             5      0   100%
/home/manu/github/cwltool/cwltool/__init__.py                    2      0   100%
/home/manu/github/cwltool/cwltool/builder.py                   190    163    14%
/home/manu/github/cwltool/cwltool/cwlrdf.py                     84     71    15%
/home/manu/github/cwltool/cwltool/docker.py                     99     85    14%
/home/manu/github/cwltool/cwltool/docker_id.py                  46     32    30%
/home/manu/github/cwltool/cwltool/draft2tool.py                476    424    11%
/home/manu/github/cwltool/cwltool/errors.py                      4      0   100%
/home/manu/github/cwltool/cwltool/expression.py                174    144    17%
/home/manu/github/cwltool/cwltool/flatten.py                    19     16    16%
/home/manu/github/cwltool/cwltool/job.py                       314    274    13%
/home/manu/github/cwltool/cwltool/load_tool.py                 151    124    18%
/home/manu/github/cwltool/cwltool/main.py                      560    395    29%
/home/manu/github/cwltool/cwltool/mutation.py                   37     25    32%
/home/manu/github/cwltool/cwltool/pack.py                      125    112    10%
/home/manu/github/cwltool/cwltool/pathmapper.py                175    135    23%
/home/manu/github/cwltool/cwltool/process.py                   548    466    15%
/home/manu/github/cwltool/cwltool/resolver.py                   40     30    25%
/home/manu/github/cwltool/cwltool/sandboxjs.py                 211    185    12%
/home/manu/github/cwltool/cwltool/software_requirements.py      74     53    28%
/home/manu/github/cwltool/cwltool/stdfsaccess.py                51     25    51%
/home/manu/github/cwltool/cwltool/update.py                    377    329    13%
/home/manu/github/cwltool/cwltool/utils.py                     104     84    19%
/home/manu/github/cwltool/cwltool/workflow.py                  635    575     9%
--------------------------------------------------------------------------------
TOTAL                                                         4501   3747    17%

tetron pushed a commit that referenced this issue Jul 23, 2017

Merge pull request #334 from StarvingMarvin/master

cbe1c8d

Explicit 'args' output for three tests in draft-2, draft-3 and v1.0

manu-chroma mentioned this issue Aug 13, 2017

remove draft-2 and draft-3 support #520

Merged

mr-c closed this as completed Feb 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make coverage.html; make coverage-report fails #334

make coverage.html; make coverage-report fails #334

manu-chroma commented Mar 14, 2017

manu-chroma commented Mar 14, 2017 •

edited

Loading

manu-chroma commented Mar 15, 2017

mr-c commented Mar 22, 2017

manu-chroma commented Mar 22, 2017 •

edited

Loading

mr-c commented Mar 22, 2017

mr-c commented Mar 22, 2017

manu-chroma commented Mar 22, 2017

manu-chroma commented Jun 18, 2017 •

edited

Loading

mr-c commented Aug 9, 2017 •

edited

Loading

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017 via email

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

mr-c commented Aug 14, 2017

manu-chroma commented Aug 14, 2017

manu-chroma commented Aug 14, 2017

make coverage.html; make coverage-report fails #334

make coverage.html; make coverage-report fails #334

Comments

manu-chroma commented Mar 14, 2017

manu-chroma commented Mar 14, 2017 • edited Loading

manu-chroma commented Mar 15, 2017

mr-c commented Mar 22, 2017

manu-chroma commented Mar 22, 2017 • edited Loading

mr-c commented Mar 22, 2017

mr-c commented Mar 22, 2017

manu-chroma commented Mar 22, 2017

manu-chroma commented Jun 18, 2017 • edited Loading

mr-c commented Aug 9, 2017 • edited Loading

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017 via email

manu-chroma commented Aug 12, 2017

mr-c commented Aug 12, 2017

mr-c commented Aug 14, 2017

manu-chroma commented Aug 14, 2017

manu-chroma commented Aug 14, 2017

manu-chroma commented Mar 14, 2017 •

edited

Loading

manu-chroma commented Mar 22, 2017 •

edited

Loading

manu-chroma commented Jun 18, 2017 •

edited

Loading

mr-c commented Aug 9, 2017 •

edited

Loading