kcov + bats + bash: lines sometimes covered, sometimes not

[sschmid]

Hi @SimonKagstrom

After we found a fix in #458 I was updating my projects and I ran into another issue: test coverage was different when running in GitHub action, docker container or locally. Most of the covered lines were the same, but in some cases some lines were not covered, even-though they should have been. I try my best to explain, but currently it's hard to know what's up, because the issue varies from project to project.

I could narrow it down and updated the minimal sample project that you already know from #458, see
https://github.com/sschmid/kcov-sample

Working test coverage:

@test "prints args" {
  run app test
  assert_success
  assert_output "args: test"
}

Not working test coverage:

@test "prints args" {
+ cd "${BATS_TEST_TMPDIR}"
  run app test
  assert_success
  assert_output "args: test"
}

Adding the line cd "${BATS_TEST_TMPDIR}" results in kcov not covering the app file at all, coverage is 0% and the file doesn't show up in index.html. Fyi: BATS_TEST_TMPDIR is a temp test dir provided by bats in a completely different location and not part of the project

What I experienced so far:

• test coverage kept working as expected on the macos-latest run
• when running locally (also macOS, with brew install kcov) test coverage was 0% and the file doesn't show up in index.html
• I added a new step called coverage to the GitHub action that uses kcov/kcov container, same result as locally: 0% coverage

see run "Break coverage", which comes with 2 uploaded coverage reports as artifacts, one from macos-latest, the other from the kcov/kcov container.
https://github.com/sschmid/kcov-sample/actions/runs/10949339328

Adding cd "${BATS_TEST_TMPDIR}" breaks it. Maybe kcov filters or strips it out, because the path is matching the project?

Now here's the weird part:
In an other projects it's the opposite way, and it's where I actually initially found the issue, caused by cd "${BATS_TEST_TMPDIR}" in a test setup.
This projects previously used the kcov/kcov container. I changed it to not use the container and to use the solution we found in #458 and coverage percentage was lower. Note: the action is running on ubuntu-latest

What I experienced there:

• kcov/kcov container worked (opposite what I described above)
• GitHub action has less coverage (opposite what I described above, but it's ubuntu-latest not macos-latest)
• locally (macOS) has less coverage (same as above)

One more strange observation:

while cd "${BATS_TEST_TMPDIR}" causes the issue described above, it helped in the second project to make it work (?!?)

old setup that produced less coverage

# is called by bats before each test
setup() {
  ...
  cd "${BATS_TEST_TMPDIR}"
}

@test "prints args" {
  run app test
  assert_success
  assert_output "args: test"
}

New and working:

setup() {
  ...
- cd "${BATS_TEST_TMPDIR}"
}


@test "prints args" {
+ cd "${BATS_TEST_TMPDIR}"
  run app test
  assert_success
  assert_output "args: test"
}

Moving cd instruction into the test helped getting the affected lines in the source script being covered again.

This is all the info I have so far. Let me know if you have a clue what's going on ir if I should try something.

direct links to the coverage reports from the run "Break coverage":

• run: https://github.com/sschmid/kcov-sample/actions/runs/10949339328
• working test coverage: https://github.com/sschmid/kcov-sample/actions/runs/10949339328/artifacts/1955535108
• broken test coverage: https://github.com/sschmid/kcov-sample/actions/runs/10949339328/artifacts/1955535027

[SimonKagstrom]

Long shot, but you use --include-path= as an option. What happens if you remove that?

[sschmid]

Without --include-path the kcov terminal output is the same and I can see the test passed as expected.
The index.html now lists many files, mostly bats related, but unfortunately not src/app.

See run (as usual, with both coverage reports attached, ready to be downloaded)
https://github.com/sschmid/kcov-sample/actions/runs/10958582883