Java System.out.print is affecting Kata tests. Bad solution can be submitted.

[dinglemouse2250]

A user encountered some weird problem running one of my Kata.

• His solution fails my test cases
• He included some Java System.out.print statements and magigically the Solution "passes". Actually it still fails same as before but CW does not recognize it anymore as failing!

--

I have reproduced his problem and the System.out.print REALLY DO affect the test result in some strange way.

Futhermore, if I change his System.out.print statements to println instead of print then the bad Kata solution fails again like it was ought to...

Something weird going on...

PSA screenshot showing the Output window when using System.out.print.

• Notice the Passed number (1) is plain wrong.
• Notice the error message shows on the screen but CW still thinks the solutions is a pass.

FYI - Here is link to his solution code:
https://www.codewars.com/kata/reviews/56f53a5b6dc4feeaa2000044/groups/58d1e2f7f4080cf53d0000bb

[kazk]

@jhoffner maybe you figured this out already, but this is probably caused by the following:

System.out.print:

user logged output<FAILED::>

System.out.println:

user logged output
<FAILED::>

[kazk]

Confirmed this reproducible in multiple languages. If the assertion failure doesn't write to stderr, and there's at least one <PASSED::> recognized, the test is considered passed.
I think this is essentially the same as codewars/codewars-runner-cli#127 where the user's output interferes with test result.

---

JavaScript behaves differently when the tests are not inside describe.

function add(a, b) {
  if (a != 1) return a + b;
  process.stdout.write(''+a);
  return a - b;
}

with describe

describe("Solution", function() {
  it("should test for something", function() {
    Test.assertEquals(add(1, 1), 2);
    Test.assertEquals(add(2, 1), 3);
  });
});

without describe

Test.assertEquals(add(1, 1), 2);
Test.assertEquals(add(2, 1), 3);

[blarosen95]

If I refactor my submission, will you all still have access to the bug related code?

[jhoffner]

The best way to handle this across all languages would be to either go through all of them and ensure that <PASSED::> and <FAILED::> are on their own line (by prefixing with \n), or updating the output parser to allow those tokens to exist anywhere in the line.

The later would be easier to make the change but I'm not certain of what edge cases that could create.

[kazk]

@jhoffner
How are blank lines handled? I tried in Kumite and it seems to be completely ignored.

console.log(`
a


b
`); // including the leading and the trailing

I don't know if this is intentional, but prefixing with \n looks safe.

Changing the output parser might be easier in short term, but I think it should be avoided because of the uncertainty. I also like the idea of "each new STDOUT line is considered a new piece of data" :)

[jhoffner]

@kazk I agree.

This change (for Java only) will be fixed within the next Coderunner deployment.

codewars/codewars-runner-cli#336

[kazk]

I can start changing other languages if you'd like.
Updating lib/runners/X.js and test/runners/X_spec.js and opening a PR per language?

[kazk]

@blarosen95 I think every distinct solution gets a permalink so no need to worry. Also the cause of this issue is found :)

[blarosen95]

I figured you guys had the issue pinpointed 👍

[jhoffner]

Changes are in preview, www is deploying now.

@kazk sounds good. Thanks!