About the New Runner

[kazk]

I'm planning to release the new runner for Codewars during this week (maybe early next week depending on how things goes) and wanted to let Codewars users know about the changes.
The preview site uses it already, so some of the features can be tested there for now.

Background

Codewars and Qualified uses codewars-runner-cli to execute submitted code.
The project started more than 4 years ago and requirements have changed significantly over the years.

• increased number of supported languages: just few to 30+ (the new runner has 37)
• need to support language versions without breaking existing contents
  • many languages are outdated because of this
• increased number of users

We needed a drop-in replacement that's more flexible. So I've refactored the runner and the server, then rebuilt the infrastructure.

We're going to eventually open source parts of this (Dockerfile, test frameworks, etc) after some planning and cleaning up.

Changes

This release focused mainly on backwards compatibility with codewars-runner-cli and the goal was to build a more flexible system that we can easily change depending on the requirements.

Note that some of the new features may not be available on Codewars until the frontend is updated.

New Features

• New Languages
  • Julia with FactCheck
    • Based on JuliaAttic/FactCheck.jl@51a5e71
    • Julia run issues codewars-runner-cli#505
  • NASM with Criterion
  • Fortran with CW2 (contributed by @DonaldKellett)
    • Add Fortran 95 Support codewars-runner-cli#553
  • PowerShell (6.0.0) with Pester
    • Add PowerShell Core 6 codewars-runner-cli#562
• Language Versions
  • Elixir 1.6 (Add Elixir 1.6 codewars-runner-cli#566)
  • Haskell GHC 8.2 (Haskell GHC 8 support codewars-runner-cli#454)
  • Ruby 2.5 (Add Ruby 2.5 support codewars-runner-cli#560)
  • Rust 1.23 (Add Rust version 1.25 codewars-runner-cli#556)
  • Solidity 0.4.19
  • Swift 4 (Add Swift 4 support codewars-runner-cli#554)
  • Crystal 0.24 (Add Crystal 0.24.2 support codewars-runner-cli#585)

Enhancements

• Python: improved cw-2.py (contributed by @Bubbler-4)
• Python: Unicode output
• Python: No concatenation (python traceback is missing line numbers, files #785)
  • breaks ~70 kata and needs some updates, see below
• TypeScript: Use @types. No more /// <reference
• Versioned images to allow language version upgrades without breaking
• Infrastructure: more computing resources to everyone

Bug Fixes

• Proper C++14 support (Missing std::make_reverse_iterator codewars-runner-cli#563)
• numpy on Python 3.6 (Can't import numpy in Python 3.6.0 codewars-runner-cli#367)

Deprecations

• Node 0.10.x; Node 6.x will be used when 0.10.x is specified for now.

Compatibility

Tested against all published kata (including beta) and it's currently compatible with ~98% of the contents.
I'll create a wiki page (will post a comment when I do) listing affected kata so power users can help fixing them. Most of them should be fixed by adjusting few things.
However, those kata that uses things that we never officially supported is not considered and needs major changes. This includes Shell kata written in Perl or Python.

Stats

• 7605 kata (data exported on 2018-03-15)
• 20594 counting each language variations
• 498 possible issues detected (compatible with 20096/20594 ~97.58%)
  • already fixed: 90
  • caused by missing package (will be fixed soon): 44
  • caused by unsupported features (invalid/wontfix): 58
  • this number includes some noise like author's solution failing random tests
  • this also includes already broken ones

[Voileexperiments]

this number includes some noise like author's solution failing random tests

Well, in this case it's still an issue (even if it's unrelated to this topic), so it'd be helpful if you list them too (probably in #1315).

[kazk]

Started List of Affected Kata. Just listing Python kata affected by avoiding concatenation for now.

I'll update it again later to include other issues.

[DonaldKellett]

Glad to hear that the new runner is finally out on preview, I can barely contain my excitement! 😄 One thing though - am I allowed to author Fortran kata/translations ATM? If authoring them will interfere with the main site negatively them I'm going to stick to plain Kumite until further notice, cheers 😃

[Voileexperiments]

@DonaldKellett well, we did have written bf katas (and got them approved I think?) while it was in beta, so...

[kazk]

@DonaldKellett I don't think that's possible yet because the frontend doesn't know about it (you can't select Fortran).
I wanted you to at least try some Kumite so I opened a PR to add Fortran support last week, but I'm not sure when it'll be merged/deployed because Jake has been busy with other tasks.

[Bubbler-4]

Fixed all of my katas, and will start fixing others in Python as soon as I get some spare time. Looks like many have problems in common (such as Linked List series or CIS series, though I really think CIS ones should be just retired asap), so it won't take too much time to fix most of them, though re-publishing a kata often times out these days unfortunately.

[kazk]

Kata with issues will show a warning message like the following on the preview site.

Once someone completes or republish successfully, this message will go away.

[Bubbler-4]

Just for reference, the "Caution" message is removed only if:

• On Preview site's edit screen, somewhere in the "code" section (usually Preloaded / Test) is updated and re-published, or
• On Preview site's trainer, a solution different from the solver's previous ones is submitted final.

Also, some katas in the affected list don't have issues raised (so not open for edit right now).

[kazk]

Just an update, we've decided to release on this Wednesday.

Sorry for the delay. I had some issues finalizing the production setup 😅
We'll make new languages and language versions available on Codwars soon if everything goes well.

[kazk]

I forgot to mention, but testing Scala will be significantly faster soon.

The new runner is able to run ScalaTest without JUnitRunner (avoiding gradle test). Using Gradle allowed adding Groovy/Kotlin/Scala all at once, but it wasn't a good choice for our use case.

So minimum kata takes (includes compiling):

• 20+s with legacy runner (non-Red) 😭
• ~16s with legacy runner (Red) 😢
• ~12s with new runner (gradle test) ☹️
• ~6s with new runner (ScalaTest) 🙂

Because Codewars only supports one test framework, existing 25 Scala kata needs to be changed. If tests are using ScalaTest, migration will be just removing lines that references JUnit 4. I hope it's not a huge pain migrating those written with JUnit 4. This won't be included in tomorrow's launch, but I'd like to switch soon because it'll help reducing the server load.

I'll look into doing similar for Kotlin and Groovy.

[kazk]

I've released the new runner! Please let me know if you find anything weird.

[kazk]

For Red users, there're some features that I didn't have the time to add at the moment. Like real time output.

[kazk]

Also, the time displayed at the top of the test results will be different from before. It includes all the time it took to execute the code, including some overhead and compiling. For most interpreted languages, the difference is few hundred milliseconds. Some compiled languages will be more different because now it includes the time it took to compile.

I'll work on improving this after other higher priority tasks.

[kazk]

Looks like there's some weird issue with Python encoding.

Python 2 doesn't allow non-ASCII characters unless the encoding is declared at the top (# -*- coding: utf-8 -*-), but there're some Python 2 only kata with non-ASCII characters :/
I don't know how those got published with Python 2 or if it was working on the old runner. The new runner supports the magic comment, but the old one didn't because of concatenation.
I don't know if python -c "code" (the old runner did this for unknown reason) handles the code differently.

[Blind4Basics]

Looking at it. Unfortunately, most of those are katas I didn't do yet, apparently. I opened a bunch of issues to get access (note: for the nonogram, I published a fork with what's needed)

[kazk]

I opened a bunch of issues to get access

Wait, so a kata needs to have some issues open or it's not editable? I didn't know that... I guess that makes sense to prevent abuse.

Thanks for your help!

[Voileexperiments]

A kata is only open for editing if it has any issues at least 1 week old.

[gbdj]

Previously tested Haskell solution for TinyThreePassCompiler kata failing now with errors:

Could not find module `Text.Parsec'
Could not find module `Text.Parsec.String'
Could not find module `Text.Parsec.Expr'
Could not find module `Text.Parsec.Token'
Could not find module `Text.Parsec.Language'

May be parsec library was forgotten after Code Execution Engine upgrade?

[kazk]

@gbdj thanks for letting me know. I'll look into it.

[kazk]

I'm not sure why the old image had parsec installed :/ The new image have all of the packages specified in the old Dockerfile:

# Install Haskell Packages
# To install additional packages use "RUN cabal install <pkg-name>"
RUN cabal install split ifelse
RUN cabal install persistent-sqlite persistent-template
RUN cabal install haskell-src-exts lens
RUN cabal install hspec hspec-core hspec-discover

https://github.com/Codewars/codewars-runner-cli/blob/d1c1d8bfb9919ee61d520370484894be27f48739/docker/haskell.docker#L30-L35

I'll just add parsec explicitly.

[kazk]

@gbdj I've deployed the fix. You should be able to use parsec now.

[gbdj]

@kazk Thanks. It works now.

[10XL]

May be parsec library was forgotten after Code Execution Engine upgrade?

I have a similar issue with a Ruby kata that uses nokogiri, where it seems to not be installed on Ruby 2.5

@kazk running this kumite shows that some Ruby gems from 2.3 are missing on 2.5.

[kazk]

@10XL Ruby 2.5 doesn't have gems installed at the moment because I want to update package versions.
Kata shouldn't have 2.5 enabled if it can't be completed. If it does, then it's another bug from the auto upgrade script :(

[10XL]

Kata shouldn't have 2.5 enabled if it can't be completed. If it does, then it's another bug from the auto upgrade script :(

Yes, that is the case. The kata has 2.5 as unavailable/unsupported in the trainer as it should.

[kazk]

OK, good.
I'll add gems for 2.5 next week. I can't guarantee all of the gems from 2.3 will be installed or if they're compatible. But I'll make sure to install commonly used ones like nokogiri.

[Voileexperiments]

@kazk Is Solidity 0.4.19 even working at all? Because I just saw this kata comment.

[kazk]

It works, but it's not compatible with 0.4.13. Also, you shouldn't need to require('web3') because Truffle injects it globally.

[kazk]

On 0.4.19 which uses Truffle 4, you don't need to set web3 provider because it starts internal server automatically.

[kazk]

I don't know why it has 0.4.19 enabled when it doesn't work. Probably just another bug from auto update script :(

[kazk]

@10XL I've added gems to Ruby 2.5. gem list also lists gems that are installed as dependency of others so don't assume all of them will continue to exist. I'll try to update the wiki soon for list of gems that we installed.

[nomennescio]

So where's the new runner repo? The main webpage are still pointing to the old repo, which is marked deprecated. I want to submit a bug to the NASM runner, which issue tracker to use?

[kazk]

@nomennescio we don't have one yet.

We're going to eventually open source parts of this (Dockerfile, test frameworks, etc) after some planning and cleaning up.

So we're still using codewars-runner-cli to track runner issues.

[nomennescio]

We're going to eventually open source parts of this (Dockerfile, test frameworks, etc) after some planning and cleaning up.

So we're still using codewars-runner-cli to track runner issues.

That's great! Marking the whole repository deprecated seems however a bit premature if the issues are still tracked there. Maybe comment on the mainpage that it is still used for issue tracking?

[nomennescio]

Duh! It does mention it on the mainpage now.

[kazk]

@nomennescio yeah, I changed it because I realized that it's confusing after you mentioned it :)

[FArekkusu]

@kazk the list of affected katas has been cleared up (except 2 katas which should be retired). With those ones resolved the wiki page could be also removed.

[kazk]

Thanks!