Feature: upgrade macro evaluation strategy to use nimscripter

[ynfle]

This upgrades the representer to use https://github.com/beef331/nimscripter.

This module is a branch between Nimscript (a subset of nim that runs in
the VM) and the compiled nim. It embeds an interpreter in to the
executable, and doesn't require a nim compiler to be in path. The
primary advantage is the the bridge is built-in, so there is no need for
recompiling to create a representation for every exercise, as the
representer uses nim's macros which only function in the VM. Without
this, only the VM is used, but the executable has to be recompiled every
time.

This also greatly simplifies the testing process, as the representation
creation doesn't need to happen at compile to and injecting in. Rather,
it can happen at runtime with the VM bridge.

CI: update to use version 1.6.6 and refractor of the representer

https://github.com/jiro4989/setup-nim-action is used to install nim
using choosenim with the desired version.

Caching is utilized to not have to redownload the dependent packages
again and invalidates the cache when the nimble file changes. This is
the example used on the setup-nim-action repo README.md

Closes #9 as not relevant

[ynfle]

Please review, although I need to look over everything again

[ErikSchierboom]

The code looks good. I'll let @ee7 do the actual Nim review. I restricted myself to commenting on the CI-related things.

[ErikSchierboom]

@ynfle Is there any performance difference between this version and the previous one?

[ErikSchierboom]

Looks great! I'm approving based on the GHA stuff.

[ynfle]

Note docopt was added as a dependency here because before there was just compilation, so runtime args weren't possible. This can be split in to a separate PR or a separate commit.

[ynfle]

@ynfle Is there any performance difference between this version and the previous one?

Yes. on my machine it is about a 2x performance boost. From ~1.6s on main and ~.75s with this latest commit on the PR.

As more normalizations are added, I think that there will be a less of a performance boost, as the primary difference isn't in the evaluation of the AST etc. but rather the startup time and "other things" that compiler does which we don't need and isn't embedded in our executable. The previous strategy was to recompile every time.

[ynfle]

A fairer comparison would be a nimscript script which wouldn't need to recompile every time. A prototype with the following script ran at about ~.95s which is only a 1.25x speed up instead of 2x

import std/[json, macros, os, strformat, strutils]
import representer/[mapping, normalizations, types]

proc createRepresentation*(file: string): tuple[tree: NimNode, map: IdentMap] =
  var map: IdentMap
  let code = parseStmt(file)
  result = (tree: code.normalizeStmtList(map), map: map)

proc getTestableRepresentation*(contents: string, switch = false): SerializedRepresentation =
  let (tree, map) = createRepresentation(contents)
  result = (repr tree, $(if switch: %map.switchKeysValues else: %map))

proc getFileContents*(fileName: string): string = readFile fileName

func underSlug(s: string): string = s.replace('-', '_')

proc main() =
  let (tree, map) = getTestableRepresentation(getFileContents(paramStr(3) / underSlug(paramStr(2)) & ".nim"))
  if paramCount() == 5:
    let outDir = paramStr(4)
    writeFile outDir / "mapping.json", $map.parseJson
    writeFile outDir / "representation.txt", tree
  echo &"{tree = }\n{map.parseJson.pretty = }"

main()

[ee7]

Great to see work on the representer - keep going! Nice idea to use nimscripter - it's an interesting application for it.

The main downside that I see is the relatively large increase in compilation time, but I think it's worth it.

This PR means that when we get a Dockerfile in this repo, we'll now add the representer binary, rather than the representer source file. And then we don't even need to have Nim installed in the image, right? So if we statically link the representer binary, we can have a tiny image that only contains enough to run run.sh, and the representer binary (looks like roughly 4.0 MiB after stripping, or 3.3 MiB with --passC:-flto).

[ee7]

Looking good - I find tests/tnormalizations.nim way more readable now - thanks.

I've created #38 and #39 to track the review comments that we could keep in mind for later.

Here's a few tiny nits - feel free to ignore.

Already approved before, but approving again for the recent changes.

[ynfle]

@ee7 @ErikSchierboom Which size level should this PR be labeled? x:size/massive?

[ErikSchierboom]

We don't yet have official guidelines, but I'm fairly certain this would count a massive.

P.S. you should probably get into the habit of using the x:rep labels to assign reputation, as they're superseding the x:size labels for reputation awarding (they both work for backwards compatibility).