🧹 Add a build tool (#4799)

We are accumulating quite a lot of build scripts that are all written in bash, and need to be implemented in a just-so order, and you need to know exactly when you need to run them. It might be time to start looking into moving to an actual tool to manage the build for us.

These come in a lot of flavors with various capabilities. I've looked around a bunch, and I think [DoIt](https://pydoit.org) looks quite nice.

Here are the features that I think make this tool quite nice:

- It's written in Python, so comes in automatically as Python dependency during `pip install`.
- The configuration file is written in Python as well, so we don't have to learn a new configuration language just for this tool.
- Tasks have dependencies between them, so if there are multiple that need to run, they automatically run in the right order.
- Tasks can (optionally) declare files that are inputs to the task itself, and if the input files have not changed the task is skipped. This makes it safe to just run the build toold and trust that you're not waiting too long for nothing.

I've started converting some build scripts over to this new tool to show how it works. You can see it in action by going into the virtual env and running (must be in the root directory, unfortunately):

```
$ doit
$ doit run typescript
$ doit help
```

The `.doit.db.db` that I've ignored is a file it uses internally to track changes to dependencies.

I've paused here because I'd like to solicit feedback.  Does the tool look okay? Is this understandable? Do we prefer this to bash, or to other build tools?

.github/workflows/cypresstests.yml

@@ -30,22 +30,9 @@ jobs:
     - name: Install Nodejs dependencies
       run: |
         npm ci
-    - name: Generate grammar files
+    - name: Run the deploy script and prepare the dev database
       run: |
-        python ./content/yaml_to_lark_utils.py
-    - name: Compiling Babel translations
-      run: |
-        pybabel compile -f -d translations
-
-    - name: Feed the dev database
-      # This replaces the json file, do it before starting the server
-      run: ./feed_dev_database.sh
-
-    - name: Bundle files
-      # Generates js and css files.
-      run: |
-        build-tools/heroku/tailwind/generate-development-css
-        build-tools/heroku/generate-typescript
+        doit run deploy devdb
 
     - name: Start the server
       # Runs in the background

.github/workflows/unittests.yml

@@ -34,13 +34,9 @@ jobs:
     - name: Install Nodejs dependencies
       run: |
         npm ci
-    # we already run this in build-tools/github/validate-tests
-    #- name: Generate grammar files
-    #  run: |
-    #    python ./content/yaml_to_lark_utils.py
-    - name: Compiling Babel translations
+    - name: Prepare backend
       run: |
-        pybabel compile -f -d translations
+        doit run backend
     - name: Calculate hedy cache key
       run: "echo value=$(ls -1 hedy.py grammars/* | sort | xargs tail -n 99999999 | sha256sum | cut -f 1 -d ' ') >> $GITHUB_OUTPUT"
       id: hedy_cache_key

.github/workflows/update-javascript-on-main.yml

@@ -40,14 +40,9 @@ jobs:
         python3 -m pip install --upgrade pip
         pip3 install -r requirements.txt
 
-    # This is necessary for the TypeScript to contain translations
-    # from Gettex.
-    - name: Compiling Babel translations
+    - name: Generating TypeScript
       run: |
-        pybabel compile -f -d translations
-
-    - name: Compile TypeScript to JavaScript
-      run: build-tools/heroku/generate-grammars-and-js
+        doit run typescript
 
     - name: Commit changed files
       uses: stefanzweifel/git-auto-commit-action@v2.3.0

.gitignore

@@ -177,3 +177,6 @@ grammars-Total/*.lark
 
 # hedy test cache
 .test-cache
+
+.doit.db
+.doit.db.*

build-tools/github/validate-typescript

@@ -8,6 +8,4 @@ set -eu
 scriptdir=$(cd $(dirname $0) && pwd)
 
 echo "------> Compiling the TypeScript"
-
-cd $scriptdir/../heroku
-./generate-typescript
+doit run typescript

build-tools/heroku/generate-lezer-parsers

@@ -1,5 +1,7 @@
 #!/bin/bash
 set -eu
+scriptdir=$(cd $(dirname $0) && pwd)
+cd $scriptdir
 
 # Generate the lezer parsers from the grammars specified in  highlighting/lezer-grammars/
 for grammarFile in ../../highlighting/lezer-grammars/level*.grammar ; do

build-tools/heroku/generate-typescript

@@ -1,83 +1,6 @@
 #!/bin/bash
 set -eu
-# Recompile the TypeScript and generate a minified JavaScript bundle using esbuild.
-#
-# We commit the generated JavaScript bundle to minimize the burden on contributors--
-# they don't need to install any of the NPM/esbuild tools and remember to run build
-# scripts if they don't need to work on the browser code.
-#
-# We minimize the generated bundle so that contributors aren't tempted to accidentally
-# edit straight in the JavaScript.
-#
-#----------------------------------------------------------------------
-#
-# This script does two similar, but different jobs:
-#
-# In normal build mode, we first compile TypeScript -> JavaScript using the
-# TypeScript compiler (TSC), and then bundle it all into one file using esbuild.
-# The reason we do this is because TSC can translate certain desirable idioms
-# that are are not supposed in old browsers, like `async`, `?.` and `??` to
-# plain old JavaScript. `esbuild` does not do that, which is why have TSC do
-# this job first.
-#
-# We can also run in `--watch` mode, in which case we bypass the TypeScript
-# compiler and only run esbuild. This is because the TypeScript compiler is
-# rather slow (think 5-10 seconds to do a compilation), and we want to
-# optimize for iteration speed in that case. That does mean that the generated
-# JavaScript is slightly different between those cases--however, the final
-# JavaScript will always be generated on deploy using the official generation
-# method.
-root=$(cd $(dirname $0)/../.. && pwd)
-
-cd $root
-
-python3 highlighting/generate-rules-highlighting.py
-python3 build-tools/heroku/generate-client-messages.py
-
-if [[ ! -f node_modules/.bin/tsc ]]; then
-    echo "👉 You need to run the following command ONCE before running 'generate-typescript':" >&2
-    echo "" >&2
-    echo "    npm ci" >&2
-    echo "" >&2
-    exit 1
-fi
-
-if [[ "${1:-}" == "--watch" ]]; then
-    echo "👀 Running TypeScript compilation in watch mode. 👀"
-    echo "Leave this window open and edit (and save) .ts files to your heart's content."
-    echo ""
-    # Watch mode, just use esbuild and nothing else
-    npx esbuild static/js/index.ts \
-        --watch \
-        --bundle \
-        --sourcemap \
-        --minify \
-        --target=es2017 \
-        --global-name=hedyApp \
-        --platform=browser \
-        --outfile=static/js/appbundle.js
-elif [[ "${1:-}" == "--tests" ]]; then
-    echo "Generating .js bundle files"
-    npx esbuild static/js/index.ts \
-        --bundle \
-        --sourcemap \
-        --minify \
-        --target=es2017 \
-        --global-name=hedyApp \
-        --platform=browser \
-        --outfile=static/js/appbundle.js
-else
-    # Use tsc to generate .js
-    npx tsc --outDir __tmp__
-
-    npx esbuild static/js/index.ts \
-        --bundle \
-        --sourcemap \
-        --minify \
-        --target=es2017 \
-        --global-name=hedyApp \
-        --platform=browser \
-        --outfile=static/js/appbundle.js "$@"
-
-    rm -rf __tmp__
-fi
+echo "🦄 This command has been replaced! To compile TypeScript, type the following":
+echo ""
+echo "    doit run typescript"
+exit 1

content/yaml_to_lark_utils.py

@@ -33,10 +33,6 @@ def extract_Lark_grammar_from_yaml():
         with open(yaml_filesname_with_path, 'r', encoding='utf-8') as stream:
             command_combinations = yaml.safe_load(stream)
 
-        # We don't want to create the grammar if there are no translations
-        if en_command_combinations == command_combinations and yaml_lang != "en":
-            continue
-
         # Create an empty dictionary -> fill with english keywords and then overwrite all translated keywords
         translations = collections.defaultdict(lambda: 'Unknown Exception')
         translations.update(en_command_combinations)