Backend improvements - BREAKING CHANGES (#178)

# Re-read the [installation instructions](https://github.com/HackSoc/hacksoc.org#installation) and use `hacksoc_org run` instead of `flask run`

* Added build time global function

* WIP changing how the CLI works

* Slightly less WIP using `argparse`

* Don't need pip-requirements any more. Documentation TODO

* Fixed unit tests

* github actions moment

* Added cmark (cmarkgfm) backend

* Updated docs

* Added command-line help strings

* Installation note

* whoops

.github/workflows/unittest.yaml

@@ -32,8 +32,11 @@ jobs:
         run: source venv/bin/activate
         if: ${{ matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest' }}
 
-      - name: "Install dependencies"
-        run: pip install -r pip-requirements.txt
+      - name: "Upgrade pip"
+        run: pip install --upgrade pip
+
+      - name: "Install package"
+        run: pip install -e .
 
       - name: "Run unit tests"
         run: python -m unittest discover -s tests/

.gitignore

@@ -1,3 +1,4 @@
 venv/
 __pycache__/
 build/
+*.egg-info/

README.md

@@ -19,30 +19,29 @@ Documentation can be found in [`docs/`](docs/), including topics such as:
 | [Development process and using git – beginner's guide](docs/development_and_git.md) |
 
 ## Running
-Preferred way to run is through the `flask` command. It can be executed with the venv (see [Installation](#installation)) activated by just running `flask` or otherwise at `venv/bin/flask`:
+Preferred way to run is through the `hacksoc_org` command. It can be executed with the venv (see [Installation](#installation)) activated by just running `hacksoc_org` (all platforms, recommended) or otherwise at `venv/bin/hacksoc_org` on Mac and Linux.
+
+Note that running through `flask run` is **not supported**.
 
 ### Starting a development server
 While you're developing, you probably want to use:
 ```
-flask run
-venv/bin/flask run
+hacksoc_org run
 ```
 Pages are served directly from the Flask routes; you shouldn't need to restart the server when changes are made, but web pages will not automatically refresh. Open your browser to [`http://localhost:5000/`](http://localhost:5000/) to see the results.
 
 ### Freezing to HTML
-To produce a folder full of static HTML and assets (images, CSS, JS, fonts), the site must be *frozen*. The resulting folder can then be used with a regular webserver (like nginx), and should look exactly the same as when running with `flask run`. You probably want to use this or `flask serve` at least once before you create a pull request.
+To produce a folder full of static HTML and assets (images, CSS, JS, fonts), the site must be *frozen*. The resulting folder can then be used with a regular webserver (like nginx), and should look exactly the same as when running with `hacksoc_org run`. You probably want to use this or `hacksoc_org serve` at least once before you create a pull request.
 ```
-flask run
-venv/bin/flask freeze
+hacksoc_org run
 ```
 The HTTP root directory is `build/`.
 
 ### Starting a static server
 ```
-flask serve
-venv/bin/flask serve
+hacksoc_org serve
 ```
-Starts a local HTTP server from the `build/` directory. Equivalent to running `flask freeze` followed by `cd build/ && python3 -m http.server 5000`. If all goes well, you should always see the same as `flask run`.
+Starts a local HTTP server from the `build/` directory. Equivalent to running `hacksoc_org freeze` followed by `cd build/ && python3 -m http.server 5000`. If all goes well, you should always see the same as `hacksoc_org run`.
 
 ## Style guide
 To keep a consistent style, the following rules are used:
@@ -58,7 +57,7 @@ Otherwise, there is no line length limit, and you are encouraged to use "soft wr
 ### Code style
 Python files should roughly follow [PEP 8](https://www.python.org/dev/peps/pep-0008/), formatted with `black`. 
 
-Note that although constants should be in uppercase, PEP 8 does not strictly define what counts as a constant. Some variables in `hacksoc_org` are assigned once and should never be reassigned, but are mutated often and generally don't behave like constants (for example `app`, `blueprint` from Flask). When considering whether to name a variable as a constant, think about the following:
+Note that although constants should be in uppercase, PEP 8 does not strictly define what counts as a constant. Some variables in `hacksoc_org/` are assigned once and should never be reassigned, but are mutated often and generally don't behave like constants (for example `app`, `blueprint` from Flask). When considering whether to name a variable as a constant, think about the following:
  - Could the variable be replaced with its literal value?
  - Does the value of the variable ever change? (this includes mutations rather than just reassignments)
 ### Years of study
@@ -85,9 +84,12 @@ venv\Scripts\activate.bat
 # Windows users using PowerShell
 venv\Scripts\Activate.ps1
 
-pip install -r pip-requirements.txt
+pip install --upgrade pip
+pip install -e .
 ```
 
+On existing installations, if Python throws `ModuleNotFoundError`s, try running `pip install -e .` again as additional dependencies may have been added since your original install.
+
 See [Running](#running) next.
 
 ## License

docs/adding_features_python.md

@@ -44,10 +44,25 @@ Currently the website uses [`python-markdown2`][pymd2] and loads the following e
 Since the server READMEs use fenced code blocks and AGMs will often use many tables, any replacment library must support at least these. 
 
 ### Choice of Markdown libraries
-At time of writing, two Markdown libraries are available, creatively named [`markdown`](https://python-markdown.github.io/) and [`markdown2`][pymd2]. While they have a similar featureset, `markdown2` was chosen as its implementation of fenced code blocks uses a nicer syntax for code highlighting (similar to GFM, Discord, etc).
+Choosing a Markdown backend is not straightforward; implementations vary in their interpretation of the spec (Gruber's `markdown.pl` or the less ambiguous CommonMark standard) and their extra features (tables, code block highlighting, smart quotes). Currently [`markdown2`][pymd2] is used, although its non-conformance with CommonMark makes a replacement desireable.
+
+To help test between Markdown backends, non-default backends can be selected with the `--markdown` command-line option. Only [`cmark`](https://github.com/commonmark/cmark) is available (provided through the [`cmarkgfm`](https://github.com/theacodes/cmarkgfm) Python bindings).
+
+```
+# Equivalent; markdown2 is the default backend
+hacksoc_org run
+hacksoc_org run --markdown markdown2
+
+# Use cmark instead
+hacksoc_org run --markdown cmark
+
+# this works with all subcommands
+hacksoc_org freeze --markdown cmark
+```
+
 
 ## Serving Flask in production
-Some of Flask's extra power (handling POST requests, HTTP redirects) require it to be run in production (as opposed to generating HTML files and serving those from a static web server). Currently the [configuration](../.flaskenv) of Flask puts it into debug mode. This is extremely unsafe to run in production. Secondly, `flask run` or `app.run()` should not be used in production as it used Flask's built-in development server, which is not suitable for production use even when debug mode is disabled. Instead, consult [Flask's documentation](https://flask.palletsprojects.com/en/2.0.x/deploying/#self-hosted-options) on options for WSGI and CGI servers.
+Some of Flask's extra power (handling POST requests, HTTP redirects) require it to be run in production (as opposed to generating HTML files and serving those from a static web server). Currently the [configuration](../.flaskenv) of Flask puts it into debug mode. This is extremely unsafe to run in production. Secondly, `hacksoc_org run` or `app.run()` should not be used in production as it used Flask's built-in development server, which is not suitable for production use even when debug mode is disabled. Instead, consult [Flask's documentation](https://flask.palletsprojects.com/en/2.0.x/deploying/#self-hosted-options) on options for WSGI and CGI servers.
 
 
 [pymd2]: https://github.com/trentm/python-markdown2/wiki

docs/creating_modifying_simple_pages.md

@@ -54,7 +54,7 @@ The site-wide stylesheet will be sufficient for most pages, but if you find your
 See [Creating & modifying complex pages](creating_modifying_complex_pages.md).
 
 ## A new page doesn't appear when building the website!
-`frozen-flask` only generates pages which have a `url_for` link pointing to them. Usually pages on the website should have a link from `index.html`, or from another page which is linked from `index.html` (and so on,). If a new page is being written, it's possible that such a link doesn't yet exist. This only happens when running `flask build` or `flask serve`, as `flask run` will find any page, even if it's not been linked to. Make sure to test with `build` or `serve`.
+`frozen-flask` only generates pages which have a `url_for` link pointing to them. Usually pages on the website should have a link from `index.html`, or from another page which is linked from `index.html` (and so on,). If a new page is being written, it's possible that such a link doesn't yet exist. This only happens when running `hacksoc_org build` or `hacksoc_org serve`, as `hacksoc_org run` will find any page, even if it's not been linked to. Make sure to test with `build` or `serve`.
 
 ### Adding a link to the navbar
 The navbar is constructed in [`templates/nav.html.jinja2`](../templates/nav.html.jinja2). The first block, titled `{% set nav|from_yaml %}` contains the items of the navbar:

hacksoc_org/__init__.py

@@ -29,23 +29,5 @@
 # importing to trigger execution; decorated functions will add themselves to the app.
 
 # random global values are put in context.yaml and will be available to all templates
-with open(path.join(ROOT_DIR, "templates", "context.yaml"), encoding='utf-8') as fd:
+with open(path.join(ROOT_DIR, "templates", "context.yaml"), encoding="utf-8") as fd:
     app.jinja_env.globals.update(dict(yaml.safe_load(fd)))
-
-from hacksoc_org.freeze import freeze
-from hacksoc_org.serve import serve
-
-
-@app.cli.command("freeze")
-def do_freeze():
-    """Called on `flask freeze`. Renders the site to HTML in the build/ directory"""
-    freeze()
-
-
-@app.cli.command("serve")
-def static_serve():
-    """Called on `flask serve`. Freezes the site and starts a local server. Should be
-    near-indistinguishable from `flask run`."""
-    freeze()
-    print()
-    serve(path.join(ROOT_DIR, "build"))

hacksoc_org/cli.py

@@ -0,0 +1,83 @@
+from typing import Callable, Dict
+from hacksoc_org import app
+from hacksoc_org.consts import *
+from hacksoc_org.freeze import freeze
+from hacksoc_org.serve import serve
+
+import argparse
+
+subcommand_handlers: Dict[str, Callable] = {}
+
+
+def subcommand(command_str: str) -> Callable[[Callable], Callable]:
+    def inner(fn):
+        subcommand_handlers[command_str] = fn
+        return fn
+
+    return inner
+
+
+def main(args=None):
+    parser = argparse.ArgumentParser(
+        allow_abbrev=False,
+        epilog="""
+SUBCOMMANDS
+
+  run
+    Starts a local development server on https://localhost:5000/. Automatically
+    reloads when templates or Python code is changed. Recommended while 
+    developing pages or features.
+
+  freeze
+    Saves all URL routes to HTML files and copies `static/` to the `build/` 
+    directory. The resulting directory can be used with any standard HTTP
+    server (nginx, Apache, etc). In development, recommend using `serve` 
+    instead for convenience.
+
+  serve
+    Calls `freeze` then starts a local HTTP server from `build/` on 
+    https://localhost:5000/. Will not automatically rebuild the website on
+    content change, you will need to re-run `serve`. Recommended to use this at
+    least once to check that a) new content is part of the "frozen" site and b)
+    no errors occur in freezing the site.
+""".strip(),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+
+    parser.add_argument(
+        "action", choices=subcommand_handlers.keys(), help="Subcommand to run (see below)"
+    )
+
+    parser.add_argument(
+        "--markdown",
+        choices=["markdown2", "cmark"],
+        default="markdown2",
+        help="Markdown backend to use (default markdown2)",
+    )
+
+    args = parser.parse_args(args=args)
+
+    app.config[CFG_MARKDOWN_IMPL] = args.markdown
+
+    subcommand_handlers[args.action]()
+
+
+@subcommand("run")
+def do_run():
+    app.run()
+
+
+@subcommand("freeze")
+def do_freeze():
+    freeze()
+
+
+@subcommand("serve")
+def do_serve():
+    freeze()
+    print()
+    serve()
+
+
+if __name__ == "__main__":
+    main()

hacksoc_org/consts.py

@@ -0,0 +1 @@
+CFG_MARKDOWN_IMPL = "HACKSOC_ORG_MARKDOWN_IMPLEMENTATION"

hacksoc_org/filters.py

@@ -10,7 +10,7 @@
 
 import yaml
 
-from datetime import date, timedelta
+from datetime import date, timedelta, timezone
 import re
 import os
 from pprint import pformat
@@ -225,6 +225,16 @@ def format_date(d: date, year=True):
         return s
 
 
+@app.template_global()
+def build_datetime():
+    """Returns the current datetime (to seconds precision)
+
+    Returns:
+        datetime: current Python datetime object
+    """
+    return datetime.datetime.now(timezone.utc).astimezone().replace(microsecond=0)
+
+
 @app.template_filter()
 def from_iso_date(s: str):
     """Converts an ISO format datestring to a date object

hacksoc_org/markdown.py

@@ -3,18 +3,55 @@
     implementation used.
 """
 
-import markdown2
+import abc
+from hacksoc_org.consts import CFG_MARKDOWN_IMPL
+from hacksoc_org import app
 
-__markdowner__ = markdown2.Markdown(
-    extras=[
-        "fenced-code-blocks",
-        "cuddled-lists",
-        "tables",
-        # Markdown2 has a `metadata` Extra to allow frontmatter parsing
-        # this is not loaded and python-frontmatter is used instead to allow the markdown parser to
-        # be changed easily if required.
-    ]
-)
+
+class AbstractMarkdown(abc.ABC):
+    @abc.abstractmethod
+    def __init__(self) -> None:
+        super().__init__()
+
+    @abc.abstractmethod
+    def render_markdown(self, markdown_src: str) -> str:
+        pass
+
+
+class Markdown2MD(AbstractMarkdown):
+    def __init__(self) -> None:
+        import markdown2
+
+        self.md = markdown2.Markdown(
+            extras=[
+                "fenced-code-blocks",
+                "cuddled-lists",
+                "tables",
+                # Markdown2 has a `metadata` Extra to allow frontmatter parsing
+                # this is not loaded and python-frontmatter is used instead to allow the markdown parser to
+                # be changed easily if required.
+            ]
+        )
+
+    def render_markdown(self, markdown_src: str) -> str:
+        return self.md.convert(markdown_src)
+
+
+class CmarkgfmMD(AbstractMarkdown):
+    def __init__(self) -> None:
+        import cmarkgfm
+
+        self.cmarkgfm = cmarkgfm
+
+    def render_markdown(self, markdown_src: str) -> str:
+        return self.cmarkgfm.github_flavored_markdown_to_html(markdown_src)
+
+
+def get_markdown_cls():
+    return {"markdown2": Markdown2MD, "cmark": CmarkgfmMD}[app.config[CFG_MARKDOWN_IMPL]]
+
+
+_markdowner = None
 
 
 def render_markdown(markdown_src: str) -> str:
@@ -26,4 +63,9 @@ def render_markdown(markdown_src: str) -> str:
     Returns:
         str: HTML text
     """
-    return __markdowner__.convert(markdown_src)
+    global _markdowner
+
+    if _markdowner is None:
+        _markdowner = get_markdown_cls()()
+
+    return _markdowner.render_markdown(markdown_src)

pyproject.toml

@@ -1,3 +1,7 @@
 [tool.black]
 line-length = 100
 target-version = ['py37', 'py38', 'py39']
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"

setup.cfg

@@ -0,0 +1,25 @@
+[metadata]
+name = hacksoc_org
+version = 1.0.0
+
+[options]
+packages = find:
+install_requires =
+    flask
+    frozen-flask
+    pyyaml
+    pygments
+    python-frontmatter
+    pygit2
+    black
+    markdown2
+    cmarkgfm
+    # markdown-it-py
+    # mistletoe
+    # commonmark
+
+[options.entry_points]
+console_scripts = 
+    hacksoc_org = hacksoc_org.cli:main
+
+# [options.extras_require]

templates/git.html.jinja2

@@ -5,4 +5,6 @@
     Author: {{ commit.author.name }}
     Date: {{ (commit|git_date).isoformat() }}
     Message: {{ commit.message|trim}}
+
+    Built: {{ build_datetime().isoformat() }}
 -->