feat: add tokio (#102)

* docs: added preliminary `define_app!` advanced docs * refactor: made plugins system thread-safe All plugin functions must now implement `Send`. * refactor: made stores use async io * fix: added `io-util` flag to tokio Magical compilation errors occur otherwise for some reason. * refactor: made builder async This should increase performance for templates with a very large number of pages, each generated path is now built concurrently. * refactor: made exporting async * refactor: removed unnecessary leftover `Arc`s * refactor: made `FsTranslationsManager` async I *think* this makes everything async, the main thing is just the file operations. * chore(deps): pinned `actix-http` Actix seems to introduce a breaking change in the deps tree on CI, though this hasn't been reported anywhere else. If people start having issues, I'll release an urgent patch. * ci: add `sudo apt update` This should hopefully fix the consistent breakages of `apt` on CI.
framesurge · Dec 30, 2021 · 150fda8 · 150fda8
1 parent e445e56
commit 150fda8
Show file tree

Hide file tree

Showing 21 changed files with 579 additions and 319 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -30,6 +30,7 @@ jobs:
             - uses: actions/checkout@v2
             - run: cargo install bonnie
             - run: cargo install wasm-pack
+            - run: sudo apt update
             - run: sudo apt install firefox firefox-geckodriver
             - name: Run Firefox WebDriver
               run: geckodriver &
@@ -43,6 +44,7 @@ jobs:
             - uses: actions/checkout@v2
             - run: cargo install bonnie
             - run: cargo install wasm-pack
+            - run: sudo apt update
             - run: sudo apt install firefox firefox-geckodriver
             - name: Run Firefox WebDriver
               run: geckodriver &
@@ -56,6 +58,7 @@ jobs:
             - uses: actions/checkout@v2
             - run: cargo install bonnie
             - run: cargo install wasm-pack
+            - run: sudo apt update
             - run: sudo apt install firefox firefox-geckodriver
             - name: Run Firefox WebDriver
               run: geckodriver &
@@ -71,6 +74,7 @@ jobs:
             - uses: actions/checkout@v2
             - run: cargo install bonnie
             - run: cargo install wasm-pack
+            - run: sudo apt update
             - run: sudo apt install firefox firefox-geckodriver
             - name: Run Firefox WebDriver
               run: geckodriver &

diff --git a/docs/next/en-US/SUMMARY.md b/docs/next/en-US/SUMMARY.md
@@ -66,3 +66,4 @@
     -   [Initial Loads](/docs/advanced/initial-loads)
     -   [Subsequent Loads](/docs/advanced/subsequent-loads)
     -   [Routing](/docs/advanced/routing)
+-   [`define_app` in Detail](/docs/advanced/define_app)
diff --git a/docs/next/en-US/advanced/define_app.md b/docs/next/en-US/advanced/define_app.md
@@ -0,0 +1,58 @@
+# The `define_app` Macro in Detail
+
+Many users of Perseus will be perfectly content to leave the inner workings of their app to `define_app`, but some may be curious as to what this macro actually does, and that's what this section will explain.
+
+Before we begin on the details of this, it's important to understand one thing: **your code does not make your app**. The Perseus engine (the stuff in `.perseus/`) makes your app, and it _imports_ your code to create the specifics of your app, but the actual Wasm entrypoint in in `.perseus/src/lib.rs`. This architecture can be a little unintuitive at first, but it allows Perseus to abstract a huge amount of work behind the scenes, minimizing the amount of boilerplate code that you need to write.
+
+The issue that arises from this architecture is making your app interface with the engine, and that's where the `define_app!` macro comes in. It defines a number of functions that are then imported by the engine and called to get information about your app. The very first version of Perseus didn't even have a CLI, and all this interfacing had to be done manually! Today though, the process is _much_ easier.
+
+Before we get on to exactly what the macro defines, it's worth mentioning that using Perseus without the `define_app!` macro is possible, but is not recommended, even for experiences users. The main reasons are twofold: you will be writing _a lot_ of boilerplate code (e.g. you have to define a dummy translations manager even if you're not using i18n) and your app may break with new minor versions, because Perseus considers changes to the engine and the internals of the `define_app!` macro to be non-breaking. If you're still determined to persevere with going macro-less, you should regularly review the Perseus [`CHANGELOG`](https://github.com/arctic-hen7/perseus/blob/main/CHANGELOG.md) to make any changes that are necessary for minor versions.
+
+## Functions Defined
+
+Now that we've got all that out of the way, let's really dig into the weeds of this thing! The `define_app!` macro is defined in `packages/perseus/src/macros.rs` in the repository, and that should be your reference while trying to understand the inner workings of it.
+
+There are two versions of the macro, one that takes i18n options and one that doesn't. This is just syntactic sugar to make things more convenient for the user, and it doesn't affect anything more. Either way, here are the functions that are defined. (Note that a lot of these are defined with secondary internal macros.)
+
+- `get_plugins` -- this returns an instance of `perseus::Plugins`, either an empty one if no plugins are provided, or whatever the user provides
+- `APP_ROOT` (a `static` `&str`) -- this is the HTML `id` of the element to run Perseus in, which is `root` unless something else is provided by the user
+- `get_immutable_store` -- this returns an instance of `perseus::stores::ImmutableStore` with either `./dist` or the user-provided distribution directory as the root (whatever is provided here is relative to `.perseus/`)
+- `get_mutable_store` -- this returns an instance of `perseus::stores::FsMutableStore` with `./dist/mutable` as the root (relative to `.perseus`), or a user-given mutable store
+- `get_translations_manager` -- see below
+- `get_locales`
+  - With i18n -- this returns an instance of `perseus::internal::i18n::Locales`, literally constructed with the given default locale, the other locales, and with `using_i18n` set to `true`
+  - Without i18n -- this does the same as with i18n, but sets `using_i18n` to `false`, provides no `other` locales, and sets the default to `xx-XX` (the dummy locale expected throughout Perseus if the user isn't using i18n, anything else here if you're not using i18n will result in runtime errors!)
+- `get_static_aliases` -- this creates a `HashMap` of your static aliases, from URL to resource location
+- `get_templates_map` -- this creates a `HashMap` out of your templates, mapping the result of `template.get_path()` (what you provide to `Template::new()`) to the templates themselves (wrapped in `Rc`s to avoid literal lifetime hell)
+- `get_templates_map_atomic` -- exactly the same as `get_templates_map`, but uses `Arc`s instead of `Rc`s (needed for multithreading on the server)
+- `get_error_pages` -- this one's simple, it just returns the instance of `ErrorPages` that you provide to the macro
+
+Most of these are pretty straightforward, they're just very boilerplate-heavy, which is why Perseus does them for you! However, the translations manager is a little less straightforward, because it does different things if Perseus has been deployed to a server (in which case the `standalone` feature will be enabled on Perseus).
+
+### `get_translations_manager`
+
+This function is `async`, and it returns something that implements `perseus::internal::i18n::TranslationsManager`. There are four cases of what the user can provide to the macro, and they'll be gone through individually.
+
+#### No i18n
+
+We provide a `perseus::internal::i18n::DummyTranslationsManager`, which is designed for this exact purpose. Perseus always needs a translations manager, so this one provides an API interface and no actual functionality.
+
+#### A custom translations manager
+
+We just return whatever the user provided. This is technically two cases, because i18n could be either enabled or disabled (though why someone would provide a custom dummy translations manager is a bit of a mystery).
+
+#### I18n
+
+If no custom translations manager is provided, we create a `perseus::internal::i18n::FsTranslationsManager` for them, the `::new()` method for which takes three arguments: a directory to expect translation files in, a vector of the locales to cache, and the file extension of translation files (which will always be named as `[locale].[extension]`).
+
+The first argument is a little challenging, because it will usually be `../translations/` (relative to `.perseus/`), in the root directory of your project. However, if Perseus has been deployed as a standalone server binary, this directory will be in the same folder as the binary, so we use `./translations/` instead. In the macro, this is controlled by the `standalone` feature flag, but that isn't provided to your app, so the best thing to do here is up to you (you might depend on an environment variable that you remember to provide when you deploy).
+
+The second argument is probably a little weird to you. Caching translations? Well, they're actually the most requested things for the Perseus server, so the `FsTranslationsManager` caches locales when it's started by default. This uses more memory on the server, but makes requests faster in the longer-term (we do the same thing with your `index.html` file). By default, Perseus runs the `.get_all()` function on the instance of `Locales` generated by the macro's own `get_locales()` function to get all your locales, and then it tells the manager to cache everything. This is customizable in the macro by allowing the user to provide a custom instance of `FsTranslationsManager`.
+
+The final argument is blissfully simple, because it's defined internally in Perseus at `perseus::internal::i18n::TRANSLATOR_FILE_EXT`. The reason this isn't hardcoded is because it's dependent on the `Translator` being used, which is controlled by feature flags.
+
+Finally, the reason this whole `get_translations_manager()` function is `async` is because it has to `await` that `FsTranslationsManager::new()` call, because translations managers are fully `async` (in case they need to be working with DBs or the like).
+
+## Conclusion
+
+If, after all that, you still want to use Perseus without the `define_app!` macro, there's an example on its way! That said, it is _much_ easier to leave things to the macro, or you'll end up writing a huge amount of boilerplate. In fact, all this is just the tip of the iceberg, and there's more transformation that's done on all this in the engine!
diff --git a/examples/basic/.perseus/builder/Cargo.toml b/examples/basic/.perseus/builder/Cargo.toml
@@ -15,6 +15,7 @@ perseus-engine = { path = "../" }
 perseus = { path = "../../../../packages/perseus", features = [ "tinker-plugins", "server-side" ] }
 futures = "0.3"
 fs_extra = "1"
+tokio = { version = "1", features = [ "macros", "rt-multi-thread" ] }
 
 # We define a binary for building, serving, and doing both
 [[bin]]

diff --git a/examples/basic/.perseus/builder/src/bin/build.rs b/examples/basic/.perseus/builder/src/bin/build.rs
@@ -1,16 +1,16 @@
-use futures::executor::block_on;
 use perseus::{internal::build::build_app, PluginAction, SsrNode};
 use perseus_engine::app::{
     get_immutable_store, get_locales, get_mutable_store, get_plugins, get_templates_map,
     get_translations_manager,
 };
 
-fn main() {
-    let exit_code = real_main();
+#[tokio::main]
+async fn main() {
+    let exit_code = real_main().await;
     std::process::exit(exit_code)
 }
 
-fn real_main() -> i32 {
+async fn real_main() -> i32 {
     // We want to be working in the root of `.perseus/`
     std::env::set_current_dir("../").unwrap();
     let plugins = get_plugins::<SsrNode>();
@@ -23,21 +23,22 @@ fn real_main() -> i32 {
 
     let immutable_store = get_immutable_store(&plugins);
     let mutable_store = get_mutable_store();
-    let translations_manager = block_on(get_translations_manager());
+    // We can't proceed without a translations manager
+    let translations_manager = get_translations_manager().await;
     let locales = get_locales(&plugins);
 
     // Build the site for all the common locales (done in parallel)
     // All these parameters can be modified by `define_app!` and plugins, so there's no point in having a plugin opportunity here
     let templates_map = get_templates_map::<SsrNode>(&plugins);
-    let fut = build_app(
+    let res = build_app(
         &templates_map,
         &locales,
         (&immutable_store, &mutable_store),
         &translations_manager,
         // We use another binary to handle exporting
         false,
-    );
-    let res = block_on(fut);
+    )
+    .await;
     if let Err(err) = res {
         let err_msg = format!("Static generation failed: '{}'.", &err);
         plugins

diff --git a/examples/basic/.perseus/builder/src/bin/export.rs b/examples/basic/.perseus/builder/src/bin/export.rs
@@ -1,5 +1,4 @@
 use fs_extra::dir::{copy as copy_dir, CopyOptions};
-use futures::executor::block_on;
 use perseus::{
     internal::{build::build_app, export::export_app, get_path_prefix_server},
     PluginAction, SsrNode,
@@ -11,17 +10,42 @@ use perseus_engine::app::{
 use std::fs;
 use std::path::PathBuf;
 
-fn main() {
-    let exit_code = real_main();
+#[tokio::main]
+async fn main() {
+    let exit_code = real_main().await;
     std::process::exit(exit_code)
 }
 
-fn real_main() -> i32 {
+async fn real_main() -> i32 {
     // We want to be working in the root of `.perseus/`
     std::env::set_current_dir("../").unwrap();
 
     let plugins = get_plugins::<SsrNode>();
 
+    // Building and exporting must be sequential, but that can be done in parallel with static directory/alias copying
+    let exit_code = build_and_export().await;
+    if exit_code != 0 {
+        return exit_code;
+    }
+    // After that's done, we can do two copy operations in parallel at least
+    let exit_code_1 = tokio::task::spawn_blocking(copy_static_dir);
+    let exit_code_2 = tokio::task::spawn_blocking(copy_static_aliases);
+    // These errors come from any panics in the threads, which should be propagated up to a panic in the main thread in this case
+    exit_code_1.await.unwrap();
+    exit_code_2.await.unwrap();
+
+    plugins
+        .functional_actions
+        .export_actions
+        .after_successful_export
+        .run((), plugins.get_plugin_data());
+    println!("Static exporting successfully completed!");
+    0
+}
+
+async fn build_and_export() -> i32 {
+    let plugins = get_plugins::<SsrNode>();
+
     plugins
         .functional_actions
         .build_actions
@@ -31,20 +55,22 @@ fn real_main() -> i32 {
     let immutable_store = get_immutable_store(&plugins);
     // We don't need this in exporting, but the build process does
     let mutable_store = get_mutable_store();
-    let translations_manager = block_on(get_translations_manager());
+    let translations_manager = get_translations_manager().await;
     let locales = get_locales(&plugins);
 
     // Build the site for all the common locales (done in parallel), denying any non-exportable features
+    // We need to build and generate those artifacts before we can proceed on to exporting
     let templates_map = get_templates_map::<SsrNode>(&plugins);
-    let build_fut = build_app(
+    let build_res = build_app(
         &templates_map,
         &locales,
         (&immutable_store, &mutable_store),
         &translations_manager,
         // We use another binary to handle normal building
         true,
-    );
-    if let Err(err) = block_on(build_fut) {
+    )
+    .await;
+    if let Err(err) = build_res {
         let err_msg = format!("Static exporting failed: '{}'.", &err);
         plugins
             .functional_actions
@@ -61,7 +87,7 @@ fn real_main() -> i32 {
         .run((), plugins.get_plugin_data());
     // Turn the build artifacts into self-contained static files
     let app_root = get_app_root(&plugins);
-    let export_fut = export_app(
+    let export_res = export_app(
         &templates_map,
         // Perseus always uses one HTML file, and there's no point in letting a plugin change that
         "../index.html",
@@ -70,8 +96,9 @@ fn real_main() -> i32 {
         &immutable_store,
         &translations_manager,
         get_path_prefix_server(),
-    );
-    if let Err(err) = block_on(export_fut) {
+    )
+    .await;
+    if let Err(err) = export_res {
         let err_msg = format!("Static exporting failed: '{}'.", &err);
         plugins
             .functional_actions
@@ -82,24 +109,11 @@ fn real_main() -> i32 {
         return 1;
     }
 
-    // Copy the `static` directory into the export package if it exists
-    // If the user wants extra, they can use static aliases, plugins are unnecessary here
-    let static_dir = PathBuf::from("../static");
-    if static_dir.exists() {
-        if let Err(err) = copy_dir(&static_dir, "dist/exported/.perseus/", &CopyOptions::new()) {
-            let err_msg = format!(
-                "Static exporting failed: 'couldn't copy static directory: '{}''",
-                &err
-            );
-            plugins
-                .functional_actions
-                .export_actions
-                .after_failed_static_copy
-                .run(err.to_string(), plugins.get_plugin_data());
-            eprintln!("{}", err_msg);
-            return 1;
-        }
-    }
+    0
+}
+
+fn copy_static_dir() -> i32 {
+    let plugins = get_plugins::<SsrNode>();
     // Loop through any static aliases and copy them in too
     // Unlike with the server, these could override pages!
     // We'll copy from the alias to the path (it could be a directory or a file)
@@ -141,11 +155,29 @@ fn real_main() -> i32 {
         }
     }
 
-    plugins
-        .functional_actions
-        .export_actions
-        .after_successful_export
-        .run((), plugins.get_plugin_data());
-    println!("Static exporting successfully completed!");
+    0
+}
+
+fn copy_static_aliases() -> i32 {
+    let plugins = get_plugins::<SsrNode>();
+    // Copy the `static` directory into the export package if it exists
+    // If the user wants extra, they can use static aliases, plugins are unnecessary here
+    let static_dir = PathBuf::from("../static");
+    if static_dir.exists() {
+        if let Err(err) = copy_dir(&static_dir, "dist/exported/.perseus/", &CopyOptions::new()) {
+            let err_msg = format!(
+                "Static exporting failed: 'couldn't copy static directory: '{}''",
+                &err
+            );
+            plugins
+                .functional_actions
+                .export_actions
+                .after_failed_static_copy
+                .run(err.to_string(), plugins.get_plugin_data());
+            eprintln!("{}", err_msg);
+            return 1;
+        }
+    }
+
     0
 }
diff --git a/examples/basic/.perseus/builder/src/bin/tinker.rs b/examples/basic/.perseus/builder/src/bin/tinker.rs
@@ -12,6 +12,8 @@ fn real_main() -> i32 {
 
     let plugins = get_plugins::<SsrNode>();
     // Run all the tinker actions
+    // Note: this is deliberately synchronous, tinker actions that need a multithreaded async runtime should probably
+    // be making their own engines!
     plugins
         .functional_actions
         .tinker

diff --git a/packages/perseus-actix-web/Cargo.toml b/packages/perseus-actix-web/Cargo.toml
@@ -16,6 +16,7 @@ categories = ["wasm", "web-programming::http-server", "development-tools", "asyn
 [dependencies]
 perseus = { path = "../perseus", version = "0.3.0" }
 actix-web = "=4.0.0-beta.15"
+actix-http = "=3.0.0-beta.16" # Without this, Actix can introduce breaking changes in a dependency tree
 actix-files = "=0.6.0-beta.10"
 urlencoding = "2.1"
 serde = "1"

diff --git a/packages/perseus/Cargo.toml b/packages/perseus/Cargo.toml
@@ -34,6 +34,7 @@ cfg-if = "1"
 fluent-bundle = { version = "0.15", optional = true }
 unic-langid = { version = "0.9", optional = true }
 intl-memoizer = { version = "0.5", optional = true }
+tokio = { version = "1", features = [ "fs", "io-util" ] }
 
 [features]
 default = []