Struggling with dynamic library plugins

[64-Tesseract]

I'm making a game in which I'd like to be able to hotload mods. As the post name implies, I'm planning to do this with Bevy plugins, which will then do their own things like registering new items and whatnot.

I managed to create a dylib with a non-Bevy function, compiled it to a .so, and then managed to call a public function via libloading from another Rust program, but I just couldn't get it to work with Bevy.
The first thing I tried was manually passing a &mut App to a library compiled with Bevy, which worked fine until I tried adding a system to the app, at which point it segfaulted.

After a lot of digging, I finally came across bevy_dynamic_plugin - which is exactly what I need... But I couldn't find any documentation for it besides a couple footnotes on docs.rs.

Anyway, here's my non-working code for the library that I scraped together:

#![crate_type = "dylib"]
use bevy::prelude::*;

#[no_mangle]
pub fn _bevy_create_plugin () -> TestPlugin {
    println!("[dylib] This is called");
    return TestPlugin;  // This compiles but I'm not sure it's right
}

pub struct TestPlugin;

impl Plugin for TestPlugin {
    fn build (&self, app: &mut App) {
        println!("[dylib] This isn't called");
        app.add_startup_system(test_plugin);
    }
}

pub fn test_plugin () {
    println!("Test plugin running");
}

And the Bevy client loading said library:

use bevy::prelude::*;
use bevy_dynamic_plugin::DynamicPluginExt;
use libloading;

fn main () {
    let mut app = App::new();

    unsafe {
        println!("[main] Loading dynamic plugin");
        app.load_plugin("./libtestmodule.so");
    }

    println!("Running");
    app.run();
}

And the only things printed are:

[main] Loading dynamic plugin
[dylib] This is called
zsh: segmentation fault  cargo run

Setting RUST_BACKTRACE doesn't do anything, because it's not technically a panic, so that helps with debugging. Has anyone tried doing something like this before? What am I doing wrong?

Oh, and as a sidenote, let's say I have a couple components defined in the main app code; how would I go about querying or accessing them from a dynamic library? Should I put #[no_mangle] on the structs and just replicate the definitions in the library? Or should I export yet another library from the main app code with component definitions, and statically link that to dynamic libraries, which I'll then be loading back into the main app? It's already getting a bit convoluted when the entire project is 40 lines long...

[SkiFire13]

The documentation for dynamically_load_plugin (which App::load_plugin links to) requires _bevy_create_plugin to have the same signature as bevy_app::CreatePlugin, which returns a *mut dyn Plugin, not any random type implementing Plugin. The API doesn't make this clear, but it also expects such pointer to have been created by calling Box::into_raw.

That said, personally I would avoid hotreloading dynamic libraries for now. There are a ton of things that can go wrong when loading a dynamic library in Rust (especially one which uses non-repr(C) stuff from both ends) and even more when unloading.

[64-Tesseract]

Thanks for the tip, but I think dynamic libraries are my only choice, even if they're unstable - I don't see another way to natively support modding... Except using embedded Lua to make plugins, but that's an absolute last resort.
I'm not planning on "hot-reloading" specifically, I'd just like to be able to load an arbitrary number of libraries, which would be loaded from when the game starts until it's closed. Hopefully I won't run into any severe issues to do with dangling symbols with this approach.

I managed to have build called by changing the return value in the library to Box::inro_raw(Box::new(TestPlugin)), so now I see the [dylib] This isn't called get printed, but now the app panicks:

[main] Loading dynamic plugin
[dylib] This is called
[dylib] This isn't called
thread '<unnamed>' panicked at 'Requested resource bevy_ecs::schedule::schedule::Schedules does not exist in the `World`. 
                Did you forget to add it using `app.insert_resource` / `app.init_resource`? 
                Resources are also implicitly added via `app.add_event`,
                and can be added by plugins.', /home/tesseract/.cargo/registry/src/github.com-1ecc6299db9ec823/bevy_app-0.10.1/src/app.rs:388:40

So, I tried to use bevy_ecs::schedule::schedule::Schedules; and add it as a resource, but it complained that bevy_ecs::schedule::schedule is a private module. I also tried deriving Resource on my TestPlugin and mucking around with trying to load it, but nothing worked. Wouldn't some sort of scheduler be loaded in with the default plugins in the first place?

Thanks for your help!

[SkiFire13]

How are you compiling the game and the mod? AFAIK TypeId may not be stable between different compilations, see #32

Edit: just to be clear, my guess is that the TypeId of Schedules in the game and the mod differs and that's why the mod thinks it's not there, even though it is created by default.

[64-Tesseract]

I see... I'm compiling both with Cargo. The mod has crate_type = ["dylib"] and a path to the source file under the [lib] section, but besides that everything's standard. I also tried including the bevy_dynamic_plugin feature in the mod's Cargo.toml, but with no luck.

I'm speculating here, but if I were to compile Bevy as a library somehow, and then compile both the game and mod with that same library, would the TypieIds be the same...?

Edit: I just built both the mod and game with --features bevy/dynamic_linking, and they both generated their own libbevy_dylib.so files. And when I try running the game, it panicks with a new error:

[main] Loading dynamic plugin
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Library(DlOpen { desc: "libbevy_dylib-a933ae31296170a9.so: cannot open shared object file: No such file or directory" })', /home/tesseract/.cargo/registry/src/github.com-1ecc6299db9ec823/bevy_dynamic_plugin-0.10.1/src/loader.rs:44:59
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

But this persists even when I copy either library to the working directory with that exact name. I might be going off on a tangent with this though, I've got no idea what that feature's intended use case is...

[SkiFire13]

I see... I'm compiling both with Cargo. The mod has crate_type = ["dylib"] and a path to the source file under the [lib] section, but besides that everything's standard. I also tried including the bevy_dynamic_plugin feature in the mod's Cargo.toml, but with no luck.

And are the game and the mod in the same cargo project/workspace? If not, do they share the same version and features for bevy (but it would be better if that was true for all the other dependencies too)

I'm speculating here, but if I were to compile Bevy as a library somehow, and then compile both the game and mod with that same library, would the TypieIds be the same...?

Probably, but I'm not sure how you would do that, except from them being in the same cargo project/workspace (in that case Cargo will automatically share the dependency between them).

[64-Tesseract]

And are the game and the mod in the same cargo project/workspace? If not, do they share the same version and features for bevy (but it would be better if that was true for all the other dependencies too)

I had them in separate project folders, but with the same Rustup/Cargo home. Compiling with the library and main app in the same workspace solved the issue, so it confirms it's the TypeId changing.

The problem now is that each mod compilation won't be distributable as a neat single-file library, so each user will need to compile mods themselves... But all that can be hidden away behind the scenes, at least until there's a development with #32 🤷
Thanks for your help!

[CyanBlob]

Just curious; would moving all struct definitions into a separate library that gets shared between the game and the mod resolve the TypeId issue? It would require exposing all the struct definitions but that may not be a bad thing, especially since you could keep "private" structs in the main project anyways

[SkiFire13]

That still doesn't guarantee that the TypeIds will be the same. There a bunch of things that need to be the same for TypeId to not change (this is empirically, nothing guaranteed) and AFAIK currently the only sane way to keep them the same is to compile game and mod in the same workspace.