In a module best way to deal with async

[Norlock]

I'm working on the following project:
https://github.com/norlock/nvim-traveller-rs

But every time i try to acquire a lock it will crash it some error about thread boundary. Is there a way I can maybe create a new Lua context from inside the app? I would like to be able to generate one:

let lua_owned = mlua::Lua::from_state(lua);

I don't know if this possible to make in code, but it would be great if it is. If you have any good suggestion on how to deal with async safely I would like to know. For now I'm basically fixing my issues by having an interval run with try_lock. See:
https://github.com/Norlock/neo-api-rs/blob/bad84fba8c3114c00d26bb8927b5bc01fc12090f/src/fuzzy.rs#L491

Keep up the good work

[khvzak]

Is there a way I can maybe create a new Lua context from inside the app? I would like to be able to generate one

You can try Lua::init_from_ptr(raw_state) which will return previously initialized state.

If this will not help, could you explain the issue more detailed? Eg. what lock you have and how you use it.

[Norlock]

Yes but where do I get the raw_state? I will try to explain my issue (for a shared library):

I have made bindings for Neovim in the neo-api-rs repository, I'm currently building a fuzzy finder.

In the code there is a text changed event for Neovim that will be called whenever the text changes.
https://github.com/Norlock/neo-api-rs/blob/bad84fba8c3114c00d26bb8927b5bc01fc12090f/src/fuzzy.rs#L591

On that event I retrieve the text:

    let search_query = NeoApi::get_current_line(lua)?;

And then I want to spawn a task asynchronously that will run the 'fd' and 'ripgrep' processes from rust. It can take in some cases around 120~ ms. For performance reasons I don't want to block the whole Lua thread so I will not block that task (which means no more access to the lua object).

I store the results of those processes in a lazy static object from once_cell, where I can retrieve that data later.

static CONTAINER: Lazy<FuzzyContainer> = Lazy::new(|| FuzzyContainer {
    all_lines: RwLock::new(String::new()),
    cached_lines: RwLock::new(vec![]),
    fuzzy: RwLock::new(None),
    rt: tokio::runtime::Runtime::new().unwrap(),
    query_meta: RwLock::new(QueryMeta {
        update_results: false,
        last_search: "".to_string(),
    }),
    preview: RwLock::new(Vec::new()),
});

Then I have created an interval from the buildin neovim api (bindings to libUv https://neovim.io/doc/user/luvref.html) .
In that interval:
https://github.com/Norlock/neo-api-rs/blob/bad84fba8c3114c00d26bb8927b5bc01fc12090f/src/fuzzy.rs#L491

I have now the try_lock functionality which is better anyway for an interval. However if I would change that to await, and type quickly it can eventually create the: Can't leave the thread C boundary error. Or it would deadlock.

So basically try_lock is the correct way of doing it in an interval, but I would like to know is there a way if I leave the lua context (the part where I will run 'fd' and 'ripgrep'), I can get back into the Lua context? To update some stuff inside my plugin.

Then I can ditch the interval all together and run that specific code after the processes have ran. I hope this makes it a bit more clear.

If your last answer is still correct, where do i retrieve the lua_state (raw_state) pointer:

Lua::init_from_ptr(raw_state)

[khvzak]

You can make a Rust async call (function created using Lua::create_async_function) directly from neovim without blocking it. You need to have a (global) tokio runtime that you can enter inside async func:

static TOKIO: Lazy<runtime::Runtime> = Lazy::new(|| {
    runtime::Builder::new_current_thread()
        .enable_all()
        .build()
        .expect("cannot start tokio runtime")
});


let my_func = lua.create_async_function(lua, |lua, args| async move {
    let _guard = TOKIO.enter();
    <async code>
})?;

Then you can poll the function through coroutine checking for Lua::poll_pending() value.
When polling you can return control back to neovim / libuv scheduler.

It would be better and safer approach.

Take a look to #76 for some early stage ideas / examples.
Now I probably would try to incorporate some event listener to signal eventloop that execution is finished.

[Norlock]

Hey thanks for the help, the comments there look a bit more indirect / verbose than what I have now, currently I don't really have problems but I will dive a bit deeper into those concepts (lua coroutine etc) this weekend, and see if it can improve my code a bit.

Something else, I have some simple macros (I don't think they are production ready yet) to implement IntoLua and FromLua on rust structs, maybe they can be interesting for your project. I will slowly improve them a over time but in a lot of use cases they work already.

https://github.com/Norlock/neo-api-rs/blob/main/crates/macros/src/into_table.rs
https://github.com/Norlock/neo-api-rs/blob/main/crates/macros/src/from_table.rs