rlua is looking for maintainers!

[kyren]

Copied from a reddit discussion here:

I don't actually use Lua for anything currently, and I also don't have much interest in returning to Lua in the future. As such, my Lua related crates rlua (and also luster, but that's much less important) are not receiving much attention currently.

That's a real shame, and I'd like to help at least rlua not become abandoned. It needs some work done to it, none of which is terribly complex but I just don't have the time or proper motivation to be the one to do it. Also, I don't actually even think I should be the one to do it, as it's very hard to make good APIs when you don't use your own API.

I'd be happy to guide people through my ideas for rlua and help especially with reviewing PRs for unsoundness, but there is quite a lot of stuff that needs doing in the pretty near term. The internals of rlua are pretty hairy, which is why I don't expect anybody who wants to take up the mantle to go it completely alone, I'll happily be around to help with the hard parts. What it needs though is better leadership and organization skills than I have or at least have to spare currently.

[kyren]

(Also copied from the same reddit discussion)

For anybody who's interested in working on this, here's a quick wishlist of things that I think rlua needs in the near future that I would work on if I had the time:

1. rlua::Lua needs a compile-time switch to either require Send or to not require Send. Not-requiring Send seems like the obvious choice, but it makes it very hard to integrate into game engines that use threaded systems or perhaps into things like web servers.

2. rlua needs to either not use the lua_State "extradata" region or have a way to turn it off so it doesn't conflict with things trying to use it to create stand-alone Lua modules. The "extradata" region is unfortunately the fastest way to store arbitrary data alongside the main Lua state, which is why this has not been addressed yet.

3. rlua needs the ability to turn off the pcall / xpcall wrappers or any other provided function wrappers to be used in module position. They exist for a reason, but in module position it's very rude to muck with the global Lua state like this.

4. rlua needs to at least support Lua 5.4 when it is released, and ideally it would also support Lua 5.1 (and LuaJIT) and 5.2. This is potentially hard to make truly sound, but rlua has at least already done the hard work to protect against memory allocation and GC errors, so this is not completely infeasible.

5. rlua needs to give up on trying to patch out the UB in PUC-Rio Lua itself, and simply admit this fact by making the Lua constructor unsafe. This is COMPLETELY different than giving up on bindings soundness, I still consider bindings soundness to be extremely important, but this would give up on things like trying to protect rlua usage from things like the debug module or its many other ways to cause UB like directly loading dlls or even just through loading bytecode. Maybe some form of truly sandboxed Lua could be provided as a safe interface, but I don't think it should be advertised as the default because it removes too much of Lua itself.

6. rlua needs some ability to use async functions as coroutines and vice versa, but it also needs a way to in general more ergonomically return via lua_yield, possibly anywhere that a callback returns.

7. rlua contains within its guts a dirty lie, and this lie permeates the entire crate and makes proving the soundness of rlua very very hard. The signature of this type alias SHOULD be:

pub(crate) type Callback<'a> =
    Box<for<'lua> dyn Fn(Context<'lua>, MultiValue<'lua>) -> Result<MultiValue<'lua>> + 'a>;

Making this change would immediately remove much of the very delicate and honestly sort of sketchy logic in the callback creation process and the entirety of the "scope" system and make it more obviously sound. However, since I screwed this up when I initially created rlua and wrote the type signature wrong, I "accidentally" found a way to make it so that callback creation did not require macros. This has prevented me from fixing this because doing so almost certainly makes the API even less convenient by requiring macro wrappers around callbacks. There might be a way to fix this but I've tried very hard and haven't found any solution, it almost certainly requires GATs to fix properly. Edit: This is the biggest blocker in my mind to a 1.0 level API, but maybe being able to generally yield in callbacks also deserves to be a 1.0 blocker.

[rustysec]

To continue the reddit discussion, it seems like there are a few of us that would be interested in taking on maintainer-ship by committee if that is an acceptable solution.

[ahmedcharles]

I'm a fan of both Rust and Lua, though I don't have the time to commit to rlua. That said, I'll take a look around when I'm bored and thanks for the great bindings.

[erlend-sh]

🗣️ to @Grokmoo of https://github.com/Grokmoo/sulis

And if someone knows how to get in touch with @khvzak of mlua that'd be great! (edit: found their email on crates, sent a ping.)

[khvzak]

Thanks @kyren for looking into my work on the rlua fork.
Initially my primary goal was to bring Rust to Lua 5.3 and LuaJIT (the versions I work with) by providing a way to write high-level Lua modules in Rust. That's way I chose rlua 0.15 as a starting point, since it has more suitable API for this (I believe).
Therefore, standalone mode to bring Lua to Rust is also important and in focus!

My long-term goals are partially the same:

• Provide Send / non-Send API or compile time switch.
• Add Lua 5.4 support (the release candidate sources are already in the lua-src crate and (probably) MoonJIT (the project looks promising to be a luajit succeeder).
• Out of the box cross compilation support to all major non x86 platforms (ARM/MIPS/etc) with CI (using qemu?)
• Much more tests
• More "batteries", like serde support (by new crates).
• Improve documentation and create more examples

Regarding your comment in the reddit discussion:

• The first mlua async implementation was built using lua_yieldk call. You can see this code. But later I postponed this idea as it does not work with Lua 5.1 (+ LuaJIT) and switched to more universal solution. Also, I'm thinking about providing support to define a custom yield function, as recently found that necessity and currently overriding the global coroutine.yield.
• I'm going to return (optional?) custom allocator to the Lua constructor to provide more resilience/soundness.

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I'd be happy to get in touch and discuss about unsoundness problem that mlua have or maybe future of both projects.

[kyren]

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I think this might be the most important point to discuss. I think there is a more restricted form of soundness that is actually what people want that is still very useful to have, and I think the main Rust <-> Lua bindings system, whatever it is, should work very hard to maintain it.

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe. That by itself is somewhat of a useless property if as I'm proposing rlua::Lua::new becomes an unsafe function, by that definition anything is at that point sound because you must write unsafe. What I really mean by "soundness" here is more subtle, that nothing you can do directly with the bindings system can cause UB. In this definition, things like the debug table and loading C libraries inside Lua are "sound" because they are simply out of scope. This sounds like a cop-out but I really don't think it is, for example is opening a file and writing to /proc/self/mem unsound? Certainly not, despite the fact that it causes data races, it's just not within the scope for our definition, and the existence of /proc/self/mem doesn't make safety guarantees of rust any less useful.

So, for this library, "soundness" would mean bindings soundness, but as a way to make sure that people understand this, Lua::new() should just simply be unsafe. If at some point there is a good enough sanctioned sandboxing technique or a different implementation of Lua, then maybe a different method could be marked as safe, but as it stands I think this is similar to problems like what the memmap crate has. Whether or not a memory map is "safe" in rust is not just a function of the API, it is a function of the entire system as a whole, entirely different processes can cause data races with certain uses of memory maps, so the memmap crate just acknowledges this with an unsafe constructor and then proceeds to cause no extra harm.

This is (AFAIK currently) the state of the rlua crate, that it provides "bindings soundness". This is a really subtle point ofc and kind of hard to talk about, so maybe I just misunderstood what your project README said, perhaps mlua is already "bindings sound"? I was also worried because the 0.16 and 0.17 releases of rlua definitely also contain soundness fixes, however maybe the 0.16 API change is not a problem since you have also removed Lua::scope, but removing the scope system is somewhat unfortunate.

What I was worried about was there being an obviously best Lua bindings crate that does not attempt bindings soundness at all. You could make the case that since Lua is so unsafe, that there's no point anymore, but I definitely don't think that's true. I think that the kinds of things you can do in Lua that are unsafe are somewhat obvious: load C libraries, the debug table, bugs in the interpreter etc, but the kinds of things you can do that are unsafe in the Lua C API are anything but obvious, because the C API is so freakishly hard to safely use.

It's possible that your goal with mlua is already bindings soundness, and if so that's great! If that's true I don't see why there needs to be a second Lua bindings system really, both crates would have more or less the same goals in mind. My next question would be how we can work together instead of splitting effort, especially since you seem to be much more actively interested in this than I am currently.

[ahmedcharles]

There's some interesting perspectives: https://docs.rs/dtolnay/0.0.9/dtolnay/macro._03__soundness_bugs.html

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe.

I don't think that's a useful definition of soundness. The link above doesn't define it precisely, but my view is that the most useful definition of soundness is that UB is impossible when all code is either safe or uses unsafe in a way that adheres to the relevant documentation.

One of the examples for making frob sound is marking it as unsafe with documentation about it's precondition.

[kyren]

I don't think that's a useful definition of soundness. The link above doesn't define it precisely, but my view is that the most useful definition of soundness is that UB is impossible when all code is either safe or uses unsafe in a way that adheres to the relevant documentation.

Yeah, that's a better definition obviously but in this case that's still not it, becasue there is no contract with Lua::new you can fulfill because every possible method would have the same contract: don't cause Lua code to execute that can cause unsafe. Marking every method as unsafe is not helpful though, and you can limit the rules you have to follow to some categories of Lua code not to run rather than anything in the bindings layer like lifetime unsoundness. I'm saying we should allow the first but disallow the second, and that is the current situation with rlua itself.

You're correct in saying that simply writing unsafe at all does not give someone carte blanche to cause UB, that's not really what I meant to say, I was just skipping a lot of the detail. If the contract of Lua::new is "you cannot misuse the API", then by a certain definition it's "sound" but it's not useful, an example of this is real life could be "by calling Lua::new you agree not to overflow the Lua stack" or "by calling Lua::new you agree not to to throw errors in a __gc metamethod" or "by calling Lua::new you agree that a certain part of API is in unsafe if you use LuaJIT".

I promise I already understand what soundness is. Obviously an API that has an unsafe function with a clear cut guarantee is not sound if you call the unsafe function correctly and it causes UB, it's just that with Lua specifically the strawman I was describing is that the contract for Lua::new is "it's Lua, anything goes". The context here is this:

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I'm just arguing for "bindings soundness" even though the contract must technically be that any method at all can cause UB, I just want to limit how that UB may occur.

Also just to be clear I'm not saying @khvzak is suggesting this strawman definition either, I'm just trying to make sure our definitions are aligned. I think that even though someone can pass a lua_State and muck with the stack themselves, we should still aggressively strive for completely safe Lua C API usage, and advertise that fact.

[ahmedcharles]

I think the mmap analogy works the best. Creating an mmap or lua_State is unsafe and has a huge set of caveats, some easy to understand and some not. Documenting all of the caveats may not even be possible, in either case. But either way, the rest of the api can be sound in the face of a correctly created mmap or lua_State and that's what matters.

[jugglerchris]

FWIW (I've separately wrapped Lua in a Rust crate but haven't got around to publishing it), my aims for safety/soundness sounds in line with to the above:

• Bindings can be written in safe Rust
• Ordinary Lua code can't cause Rust UB but I consider loading C libraries, using the debug library, and interpreter bugs (though I'd want to fix any I knew about) as out of scope.

My use cases are Lua as configuration - the user isn't malicious but may make mistakes and may be handling malicious data (e.g. from the web or e-mails).

I think rlua is doing a better job than my own bindings and is likely to be better maintained and has had more thought put into safety, so I may look into switching over. I'm interested in being involved but do not have time to be a full maintainer.

[khvzak]

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe.
...
What I really mean by "soundness" here is more subtle, that nothing you can do directly with the bindings system can cause UB. In this definition, things like the debug table and loading C libraries inside Lua are "sound" because they are simply out of scope.

Am I correctly understanding, that the following 100% rust safe code using rlua, which 1) modify RO variable 2) causes UB; is sound?

fn main() -> LuaResult<()> {
    let lua = Lua::new();
    lua.context(|ctx| {
        let memcpy: LuaFunction = ctx.load(r#"
            local ffi = require("ffi")
            ffi.cdef[[
            void * memcpy(void *, const void *, int);
            ]]
            return ffi.C.memcpy
        "#).eval()?;

        let a: i32 = 0;
        let b: i32 = 255;

        // Set "a" to "b"
        memcpy.call((&a as &i32 as *const i32 as i64, &b as &i32 as *const i32 as i64, 4))?;
        println!("{}", a); // Prints 255

        memcpy.call((&a as &i32 as *const i32 as i64, LuaNil, 4))?; // UB; Segfault

        Ok(())
    })
}

ffi is installed from this repo.

I was also worried because the 0.16 and 0.17 releases of rlua definitely also contain soundness fixes, however maybe the 0.16 API change is not a problem since you have also removed Lua::scope, but removing the scope system is somewhat unfortunate.

Could you please provide an example(s) of unsoundness in rlua 0.15, for better understanding ? The Scope module was removed in 0.3 since my higher priority was releasing async/await support which does not play nice with Scope system. I'm planning to return Scope support in the next release, but without create_nonstatic_userdata which is pain.

It's possible that your goal with mlua is already bindings soundness, and if so that's great!

I definitely not against bindings soundness, if it's possible and does not affect API usability. Soundness is a great advantage!
However, the example above shows the Rust "safety" rule violation even in rlua
From https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html

The need for all of this separation boils down a single fundamental property of Safe Rust, the soundness property:
No matter what, Safe Rust can't cause Undefined Behavior.

[kyren]

Am I correctly understanding, that the following 100% rust safe code using rlua, which 1) modify RO variable 2) causes UB; is sound?

I'm suggesting that Lua::new be marked as unsafe, but yes it is "bindings sound". The UB comes from Lua violating one of the rules I laid out, that it loads a dll to run C (asm?) code.

However, the example above shows the Rust "safety" rule violation even in rlua
From https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html

That's not bindings unsoundness, that's the trap I don't want to fall into, not to let perfect be the enemy of good. Since nothing in Lua can be perfectly sound without a complete sandboxing solution which is just not how people use Lua, it's not useful to mark every method that can trigger Lua code to run (which is nearly every method) as unsafe. Just mark Lua::new as unsafe and explain the situation.

Could you please provide an example(s) of unsoundness in rlua 0.15, for better understanding ? The Scope module was removed in 0.3 since my higher priority was releasing async/await support which does not play nice with Scope system. I'm planning to return Scope support in the next release, but without create_nonstatic_userdata which is pain.

Yeah the scope system was maybe the main one, but also there were other bugfixes, and I re-designed some of the lifetimes at some point after better understanding the mess I had gotten myself into with the dirty lie. If you didn't include the code that removes the need for the gc_guard hack, then there's probably soundness issues there too. Sorry I can't give you a more concrete answer I don't know when you forked and I don't have the energy right now to go digging that closely, 0.15.3 was released Oct 13, 2018.

Like I said it's possible that mlua already intends to be "bindings sound" and I just misunderstood the README changes.

[Grokmoo]

I appreciate the shout out @erlend-sh. I will make more of an effort to understand the internals of rlua and see if that takes me to where I could make a useful contribution.

I'm also available to help out with more administrative or devops type stuff if the need arises.

[khvzak]

I'm suggesting that Lua::new be marked as unsafe, but yes it is "bindings sound". The UB comes from Lua violating one of the rules I laid out, that it loads a dll to run C (asm?) code.

Maybe to mark Chunk::exec() as unsafe would be enough, but unsafe Lua::new seems better. I doubt that is possible to disable functionality like package.loadlib in Lua :(

Yeah the scope system was maybe the main one, but also there were other bugfixes, and I re-designed some of the lifetimes at some point after better understanding the mess I had gotten myself into with the dirty lie. If you didn't include the code that removes the need for the gc_guard hack, then there's probably soundness issues there too.

After forking rlua 0.15 I spent many time reading the code and changing it. I didn't include gc_guard and other (now legacy) code. The Scope subsystem in mlua 0.2 came with new lifetimes. I also refactored the internals (and found couple of bugs in rlua, but forgot the details), changed error handling, added metatables cache instead of static global u8 and did some other changes.

Personally I don't use Scope subsystem and doubt a bit about use cases. All the tables/strings/objects/etc created inside executed scoped functions will be alive. Only scope.create_(function|userdata) will be destroyed with tails remaining in Lua. Also it was quite hard for me to understand Scope lifetime design and why they were chosen as they are.
However I finished writing async_scope() variant of it to allow execution of scoped asynchronous code to return Scope in next releases.

I also doubt a bit the appropriateness of adding Context and radically changes in the API.
As I understood the key point was to catch at compile time cross-usage of multiple Lua states? (where object in Lua stateA pushed to a stateB). This check is performed at runtime in mlua (as in 0.15).
The Context API seems making modules creation more harder.

Like I said it's possible that mlua already intends to be "bindings sound" and I just misunderstood the README changes.

I hope so :) I believe there is no unsoundness at the moment (with the exception of UB caused by loading C code). I need to review the README.

[kyren]

(and found coupe of bugs in rlua, but forgot the details)

It would be great if you could share the details there. We've probably both fixed different bugs since then :(

Personally I don't use Scope subsystem and doubt a bit about use cases

To safely use non-Send and non-'static data from Lua

Only scope.create_(function|userdata) will be destroyed with tails remaining in Lua.

Those are the only things that can (directly) store non-Lua data.

Also it was quite hard for me to understand Scope lifetime design and why they were chosen as they are.

because of the bad decision in the lifetimes of the callbacks, ultimately.

As I understood the key point was to catch at compile time cross-usage of multiple Lua states?

And to fix the soundness of the scope system.

I think at this point rlua is just hard forked, oh well.

[kyren]

So I looked into it some more, mlua is unsound in the same ways rlua was unsound pre-0.16, even with the scope system removed.

The following segfaults on mlua master (in release):

use std::cell::RefCell;

use mlua::{Lua, Table};

fn main() {
    thread_local! {
        static BAD_TIME: RefCell<Option<Table<'static>>> = RefCell::new(None);
    }

    let lua = Lua::new();

    lua.create_function(|_, table: Table| {
        BAD_TIME.with(|bt| {
            *bt.borrow_mut() = Some(table);
        });
        Ok(())
    }).unwrap()
    .call::<_, ()>(lua.create_table().unwrap())
    .unwrap();

    // In debug, this will panic with a reference leak before getting to the next part but
    // it segfaults anyway.
    drop(lua);

    BAD_TIME.with(|bt| {
        println!(
            "you're gonna have a bad time: {}",
            bt.borrow().as_ref().unwrap().len().unwrap()
        );
    });
}

@khvzak You should really read #97 and understand what I was saying there, it describes the "dirty lie" I keep talking about. The root issue of this, the difficulty with proving the soundness of the scope system and arguably the need to create the context system all come from the fact that I chose the "wrong" lifetimes for callbacks way back when. Fixing this bug either requires making all callbacks into macros OR the context system, and since the context system came with the benefit of not requiring ownership checks on references I went that route. If I could pick one again it would be to have the correct lifetimes on callbacks and to just require macros to create callback functions, or to find SOME other way.

I'm sorry I left you with this mess of a problem, I don't really like any part of it. I wish rust had GATs and it could be fixed properly, I think actually if you're going to remove the scope system and context system that the next step is to make that change and just require macro wrappers on callbacks, and then restore some kind of lifetime sanity. It's unfortunate that rust can't express this right now, but I think the correct solution is to just live with it and require macros until such a time where rust can express it.

I also wish we could have worked together, I would have discussed alternatives to the context system if I had known it was problematic, though I think every solution I found was problematic in one way or another. I don't blame you for forking rlua, I mean I've done exactly the same thing... it's hard sometimes if you want to make a lot of changes to a project that might be controversial to work to find common solutions, I mean I'm definitely guilty of it myself.

I dunno what to do now. mlua has a lot of nice new features and it's clear that you've done a lot of really good work on it. It would be great to leave people with a single, sound bindings system to use rather than the current situation of two bindings systems with different trade-offs and features and levels of safety.

[erlend-sh]

@khvzak could you say more about your main use case for Lua? I see you're also doing stuff with HAProxy; are you using Lua primarily for web & server stuff?

It is quite likely that rlua and mlua have just diverged too far at this point to make a merger feasible. But if that's not actually the case, and your goals are in fact well enough aligned that a collaborative effort could be possible, then an org-managed Lua library would clearly be the best of both worlds:

No userbase fragmentation (i.e. more prospective contributors), active maintenance by @khvzak as long as he has itch to scratch, backed up by @kyren in an advisory capacity drawing from her several years of production experience with Lua, plus the existing rlua community helping to keep things tidy.

[khvzak]

You should really read #97 and understand what I was saying there, it describes the "dirty lie" I keep talking about.

Thanks @kyren ! Seems I made a big mistake and didn't went through the closed issues and history of rlua :( I now understand the depth of the pre-0.16 issues and feeling quite guilty for starting to make changes without a sufficient understanding of the rlua complexity and motivation behind the 0.16 changes.

If I could pick one again it would be to have the correct lifetimes on callbacks and to just require macros to create callback functions, or to find SOME other way.

In the old reddit discussion you're saying that "This is not a practical problem for me anymore, because I have since moved onto using Rc / Weak". Just curious, what's happened with this solution? There is definitely a high chance of memleaks while using Rc.

It would be great to leave people with a single, sound bindings system to use rather than the current situation of two bindings systems with different trade-offs and features and levels of safety.

It is quite likely that rlua and mlua have just diverged too far at this point to make a merger feasible. But if that's not actually the case, and your goals are in fact well enough aligned that a collaborative effort could be possible, then an org-managed Lua library would clearly be the best of both worlds.

I definitely like the idea of creating an org-managed library. Divergence between the libraries should not be a big issue I hope.

For now I think I need to make step back and take some time to properly think about how to get out of this trap. This kind of problem is quite new to me. I'm going to play with multiple approaches to find a solution. Personally, I tend to move to use macros, as benefit this would allow to pass function arguments as an unpacked list rather than tuple.
Very likely I will introduce breaking changes to the API and it will be a new epoch of rlua/mlua development.

could you say more about your main use case for Lua? I see you're also doing stuff with HAProxy; are you using Lua primarily for web & server stuff?

You're right, my use cases are quite simple and I use Lua (Rust) primarily for server stuff. Single Lua state, mostly module position. Running code inside haproxy (Lua 5.3) and openresty (LuaJIT) platforms.
My area of activity has nothing to do with game development but I believe that bringing Rust to Lua (or visa versa) would open the door to quality and high-performance libraries for Lua.
This is a big pain to write libraries in C supporting the compatible Lua 5.1/5.2/5.3 API.

[kyren]

Thanks @kyren ! Seems I made a big mistake and didn't went through the closed issues and history of rlua :( I now understand the depth of the pre-0.16 issues and feeling quite guilty for starting to make changes without a sufficient understanding of the rlua complexity and motivation behind the 0.16 changes.

That's okay, I'm glad we're on the same page now!

In the old reddit discussion you're saying that "This is not a practical problem for me anymore, because I have since moved onto using Rc / Weak". Just curious, what's happened with this solution? There is definitely a high chance of memleaks while using Rc.

I don't even remember, it's been too long, but that solution didn't go anywhere.

I definitely like the idea of creating an org-managed library. Divergence between the libraries should not be a big issue I hope.

Yeah I agree here, it seems like the two libraries should merge and be managed by a separate organization.

For now I think I need to make step back and take some time to properly think about how to get out of this trap. This kind of problem is quite new to me. I'm going to play with multiple approaches to find a solution. Personally, I tend to move to use macros, as benefit this would allow to pass function arguments as an unpacked list rather than tuple.

Yeah I think I agree, any solution that keeps the incorrect lifetimes is fighting the the borrow checker in deeply wrong ways and is really unsustainable. A macro solution is inconvenient but is correct and forwards compatible with a better solution involving GATs, also you're correct that you also don't have to pack all your arguments in tuples anymore.

My area of activity has nothing to do with game development but I believe that bringing Rust to Lua (or visa versa) would open the door to quality and high-performance libraries for Lua.
This is a big pain to write libraries in C supporting the compatible Lua 5.1/5.2/5.3 API.

Just so you're aware of the use case, if you use Lua for game development you need both a Lua object that is Send and also the scope system, that's more or less why they exist. The Send requirement allows you to write "systems" in ECS terms as a Lua script in a multi-threaded environment, and the "scope" system allows you to borrow resources / components without breaking your brain with rental and having to do the same tricks the scope system does anyway.

[NotMoni]

Hey, mate I can help. Feel free to contact me. 🦄

[jugglerchris]

Hi @kyren,
I'm thinking of raising issues for the list of items in #172 (comment) above to make them easier to refer to and discuss (or for people to pick up and work on). Any objections?

Also, when talking about supporting Lua 5.4 and luaJIt/5.2: were you thinking of Cargo features to select one at a time or something else?

[kyren]

Also, when talking about supporting Lua 5.4 and luaJIt/5.2: were you thinking of Cargo features to select one at a time or something else?

I think using cargo features for this is basically fundamentally wrong, since cargo features are not supposed to be mutually incompatible. HOWEVER, I think everyone in similar situations does exactly this anyway, and I'm already guilty of it too with the system-lua and builtin-lua features, so I don't really know of another alternative. The fact that this is not ideal is part of the reason I hadn't worked on it yet.

[jugglerchris]

I think using cargo features for this is basically fundamentally wrong, since cargo features are not supposed to be mutually incompatible. HOWEVER, I think everyone in similar situations does exactly this anyway, and I'm already guilty of it too with the system-lua and builtin-lua features, so I don't really know of another alternative. The fact that this is not ideal is part of the reason I hadn't worked on it yet.

Darn, I was hoping you'd thought of a better way. :-) You'd want something a bit like Cargo features but which are more like an enum than a bool ("lua=lua5.3"). I hadn't really thought of anything better than having an ugly bunch of nested #[cfg(..)] to assert that exactly one is enabled at compile time, but it doesn't scale well either.

[kyren]

There are really "bold" solutions like using environment variables at build time or something. I guess maybe that's not a completely ridiculous idea since other -sys crates do use environment variables to find specific system libraries. This would be the first time I've seen it used to determine actual library features and API though, but I GUESS you could make the argument that it's not wrong?

Edit: If you actually think that's a workable idea what I'd think you'd do is use environment variables for all of it, point to a lua library with an environment variable and either auto-detect the lua version or specify a hard-coded lua version also in an environment variable, and which lua version you link to determines specific parts of the API. Also this would allow you to link to any arbitrary API compatible implementation of Lua.

[kyren]

I'm thinking of raising issues for the list of items in #172 (comment) above to make them easier to refer to and discuss (or for people to pick up and work on). Any objections?

Oh I forgot to say, this is a wonderful idea and I have no objections.

[jugglerchris]

Environment variables don't sound good - I'm not sure how that would work with dependencies (especially nested dependencies). What happens if you have two dependencies which both depend on rlua (say because they add new types/functions)? But I admit I haven't looked at how -sys crates use environment variables - I'll look into that before making any real judgements.

And I'll raise those issues when I get a few minutes.

[jugglerchris]

I have created those issues. Let me know (or just fix) if I've misrepresented anything or messed up the copy and paste!

[jugglerchris]

There hasn't been much activity here recently, but I've started thinking about making some progress (if slowly) in rlua.
I plan to:

1. Go through the open issues and PRs, and decide what I think should be done with them. The main options are:
   • Close the issue if it looks no longer relevant, or a question, or something that isn't likely to be done for various reasons
   • Add the issue to a Version 1.0 milestone, if I think it's something that really ought to be done before declaring stability.
   • Leave it for later, e.g. for features that look desirable but which could be added backwards compatibly after 1.0.
2. Gradually work through the Version 1.0 issues (and welcome other contributions. :-) )

I'm definitely not going to have a deadline - no guarantees here, I mainly wanted to establish a direction (if not velocity).

[jugglerchris]

Well, I've got to this issue in my triage. I don't think there's any reason to keep this ticket open - while more maintainers would be welcome, I think the "looking" for maintainers has happened and I'm it. :-)

[jugglerchris]

@kyren I was just thinking about doing a minor release (0.17.1 with #211) and realised I probably need your help. First to check if you have a release checklist/process (though I'm sure I can figure something reasonable out), but mainly because presumably it currently needs your key to publish to crates.io. Are you able to help sort something out for that?
Thanks!

[azdle]

@jugglerchris, I can't help with the release checklist, but Kyren also made me an owner of the rlua pacakge on crates and I can help with getting a release up.

[jugglerchris]

@azdle Thanks! So when I think things are ready, I can get you to run cargo publish on the right branch?

[azdle]

Definitely. You can @ me on the PR (if you do one) or send me an email (in my GH profile).

…

On Thu, Sep 2, 2021, at 16:06, Chris Emerson wrote: @azdle Thanks! So when I think things are ready, I can get you to run `cargo publish` on the right branch? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #172 (comment)

[makspll]

@khvzak have the unsound parts of mlua addressed here been fixed? If not could someone summarize what's unsound about mlua ?

[khvzak]

@makspll it's fixed. I don't have any open issues about mlua unsoundness. mlua moved far way forward since beginning of this discussion.

[makspll]

Hmm that's fantastic, in this case I would propose that rlua be retired to focus community efforts on a single package. Are there any benefits rlua has over mlua at this point in time ? @jugglerchris

[jugglerchris]

@makspll I don't know enough about mlua to answer that - I'm afraid I haven't been following it. From scanning https://crates.io/crates/mlua/ it sounds like lots of good stuff has been done (and it's getting more downloads). There are obviously some big design differences, for example no sign of rlua's Context in the examples - I guess I'd like to understand how that's different and why it's still safe.

[makspll]

@jugglerchris that's very fair. @khvzak would you mind elaborating on the safety without Context?

[khvzak]

mlua uses bounded lifetimes for functions and a single coroutine-aware Lua instance.

Neither mlua nor rlua can guarantee absolute safely and Context is not a silver bullet here. If you discover any unsoundness in mlua I would appreciate a bug report.

[makspll]

@khvzak I am not saying mlua is unsound but rather wanting to spark discourse around the possibility of unifying the community behind one package. If you don't believe it possible or don't think it to be right direction I totally understand.

It just seems that there items in the roadmap for both packages (for example wasm support) which overlap, but I feel like development time would be more effectively spent if the workload was reduced and collaborators increased!

Do let me know if either of you disagree though!

[khvzak]

@makspll I agree that unifying and expanding a community around one package would be beneficial, but I don't see a way how this can done.
Unfortunately I don't share some rlua decisions, for example Context-based APIs (by the way, it was a reason to make a fork). Also our internals are very different.
Plus, I don't want mlua to be a part of any existing organization apart from a new one created for this project only.

[lenscas]

rlua only joined the amethyst organization because they had experience with managing open source projects and it only happened because rlua lost its original maintainer. So, I don't see a reason for mlua to join the amethyst organization.

the context API was (as far as I know) made to solve the problems that @makspll referred to. With mlua having solved those same problems in others ways I don't see why mlua should get the same context based api.

Now, the question if one should be deprecated over the other is a totally different question which I don't really have an opinion about. I also doubt that this issue is even the correct place to discuss that. Maybe it is best to move that to something dedicated to that to increase the chance of people seeing it and chime in compared to having that happen in a long closed issue?

[jugglerchris]

Agreed that this isn't the best place to discuss it. I think probably the answer is that the two projects have slightly different aims and are complementary. I'd like to learn more about how mlua differs, though, in any case. Is there a good starting point to learn about the big differences?