rlua is looking for maintainers!

[kyren]

Copied from a reddit discussion here:

I don't actually use Lua for anything currently, and I also don't have much interest in returning to Lua in the future. As such, my Lua related crates rlua (and also luster, but that's much less important) are not receiving much attention currently.

That's a real shame, and I'd like to help at least rlua not become abandoned. It needs some work done to it, none of which is terribly complex but I just don't have the time or proper motivation to be the one to do it. Also, I don't actually even think I should be the one to do it, as it's very hard to make good APIs when you don't use your own API.

I'd be happy to guide people through my ideas for rlua and help especially with reviewing PRs for unsoundness, but there is quite a lot of stuff that needs doing in the pretty near term. The internals of rlua are pretty hairy, which is why I don't expect anybody who wants to take up the mantle to go it completely alone, I'll happily be around to help with the hard parts. What it needs though is better leadership and organization skills than I have or at least have to spare currently.

[kyren]

(Also copied from the same reddit discussion)

For anybody who's interested in working on this, here's a quick wishlist of things that I think rlua needs in the near future that I would work on if I had the time:

1. rlua::Lua needs a compile-time switch to either require Send or to not require Send. Not-requiring Send seems like the obvious choice, but it makes it very hard to integrate into game engines that use threaded systems or perhaps into things like web servers.

2. rlua needs to either not use the lua_State "extradata" region or have a way to turn it off so it doesn't conflict with things trying to use it to create stand-alone Lua modules. The "extradata" region is unfortunately the fastest way to store arbitrary data alongside the main Lua state, which is why this has not been addressed yet.

3. rlua needs the ability to turn off the pcall / xpcall wrappers or any other provided function wrappers to be used in module position. They exist for a reason, but in module position it's very rude to muck with the global Lua state like this.

4. rlua needs to at least support Lua 5.4 when it is released, and ideally it would also support Lua 5.1 (and LuaJIT) and 5.2. This is potentially hard to make truly sound, but rlua has at least already done the hard work to protect against memory allocation and GC errors, so this is not completely infeasible.

5. rlua needs to give up on trying to patch out the UB in PUC-Rio Lua itself, and simply admit this fact by making the Lua constructor unsafe. This is COMPLETELY different than giving up on bindings soundness, I still consider bindings soundness to be extremely important, but this would give up on things like trying to protect rlua usage from things like the debug module or its many other ways to cause UB like directly loading dlls or even just through loading bytecode. Maybe some form of truly sandboxed Lua could be provided as a safe interface, but I don't think it should be advertised as the default because it removes too much of Lua itself.

6. rlua needs some ability to use async functions as coroutines and vice versa, but it also needs a way to in general more ergonomically return via lua_yield, possibly anywhere that a callback returns.

7. rlua contains within its guts a dirty lie, and this lie permeates the entire crate and makes proving the soundness of rlua very very hard. The signature of this type alias SHOULD be:

pub(crate) type Callback<'a> =
    Box<for<'lua> dyn Fn(Context<'lua>, MultiValue<'lua>) -> Result<MultiValue<'lua>> + 'a>;

Making this change would immediately remove much of the very delicate and honestly sort of sketchy logic in the callback creation process and the entirety of the "scope" system and make it more obviously sound. However, since I screwed this up when I initially created rlua and wrote the type signature wrong, I "accidentally" found a way to make it so that callback creation did not require macros. This has prevented me from fixing this because doing so almost certainly makes the API even less convenient by requiring macro wrappers around callbacks. There might be a way to fix this but I've tried very hard and haven't found any solution, it almost certainly requires GATs to fix properly. Edit: This is the biggest blocker in my mind to a 1.0 level API, but maybe being able to generally yield in callbacks also deserves to be a 1.0 blocker.

[rustysec]

To continue the reddit discussion, it seems like there are a few of us that would be interested in taking on maintainer-ship by committee if that is an acceptable solution.

[ahmedcharles]

I'm a fan of both Rust and Lua, though I don't have the time to commit to rlua. That said, I'll take a look around when I'm bored and thanks for the great bindings.

[erlend-sh]

🗣️ to @Grokmoo of https://github.com/Grokmoo/sulis

And if someone knows how to get in touch with @khvzak of mlua that'd be great! (edit: found their email on crates, sent a ping.)

[khvzak]

Thanks @kyren for looking into my work on the rlua fork.
Initially my primary goal was to bring Rust to Lua 5.3 and LuaJIT (the versions I work with) by providing a way to write high-level Lua modules in Rust. That's way I chose rlua 0.15 as a starting point, since it has more suitable API for this (I believe).
Therefore, standalone mode to bring Lua to Rust is also important and in focus!

My long-term goals are partially the same:

• Provide Send / non-Send API or compile time switch.
• Add Lua 5.4 support (the release candidate sources are already in the lua-src crate and (probably) MoonJIT (the project looks promising to be a luajit succeeder).
• Out of the box cross compilation support to all major non x86 platforms (ARM/MIPS/etc) with CI (using qemu?)
• Much more tests
• More "batteries", like serde support (by new crates).
• Improve documentation and create more examples

Regarding your comment in the reddit discussion:

• The first mlua async implementation was built using lua_yieldk call. You can see this code. But later I postponed this idea as it does not work with Lua 5.1 (+ LuaJIT) and switched to more universal solution. Also, I'm thinking about providing support to define a custom yield function, as recently found that necessity and currently overriding the global coroutine.yield.
• I'm going to return (optional?) custom allocator to the Lua constructor to provide more resilience/soundness.

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I'd be happy to get in touch and discuss about unsoundness problem that mlua have or maybe future of both projects.

[kyren]

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I think this might be the most important point to discuss. I think there is a more restricted form of soundness that is actually what people want that is still very useful to have, and I think the main Rust <-> Lua bindings system, whatever it is, should work very hard to maintain it.

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe. That by itself is somewhat of a useless property if as I'm proposing rlua::Lua::new becomes an unsafe function, by that definition anything is at that point sound because you must write unsafe. What I really mean by "soundness" here is more subtle, that nothing you can do directly with the bindings system can cause UB. In this definition, things like the debug table and loading C libraries inside Lua are "sound" because they are simply out of scope. This sounds like a cop-out but I really don't think it is, for example is opening a file and writing to /proc/self/mem unsound? Certainly not, despite the fact that it causes data races, it's just not within the scope for our definition, and the existence of /proc/self/mem doesn't make safety guarantees of rust any less useful.

So, for this library, "soundness" would mean bindings soundness, but as a way to make sure that people understand this, Lua::new() should just simply be unsafe. If at some point there is a good enough sanctioned sandboxing technique or a different implementation of Lua, then maybe a different method could be marked as safe, but as it stands I think this is similar to problems like what the memmap crate has. Whether or not a memory map is "safe" in rust is not just a function of the API, it is a function of the entire system as a whole, entirely different processes can cause data races with certain uses of memory maps, so the memmap crate just acknowledges this with an unsafe constructor and then proceeds to cause no extra harm.

This is (AFAIK currently) the state of the rlua crate, that it provides "bindings soundness". This is a really subtle point ofc and kind of hard to talk about, so maybe I just misunderstood what your project README said, perhaps mlua is already "bindings sound"? I was also worried because the 0.16 and 0.17 releases of rlua definitely also contain soundness fixes, however maybe the 0.16 API change is not a problem since you have also removed Lua::scope, but removing the scope system is somewhat unfortunate.

What I was worried about was there being an obviously best Lua bindings crate that does not attempt bindings soundness at all. You could make the case that since Lua is so unsafe, that there's no point anymore, but I definitely don't think that's true. I think that the kinds of things you can do in Lua that are unsafe are somewhat obvious: load C libraries, the debug table, bugs in the interpreter etc, but the kinds of things you can do that are unsafe in the Lua C API are anything but obvious, because the C API is so freakishly hard to safely use.

It's possible that your goal with mlua is already bindings soundness, and if so that's great! If that's true I don't see why there needs to be a second Lua bindings system really, both crates would have more or less the same goals in mind. My next question would be how we can work together instead of splitting effort, especially since you seem to be much more actively interested in this than I am currently.

[ahmedcharles]

There's some interesting perspectives: https://docs.rs/dtolnay/0.0.9/dtolnay/macro._03__soundness_bugs.html

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe.

I don't think that's a useful definition of soundness. The link above doesn't define it precisely, but my view is that the most useful definition of soundness is that UB is impossible when all code is either safe or uses unsafe in a way that adheres to the relevant documentation.

One of the examples for making frob sound is marking it as unsafe with documentation about it's precondition.

[kyren]

I don't think that's a useful definition of soundness. The link above doesn't define it precisely, but my view is that the most useful definition of soundness is that UB is impossible when all code is either safe or uses unsafe in a way that adheres to the relevant documentation.

Yeah, that's a better definition obviously but in this case that's still not it, becasue there is no contract with Lua::new you can fulfill because every possible method would have the same contract: don't cause Lua code to execute that can cause unsafe. Marking every method as unsafe is not helpful though, and you can limit the rules you have to follow to some categories of Lua code not to run rather than anything in the bindings layer like lifetime unsoundness. I'm saying we should allow the first but disallow the second, and that is the current situation with rlua itself.

You're correct in saying that simply writing unsafe at all does not give someone carte blanche to cause UB, that's not really what I meant to say, I was just skipping a lot of the detail. If the contract of Lua::new is "you cannot misuse the API", then by a certain definition it's "sound" but it's not useful, an example of this is real life could be "by calling Lua::new you agree not to overflow the Lua stack" or "by calling Lua::new you agree not to to throw errors in a __gc metamethod" or "by calling Lua::new you agree that a certain part of API is in unsafe if you use LuaJIT".

I promise I already understand what soundness is. Obviously an API that has an unsafe function with a clear cut guarantee is not sound if you call the unsafe function correctly and it causes UB, it's just that with Lua specifically the strawman I was describing is that the contract for Lua::new is "it's Lua, anything goes". The context here is this:

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I'm just arguing for "bindings soundness" even though the contract must technically be that any method at all can cause UB, I just want to limit how that UB may occur.

Also just to be clear I'm not saying @khvzak is suggesting this strawman definition either, I'm just trying to make sure our definitions are aligned. I think that even though someone can pass a lua_State and muck with the stack themselves, we should still aggressively strive for completely safe Lua C API usage, and advertise that fact.

[ahmedcharles]

I think the mmap analogy works the best. Creating an mmap or lua_State is unsafe and has a huge set of caveats, some easy to understand and some not. Documenting all of the caveats may not even be possible, in either case. But either way, the rest of the api can be sound in the face of a correctly created mmap or lua_State and that's what matters.

[jugglerchris]

FWIW (I've separately wrapped Lua in a Rust crate but haven't got around to publishing it), my aims for safety/soundness sounds in line with to the above:

• Bindings can be written in safe Rust
• Ordinary Lua code can't cause Rust UB but I consider loading C libraries, using the debug library, and interpreter bugs (though I'd want to fix any I knew about) as out of scope.

My use cases are Lua as configuration - the user isn't malicious but may make mistakes and may be handling malicious data (e.g. from the web or e-mails).

I think rlua is doing a better job than my own bindings and is likely to be better maintained and has had more thought put into safety, so I may look into switching over. I'm interested in being involved but do not have time to be a full maintainer.

[khvzak]

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe.
...
What I really mean by "soundness" here is more subtle, that nothing you can do directly with the bindings system can cause UB. In this definition, things like the debug table and loading C libraries inside Lua are "sound" because they are simply out of scope.

Am I correctly understanding, that the following 100% rust safe code using rlua, which 1) modify RO variable 2) causes UB; is sound?

fn main() -> LuaResult<()> {
    let lua = Lua::new();
    lua.context(|ctx| {
        let memcpy: LuaFunction = ctx.load(r#"
            local ffi = require("ffi")
            ffi.cdef[[
            void * memcpy(void *, const void *, int);
            ]]
            return ffi.C.memcpy
        "#).eval()?;

        let a: i32 = 0;
        let b: i32 = 255;

        // Set "a" to "b"
        memcpy.call((&a as &i32 as *const i32 as i64, &b as &i32 as *const i32 as i64, 4))?;
        println!("{}", a); // Prints 255

        memcpy.call((&a as &i32 as *const i32 as i64, LuaNil, 4))?; // UB; Segfault

        Ok(())
    })
}

ffi is installed from this repo.

I was also worried because the 0.16 and 0.17 releases of rlua definitely also contain soundness fixes, however maybe the 0.16 API change is not a problem since you have also removed Lua::scope, but removing the scope system is somewhat unfortunate.

Could you please provide an example(s) of unsoundness in rlua 0.15, for better understanding ? The Scope module was removed in 0.3 since my higher priority was releasing async/await support which does not play nice with Scope system. I'm planning to return Scope support in the next release, but without create_nonstatic_userdata which is pain.

It's possible that your goal with mlua is already bindings soundness, and if so that's great!

I definitely not against bindings soundness, if it's possible and does not affect API usability. Soundness is a great advantage!
However, the example above shows the Rust "safety" rule violation even in rlua
From https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html

The need for all of this separation boils down a single fundamental property of Safe Rust, the soundness property:
No matter what, Safe Rust can't cause Undefined Behavior.

[kyren]

Am I correctly understanding, that the following 100% rust safe code using rlua, which 1) modify RO variable 2) causes UB; is sound?

I'm suggesting that Lua::new be marked as unsafe, but yes it is "bindings sound". The UB comes from Lua violating one of the rules I laid out, that it loads a dll to run C (asm?) code.

However, the example above shows the Rust "safety" rule violation even in rlua
From https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html

That's not bindings unsoundness, that's the trap I don't want to fall into, not to let perfect be the enemy of good. Since nothing in Lua can be perfectly sound without a complete sandboxing solution which is just not how people use Lua, it's not useful to mark every method that can trigger Lua code to run (which is nearly every method) as unsafe. Just mark Lua::new as unsafe and explain the situation.

Could you please provide an example(s) of unsoundness in rlua 0.15, for better understanding ? The Scope module was removed in 0.3 since my higher priority was releasing async/await support which does not play nice with Scope system. I'm planning to return Scope support in the next release, but without create_nonstatic_userdata which is pain.

Yeah the scope system was maybe the main one, but also there were other bugfixes, and I re-designed some of the lifetimes at some point after better understanding the mess I had gotten myself into with the dirty lie. If you didn't include the code that removes the need for the gc_guard hack, then there's probably soundness issues there too. Sorry I can't give you a more concrete answer I don't know when you forked and I don't have the energy right now to go digging that closely, 0.15.3 was released Oct 13, 2018.

Like I said it's possible that mlua already intends to be "bindings sound" and I just misunderstood the README changes.

[Grokmoo]

I appreciate the shout out @erlend-sh. I will make more of an effort to understand the internals of rlua and see if that takes me to where I could make a useful contribution.

I'm also available to help out with more administrative or devops type stuff if the need arises.