feat: support recursive stubgen

[cemlyn007]

To support producing stubs for submodules.

#43

[nicholasjng]

Thanks for taking this up! Here are a few comments.

If I may ask: In which setup do you use recursive stubgen? I'm wondering if for multiple extensions, you need to make all extension labels available to the stubgen binary as data, cf. L198 in the diff.

[nicholasjng]

This is potentially very expensive if many packages appear under runfiles, and if they have stubs in them, they might be erroneously copied into $(BINDIR).

I wonder if there is an option for stubgen to put all recursively generated stubs into the same directory? Maybe -O $(BINDIR)?

[cemlyn007]

I can't believe I didn't see that option XD, yeah I'll get it done! Thank you so much

[cemlyn007]

Thanks for taking this up! Here are a few comments.

If I may ask: In which setup do you use recursive stubgen? I'm wondering if for multiple extensions, you need to make all extension labels available to the stubgen binary as data, cf. L198 in the diff.

I started a project based off the Nanobind examples bazel branch, but my code was getting a bit chunky so I was looking to modularise the code but I still only wanted a single Nanobind extension. I saw in the Nanobind documentation they have support for sub-modules.

I was hoping ultimately have source code looking something like:

src/packageA/moduleA_0/<some_python_files>
src/packageA/ext_moduleA_0/ext_moduleA_0.cpp <- The Nanobind extension
src/packageA/ext_moduleA_0/sub_module_A_0_a/<some_cpp_files_included_by_ext_moduleA_0.cpp>
src/BUILD.bazel

And produce a wheel that looks something like:

packageA/moduleA_0/<some_python_files>
packageA/moduleA_0/ext_moduleA_0.so
packageA/moduleA_0/__init__.pyi
packageA/moduleA_0/sub_module_A_0_a/__init__.py

But my MR is not entirely correct with naming the stubs correctly, at the moment packageA/moduleA_0/__init__.pyi looks like packageA/moduleA_0/_main.src.__init__.pyi (I think but AWOL from my home desktop), and they're not being picked into the wheel, although that might be resolved when I try your comment with using the -O flag in the stubgen script 🤷🏻‍♂️

[cemlyn007]

I had another go, I also have modified the Bazel Nanobind example which can be found here for a minimal example. Unfortunately the root stub is named like _main.src.pyi which led me to do some awful hackery, would appreciate any advise on how to better tackle it!

If you use this branch with the this example branch, you should get a working (brittle) example.

[nicholasjng]

I remember encountering this when attempting to write stubs relative to the working directory in a Bazel project - I think you cannot avoid botched stub names when specifying the output directory and it's not the same as the current working directory.

What I would like is this:

src/nanobind_example/__init__.py
src/nanobind_example/nanobind_example_ext.pyi
src/nanobind_example/sub_ext/__init__.py
src/nanobind_example/sub_ext/__init__.pyi # <- this would contain stubs for sub()

I'll still have to check what stubs are actually produced with your changes, but tl,dr: I think the correct solution is that top-level module extensions get named stubs, and submodules get __init__.pyi files.

Maybe we can avoid the move hack by tweaking some logic in nanobind.stubgen, I'll check it out.

[nicholasjng]

Could you explain this comment? Since the output dir is relative to BINDIR, it cannot be absolute.

[cemlyn007]

I was copying a similar comment that can be found on line 108, but have now removed it

[nicholasjng]

Hi @cemlyn007, how is the situation?
Since a new nanobind release was just made, I'm going to need to put one out as well on the BCR. I'd still love to see this go through. Could you address the remaining comments? Then I'd take care of your PR in nanobind_example.

(I can also take over from here if you like.)

[cemlyn007]

Hey @nicholasjng, sorry for the delay, I've made some changes based on your comments, I do still feel like the top-level stub renaming is hacky but I'm not sure how we can do it in a better way.

FYI for testing I was pip installing and inspecting the file structure in site packages, but this wasn't great because for reasons unknown to me that file structure looked dirty (had old files being seen in). I thought a simple bazel clean --expunge would work but at least for me it didn't.

I think for testing this PR, one working way is to use the python build module (pip install build), but I also needed to create MANIFEST.in and paste the following to get python -m build to run successfully:

include MODULE.bazel
include MODULE.bazel.lock
include .bazelrc
recursive-include src *

If you have a better strategy for testing I am all ears, also let me know if you would like any further changes

[nicholasjng]

Hey! I merged your additions in the nanobind_example. Could you rebase on current master, so that the CI changes are reflected here?