pyosys: rewrite using pybind11

[donn]

What are the reasons/motivation for this change?

pybind11 is a reasonably more lightweight library that is subjectively a bit more intuitive to use. But more importantly, it does not require an external library (boost) to be installed on the machine, and does not require boost to be compiled for every version of Python.

Explain how this is achieved.

• Rewrite all Python features to use the pybind11 library instead of boost::python. Unlike boost::python, pybind11 is a header-only library that is just included by Pyosys code saving a lot of compile time on wheels [EDIT: as it turns out not really, I used enough C++ templates to offset the time gain from not compiling boost]
• Wrapper generator entirely rewritten using a fault-tolerant c++ header-parsing library
  • Most of the "translation" moved outside of the file
• Move Python-related elements to pyosys directory at the root of the repo
• Create really short test set for pyosys that just exercises basic functionality

Pyosys API Breaks

• The embedded interpreter now requires from pyosys import libyosys as ys like wheels
• Changes in bridging semantics for some types
  • Containers are declared as "opaque types" and are passed by reference to Python - many methods have been implemented to make them feel right at home without the overhead/ambiguity of copying to Python and then copying back after mutation
  • Only classes that handle virtual methods are Monitor and Pass, which now use "trampoline" pattern to support overriding virtual methods in Python: virtual methods no longer use (or support) py_ prefix
• Overloaded methods no longer include type information in the function name, e.g. Pass.call__YOSYS_NAMESPACE_RTLIL_Design__std_vector_string_ is just Pass.call (pybind11 handles overload resolution)
• ys.Yosys has been renamed ys.Globals to more adequately communicate its purpose

If applicable, please suggest to reviewers how they can test the change.

pip3 install -v .
python3 ./tests/pyosys/run_tests.py

---

Resolves #4591

---

TODO:

• rebase on main
• set operations (union, intersection, difference, etc)
• equivalence operators (need constexpr to determine if containers are comparable)
• global variables
• study and address memory concerns raised by @jix

[gatecat]

Just as a data point, some years ago nextpnr moved from boost::python to pybind11 and it was a very good decision, the number of weird Python-related build issues we had dropped significantly.

[jix]

@donn During the Dev JF where you proposed this, I brought up the inherent issue with the object lifetime of Cells and Wires being determined by RTLIL and no way for a binding to extend this. What I didn't realize during that call is that a mechanism I added to extend the lifetime for Cells tracked by bufnorm could be generalized to provide safe Cell and Wire references for python bindings as well.

The idea would roughly be to add a dict<RTLIL::NamedObject *, int> external_refcounts together with some retain/release API. If a retained object gets module->remove(..)d, instead of calling delete object right away, we would clear the name and all other data (marking it as deleted) and move it to a deferred deletion area (so it doesn't show up when iterating over module items) until all external references are gone. This would require checking external_refcounts for each deleted object, but this is essentially free when there are no external references and relatively cheap when there are few. While what bufnorm currently does is more limited, we might be able to use the same mechanism across bufnorm and the overhauled python bindings, so in that case there would be no additional overhead at all if bufnorm was already in use, which should become more common over time.

(I still think what I suggested during the call, eventually changing the API to track module item references by name instead of by reference would be a viable strategy, but I think it makes sense to consider both options and pick whatever what would make maintaining the bindings easier)

[donn]

@jix Thanks for writing it down, will investigate before I mark this PR ready 🫡

[jix]

Note that I don't want to make fixing all currently present lifetime issues a requirement for merging a first pybind11 based version. It's the other way around in that I don't think we could realistically address the lifetime issues with the current python bindings setup, so I see this more as an opportunity for follow up improvements this would enable.

[donn]

@jix So I looked into it - noting it down for myself in the future but your implementation is basically this code block here:

yosys/kernel/rtlil.cc

Line 2847 in 7ebd972

if (design && design->flagBufferedNormalized && buf_norm_cell_queue.count(cell)) {

I think if that's not a blocker then yeah maybe I can tackle Wire/Cell lifecycles in a separate PR.

[donn]

I tested it with my projects, librelane and difetto. Both make heavy usage of pyosys and yosys plugins, and everything's working! Some notes:

• import libyosys as ys fails, except sometimes it also tries importing libyosys.so if that's somewhere in PYTHONPATH. this is worse! I wonder if I can add something to InitTab to intercept this and force users to change their import… fixed

• Needed to modify my Pass::call convenience method as follows:

  def _Design_run_pass(self, *command):
      if hasattr(ys.Pass, "call"): # new API
          ys.Pass.call(self, command)
      else:
          ys.Pass.call__YOSYS_NAMESPACE_RTLIL_Design__std_vector_string_(self, list(command))
  
  
  ys.Design.run_pass = _Design_run_pass  # type: ignore

---

@georgerennie Would also appreciate testing from your side.

[KrystalDelusion]

@donn how would you feel about writing some documentation for using pyosys? If you're not familiar with the rst format that we use, markdown would also be fine and then I can convert it. At the moment it looks like the only thing we have is the pass.py and script.py example scripts and the pyosys tests.

[donn]

@KrystalDelusion I can try putting something under "Using Yosys (advanced)."

[KrystalDelusion]

That would be great, thanks!

[mmicko]

Thanks for your hard work, tested locally and that works fine.
For cross compiling I would probably need to touch more generator.py and Makefile to make sure that part works as expected. Anyway with pybind11 used we could distribute python module as part of oss-cad-suite as well. Using templates tend to make compiling time much longer, but that is expected and think nothing much can be done there.
We would probably need also to update/add build instructions for those building it locally in README.md, but that is something for later when when things are stable and can be recommended to be used as such.

[jix]

your implementation is basically this code block here:

That's correct. Right now it just does what bufnorm needs, but ideally we would end up with a single mechanism that allows different parts of yosys to "retain" and "release" named objects within a module. That can then be shared by pyosys and bufnorm. Let me know when you want to work on this, so we can discuss the requirements for a more general way to do this and how to best implement that.

[donn]

@KrystalDelusion Done, I wrote a small guide. Didn't go too in-depth because it kind of requires knowledge of Yosys internals and that overlaps with other parts of the documentation.

[KrystalDelusion]

Thanks for the documentation! I have left a couple minor comments on some of the wording, but otherwise it looks great.

[KrystalDelusion]

Interesting. I think officially we require at least 3.11, but I'm not sure how much of that is for Yosys and how much is for the frontends like SBY.

[donn]

Tbh it might be less of a headache to drop 3.8 and 3.9 especially, the problem is people on like Debian Bullseye (Python 3.9/not EOL) and Ubuntu 20.04 (Python 3.8/EOL) start complaining...

pybind11 3.0.0 officially supports CPython 3.8+, so it seems prudent to just match its compatibility unless we specifically need functionality that's not 3.8-compatible