Multiple cdos

[makslevental]

The gist is the PR implements two ways to merge xclbins:

1. by binding the xclbinutil APIs directly (in xaiepy/xclbinutil.cpp, which is a pybind extension)
2. by shelling out (in examples/dump.py)

Currently it does work in that can run both kernels and you get numerical accuracy (the right numbers come out) but there is some kind of race condition or some lock beind held or something somewhere because it's slow (wouldn't be suprised if it were slower than actually loading individual xclbins). mlir-aie has a PR up (that I emulated) where the same thing does not occur but dump.py is literally their same code 🤷.

FYI I spent like 20 hours on this and it was horrible but what I learned was that the xclbins do literally nothing that isn't immediately undone by the driver. That means eventually we can get rid of all of this.

cc @powderluv

[makslevental]

@nirvedhmeshram you can ignore basically everything in this PR except for this line.

[makslevental]

I tested just running the same kernel twice from an xclbin that only has one kernel (in examples/xrt.py) and I see the same thing (actually that just fails). I have no clue but it would not surprise me if the xrt python bindings are just sloppy with respect to ref counting (and hence direct use in C++ doesn't face the problem). Obviously at this point it's safe to say the problem could be absolutely anywhere in the stack 🤷.

[makslevental]

refactored to drop the context every time and now we have this

ert_cmd_state.ERT_CMD_STATE_ERROR
3.14

ert_cmd_state.ERT_CMD_STATE_COMPLETED
6.28

ert_cmd_state.ERT_CMD_STATE_COMPLETED
3.14

ert_cmd_state.ERT_CMD_STATE_COMPLETED
6.28

ert_cmd_state.ERT_CMD_STATE_COMPLETED
3.14

ert_cmd_state.ERT_CMD_STATE_COMPLETED
6.28

ert_cmd_state.ERT_CMD_STATE_COMPLETED
3.14

ert_cmd_state.ERT_CMD_STATE_COMPLETED
6.28

So something goes wrong the first time but not subsequent times. Note, I made a similar sort of change to the single cdo example and this isn't the behavior (no fails there).

[makslevental]

I'm going to merge even though it's not perfect so that the wheels get built and so you can pip install from here https://github.com/nod-ai/prototype-aie-toolchain/releases/tag/release

[nirvedhmeshram]

Thanks Maks! Will try it out.