Pre computed display lists

[lambertjamesd]

Are there any plans for rdpq pre computed display list? I want to be able to load display lists from the cartridge with minimal code to process them into usable display lists by the RCP. I know there is glGenLists for the opengl implementation but I am wondering if there are plans to have a lower level and faster way of achieving the same thing. I can help out with the implementation but I figured I would ask here if there were any plans for it and ideas on how a potential interface for usage could look

[anacierdem]

@snacchus have plans for a new graphics api, but I am not sure if it will have a pre-baked display lists. The original reason why we didn't want it was that it requires locking in the DL format as it becomes data at that point and doesn't allow for dynamic changes at runtime. I am sure they will provide more information around the topic.

Also genereting lists at load time is not that bad in general. OTOH, I think the ability to also baking in the transformations might make sense in sone situations. I mean if a transformation is in a dl, you can pre compute all of the RSP commands, ogl api allows for that. The use case I had in mind is precomputing the environment of a shoot'em up ahead of time and then paying only the RDP cost at runtime, freeing up a lot of room for dynamic stuff.

Still, want to hear opinions around this?

[snacchus]

The new graphics API won't include precomputed display lists. In fact, the concept of display lists will be entirely orthogonal to it, as opposed to GL. That means you can put calls from the new API into DLs if you want (note that the libdragon term for DL is "block"), regardless of being precomputed or not.
That means we could implement precomputed blocks at the rspq level. However, as @anacierdem already hinted at, that could potentially be a dangerous move. Imagine we implemented precomputed blocks today. We could even offer a tool that will generate them for you (or a PC-side library) so you don't even have to worry about the gritty details of the binary format. Each time we change something in libdragon that affects this binary format, we'd update the tools as well. That sounds all very good.
But now imagine some third party wrote a Blender plug in that outputs these precomputed blocks. Even if they used the tools provided by libdragon, they'd have to stay on top of updating them each time there are changes. And users would have to keep downloading new versions of the plug in. Maybe someone locks the libdragon version for their project, so now they are stuck with a specific version of the plug in, which might contain other bugs. And so on... You could imagine this turning into a big mess of compatibility and maintenance.
Now this isn't to say that we never want to implement this feature. Libdragon generally tries to offload as much computation to build time as possible. But in this particular case it's difficult, especially because in the current stage of libdragon's development, there might be changes to the binary format of commands at any time. In the future this will probably change as features become more stable. But freezing the format forever is something we want to avoid as much as possible, as we are concerned it will slow down development of libdragon.
I'm open to hear your opinion on this. Maybe there is a solution to this problem that we simply haven't thought of.

[lambertjamesd]

That makes sense. I guess a good compromise would be a block format that can be loaded quickly, so maybe the format is close to its final representation in memory but needs only minimal work to finish formatting it. I can probably be content with gl lists for now. Maybe if I find resource loading to be a bottle neck do I try to optimize it further.

[lambertjamesd]

We could borrow the idea from libultra where display lists are generated from macros. That way the macro becomes the interface that tools can target but the implementation of the macro could change.

[snacchus]

Right, a set of macros certainly seems like a more maintainable public API than pure binary data. I think it could be worth experimenting with.
Out of interest, since you mentioned GL display lists specifically: did you find that they are too slow to generate at runtime for your needs? Maybe we could try to improve performance there?
And maybe @rasky could chime in on this as well.

[lambertjamesd]

I just like the idea of faster, closer to hardware control. Maybe after using it more extensively my opinion will change.

[snacchus]

That's completely understandable, since GL as an API is quite detached from the hardware and induces a non-negligible amount of overhead. This "new API" that has been mentioned a couple of times in this thread will hopefully help in this regard. It's still in the design phase, but it will be a completely custom graphics API that is tailored to the hardware with the goal of allowing more efficient utilization. It's called "magma" and I'm currently working on it on my fork: https://github.com/snacchus/libdragon/tree/magma
There is also a discussion thread on the N64brew Discord if you're interested: https://discord.com/channels/205520502922543113/1196576447058432090
Feedback is very much appreciated!