Roadmap

#Roadmap

XLE is still very much a work in progress. It's not a finished engine right now.

I've found that part of the difficulty of building an engine is the chicken-and-egg problem.

It's hard to build the code and features you want, without having the data and practical usages for them. But we can't create data for the engine (or build practical uses of it) until the engine is ready.

There's a kind of co-dependency between the engine and it's clients. What it means is that things can progress most naturally if they happen in a incremental way. A little bit of technology over here. Then a bit over there. Then a bit over way over there. Then returning to the first.

Have you about how Michelangelo created his sculptures? Parts would emerge fully-formed from the marble as he went along. But part of the marble always remained unshaped block of marble, until the very end. But his work remained just a collection of parts, emerging at angles from untouched stone -- until the final day when it was all finished.

##What's next for XLE

There are many areas of work that need to be started or improved. I'm not going to draw up a long-term roadmap. But here are a few important areas of work to come.

###GUI tools!

How to do GUI tools? The shader editor provides a proof-of-concept of one method: building using a combination of reusable C# parts, C++/CLI and native C++ engine code. It works surprisingly well, I think.

Another way, is to use some third-party app with plugins for XLE. For example, a system of scripts and plug-ins for 3DS Max could provide the majority of features an engine like XLE would need. Maybe this path would allow for better results with less dev time?

There are advantages and disadvantages to both.

But GUI tools will become a bottleneck for engine development at some point. We need to have a lot of custom data, and methods for artists to author XLE-specific settings. Without that, building world-rendering technologies feels like working in a vacuum.

###Streaming!

XLE was always imagined to do a lot of data streaming in background threads. Some of the underlying structure required is there, but a lot of the work remains.

In particularly, XLE still uses the utility "thread-pump" from DirectX for loading shaders and textures in the background. It's convenient. But there are some big issues at the moment (eg, the SRGB flags aren't really right for textures currently). And, architecturally, there's some overlap between that and the buffer uploads system.

Along the same lines, we need support for mounting archives of some sort, and maintaining a virtual file system.

###Asset processing pipeline

All asset processing is currently done by the sample executable itself. But this isn't a good solution for large scale work. XLE is designed to work with a large database of input data. We want tools that can process exported data, with the conveniences we would normally expect: distribution, error checking, metric recording, continuous integration.

###Practicality!

There's a big difference between samples and a real-world application. It takes some time to get all the little practicality issues right, so that clients can get the behaviour they want out of the engine. There's that old proverb -- no army survives it's first encounter with the enemy. It's the same with a large codebase. The first few practical uses of XLE are likely to find that there are some rough edges to smooth out.

###Threaded rendering

XLE was also imagined to be naturally multi-threaded. Currently most work is done with a single dominant thread. But my hope is that we will build multi-threading technology that can support 1 or 2 different schemes, and aggressively distribute the workload across threads. It's hard to do this without enough data to stress the engine, though. So it's been on hold until there is a very full world to render.

###More rendering technologies

There's lot of rendering technologies to incorporate:

• complex materials
• occlusion culling / potentially visible culling algorithms
• second-bounce GI schemes
• motion blur & more post processing
• voxel hull rendering
• & many other things

###Debugging and profiling tools

I like to have a lot of visual debugging and profiling tools built into an engine. There's some framework for that in XLE, but it still feels like there's a lot to come.

##Extensibility!

XLE is intended to be extensible. That is, it should be possible to integrate code and techniques from a variety of sources. Or even to replace XLE's native solutions with alternative solutions. We want to make it possible to integrate XLE with other large scale codebases. So if someone wanted to integrate XLE with a given 2d engine, or ui engine -- that should be possible. Or use a certain asset processing path. Or mix and match asset formats with another engine. Or use just one feature from XLE with some other large codebase. Ideally these should be possible. Since XLE is open-source maybe the approach it should take for large-scale extensibility is different from a traditional engine. That is, maybe it's not just a matter of providing APIs for and interfaces for extensibility. Anyone can change any code, anyway. Perhaps the real key is in the architecture -- building a design that can be violently cracked open without destroying the whole. Like a Jenga that stays up, even after all of the pieces have been removed. With this in mind, we should see continual improvements for extensibility as time goes on.

###Optimisation

XLE has only minor optimisation so far. There are some major performance problems at the moment, and plenty of optimisation to come.

In particularly, there still needs to be a lot of work related to run-time memory manager. Due to the development time restrictions, it feels like we can't afford to have too many limits on use of the heap. But to get the performance we want, we will need to have smart methods for managing run-time allocations (including more custom heaps, etc).

But while there hasn't been a lot of optimisation yet, that isn't to say there hasn't been thought towards performance. It's impossible to build a successful game engine without keeping performance in mind from day one. But part of planning for performance means recognising that the optimisation of certain parts can (and should) be delayed for a later date.