Implement support for off-screen rendering

[Calinou]

Related to #1725.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

When using Movie Maker mode (or anything else that depends on the output of a rendered scene, such as taking screenshots), a window must be spawned by Godot. This is not desired in some scenarios, such as render farms or automated benchmarking/regression testing.

In the case of Movie Maker mode, the window must also not be resized to avoid affecting the video output in any way. When taking screenshots, this only applies if using the disabled or canvas_items stretch mode – the window can be resized without consequences if using the viewport stretch mode.

The --headless command line argument allows Godot not to spawn a window (and not require a GPU), but it disables all rendering code.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Implement support for off-screen rendering with a --offscreen CLI argument. Window position/size CLI arguments should still have an effect to set the initial virtual window size (which affects the viewport size if using the disabled or canvas_items stretch modes).

If both --offscreen and --headless are passed, it should exit with an error as these arguments are incompatible by design. --offscreen should also exit with an error in binaries that don't have any rendering driver built into them.

Note that this proposal does not lift the requirement of having a GPU accessible, unless you use Lavapipe or SwiftShader on your end to provide a software Vulkan implementation.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

This requires OS-specific wrangling to be implemented – perhaps a DisplayServerOffscreen that can be used on desktop platforms (and maybe Android). See this sample Vulkan code: https://github.com/SaschaWillems/Vulkan/blob/master/examples/renderheadless/renderheadless.cpp

Implementing off-screen rendering is technically feasible with OpenGL as well, but an initial implementation will likely focus on Vulkan first.

If this enhancement will not be used often, can it be worked around with a few lines of script?

On Linux, startx can be used to start a dummy X server, but it involves various complications that could otherwise be avoided with off-screen rendering.

Windows and macOS aren't as stringent with GUI requirements, but you may still want to prevent spawning a window during movie rendering (so that it doesn't take up space in your task bar).

Is there a reason why this should be core and not an add-on in the asset library?

This is low-level rendering functionality, which can't be implemented with an add-on.

[havi05]

@Calinou 1. Would it then be possible to take a screenshot in a higher resolution than that of the monitor? (e.g. for 4k or 360° pictures)
2. Would it be possible to make an module/extension to send the images in realtime to OBS without any open windows ? (#4783)

[Calinou]

1. Would it then be possible to take a screenshot in a higher resolution than that of the monitor? (e.g. for 4k or 360° pictures)

Yes, you should be able to use any window size with off-screen rendering (assuming the GPU supports a framebuffer of the given size, usually limited to 16384×16384). It's already possible to resize a window larger to make it larger than the screen size, but it requires third-party tools to do so and it's not very convenient.

Using desktop-wide DSR (NVIDIA) or VSR (AMD) is also an option to allow you to use a desktop resolution that is larger than what your monitor allows. You can then run your project in fullscreen with the output resolution matching the virtual desktop resolution.

That said, if you're only interested in supersampling (and not actually increasing final screenshot resolution), you can increase Scaling 3D > Scale to 2.0 for 4× SSAA. This can be combined with other antialiasing methods as well for even greater antialiasing quality.

2. Would it be possible to make an module/extension to send the images in realtime to OBS without any open windows ? (Realtime Rendering to OBS (NDI) #4783)

This should be achievable with a custom MovieWriter with GDExtension. Note that MovieWriter forces non-realtime simulation, so you must be able to achieve the target framerate consistently and have the rendering FPS limited to the same value to ensure the project doesn't speed up or slow down during gameplay. It also prevents you from hearing any audio output while the project is running.

[Calinou]

@Gamemap Regarding rendering at a higher resolution than the screen, it's already feasible 🙂

The same approach can be used outside of Movie Maker mode when getting the root viewport's data, if using the viewport stretch mode. For example, this applies to screenshots taken with Godot's Image class, etc.

[sosasees]

It's already possible to resize a window larger to make it larger than the screen size, but it requires third-party tools to do so and it's not very convenient.

the window size can be changed with DisplayServer.window_set_size() or Window.size — even to sizes bigger than the screen resolution.

script that changes the game's window size when it starts:

extends Node


@export var override_window_size : Vector2i = Vector2i(2048, 2048)


func _ready() -> void:
	get_tree().get_root().set_size(override_window_size)

[Calinou]

the window size can be changed with DisplayServer.window_set_size() or Window.size,
though i have never tried making them larger than the screen before. i will test this later today.

The OS will clamp the window size to the screen size. Either way, this isn't a viable solution as manipulating this kind of oversized window is very difficult without keyboard shortcuts or third-party tools.

[sosasees]

The OS will clamp the window size to the screen size

not on my Linux computer with Openbox and Picom. i just tested this (you caught the previous version of my comment in your quote, right when i clicked Update), and the window was 9999×9999px large when i set this size in GDScript.

because your computer's OS does clamp the window size, then off-screen mode might be necessary for recording 4320p movies on a 1080p screen — unless we find a way to capture an embedded subwindow.

[dreadpon]

This would be pretty cool to have, as it unlocks the potential for command-line-only projects that still need to provide some form of rendering
Say, an additional game instance can be started from the main one, rendering a replay of an online match to an .avi file (based on downloaded match logs).
Or some visualization tool that renders still images to files using Godot's renderer (instead of writing your own)

[fire]

Do we have an proposal for 2x or more rendering?

[Calinou]

Do we have an proposal for 2x or more rendering?

Once offscreen rendering is implemented, you'll be able to render at any window size, including those larger than the display (without needing the viewport stretch mode or third-party window manipulation tools).

Also, you can already perform supersampled rendering in 3D by setting Scaling 3D > Scale to a value above 1.0 in the Project Settings.

[revpriest]

Expressing an interest: I too would like offscreen rendering, in my case for render-farms. I'm making 360-stereo renders at 4k and each frame is taking about two minutes. Would be really helpful to be able to farm that all out to a hundred AWS instances overnight.

Intrigued to hear there might be some way to run a dummy x server to get it done. Will look into that.

[Calinou]

I'm making 360-stereo renders at 4k and each frame is taking about two minutes

I'm surprised it's taking so long. Are you sure you're not spending most time in writing PNGs? In this case, godotengine/godot#91263 will help significantly.

(You can call ImageMagick or similar afterwards to convert the QOIs to another format, but QOI should be preferred as an intermediate format as it's faster to write.)

[revpriest]

Trouble with the 360 sterographic render is that you can't just point one camera at it and hope that'll work, you gotta move the cameras as though they were eyes inside a head, panning a one-pixel column across the whole image. I built a rig with like 50 cameras but it still needs considerable time rotating them around to build that stereo image.

https://github.com/revpriest/godotPanoRenderer

If there's a better way to do that then I'm all ears, but the standard camera don't seem capable of moving the origin point relative for each pixel column in the render.

[revpriest]

To report back:

I can indeed just install NVidea drivers, X11 and Vulkan on a AWS G3 machine, then run my compiled Godot app on it with command-line args to tell it what to render out and a faked DISPLAY env var.

I dunno if it's making a window, certainly no hardware displays any windows it creates, and there isn't even any window-manager software, but my rendered output files are produced and console text appears as hoped.

So this off-screen-rendering ticket doesn't really matter to my use-case. Maybe this is about the editor and I don't need the editor just my app.

Interestingly the scene loads way slower on the remote machines than my desktop, but the actual rendering happens about twice as quickly in the cloud as on my desktop GPU. About 75 seconds on amazon vs 150 or so at home.

More strangely, the 8x large instance, which claims to have 32 vgpu vs the xlarge with only 4, renders in basically the same time. So no point at all paying extra for the more GPU units there. Guess I'd have to run multiple copies of my app or something.

[EDIT: Can't run multiple copies of the app either: OpenXR: Failed to enumerate number of extension properties]

[Calinou]

Guess I'd have to run multiple copies of my app or something.

Vulkan does not have multi-GPU support in the form of SLI or similar, so you need to run Vulkan instances. This can be done within a single process, but it needs to be designed specifically for this. (Godot supports local RenderingDevices, which can technically achieve this. However, it doesn't allow for SLI-like functionality.)

[EDIT: Can't run multiple copies of the app either: OpenXR: Failed to enumerate number of extension properties]

That's strange, I don't see why OpenXR would need to be initialized (unless your project is using XR functionality).

cc @BastiaanOlij

[BastiaanOlij]

That's strange, I don't see why OpenXR would need to be initialized (unless your project is using XR functionality).

cc @BastiaanOlij

Indeed this is weird, OpenXR is only required and should only be initialised if you're outputting to a headset through OpenXR. Stereo rendering is completely separate.

That said, I don't know why running multiple copies of Godot would fail if OpenXR is enabled, the OpenXR runtime will treat everything but the last app as background processes. But it may be that not all XR runtimes are that flexible.

[revpriest]

Oh yeah. That won't be the error that stopped it running multiple times then.

It does indeed use XR usually, but obviously not in the context where it's rendering on a remote machine. That error probably came up with the fist time running it, when it worked, too.

Sounds like Vulkan drivers don't allow multiple-instance shenanigans then anyway then, at least not without significantly more faffing about than just ordering another instance of the 4vgpu type instead.

Adjustments to the numbers of cameras and things in the 360-degree sweep have the render time optimized down to about a minute, and even faster on my local machine. Might get away with about 20 machines overnight for the final run.

[blubberdiblub]

@revpriest

Trouble with the 360 sterographic render is that you can't just point one camera at it and hope that'll work, you gotta move the cameras as though they were eyes inside a head, panning a one-pixel column across the whole image. I built a rig with like 50 cameras but it still needs considerable time rotating them around to build that stereo image.

https://github.com/revpriest/godotPanoRenderer

If there's a better way to do that then I'm all ears, but the standard camera don't seem capable of moving the origin point relative for each pixel column in the render.

Intuitively to me that appears to be the wrong approach. From a graphics hardware perspective, the hardware doesn't care about the projection, as you do the projection "yourself" (in case of using stock Godot stuff, the engine provides the necessary projection in its stock shaders). So in theory, given the right shader (or multiple ones), it should be possible to render a full panoramic scene in one go rather than in n columns.

So personally if I were to do such a thing, I would take the stock shaders, modify them to do panoramic projection, somehow make sure that no frustum culling is performed and then render the scene with the modified shaders. Not sure if that's workable in practice with Godot, but worth an attempt, I'd say.

[revpriest]

I dunno what most of that means. Isn't a shader the thing that puts a material surface onto a triangle rather than the thing which decides where each triangle is drawn? Perhaps there's two uses of the word.

Is this a suggestion to start writing a new renderer in Godot's source-code or to do something with a change to existing materials?

Either way I wouldn't know where to start. I did start looking at changing the source-code so that the lines it projects to determine which triangle is in which pixel could be altered for each column in the output image but it seemed like it was impossible to change the origin from the code I was looking at, only the angle, and the origin of the projection needs to be different for each pixel-column as far as I can tell.

Probably we're on the wrong ticket for that sort of discussion. Dunno if one exists for adding a stereo 360 panoramic render. Couldn't find one earlier in the year when I looked.

[sosasees]

yes, there are 2 kinds of shaders.
oversimplified summary:

• the vertex shader decides where each vertex is and how they connect to triangles
• the fragment shader decides how each pixel looks

[apples]

Has anyone looked into implementing this?

I have a need for an interactive console application which needs to run both on Windows desktop and Linux headless servers, and it needs to be able to render images.

Obviously the workaround of simply creating a main window anyways is... fine, but less than ideal for user experience, at least on Windows desktop where the window is annoying visual clutter.

I'm not very familiar with these systems, but it seems like the most straightforward implementation would be:

1. Create a DisplayServerOffscreen and register it in the same manner as the headless driver.
   • This would essentially be a dummy implementation, except that the rendering context would still be initialized, and windows would pretend to be functional.
   • Platform-specific features such as clipboard operations would not be implemented, but could be added in the future by creating e.g. DisplayServerOffscreenWindows.
2. Create a RenderingContextDriverVulkanOffscreen which would be used by the DisplayServerOffscreen.
   • This would be responsible for creating the offscreen surfaces by implementing surface_create().
3. Tie the offscreen implementation to a --offscreen CLI argument.

(Vulkan obviously being the main focus, I'm not even sure if this is possible or even desired in OpenGL.)

It seems like the hardest part will be simulating the various window functions e.g. window resizing and all that.

Does that sound like the way forward, or am I completely off base here?

I don't have a lot of spare time lately, but I am interested in implementing this. Will report back if I make any progress.