BulletAppState debug not working in VR

[marcelfalliere]

Hello.

A BulletAppState has a nice bulletAppState.setDebugEnabled(true); really handy to debug physics. Unfortunately, it doesn't work in VR. Below is a reproductive test case. It's the VR example from the wiki, with 1/ A bullet app state 2/ a green cube, not a blue one, because bullet wireframe view is either pink-ish or blue.

public class MainVRxBulletDebugBug  extends SimpleApplication {

    public static void main(String[] args) {
        AppSettings settings = new AppSettings(true);
        settings.put(VRConstants.SETTING_VRAPI, VRConstants.SETTING_VRAPI_OPENVR_LWJGL_VALUE);
        settings.put(VRConstants.SETTING_ENABLE_MIRROR_WINDOW, true);

        VREnvironment env = new VREnvironment(settings);
        env.initialize();

    	// Checking if the VR environment is well initialized
    	// (access to the underlying VR system is effective, VR devices are detected).
    	if (env.isInitialized()){
            VRAppState vrAppState = new VRAppState(settings, env);
            vrAppState.setMirrorWindowSize(1024, 800);
            MainVRxBulletDebugBug app = new MainVRxBulletDebugBug(vrAppState, new BulletAppState());
            app.setLostFocusBehavior(LostFocusBehavior.Disabled);
            app.setSettings(settings);
            app.setShowSettings(false);
            app.start();
        }
    }

    public MainVRxBulletDebugBug(AppState... appStates) {
        super(appStates);
    }

    @Override
    public void simpleInitApp() {
        BulletAppState bulletAppState = getStateManager().getState(BulletAppState.class);
        bulletAppState.getPhysicsSpace().setGravity(new Vector3f(0, 2f, 0));
        bulletAppState.setDebugEnabled(true);
        
        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);

        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setColor("Color", ColorRGBA.Green);
        geom.setMaterial(mat);
        
        RigidBodyControl rigidBodyControl = new RigidBodyControl(1.0f);
        geom.addControl(rigidBodyControl);
        bulletAppState.getPhysicsSpace().add(rigidBodyControl);

        rootNode.attachChild(geom);
    }

}

[stephengold]

Off the top of my head...

1. What does the test case do on your system?
2. Which version of JMonkeyEngine are you using?
3. Which physics library are you using?

[marcelfalliere]

What does the test case do on your system?

The green cube is visible in VR, and is affected by gravity (it fails upward, with a gravity vector of Vector3f(0, 2f, 0)). As intented.

This line bulletAppState.setDebugEnabled(true); seems ignored. In a non-VR environment, it displays a wireframe, either Blue or Magenta around a physic object. This is the bug.

Which version of JMonkeyEngine are you using?

3.5.0-stable

Which physics library are you using?

org.jmonkeyengine:jme3-jbullet:3.5.0-stable

[marcelfalliere]

To illustrate, what is expected :

And in VR :

The magenta line in the first pic are the wireframe that Bullet adds when bulletAppState.setDebugEnabled(true);

[stephengold]

Thanks for documenting this issue.

I don't have VR hardware. I'm curious whether you can reproduce the issue with Minie in place of jme3-jbullet.

[marcelfalliere]

I also tried with 3.5.1-stable : same issue.

[marcelfalliere]

Just tried with com.github.stephengold:Minie:4.8.1, got same behaviour.

The only difference is the BulletAppState needs to be created after Application.start().

I don't have VR hardware.

Some games are really nice :)

Oh and also, just to be extra sure debug is enable :

    @Override
    public void simpleUpdate(float tpf) {
        super.simpleUpdate(tpf);
        
        System.out.println(this.stateManager.getState(BulletAppState.class).isDebugEnabled());
    }

Which ouputs true...

[stephengold]

Since you don't report any diagnostic messages, I hypothesize that the debug scene gets added to the wrong ViewPort.

Minie's DebugConfiguration initializes its list of viewports using Application.getViewPort(). I expect there are multiple viewports in a VREnvironment. A quick glance at the code suggests they would be accessed via VRViewManager.

At this point, I suspect your best workaround will be to use Minie's BulletAppState.setDebugViewPorts() method to tell Minie where you want the debug visualization to appear. (Unfortunately, jme3-jbullet doesn't yet include this feature.)

[marcelfalliere]

I tried Minie in the actual project, not the test case in OP ; I get this error :

java.lang.IllegalArgumentException: Must have mode=Triangles/TriangleFan/TriangleStrip.
	at jme3utilities.Validate.require(Validate.java:929)
	at com.jme3.bullet.collision.shapes.infos.IndexedMesh.<init>(IndexedMesh.java:171)
	at com.jme3.bullet.collision.shapes.infos.CompoundMesh.<init>(CompoundMesh.java:103)
	at com.jme3.bullet.collision.shapes.MeshCollisionShape.<init>(MeshCollisionShape.java:174)
	at com.jme3.bullet.util.CollisionShapeFactory.createSingleMeshShape(CollisionShapeFactory.java:469)
	at com.jme3.bullet.util.CollisionShapeFactory.createCompoundShape(CollisionShapeFactory.java:379)
	at com.jme3.bullet.util.CollisionShapeFactory.createCompoundShape(CollisionShapeFactory.java:369)
	at com.jme3.bullet.util.CollisionShapeFactory.createCompoundShape(CollisionShapeFactory.java:369)
	at com.jme3.bullet.util.CollisionShapeFactory.createMeshShape(CollisionShapeFactory.java:246)
	at com.fmf.dedale.world.Chunk.buildAndAttachToNode(Chunk.java:131)
        ....

The line in question (Chunk.java:131) is calling CollisionShape chunkCollisionShape = CollisionShapeFactory.createMeshShape(node); where node contains many children, that are all Boxes geometries.

Weird thing is I don't find Validate.java source code anywhere 🤔

[stephengold]

Validate is found in the Heart library, which Minie depends upon: https://github.com/stephengold/Heart/blob/master/HeartLibrary/src/main/java/jme3utilities/Validate.java

The IllegalArgumentException could be caused by trying to create a MeshCollisionShape from a mesh composed of lines or points. MeshCollisionShape is only for triangle meshes.

[marcelfalliere]

Oh indeed, I've added debug arrows, as in com.jme3.scene.debug.Arrow. Maybe that's the cause.. I'l try that and update.

[stephengold]

That would make sense.

I'm glad you reported the crash. For the next release of Minie, I've added code to CollisionShapeFactory to ignore any non-triangle meshes.

I'm curious whether BulletAppState.setDebugViewPorts() enables debug visualization with jme3-vr.

[marcelfalliere]

Arrows were the problem 👍

VRAppState vrAppState = stateManager.getState(VRAppState.class);
bulletAppState.setDebugViewPorts(vrAppState.getLeftViewPort(), vrAppState.getRightViewPort());

I'm calling this right at the end of the simpleInitApp, after every other state is attached.

Does not do it ... but there seems to be a difference. There is a FOUC that displays the bullet debug wireframes. It's at the start. It's really fast. Then it disappears. Weird ! I would like to help but I have not enough JME knowledge.

That is the behaviour in my game.

When adding these 2 lines to the test case in my OP, I get this error :

java.lang.NullPointerException
	at com.jme3.bullet.debug.BulletDebugAppState.initialize(BulletDebugAppState.java:667)
	at com.jme3.app.state.AppStateManager.initializePending(AppStateManager.java:332)
	at com.jme3.app.state.AppStateManager.update(AppStateManager.java:362)
	at com.jme3.app.SimpleApplication.update(SimpleApplication.java:258)
	at com.jme3.system.lwjgl.LwjglWindow.runLoop(LwjglWindow.java:580)
	at com.jme3.system.lwjgl.LwjglWindow.run(LwjglWindow.java:669)
	at java.base/java.lang.Thread.run(Thread.java:834)

It seems the viewports are null. Moving these two lines in start() fixes that error.

But the debug wireframe of bulet are still not visible.

Here is the debug test case updated :

public class MainVRxBulletDebugBug  extends SimpleApplication {

    public static void main(String[] args) {

        AppSettings settings = new AppSettings(true);
        settings.put(VRConstants.SETTING_VRAPI, VRConstants.SETTING_VRAPI_OPENVR_LWJGL_VALUE);
        settings.put(VRConstants.SETTING_ENABLE_MIRROR_WINDOW, true);

        VREnvironment env = new VREnvironment(settings);
        env.initialize();

    	// Checking if the VR environment is well initialized
    	// (access to the underlying VR system is effective, VR devices are detected).
    	if (env.isInitialized()){
            VRAppState vrAppState = new VRAppState(settings, env);
            vrAppState.setMirrorWindowSize(1024, 800);
            MainVRxBulletDebugBug app = new MainVRxBulletDebugBug(vrAppState);
            app.setLostFocusBehavior(LostFocusBehavior.Disabled);
            app.setSettings(settings);
            app.setShowSettings(false);
            app.start();
        }
    }

    public MainVRxBulletDebugBug(AppState... appStates) {
        super(appStates);
    }

    @Override
    public void simpleInitApp() {
        
        BulletAppState bulletAppState = new BulletAppState();
        getStateManager().attach(bulletAppState);
        bulletAppState.getPhysicsSpace().setGravity(new Vector3f(0, 2f, 0));
        bulletAppState.setDebugEnabled(true);
        
        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);

        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setColor("Color", ColorRGBA.Green);
        geom.setMaterial(mat);
        
        RigidBodyControl rigidBodyControl = new RigidBodyControl(1.0f);
        geom.addControl(rigidBodyControl);
        bulletAppState.getPhysicsSpace().add(rigidBodyControl);

        rootNode.attachChild(geom);
    }

    @Override
    public void start() {
        super.start();
        
        VRAppState vrAppState = stateManager.getState(VRAppState.class);
        BulletAppState bulletAppState = stateManager.getState(BulletAppState.class);
        bulletAppState.setDebugViewPorts(vrAppState.getLeftViewPort(), vrAppState.getRightViewPort());
    }
}

With com.github.stephengold:Minie:4.8.1 and jme 4.5.1

[marcelfalliere]

Quick update : I figure maybe calling setDebugViewPorts inside the update would maybe "force" it, but it does not.

@Override
    public void simpleUpdate(float tpf) {
        super.simpleUpdate(tpf);
        
        VRAppState vrAppState = stateManager.getState(VRAppState.class);
        BulletAppState bulletAppState = stateManager.getState(BulletAppState.class);
        bulletAppState.setDebugViewPorts(vrAppState.getLeftViewPort(), vrAppState.getRightViewPort());
    }

[neph1]

The VR lib does some tricks to the viewports. I think it ignores all but the main view. You can see the details in LWJGLOpenVRViewManager. BulletDebugAppState creates its own viewport, which the VR lib is ignorant of.
viewPort = rm.createMainView("Physics Debug Overlay", app.getCamera());
Unfortunately it's not accessible outside the package, same with physicsDebugRootNode. I think getting the debug view into vr is not trivial. They were never written to be supportive of each other.
If you could render it into a post filter, that might be one way of doing it.

Disclaimer: This is just coming from me, glancing at the Bullet code, though. I know a fair bit about the VR lib, but not much about the Bullet implementation.

[stephengold]

Thanks, @neph1. I'm sure that, with moderate effort, we can make them work together.

Since it's easier to create new releases of Minie, I'd like to address the issue there before tackling jme3-jbullet. But without VR hardware at my end, it's difficult for me to debug/test anything.

[neph1]

I would recommend having a regular application class along side the VR one for debugging. When doing quick tests, putting the VR equipment on and off is cumbersome. What case requires the physics to be debugged in VR?

Mid term, perhaps a BulletDebugAppStateVR could be developed for specialized cases.

Long term, the VR lib needs to be rewritten, IMO. A lot of things have happened since it was developed.

[marcelfalliere]

What case requires the physics to be debugged in VR?

VR specific controls where the player head and hands interact with the world in a way (the VR way) not doable with KBM.

But without VR hardware at my end, it's difficult for me to debug/test anything.

I could debug/test snapshots for you if you'd like.

In the meantime, I found a way around, that is to display a wireframe duplicate of the mesh. Advantage of this is that the VR view is not cumbersome with physics-debug artifacts, since I add this to the mesh I want.

Also, @neph1 have you looked at the Tamarin lib ? It's a jme plugin made by @richardTingle, pretty great stuff 🚀

[stephengold]

I'm glad you found a workaround. However, the main benefit of debug visualization comes when the collision shapes don't precisely match the JME meshes. So I think we need a better solution.

[richardTingle]

@stephengold I've been able to fix this, but I wanted to talk though my solution (mostly because I don't understand why the non-VR version doesn't do the same thing I'm doing for VR).

Within BulletDebugAppState

@Override
public void initialize(AppStateManager stateManager, Application app) {
    super.initialize(stateManager, app);
    this.app = app;
    this.rm = app.getRenderManager();
    this.assetManager = app.getAssetManager();
    setupMaterials(app);
    physicsDebugRootNode.setCullHint(Spatial.CullHint.Never);

    if (isVr()){
        VRAppState vrAppState = stateManager.getState(VRAppState.class);
        vrAppState.getLeftViewPort().attachScene(physicsDebugRootNode);
        vrAppState.getRightViewPort().attachScene(physicsDebugRootNode);
    }else{
        viewPort = rm.createMainView("Physics Debug Overlay", app.getCamera());
        viewPort.setClearFlags(false, true, false);
        viewPort.attachScene(physicsDebugRootNode);
    }
}

I'm just attaching the scene to the two viewports for the two eyes and it all just works. But the normal view could do that as well but goes through the trouble of creating a whole new viewport (which doesn't work in VR, because of reasons I only half understand¹), and that has got me worried as people don't usually do a bunch of complicated stuff for no reason.

¹ I think the reason it doesn't work for VR is that each eye has a single viewPort that is copied into the texture going to that eye, not a layering of viewports the way that normal JME works.

[stephengold]

@stephengold I've been able to fix this, but I wanted to talk though my solution (mostly because I don't understand why the non-VR version doesn't do the same thing I'm doing for VR).

I'm paying attention.

I'm just attaching the scene to the two viewports for the two eyes and it all just works. But the normal view could do that as well but goes through the trouble of creating a whole new viewport (which doesn't work in VR, because of reasons I only half understand¹), and that has got me worried as people don't usually do a bunch of complicated stuff for no reason.

¹ I think the reason it doesn't work for VR is that each eye has a single viewPort that is copied into the texture going to that eye, not a layering of viewports the way that normal JME works.

One reason to put debug visualization in a separate viewport is to prevent it from being obscured by ordinary geometries. For instance, if your avatar is exploring a building, you might want to visualize physics objects that are interior to (or on the other side of) walls. If you have obscuring filters in your main scene, you might not want them to apply to debug visualization. Layered viewports provide additional flexibility to handle such situations.

If jme3-vr can't handle layered viewports for each eye, that seems like a serious limitation. For instance, I wonder how GUI geometries are handled...

[richardTingle]

One reason to put debug visualization in a separate viewport is to prevent it from being obscured by ordinary geometries. For instance, if your avatar is exploring a building, you might want to visualize physics objects that are interior to (or on the other side of) walls. If you have obscuring filters in your main scene, you might not want them to apply to debug visualization. Layered viewports provide additional flexibility to handle such situations.

O yeah, I've constructed a more complex scene and I can see that now, nice effect.

If jme3-vr can't handle layered viewports for each eye, that seems like a serious limitation.

I don't think it can, what leads me to believe that is LWJGLOpenVRViewManager#setupVRScene where it grabs a texture for (the one and only) view port per eye:

            leftEyeTexture = (Texture2D) getLeftViewPort().getOutputFrameBuffer().getColorBuffer().getTexture();
            rightEyeTexture = (Texture2D) getRightViewPort().getOutputFrameBuffer().getColorBuffer().getTexture();
            leftEyeDepth = (Texture2D) getLeftViewPort().getOutputFrameBuffer().getDepthBuffer().getTexture();
            rightEyeDepth = (Texture2D) getRightViewPort().getOutputFrameBuffer().getDepthBuffer().getTexture();

And then (and this is the bit I only sort of understand), in LWJGLOpenVRViewManager#postRender the id of that texture is used as the submission to the VR compositor

@Override
public void postRender() {
        ....
        ....
        ....
 
        leftTextureType.set(getLeftTexId(), leftTextureType.eType(), leftTextureType.eColorSpace());
        rightTextureType.set(getRightTexId(), leftTextureType.eType(), leftTextureType.eColorSpace());
        ....
        ....
        errr = VRCompositor.VRCompositor_Submit(VR.EVREye_Eye_Right, rightTextureType, null, submitFlag);
        errl = VRCompositor.VRCompositor_Submit(VR.EVREye_Eye_Left, leftTextureType, null, submitFlag);

}

For instance, I wonder how GUI geometries are handled...

GUIs usually hang as 3D floating windows in amongst the normal scene, although I can see how forcing the floating window to always be visible would be nice.

Perhaps another thing to think about when the JME OpenXR VR rewrite happens, I did have a go, but my knowledge of LWJGL is still a little rudimentary so I didn't get very far

[stephengold]

Thanks for digging into this. I'm glad you found a suitable workaround.

In my mind the real issue is much deeper than physics debug. There seems to be a fundamental disconnect between the jme3-core concept of viewports and that of jme3-vr. Not surprising given what I know of the history, but it's unfortunate.

[stephengold]

I've reviewed the changes, and I approve this for integration.