Feature openxr

[Starcommander]

This pull-request adds the feature for using openxr in jme3.
It is a working first implemention and some improvements are still necessary.

• Moving HMD (currently only rotating)
• Controllers
• View perspective is not correctly

I tested it (and works) with OSVR-HMD on Debian 11 (Bullseye) with running monado-service.
I also tested it (and works) with my HTC-VIVE.

To get it running:

• In main function modify app settings with: XrHmd.setRendererForSettings(appSettings);
• In simpleInitApp(): XrHmd.initHmd(this);
• Add your stuff into rootNode
• Start monado-service
• Start app.

[stephengold]

Given JME's staffing constraints, I worry about adding to the project's maintenance burden.

• Has adding OpenXR support been discussed at the JME Hub/Forum?
• Is there a good reason develop the jme3-xr library in the jmonkeyEngine repo, instead of as a separate project (like Blocks, Lemur, Minie, and ShaderBlow)?

[Starcommander]

@stephengold
Here is a Forum, where openxr was discussed.
https://hub.jmonkeyengine.org/t/solved-vr-launch-title-and-icon-in-steamvr/43706/14

When we have openvr included in jme3-vr (that needs steam binary blob "steamvr"), then why not openxr that is open source, and should be the successor of openvr?

[neph1]

I think it would be a great contribution to add openxr. Adding just openxr as a separate project is probably not feasible as it relies on jme-vr.

At the same time I share @stephengold 's concerns. The vr package hasn't had a proper maintainer since 2015 and has relied on sporadic contributions since.

I wonder if it would make sense to break it put of core to a separate project. It could still reside in the jme project, but not alongside the core packages. That way, updating it would be a little less sensitive.

Commenting on the pr itself, I think proper hmd translation and controller support would be minimum requirements for it to be considered. Otherwise the use case is very limited.

[jseinturier]

Integrating OpenXR to JMonkey is a good idea for me. However, i've some concerns regarding the proposed method:

• jme3-vr module has been thinked in order to provide a high level abstraction of existing VR API (at the beginig OpenVR and VRAPI). Integrating OpenXR should be achieved by extending the current jme3-vr module instead of removing it and providing an OpenXR exclusive module. At this time, OpenVR and VRAPI are supported and are binded by LWJGL. Removing these functionnalities from JMonkey can be problematic for end users.

• OpenXR provide Virtual Reality and Augmented Reality functionnalities. In the "real life" we are not offen use VR / AR at the same time so for me it is not relevant to create a new module that embed all the functionnalities. I think we could split VR / AR functionnalities within separated modules (jme3-vr, jme3-ar ?)

[richardTingle]

Very cool, I will take a look at this PR properly on Friday (and can test on an occulus quest). But I would vote yes to having openXR be an engine feature rather than a seperate library. It's a fundamental capability like android launch that might then have libraries build around it to produce advanced functionality.

I'm also pro this being seperate from the existing VR stuff because that was written in a very different time where there were many competing standards that needed to be abstracted over

[Ali-RS]

Adding just openxr as a separate project is probably not feasible as it relies on jme-vr.

From a quick look at this PR (in the gradle build), I do not see how the jme3-xr project relies on jme3-vr.

[neph1]

Adding just openxr as a separate project is probably not feasible as it relies on jme-vr.

From a quick look at this PR (in the gradle build), I do not see how the jme3-xr project relies on jme3-vr.

Damn. I thought I saw it was when I checked on my phone last night. Disregard that part.

[Starcommander]

@Ali-RS Yes, you are right, jme3-vr is not necessary to launch an app with jme3-xr.
So, maybe jme3-xr can be a successor of jme3-vr in future.
I will also still work on improvements of this project, for example controller support, and so on.

[richardTingle]

If we do want it in a seperate library I'd be happy to have it in Tamarin (as a Tamarin 2 that is only openXR and Tamarin 1 keeping legacy support for openVR) and then I'd maintain it

[richardTingle]

I've been working on actions and am making a fair bit of progress. Have got basic button presses registered and working (I mention this to avoid us duplicating work).

I have no idea why I'm getting the weird lensing effects when I turn my head and have given up trying to understand why (at least for now)

[stephengold]

Could you please convert this PR to a "draft"?
That would make it obvious that the PR isn't ready to be integrated.

[Starcommander2]

Could you please convert this PR to a "draft"? That would make it obvious that the PR isn't ready to be integrated.

Yes, when I am back on Friday I will switch to draft.
Unfortunately I have no access to my account currently.

[Starcommander]

@richardTingle
You made some interesting progress regarding actions and some little changes in hmd-moving
How would you suggest, we should continue the work?
I would really like to see jme3-xr in next release of jmonkeyengine 😉

[richardTingle]

@richardTingle You made some interesting progress regarding actions and some little changes in hmd-moving How would you suggest, we should continue the work? I would really like to see jme3-xr in next release of jmonkeyengine 😉

@Starcommander yes, actions are going well, I have basic button presses and haptics working. Getting hand skeletons and hand poses working next; then I'll try to get tamarin's bound hands and grab interactions working with it. I much prefer openXR's approach to actions; suggested bindings specified in code, rather than openVRs manifest file that must be specified by an absolute path.

I found a method to get openXR to suggest eye positions and angles which is what I used in that hmd change.

On camera stuff I'm much less good, are you happy to tackle that? I couldn't get to the bottom of the lensing effects I was seeing. Do you see that too? That if you look directly at something it's in the right place but as you move your head it seems to move.

I'm fairly relaxed on it being a jme3-xr or incorporated into Tamarin (im less keen on a non Tamarin non jme library but thats just me being selfish as Tamarin is my library so if other people want that I'll go with the flow). 2024 does sound like a long time to wait though; that said an OpenVr to OpenXR migration looks like it would be quite easy for existing projects. I will at the very least want to keep the more opinionated bound hand, grab and lemur support within Tamarin.

[richardTingle]

Should this be XR_REFERENCE_SPACE_TYPE_LOCAL or XR_REFERENCE_SPACE_TYPE_STAGE. I found that it was surprising that my face was in the floor when I tried this; because XR_REFERENCE_SPACE_TYPE_LOCAL puts the origin at the headset position when the application is started but XR_REFERENCE_SPACE_TYPE_STAGE puts it at the origin of the play area (usually at floor level). Or maybe we should let the applications choose.

I thought this was a useful explanation

The difference between XR_REFERENCE_SPACE_TYPE_STAGE and XR_REFERENCE_SPACE_TYPE_LOCAL in OpenXR comes down to the origin and orientation of the coordinate system that the application will use to represent the position and orientation of devices, like headsets and controllers, within the virtual world.

In XR_REFERENCE_SPACE_TYPE_LOCAL, the origin is typically located at the position where the XR system was initialized. For example, in a seated VR experience, this could be the location of the headset at the time the application was started. The Y-axis is typically up, and the orientation is typically such that the forward direction (negative Z-axis) is facing in the initial direction of the user.

XR_REFERENCE_SPACE_TYPE_STAGE, on the other hand, is designed for standing, room-scale experiences. The origin is typically located at the center of the user's designated play area (if one has been defined), with the Y-axis pointing up, and the forward direction (negative Z-axis) is often facing a user-defined forward direction in the play area (like the direction facing the computer or the front of the room).

The practical differences a user might experience would depend on the application, but here are some examples:

In a room-scale VR game, if XR_REFERENCE_SPACE_TYPE_LOCAL is used, the user's initial position in the game might not be in the center of the play area, but somewhere near the edge or corner, because the origin is where the system was initialized, not the center of the play area. If the user moves around in the play area, they might move out of the game's designated area. Using XR_REFERENCE_SPACE_TYPE_STAGE in this case would be better, because it ensures the user starts in the center of the game's area and can freely move around within it.

If the user is sitting at a desk and not moving around (like in a cockpit simulator game), the difference between XR_REFERENCE_SPACE_TYPE_LOCAL and XR_REFERENCE_SPACE_TYPE_STAGE might not be noticeable, because the user is not using the play area. But if XR_REFERENCE_SPACE_TYPE_STAGE is used and the user's chair is not in the center of the play area, they might find themselves starting off-center in the game, which could be disorienting.

In general, the choice between XR_REFERENCE_SPACE_TYPE_LOCAL and XR_REFERENCE_SPACE_TYPE_STAGE should be based on the type of experience the application provides (seated vs. room-scale) and the user's physical environment.

Personally I always use stage even for seated applications because its easy to move the origin of the environment (what OpenVR era called the observer) to place the players head; but the local frame makes it really hard to figure out where to put the floor

[Starcommander]

At least it should be configurable for app-developers.
I tried it with XR_REFERENCE_SPACE_TYPE_STAGE, and still moving does not work.
Just the rotation starts initial in wrong direction.

[Starcommander]

I found a method to get openXR to suggest eye positions and angles which is what I used in that hmd change.

On camera stuff I'm much less good, are you happy to tackle that? I couldn't get to the bottom of the lensing effects I was seeing. Do you see that too? That if you look directly at something it's in the right place but as you move your head it seems to move.

When you found a method, then where do you use it?
I only see changes, where you just removed the rotation+distance code.
So, I also tested your modifications:

• No eye-rotation/eye-distance, so I can only see the same on left and right eye
• Moving HMD does still not work (does nothing)

Regarding the "lensing-effect" you mentioned:

• I think I can see it now, since the fieldOfView is now set to a value, that makes a better sense.
• But it has nothing to do with moving HMD, instead it appears when rotating the HMD, right?

[richardTingle]

@Starcommander That's odd. I definitely can move my head and the cameras move (and closing and opening alternating eyes gives different angles on scene, as you'd expect. This is all on the openxr-actions branch.

And in HelloOpenXRGL it sets the position and rotation for each eye

        viewPos.set(pos.x(), pos.y(), -pos.z());
        viewRot.set(orientation.x(), orientation.y(), -orientation.z(), orientation.w());
        
        Eye eye = viewIndex == 0 ? xrHmd.getLeftEye() : xrHmd.getRightEye();

        eye.setRotation(viewRot.inverse());
        eye.setPosition(viewPos);

I'm not very happy with those coordinate transformations, as JME and OpenXR are supposed to have the same coordinate system. I might try putting in the observer (the reference node from OpenVR implementation) and see if I can use that instead to get consistent directions with the axes.

But it has nothing to do with moving HMD, instead it appears when rotating the HMD, right?

True, its when you rotate your head that the lensing becomes apparent.

This is the test application I have been using (It only has suggested bindings for Oculus because that's what I've been using to test)

public class XRTestMain extends SimpleApplication{

    public static void main(String[] args){

        XRTestMain app = new XRTestMain();
        AppSettings settings = new AppSettings(true);
        XrHmd.setRendererForSettings(settings);
        app.setSettings(settings);
        app.start();
    }

    @Override
    public void simpleInitApp() {
        XrHmd.initHmd(this);

        for(int i = 0; i < 5; i++){
            Box b = new Box(0.1f, 0.1f, 0.1f);
            Geometry geom = new Geometry("Box", b);
            Material mat = new Material(assetManager,
                    "Common/MatDefs/Light/Lighting.j3md");
            geom.setMaterial(mat);
            geom.setLocalTranslation(i,0,0);
            rootNode.attachChild(geom);
        }
        for(int i = 1; i < 5; i++){
            Box b = new Box(0.05f, 0.1f, 0.1f);
            Geometry geom = new Geometry("Box", b);
            Material mat = new Material(assetManager,
                    "Common/MatDefs/Light/Lighting.j3md");
            geom.setMaterial(mat);
            geom.setLocalTranslation(0,i,0);
            rootNode.attachChild(geom);
        }
        for(int i = 1; i < 5; i++){
            Box b = new Box(0.1f, 0.1f, 0.1f);
            Geometry geom = new Geometry("Box", b);
            Material mat = new Material(assetManager,
                    "Common/MatDefs/Light/Lighting.j3md");
            geom.setMaterial(mat);
            geom.setLocalTranslation(0,0,i);
            rootNode.attachChild(geom);
        }

        rootNode.attachChild(checkerboardFloor(assetManager));

        DirectionalLight directionalLight = new DirectionalLight(new Vector3f(0.4f,-0.3f,0.6f).normalizeLocal());
        AmbientLight al = new AmbientLight();
        al.setColor(ColorRGBA.White.mult(0.3f));
        rootNode.addLight(al);
        rootNode.addLight(directionalLight);
    }

    boolean started = false;

    @Override
    public void simpleUpdate(float tpf) {
        OpenXRActionState state = getStateManager().getState(OpenXRActionState.ID, OpenXRActionState.class);

        if (!started) {
            Action teleport = Action.builder()
                    .actionName("teleport")
                    .translatedName("teleportTranslation")
                    .actionType(ActionType.BOOLEAN)
                    .withSuggestedBinding(OculusTouchController.PROFILE, OculusTouchController.pathBuilder().leftHand().xClick())
                    .build();


            Action trigger = Action.builder()
                    .actionName("fire")
                    .translatedName("fireTranslation")
                    .actionType(ActionType.BOOLEAN)
                    .withSuggestedBinding(OculusTouchController.PROFILE, OculusTouchController.pathBuilder().leftHand().triggerValue())
                    .withSuggestedBinding(OculusTouchController.PROFILE, OculusTouchController.pathBuilder().rightHand().triggerValue())
                    .build();

            Action haptic = Action.builder()
                    .actionName("haptic")
                    .translatedName("haptic")
                    .actionType(ActionType.HAPTIC)
                    .withSuggestAllKnownHapticBindings()
                    .build();

            Action pose = Action.builder()
                    .actionName("pose")
                    .translatedName("pose")
                    .actionType(ActionType.POSE)
                    .withSuggestAllKnownGripPoseBindings()
                    .build();

            ActionSet testActionSet = ActionSet.builder()
                    .name("test")
                    .translatedName("testTranslation")
                    .withAction(teleport)
                    .withAction(trigger)
                    .withAction(haptic)
                    .withAction(pose)
                    .build();

            ActionManifest manifest = ActionManifest.builder()
                    .withActionSet(testActionSet)
                    .build();


            state.registerActions(manifest, "test");


            started = true;
        }else{
            {
                BooleanActionState booleanActionState = state.getBooleanActionState("test", "teleport", null);
                if(booleanActionState.hasChanged()){
                    System.out.println(booleanActionState.getState());
                }
            }
            {
                BooleanActionState booleanActionState = state.getBooleanActionState("test", "fire", null);
                if(booleanActionState.hasChanged()){
                    System.out.println("fire"+ booleanActionState.getState());
                }
            }
            for(HandSide handSide : HandSide.values()){
                BooleanActionState booleanActionState = state.getBooleanActionState("test", "fire", handSide.restrictToInputString);
                if(booleanActionState.hasChanged()){
                    System.out.println("fire " + handSide + booleanActionState.getState());
                    state.triggerHapticAction("test", "haptic", 0.1f, 50, 0.2f, handSide.restrictToInputString);
                }
            }
        }
    }

    public static Geometry checkerboardFloor(AssetManager assetManager){
        Quad floorQuad = new Quad(10,10);
        Geometry floor = new Geometry("floor", floorQuad);

        Texture floorTexture = assetManager.loadTexture("Textures/checkerBoard.png");
        floorTexture.setMagFilter(Texture.MagFilter.Nearest);
        Material mat = new Material(assetManager,"Common/MatDefs/Misc/Unshaded.j3md");
        mat.setTexture("ColorMap", floorTexture);
        mat.getAdditionalRenderState().setFaceCullMode(RenderState.FaceCullMode.Off);

        floor.setMaterial(mat);
        Quaternion floorRotate = new Quaternion();
        floorRotate.fromAngleAxis(-FastMath.HALF_PI, Vector3f.UNIT_X);
        floor.setLocalRotation(floorRotate);
        floor.setLocalTranslation(-5,-1,5);

        return floor;
    }
}

There is definitely something wrong with the scene, the boxes seem like they are being looked at from the inside

[richardTingle]

I've got hand skeletons more or less working. Weirdly they are mirror images (So the left hand has a right hand and vice versa). Might be more coordinate incompatibilities between JME and OpenXR.

This shows the hand joints (each cube is a joint), the large green cube is the overall pose location

I don't want to go too much further without getting the cameras sorted out (especially given the coordinate transformations that seem to be being needed). Do you mind if we look at that in parallel and see what we both find?

[richardTingle]

I've asked a question about the lensing at https://hub.jmonkeyengine.org/t/openxr-vr-rendering-cameras-into-eyes/46837

[Starcommander]

@richardTingle I found out, why we see the boxes from inside.
See at HelloOpenXRGL.java: glFrontFace(GL_CW); //Clockwise
This should be: glFrontFace(GL_CCW); //Counter-Clockwise as in jme3
I will commit this modification now.

[richardTingle]

@Starcommander Nice! I bet that was why the hands were showing mirror imaged. I should be able to remove a bunch of nasty fudge factors from the actions stuff now which will put me in a much better place for getting tamarin hands working.

[Starcommander2]

@richardTingle: Hint: You have to disable GL_CULL_FACE in HelloOpenXRGL.java. Otherwise you will have a black screen, as offGeo is not rendered. Am 18. Juni 2023 13:54:42 MESZ schrieb richardTingle ***@***.***>:

…

@Starcommander Nice! I bet that was why the hands were showing mirror imaged. I should be able to remove a bunch of nasty fudge factors from the actions stuff now which will put me in a much better place for getting tamarin hands working. -- Reply to this email directly or view it on GitHub: #2012 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[stephengold]

What's the status of this PR? Is this still a work in progress, or has it stalled out?

[richardTingle]

I tried to figure out where the lensing effect was coming from, but in the end I gave up. I think to solve that I'd have to start from scratch to really understand what's going on. I'm not actively working on it, not sure if @Starcommander2 is

[Starcommander]

Is still work in progress.
Last time i fixed the issue, that we see the boxes from inside.

Also I am very interested in the configuration (Hardware, OS, xr-runtime) that @richardTingle uses.
Because I have moving not working currently, where it works for richardTingle.
Seems there is a difference in xr-runtime, where I use monado-service on Debian11.

[richardTingle]

I'm on and Oculus Quest 2, over virtual desktop to a windows 10 PC using Steam VR as my xr-runtime

[richardTingle]

I have had another go at this. I started again from the openXR HelloOpenXRGL example making sure I understood what it was doing but taking what we learned from this attempt as well.

I think I've solved it this time. I found why I was getting lensing effects; the OpenXR runtime was requesting an asymmetrical field of view (with angleLeft != angleRight) but I was using the standard camera#setFrustumPerspective which doesn't support asymmetrical field of view but when I instead calculated a projection matrix and used camera#setProjectionMatrix it stopped lensing. It makes sense to me now that @Starcommander didn't see the lensing as presumably your headset didn't request an asymmetrical field of view (or al least not so heavily asymmetric) and so the problem didn't occur for you.

We were talking before about whether this should be in JMonkey proper or in a library. I did my attempt in a tamarin 2.x branch. Are we ok with that as an approach @Starcommander & @stephengold ?

My approach adopts the existing JMonkey OpenGL context rather than creating its own which means it can be a pure app state attached "whenever" in the applications running process rather than needing to be booted up before JMonkey.

An ultra minimal example application would look like this

public class MinimalOpenXr extends SimpleApplication {
    public static void main(String[] args) {
        AppSettings settings = new AppSettings(true);
        settings.put("Renderer", AppSettings.LWJGL_OPENGL45);
        MinimalOpenXr app = new MinimalOpenXr();
        app.setSettings(settings);
        app.start();
    }

    @Override
    public void simpleInitApp(){ 

        getStateManager().attach(new XrAppState());
        // below is optional to get actions working (including hand bones). A manifest needs to be created to describe the available actions
        getStateManager().attach(new OpenXrActionState(manifest(), ActionSets.TEST_SET));

        Geometry geom = new Geometry("Box", new Box(1, 1, 1)); 
        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setColor("Color", ColorRGBA.Blue); 
        geom.setMaterial(mat);
        geom.setLocalTranslation(new Vector3f(3,0,0));
        rootNode.attachChild(geom);  
    }
}

I already have these things working:

• Non distorted visuals
• Cameras tracking headset position
• Button press actions
• Hand Pose actions (aka hand positions)
• Hand Bone positions

I am now working on these

• Moving the observer (aka moving the offset between where the player is in the real world to where in the virtual world - teleporting etc)
• Getting the hand models to pair correctly to the bones. The OpenXR bones seem slightly different to the OpenXR ones so despite already having the bone positions this doesn't seem to immediately give me working hand models. I always thought the OpenVR bones were eccentric so if openXR has fixed this I think that's fine and I'll update my model's bones.

Once I get those working I'll start updating the TamarinTestBed examples to use this. Once that all works we'll be approaching parity with OpenVR

[stephengold]

We were talking before about whether this should be in JMonkey proper or in a library. I did my attempt in a tamarin 2.x branch. Are we ok with that as an approach @Starcommander & @stephengold ?

As far as I'm concerned, the decision whether to be an Engine library or a 3rd-party library is up to you. I see a lot of advantages to being a 3rd-party library, especially if the codebase is changing rapidly. But in principle I have no objection to making jme3-xr part of the Engine.

[Starcommander]

@richardTingle Thanks for working this out.

I found out, why moving does not work with my configuration.
Seems monado-service has to be compiled with libsurvive to support moving.
https://monado.freedesktop.org/libsurvive.html
https://github.com/cntools/libsurvive
I will try this out soon next time.

When this project is working well, I still prefer to include it into the engine.

[pspeed42]

re: "When this project is working well, I still prefer to include it into the engine."

...better get it 100% perfect then since then it will go into 1.5 year release cycles and be painful to update as VR tech changes. "Sorry, that bug will be fixed next time the giant monolithic JME releases again..."

[richardTingle]

I do think I agree with @pspeed42 and @stephengold ; I think I plan to continue with a Tamarin 2. But that doesn't preclude other XR projects like this one. It might make sense to have a move-fast-and-break-things Tamarin 2 and a more stable JME3-XR

[Starcommander]

I have now compiled monado-service with libsurvive support.
Now moving is also working well in my configuration.