Babylon.js version of ARnft

[kalwalt]

I'm working also on this, you can test it here: https://github.com/kalwalt/ARnft/tree/babylonjs-renderer

[eddiecarbin]

Great work!!! I like how you implemented BabylonJS renderer. I am taking notes and will compare them to what I did. I have a project to share with you soon. Good job!

[kalwalt]

Thank you @eddiecarbin ! But still the model is not displayed, i can not find the reason. If you have an idea let me know. Please post here when you finish your project. 🙂

[eddiecarbin]

haha yeah, I downloaded it and noticed that. Though your great new design pattern made me rethink a few ways that I am doing things. I need to finish one more feature which should be today and I will share it. :)

[eddiecarbin]

Here is the project I created with your framework.

https://sites.google.com/view/science-of-pixar/home

I built a proper framework around ARnft using typescript and it requires a bundler. I'll try to extract the necessary Babylon code into a single page app. We are trying to make a product. Most AR frameworks give small examples or experiments, at least the open-sourced ones. Except for AFrame is it ready for a production environment, but I was not too fond of the idea of creating a game around a markup language.

First thing I noticed you were setting the camera matrix with the response from the Worker. I found that the returned matrix doesn't work for the Babylonjs camera. The matrix corrupted the renderer. I asked on the Babylonjs forums, but no answer. I ended up not setting the camera matrix. I wanted to talk to you about this.

https://forum.babylonjs.com/t/problem-rendering-normals-polys-flipping-around/15897

I see that you are using the Universal Camera. I stuck with the simple Camera object.

As for your 3d model, I would start with rendering Babylon primitive to work before you begin importing Gltf models. That's what I did.

The other thing you do is spin the camera around one object. I had to reverse the method and spin the object with the calculated worker matrix. My project had a requirement of detecting multiple NFT cards in a scene. The way ARnft examples currently work, this would be a problem. It is great that it works with Workers, so I was able to build multiple NFT detections.

Let me work on your examples and ill give you more feedback soon.

[kalwalt]

Here is the project I created with your framework.

https://sites.google.com/view/science-of-pixar/home

Nice project indeed, thanks for sharing!

I built a proper framework around ARnft using typescript and it requires a bundler. I'll try to extract the necessary Babylon code into a single page app. We are trying to make a product. Most AR frameworks give small examples or experiments, at least the open-sourced ones. Except for AFrame is it ready for a production environment, but I was not too fond of the idea of creating a game around a markup language.

I think Babylon.js has a lot of examples, or you can't find examples that fits your needs?

First thing I noticed you were setting the camera matrix with the response from the Worker. I found that the returned matrix doesn't work for the Babylonjs camera. The matrix corrupted the renderer. I asked on the Babylonjs forums, but no answer. I ended up not setting the camera matrix. I wanted to talk to you about this.

https://forum.babylonjs.com/t/problem-rendering-normals-polys-flipping-around/15897

I see that you are using the Universal Camera. I stuck with the simple Camera object.

Maybe some values of the Camera matrix need to be reversed, but i can not be sure about this. So you set the Camera by your means. Maybe @ThorstenBux may have an idea?
I'm using the UniversalCamera, because i red that has more settings.

As for your 3d model, I would start with rendering Babylon primitive to work before you begin importing Gltf models. That's what I did.

Yes, to keep simplicity.

The other thing you do is spin the camera around one object. I had to reverse the method and spin the object with the calculated worker matrix. My project had a requirement of detecting multiple NFT cards in a scene. The way ARnft examples currently work, this would be a problem. It is great that it works with Workers, so I was able to build multiple NFT detections.

Yes, that it is a similar approach used also in AR.js: you have two possibilities, setting the CameraMatrix or ModelMatrix. If your project needs Multiple NFT markers i can implement it in JsartoolkitNFT and then in ARnft. We have this feature in our jsartoolkit5 see my PR for the example webarkit/jsartoolkit5#10 I need only to complete another PR #143 and then i will make that changes.

Let me work on your examples and ill give you more feedback soon.

Super!

[kalwalt]

@eddiecarbin WIP PR #150 if you want take a look at.

[eddiecarbin]

I think Babylon.js has a lot of examples, or you can't find examples that fits your needs?

I was referring to this project. The examples are more of a showcase for jsartoolkit.

Okay, I was able to look closer at what you have. To take a step back and remember why I chose this framework out of all the other open-sourced AR projects out there. What I remember was that this project was active, and all I needed was NFT. This project used a revived jsartoolkit and worked well. You can look at my project code. I followed some of the code you did to learn how it works. I just needed the artoolkit.wasm_worker, and that was it. If this project is just to show the capabilities of jsartoolkit? It works then. I believe you can make a better interface or wrapper for jsartoolkit.

I think you are going in the right direction with the new ARnft and Utils class. They don't follow solid OOP design patterns, making it hard to scale for a more significant project. Next weekend my time will free up, and I can write a proposal of changes for you.

The classes are handling too many features. For example, ARnft shouldn't have references to the ThreeJS or Babylon libraries. Keep it ARnft agnostic from the application. It should just do the AR math work. I should pass in an image to ARnft, and it returns a model matrix. Let the game/application do the 3d rendering, web camera, and even update loop. You will find many benefits to doing this, like allowing us to make multiple instances of ARnft and track more than one image. This concept will make it easier to use your framework.

Right now, the WebXR image tracking is in the experimental stage in chrome. I am guessing in 6 months, it will be released, and I will be implementing that into my projects. Besides being native to the browser, it has a ton of advantages. I will probably still use an AR framework like ARnft because IOS has not adopted WebXR, and I have to support both platforms.

I would also suggest not naming the class Utils. Every engineer makes their own utils class. Unless you hide your utils behind a namespace it will cause problems. Name it ARUtils or something, but after reading it doesn't read like it's a Utils class. It does to many core AR functions.

[kalwalt]

Hi @eddiecarbin thank you for your in-depth reply. I started this project because i wanted to develop a lighter and simpler version of jsartoolkit with only NFT as you know. consider that i am not a professional developer and my formation came for another field. Probably another develop with more experience would start with other basis this project. This only to say that i understand your point and i'm open to a wide sight of the project.

I think you are going in the right direction with the new ARnft and Utils class. They don't follow solid OOP design patterns, making it hard to scale for a more significant project. Next weekend my time will free up, and I can write a proposal of changes for you.

You are welcome, and if you are intereted you can join our WebARKit team. ARnft is part of WebARKit now and it is a pleasure that other people want to collaborate. We have a slack channel also, i will post the invite link.

The classes are handling too many features. For example, ARnft shouldn't have references to the ThreeJS or Babylon libraries. Keep it ARnft agnostic from the application. It should just do the AR math work. I should pass in an image to ARnft, and it returns a model matrix. Let the game/application do the 3d rendering, web camera, and even update loop.

Yes that could be interesting. but i started ARnft to simplify the code, and to offer some simple method to quicly have some result. I will wait your proposal plan to understand better your idea.

You will find many benefits to doing this, like allowing us to make multiple instances of ARnft and track more than one image. This concept will make it easier to use your framework.

We should make the changes that are in our jsartoolkit5 repository: https://github.com/webarkit/jsartoolkit5/blob/7f998d00582d86fe95d573e539ddeea9380ccc4e/js/artoolkit.api.js#L579-L585
and PR for examples webarkit/jsartoolkit5#10

I would also suggest not naming the class Utils. Every engineer makes their own utils class. Unless you hide your utils behind a namespace it will cause problems. Name it ARUtils or something, but after reading it doesn't read like it's a Utils class. It does to many core AR functions.

We can change it. Part of the code came from artoolkit5-js and it is very similar. i haven't the time to think about this aspect.
To conclude, me and @ThorstenBux are developing the new WebARKit, not based on Artoolkit5/Jsartoolkit5 anymore. ARnft is a transitional form. Experimentig repository: webarkit-testing. It's not yet working, but just to say that we are in that direction.
I hope to find you well, waiting for other news.... 🙂

[kalwalt]

I think slack link in our website is expired, join us at this link

[eddiecarbin]

OK, so I was able to make a quick sketch and propose a new framework for ARnrt. I already started a new project I can show you soon. This is what your one-page sample HTML script would look like when it is done.

async function init() {

  //  pick your renderer,  babylonjs or threejs
  const renderer = new BabylonJSRenderer();
  await renderer.initialize();
  
  // init the cameraview
  const cameraView = new CameraViewRenderer();
  await cameraView.initialize();
  
  // init arnft and add video.
  const arnft = new ARnft(cameraView.video);

  // select ar marker targets
  const duckModelMarker = new ModelARMarker ( renderer, duckJsonData )
  arnft.addMarker ( duckModelMarker );
  
  const videoMarker = new VideARMarker ( renderer, videoJSONData );
  arnft.addMarker ( videoMarker );

  const imageMarker = new ImageARMarker ( renderer, imageJSONdata );
  arnft.addMarker ( imageMarker );

  await arnft.initialize();

 // we are done loading now start update loop.    

  var tick = () => {
        arnft.update();
        setTimeout(tick, 1000 / 60);
  };
  tick();      

}

I have a little more work to do. I removed the 3d renderer, camera setup, and update loop giving the user more control over what they want to do. I also designed ARNft to be more flexible in use, it only manages the AR work. This single responsibility will give the ability to do more. For example, working with multiple marker targets and media.

[kalwalt]

Ok @eddiecarbin let me know when you post the whole code, thank you.

[JunkYardRobotBoy]

Hello @kalwalt and @eddiecarbin
Wanted to politely ask when you think a Babylon HTML Sample for the latest version of this might be released?
Been following your progress for a few months now and would love to use this in Babylon JS
Thanks

[kalwalt]

Hello @kalwalt and @eddiecarbin
Wanted to politely ask when you think a Babylon HTML Sample for the latest version of this might be released?
Been following your progress for a few months now and would love to use this in Babylon JS
Thanks

I can't say when It willl be available, but i can confirm that Is on my plans. For now i have the priority to release the typescript version, and then i willl work again on the Babylonjs feature. Anyway It will be a separate package: ARnft- babylonjs

[albjeremias]

i really love the discussion, but would rather see pull requests on more clear issues :) i'd close this one too!

[kalwalt]

i really love the discussion, but would rather see pull requests on more clear issues :) i'd close this one too!

We can move this to the discussions board, i think it is more convenient.