Final Article
I wrote a Medium article about my findings:
https://medium.com/@ava.mirzaee.cheshmeh/what-ar-my-options-859089cb3ab2
Picture from one of my AR.js demo projects.
As a student developer, I often depend on resources created by other developers, when diving into a new topic. So, for the final project in my development course, I wanted to try and return the favour.
Specifically, I wanted to start experimenting with image tracking in Augmented Reality (AR). The final result is exactly what you are reading right now: the summary of the findings I made along the way. Hopefully, this article can help as a guide for other developers who want to dive into AR for the first time.
NOTE: This article serves as a short summary of the findings I made during the 3,5 weeks I spent on this project. To read about this in more detail you can look at my github wiki blog, or to look at the code of my demo projects, you can look on my github repo.
By means of making small example demos, I explored the image tracking functionalities of multiple AR frameworks. My goal here was to find a clear way to compare similar functionalities between the frameworks. To do this in a structured way, I made a list of which features or aspects I found most important to compare:
- DEVELOPMENT EXPERIENCE
- How is the documentation, is there a big community, is it easy to use/learn? Do you need a lot of prerequisite knowledge and can you easily incorporate it in other frameworks and projects?
2. AR SPECIFIC QUALITIES
- How accurate is the image tracking itself? Is there any lagging? Are there any limitations to the target images you can use?
Furthermore, in my preliminary research into the topic of AR, I found two important categories: web AR and native AR. So, apart from more specific differences between AR frameworks, I wanted to see from my experiments how these two AR options in general compared to each other as well.
For context, I am a third year Devine student majoring in development, with no prior experience with AR or native app development. I am most familiar with css, html and JavaScript and I have worked a lot with React before.
Some new things I experimented with throughout this project are:
- React Native and native app development in general.
- Next JS
- Unity
There are a lot of people who can and have explained already what AR is, much better than I could, such as here. In short, Augmented Reality or AR, is a way to add digital layers to your physical environment. Within AR, you can also make the distinction between marker-based and marker-less AR, which basically refers to whether the AR is triggered by some physical markers that are used as anchors to place the digital effects on, or whether the AR uses the geometric properties in the room, so that the effect will be placed on a random flat surface around you, for example. Thus, image tracking falls under the category of marker-based AR.
To find out which AR framework is best suited for you and your project, one of the first and most important distinctions you will need to make is whether you want to go for a web-based application or a native one. First and foremost, this choice will depend on the nature of your project. If you already have an existing web or native project that you need to add an AR component to, you might not really have a choice.But, if you are still on the fence or don’t have any preliminary restrictions set up, these are some of the things to keep in mind to make your decision:
- Download threshold
One of the benefits of going for a web-based solution over a native one is the elimination of the downloading threshold. At the current time, people are less and less prone to download an app. It makes for an extra added step, before reaching your desired content and takes up precious storage space on their device. AR also often starts from a QR code that needs to be scanned. So, if you have your application running on the web, this QR code can take you directly to the AR experience, without the hassle of needing to download an app first.This can also be important if your service relies on a big part of the public who are not yet familiar with your business or product. By going web, you will increase the share-ability of your experience and have more chance of engaging with a part of the public that is unaware of your presence yet.
2. Device range
Native AR-based apps are very limited in which devices can run them. You do not have this same limitation for the web. If you have a device with a camera that can run a browser with webRTC and webGL support, you can use most AR web application.
The limitations for native AR apps are much more restrictive. In general, it comes down to the support of the Android and iOS AR software development kits, ARCore and ARKit respectively. To know which devices specifically are supported, you can look here for ARCore and here for ARKit.The one benefit that native apps have in this regard over web-based applications, is the fact that for native apps, you can eliminate devices beforehand and make sure that your app can not even be downloaded by devices for which it won’t work, which makes sure that the person behind the screen does not get ‘disappointed’ by a message stating ‘Sorry, this will not work for your device’.
3. Development experience
In general, native app development is much more complicated than web-based. You need different versions for each operating system. Updating applications is also much more difficult. You need to rely on the user itself updating your app. Furthermore, each update needs to be accepted and published to the app platforms where your app can be downloaded.
This also has an effect on testing out your applications. Specifically for AR, as the device range is very limited and relies on camera access, you will need to own the right devices to be able to test your native application correctly. In some cases device emulators can be possible, but in my experience the Android Digital Devices from Android Studio were a hassle to set up and were really slow to run on my laptop. As for iOS, you do not even have the option to use an emulator if you are dependent on camera usage.
4. Performance
The point where native does shine is AR performance. As a native application runs on your device’s operating system, it directly has access to your camera and will run your AR application in the most efficient way. This results in better performance with almost no lagging for the image tracking functionality. Though this is not a feature of all native AR frameworks, in some cases the image tracking is done in such a way that the 3D or other digital elements you add to your tracking image still remain visible, even when your target has gone out of view of the camera. This makes the effects feel more real, as if they are really part of your environment. So, for the most immersive and reality based AR experience, I would say native wins!
5. Camera access
Whether you are working with a native application or a web-based experience, you will need to ask camera permissions from your user. However, how many times you need to get that permission does differ. For native applications, accepting once is enough. For a web-based AR experience, dependent on your browser, it will ask you from once per session, up to once per page refresh.
Of course, there are a lot of other aspects that you can take into account for your choice between web or native, such as SEO, offline use, notifications, … However, in the context of this article, I wanted to mention the most prevalent ones I encountered, specifically related to my AR based exploration.
If you really want to know which AR framework is right for you, I hope the following results can help you out. I compared the basic image tracking functionality of 5 frameworks in total: 3 web based (MindAR, AR.js and 8th wall and 2 native solutions (ViroReact, Unity’s AR Foundation). I hope these results can help you decide which option is best for you.
WEB AR FRAMEWORKS
Comparison of web AR frameworks
NATIVE AR FRAMEWORKS
Comparison of native AR frameworks
For a more detailed comparison table, check out my github wiki page. If you are more interested in a quick look via videos, live demos or code, check out this blog post.
If you don’t want to read my whole article and just want to know quickly what my thoughts are, here are the most important findings according to me.
WEB vs NATIVE
Web AR vs native AR comparison summary
To end, I would like to add some tips for other starting developers like me who would like to start exploring a development topic they haven’t tried before.
- You will get stuck. Many times. Sometimes you will find a way out, and sometimes you will reach the limitations of your own knowledge or the framework itself. Or both.
- Development is about more than performance. In the beginning, when I wanted to make a comparative analysis of AR technologies and frameworks, I assumed most of my findings would relate to the actual functionalities of the frameworks or the specifics of how well the image tracking worked and how fast. And while that is definitely a part of it, I found the ‘development experience’ just as important as a comparative factor.
- Don’t go too fast. To give myself some direction in the beginning, I worked on a small concept demo that incorporated AR image tracking in some way. So, in the beginning, I wanted to work too fast to get that AR functionality immediately embedded in my full example demo. However, this meant that I would often want to jump a few steps ahead. I needed to take a step back and first try out the very most basic examples I could find, and then work my way up to adding elements and seeing whether I could incorporate it more easily in my project. Sometimes I was able to do this and sometimes I wasn’t.
- Testing an AR application can become tricky if you want to try it out with your own target images. It helps if you already know that your image can be tracked or recognised. If you are not sure of this, you will need to check sufficiently if the problem is caused by the target image itself not being recognised, or your AR functionalities not working correctly.
- Almost all web AR frameworks depend on A-frame and/or THREE.js. So, having prior experience with these frameworks will be a big help. Though it is not necessary for all of the frameworks I tried out, just looking at the basics can save you some time.
- At first, I thought that the result of my comparison would be ‘framework A is the best, because of reasons X, Y and Z, so use this one’. The reality is more nuanced. A comparative study mostly has the goal to get a clear overview of the differences and aspects of multiple frameworks, so that you can hand the tools to someone to make an informed decision of what is best suited for their specific case.
I hope you enjoyed reading my first Medium article! If you want to, you can check out some of my other work over here.
GOAL
This article is meant to give a comparative analysis of different AR frameworks, focussing on the image-tracking functionality.
WHICH FRAMEWORKS DID I TRY?
For WebAR:
- mindAR
- AR.js
For native AR:
- Viro-React
(THE DESIRED RESULT
For my analysis, I focused on a small application goal I wanted to hypothetically be able to achieve: a Mystery Mail service, that shows a secret AR message on top of the front image of a physical, printed out post-card. With this goal in mind, I wanted to find the best way to create such an AR project.)
INTRO
As a student developer, I often depend on resources created by other developers, when diving into a new topic. So, for my final project for my development course, I wanted to try and return the favour.
Specifically, I wanted to start experimenting with image tracking Augmented Reality (AR). So, instead of just delivering a single, fully fleshed out end-result, I made the research itself my project. The final result is exactly what you are reading right now: the summary of the findings I made along the way.
Hopefully, this article can help as a guide for some other unexperienced developers when they want to dive into AR for the first time.
PREREQUISITES & LIMITATIONS
I am writing this article from the standpoint of a student developer. My pre-existing knowledge is mostly
- html, css
- JavaScript
- React
Furthermore, I was working within a limited time frame of 3,5 weeks. Within this time frame, I also experimented with some frameworks that were new for me:
- React Native
- Next.js
I had no previous experience with AR.
This is just mentioned to give some background about the pre-required knowledge for this article.
If you didn’t know already, Augmented Reality or AR, is a way to add digital layers to your physical environment.
Initially, I was under the impression that Image Tracking is a subcategory of Marker-based AR, as it often described this way.
When you want to develop an AR project, one of the most important choices you need to make is: do I need a native app for this or can I do this on the web?
Currently, there are already a lot of working options for Augmented Reality on the web. So, you do not need to rely on native applications anymore to be able to make an AR based experience.
These are some of the benefits I encountered for webAR
-
No need to download an app:
At the current time, people are less and less prone to download an app. It makes for an extra added step, before reaching your desired content and takes up precious storage space on their device. AR also often starts from a QR code that needs to be scanned. If you have your application running on the web, this QR code can take you directly to the AR experience, without the hastle of needing to download an app first.
-
Share-ability of the experience
Especially in my specific case, I relied on a user group receiving an AR experience, that might have been unfamiliar with the service providing the experience. This means that there is a smaller chance that the intended user already has the app pre-installed. So opting for web in this case, can more easily reach new users, who did not know about your service before.
-
Better SEO
Discoverability of apps is still very difficult via the app store, so a web experience will have much better SEO (Search Engine Optimisation) and will more easily be discovered by people.
-
Bigger device range
Apps are very limited in which devices can run them. You do not have this same limitation for the web. If you have a device with a camera that can run a browser with the right support, you can use the AR web application.
-
Cheaper and easier development
Since native apps differ and depend on the operating system of your device, it is harder and more resource intensive than developing a website. But not only is the cost much lower for a web experience, it also requires much less maintenance: you only need to update one project and you can update it automatically for all your users at once.
However, there are also some reasons why a native app might be a better choice for you
- Performance
- Security
- Offline use
- mindAR
I found mindAR via the official AR.js github. It is mentioned as a new, super easy framework, specifically for Image tracking and Face tracking. They position themselves as a framework that is as easy as writing plain html.
-
open source
-
Free
-
Limited in quantity, but clear documentation. Very limited examples.
-
Small community
-
Very easy in plain HTML, difficult to implement in other frameworks, such as React.
-
Only Image tracking and Face tracking
-
Options for tracking images is not limited. Easy to convert your own image via their Image Target Compiler tool: https://hiukim.github.io/mind-ar-js-doc/tools/compile/
-
Pre-required knowledge:
- Can be understandable with just html knowledge
- A-frame or THREE.js knowledge is a plus, but not needed.
-
‘Google-ability of your problems’: Very low. The community is still very small. At most you will encounter someone who has the same issue as you, but without any solution for it for the time being.
-
My opinion: Despite the limited community, I found this library very promising for those who want to quickly set-up a simple webAR project with just some basic html knowledge. The image tracking accuracy is good, you can use your own images and the structure is very self-explanatory and clear, even without any previous background in A-frame or THREE.js. There is however still a lot of work to be done if you want to expand upon the basic simple examples they offer, such as integrating it in other frameworks or via ES6 modules.
-
Code simplicity
The set-up for a simple Image tracking example is as short as a few lines in html. It has the following basic structure (dependent on some A-frame syntax)
<a-scene mindar-image="imageTargetSrc: assets/custom/multi.mind; uiScanning: #example-image"> <!-- A scene object to group your AR set-up. Here you specify which image target you use. This .mind file can contain the info of multiple images together, you will later need to specify the index of your target --> <a-assets> <!--Option to preload the assets you want to use --> </a-assets> <a-camera></a-camera> <!-- This adds the camera object --> <a-enitity mindar-image-target="targetIndex: 0"> <!-- The object that groups the AR effect, linked to a certain image target. You specifiy the index here. --> <!-- Here you add the elements that you want to show on top of the tracking image. These can be any A-frame elements, such as gltf 3D models, images, text --> </a-entity> </a-scene>
- AR.js
I encountered AR.js as one of the biggest currently free AR frameworks for the web. They position themselves as the open-source, free AR framework that should be able to do everything from marker-based, Image tracking, Location based and Face tracking AR on the web.
- open source
- Free
- Limited documentation, both in quality and quantity. No clear starting guide. Demo examples do not work currently.
- Fairly large community.
- Marker tracking is fairly easy. Nft tracking did not work for me within my time-frame and knowledge.
- Marker tracking, NFT image tracking, Location based AR, Face tracking
- Since only Marker based tracking worked out for me, I can only talk about the requirements for these types of images. You can create your own custom markers via a convertor (https://ar-js-org.github.io/AR.js/three.js/examples/marker-training/examples/generator.html) mentioned in their documentation. However, I did notice that for your custom marker to be recognised within the tracking functionality, you really need to test it out and make sure the thickness of the border has the right ratio and the contrast of your image is high enough. For me, I found that a 0.5 ratio of border to image works best, combined with a simple illustration or text, with hight contrast between the background and the simple elements.
Some tips/things I learned in this process
-
Take it step-by-step. In the beginning, I wanted to work too fast. I wanted to immediately try to implement the AR logic into an existing react or Next.js project. When hitting a wall in this way, I took a step back and started from separate plain html examples, and worked on that until I had a working example and then looked from there how it could be incorporated within my existing demo project. Sometimes this worked, and sometimes it didn’t. But then at least I knew whether it worked in a simple example or not and I could more easily figure out my mistakes.
-
SSR and AR
For this project, I wanted to explore Next.js as well, as a commonly used framework for making web-based applications. One of the features of Next is server side rendering (SSR), which has its benefits for creating websites. However, you do need to be careful when combining it with AR. Since most AR frameworks depend on client side properties, in my experience, I needed to dynamically load my AR component, without SSR, to make sure there weren’t any issues of the AR framework needing to access client-specific properties, which are not accessible yet, when the content is first rendered on the server.
Things I did not have the chance to look into: interaction events and UI that can
Unity,
even more of the existing frameworks
Aframe and THREE js dependency!!
GOALS
So, what did I want to achieve in
WebAR vs native AR
You might think your question will be, which is the best way to do AR? Well, because of the current progress, web AR has come to a similar level of quality as native AR frameworks, so your question might actually first rather be: do I need a native app for this or not? And to be truthful, if an app is not absolutely necessary for your product, I believe a web app alternative will be able to respond to all your needs.
Web vs native
- permissions
- notifications
- download threshold
- local storage
- current ‘app’ culture (remove after a month)