[WIP] ARKit support #9967
[WIP] ARKit support #9967
Conversation
|
Owh one thing I forgot to add here. ARKit requires iOS 11 and XCode 9 beta to run. That is a bit of an issue as well, it needs to be possible to make ARKit an optional compile. |
|
This is the test project I used in the video that I posted: Note at this time the "green screen" background that I abuse to overlay the camera. That is only temporary until I get the camera to render properly as the background of our main render buffer. |
|
Pretty much got everything working at the moment. I haven't had time to rebase it yet, planning to do that over the weekend. Plane detection now works, the new ARVRAnchor object will probably get its own PR soon and needs to be enhanced to extract the scaling. Discussed a possible solution to improve the way the camera image is implemented with Reduz today, we'll see how easy that will turn out to be implemented. |
BastiaanOlij
force-pushed
the
arkit
branch
3 times, most recently
from
d8436b7 to
3bff1c2
Compare
|
Sorry for the little movement on this. Work has been keeping me busy, I'm slowly working on a new framework so Godot can capture video feeds and use those with one of the goals of feeding ARKits camera frames into this (as ARKit already does this, no use in doubling up and using the camera directly). Once I have finished that I'll round of the ARKit implementation. |
|
This is still quick and dirty but I redid some of the code so this now works with my camera server implementation. Its rebased to todays master and provided you have XCode 9 installed (which is no longer in beta) this should compile. I'll be spending more time polishing this up as I've still cut a few corners, but it is really coming together now! |
BastiaanOlij
force-pushed
the
arkit
branch
2 times, most recently
from
54ba3d8 to
35eae67
Compare
|
Just a few bug fixes and a rebase.. |
BastiaanOlij
force-pushed
the
arkit
branch
3 times, most recently
from
0a3fd1e to
bc8105d
Compare
|
Rebased, and applying the tweaks we're working on in the arvr server. |
|
Just a rebase, no changes. |
BastiaanOlij
force-pushed
the
arkit
branch
3 times, most recently
from
18685e5 to
6dafdd7
Compare
|
Thanks a lot for your contribution! Moving this PR to the 3.1 milestone, to be reviewed once the release freeze is lifted. It could eventually be cherry-picked for a future 3.0.1 maintenance release if it doesn't change the user-facing APIs and doesn't compromise the engine's stability. |
|
Rebased but can't test yet, issues with exporting |
|
PR #24227 has superseded this now that all code builds for Godot 3.1+ |
This is far from finished and very, very rough, but I figured I'd submit it for people to start taking a look at and coming up with better solutions for a few of the issues.
Functionality everything is there except for plane detection, that shouldn't be hard to add in. For me the main goal was to get the tracking up and running and the camera.
The camera isn't implemented the way I would like to, right now I'm taking the render buffer created by Godot and overlaying it ontop of the camera image during the final copy to screen. I don't believe this is the right way about it. I need to find a way to make the camera image available to the render engine so it can render the geometry on top of the image. The really annoying part here is that the image is provided by the camera as a YCbCr image so it needs to be converted to RGB. Since we get a new image almost every frame I'm doing this in the shader itself.
Basically everything in arkit_shader.h and arkit_shader.cpp should be solved differently
The other thing which right now is fairly dirty is the way everything hangs together. For 90% ARKit actually works with a pull mechanism but the remaining 10% required some ugly hacks. I'll be cleaning that up soon.
Finally there are a bunch of rendering issues but they are issues on iPhone in general. One specific to ARKit is that without a background (it being the camera image of the phone) there is no HDR image to do nice reflections with. I'm uploading a video to YouTube to show off the tracking, will post the link as soon as it finishes.
https://youtu.be/QCH5wOYeIdQ