ARKit Photogrammetry

3
votes

For the last month, I have been researching ways to create 3D reconstructions from live video. It seems to me like the only option is to use a 3rd party service like Agisoft or Autodesk Capture. However, after the release of ARKit 2.0, I saw this tweet. In this tweet, Tim Field claims that he used ARKit photogrammetry to create 3D models. From what I know, this is impossible. I have only used ARKit to scan 3D objects and create point clouds. In fact this is what the ARKit documentation says exactly:

An ARReferenceObject contains only the spatial feature information needed for ARKit to recognize the real-world object, and is not a displayable 3D reconstruction of that object.

If what Tim Field is saying is true, how can I use ARKit to create 3D models using photogrammetry?

1
computer-visionaugmented-realityarkit3d-reconstructionphotogrammetry
You might notice that said tweet predates the announcement of ARKit 2. Most likely, that demo is using a fair amount of custom processing on top of what ARKit provides, attempting to match per-frame pixel buffers to feature points and integrate them over time. – rickster

1 Answers

1
votes

Apple RealityKit 2.0 got Object Capture API. It provides us with the long-awaited photogrammetry tools. At the output we get USDZ model with a corresponding texture.

ARKit 5.0 and 4.0 allows you make a Scene Reconstruction in real time if you're using a device with a LiDAR scanner. Scene Reconstruction occurs in nanosecond speed. A distance range is up to 5 meters (sometimes even more). For details watch this video.

One of the key features of a LiDAR scanner is: it helps you eliminate everyday problems that you encounter at tracking and scene understanding stages. For instance: there's no need for physical movement and quality lighting conditions when tracking using LiDAR.

When reconstructed scene is ready, you get a polygonal geometry with Occlusion Material that you can immediately use for object occlusion, physics simulation and light estimation.

In addition to the above ARKit 5.0/4.0 gives you Depth API for managing a high-quality Depth channel, that now runs at 60 fps. Read this webpage for more information.