Tracky

Record an ARKit session for reconstruction in Blender

Background

We developed this tool because we wanted to experiment with new AR ideas
faster. By capturing an ARKit session (including its depth data, its
segmentation masks, all of its positional and orientation streams, stereo
microphone audio, and any detected planes) we can reconstruct it in Blender.
Once in Blender, we can quickly iterate on concepts and see the results as they
would look in an ARKit session.

Prerequisites

• XCode 14.2 or higher
• Blender 2.92 or higher
• An iPhone or iPad with LiDAR

Basic Use

1. Launch Tracky on your iOS device:

2. Tap the record button (after waiting for the app to find tracking planes):

3. Use the Files app to navigate to the Tracky directory, and transfer your latest recording to a computer with Blender installed:

4. Import the .bren file in Blender:

Build

1. Open the Tracky.xcodeproj XCode Project:

2. Plug in a device and select it as the target:

3. Hit play to build it:

Install (Blender)

Tracky outputs .bren files with all of the recorded data from the ARKit
capture. This file can be imported into Blender via the use of the included
Blender plugin.

1. Open Blender Preferences:

2. Navigate to the Add-ons section:

3. Click the "Install..." button:

4. Navigate to the BrenImporter/bren_importer.py file from this checkout:

5. Click the "Install Add-on" button:

6. Important: check the box to enable the add-on:

You're all set! Now you can import your .bren files output by the Tracky app.

Sample Tracky Output

• Sample Clip 1
• Sample Clip 2

Fun One-Minute Tutorial

https://user-images.githubusercontent.com/1228/223236400-98469d59-504d-4276-8221-45e035c031fb.mp4