Porting a content summarization mobile application to Apple Vision Pro

Exploring use cases for Apple Vision Pro, we decided to bring our mobile AI-powered summary generator to the immersive realm.

Porting a mobile application to Apple Vision Pro has many considerations, starting with defining the spatial moment and adjusting the level of animation detail. The following is an overview of the steps we took to port an application from iOS to Apple Vision Pro.

What Is Taim Spatial?

Taim is a mobile application that simplifies how users consume digital content. The app uses ChatGPT (GPT 3.5) to create brief summaries of articles and YouTube videos. Offering summaries in two formats and three length types, Taim caters to different user preferences for detail. Taim is an efficient tool for accessing key information and decreasing information overload.

By extending Taim’s operating platforms to include Apple Vision Pro, we could deliver the app's decluttering capabilities to a broader audience.

Step 1: Defining the Spatial Moment

When porting an existing application to Apple Vision Pro, the first step is often the identification of the spatial moment.

The spatial moment is an experience that deeply resonates with users and can only be realized in an immersive XR environment. The counterpart of the spatial moment in the mobile app world is the mobile moment.

While identifying the spatial moment usually commences an XR project, it isn’t the only way to start the process.

For example, with Taim Spatial, we didn’t need to figure out the spatial moment because we could focus on solving the pain point instead: quick access to summaries of lengthy content while multitasking within the XR environment.

Step 2: Determining the Type of Application

Applications on Apple Vision Pro have three types of views with different approaches to content presentation:

• Windows
• Volumes
• Spaces

Each view type has different features and availability of user interaction methods. The views also have defined options in how they interact with the user environment (e.g., a Windows app view can be pinned in a specific location in an apartment).

Taim Spatial needed to be viewed in two application types: Windows and Volumes. Here’s how each app type of Taim Spatial works on Apple Vision Pro:

• The 2D UI Windows app is where the user gets the written summary.
• The Volumes app is an animated assistant where the user asks questions and receives answers in real time.

Step 3: Choosing the Technology Stack

Now that we knew how Taim Spatial would work on Apple Vision Pro, we could pick the technology to build it on. To add immersive 3D elements we used RealityKit, with the app’s UI created in SwiftUI.

UI design

We chose SwiftUI because Apple has created a very comprehensive design system with detailed documentation for its headset. The guidelines helped us apply the spacing, fonts, and contrasts for the most optimal user experience. The integrated design system let us build the user interface quickly, resulting in a pleasant native UI.

Challenges During Development

The biggest challenge was designing the experience for an entirely new platform. Since Apple Vision Pro isn’t commercially available yet, we had to rely on our experience in XR development and the simulator to build the application.

For the first on-device testing, we signed up for the official Apple Vision Pro developer labs held on October 23 in Munich. During developer labs, we could launch the application and identify potential UX issues and further adjust the app to Apple’s headset. Having a couple of hours with the headset let us do real-time fixes.

Another challenge was with the animations and the level of detail that could be achieved with RealityKit. In essence, the more sophisticated the animations, the harder it is to implement the details using RealityKit. Conversely, Unity supports complex animations flawlessly.