Apple Vision Pro Features List for App Creators — A Deep Dive

This comprehensive guide is designed for people looking to envision a new Apple Vision Pro application or service for existing or new business. It's a summary at a level you won’t find anywhere else online, perfect for product managers, UX designers, and entrepreneurs.

Job to Be Done

Here, I delve into the feasibility aspect of the Apple Vision Pro app development, highlighting what's possible and what's not.

As you embark on the ideation journey for your application or service, this content will be a beacon, ensuring you're well-informed and equipped. It can also simply help you to better understand AVP and add it to your toolkit.

I've poured over Apple's documentation to provide you a condensed high-level summary of the features and building blocks available to Vision Pro developers. This ensures that as you conceptualize, you're fully aware of the tools at your disposal, optimizing your app's value to users and your audience.

Some feature descriptions come directly from Apple's documentation, while others are edited for clarity. Always make sure to test the specifics of these features in a simulator or on the device before making significant commitments. For every building block, the supporting Apple technologies are provided, making further research a breeze. This list is curated based on version 1.0 of visionOS.

Unsure about a feature or need clarity? Don't hesitate to ask us via our contact form. We prioritize your queries, aiming to respond swiftly (often within hours) without any strings attached!

List of contents

The details of the available visionOS features are organized in the following sections:

1. General Platform Features

2. Traditional 2D UI Support — Windows

Standard 2D UI components are necessary for most of the apps. VisionOS provides beautiful high usability 2D Windows with familiar UI controls.

These Windows are mostly 2D, with two exceptions:

• It’s possible to display static 3D objects on top of those windows — great opportunity for product presentations!
• Those 2D layouts entertain beautiful depth effects (for example: a glass panel is not flat, has depth). I recommend checking it out by watching them from the side.

It’s worth mentioning here that native UI on the Vision Pro headset is built with SwiftUI, which provides a cross-platform (yes!) and declarative programming paradigm (programmer says what he wants, not how to get/draw it).

Other less obvious possibilities that the programmer can use:

• High-fidelity path-based drawing, antialiased rendering, gradients, images, color management, PDF documents.
• Create text layouts, optimize font handling and access font metrics and glyph data.
• Compatible with Core Haptics, but sounds are played instead of haptics!
• There is a native support for 2D charts (Swift Charts) that works across platforms.
• There is a library of animated symbols (for numerous cases, example: weather symbols).
• Interestingly, there is support (inherited from 2d platforms) for high-performance 2D content with smooth animations and games (SpriteKit).

Technologies: SwiftUI, Core Animation, Core Graphics, Core Text, Core Haptics, Observation, Swift Charts, Symbols, SpriteKit

3. Rich 3D Content — Volumes

Volumes are a kind of 3D Windows application (not to be confused with Microsoft’s Windows). They can use all bells and whistles of 3D content: animations, physics simulation, particle effects, lighting and realistic physically based rendering, sounds, and custom behaviors. All these are supported by the native platform framework called RealityKit, which mainly uses assets created within the new 3D scene authoring tool from Apple: Reality Composer.

Interestingly, lighting used by RealityKit in 3D volumes cannot be set custom. Instead the system estimates the real environment light and applies the same to the volume.

Alternatively, the 3D content can be provided with the Unity framework, so you can use the goodies of the large Unity ecosystem.

Technologies: RealityKit, SceneKit, Model I/O, SwiftUI, Spatial, Unity

4. Full Environment Takeover — Spaces

Spaces are equivalent to full screen mode on a desktop computer — only one application is displayed to the user and has almost full control of everything around it. For AR (not virtual reality) apps, the user can still control immersion level with Digital Crown. Note: the app needs to declare upfront whether it works in full passthrough mode, full VR mode or progressive mode with Digital Crown control. This is the only way to control the immersion level from the app itself, i.e., after entering the progressive mode, the app has no control and no way to check the current immersion level.

Read more about Windows, Volumes, and Spaces on Apple Vision Pro.

Technologies: SwiftUI, Unity

5. Advanced Non-Standard 3D Graphics

The new visionOS Metal framework, as the name suggests, allows the visionOS application developer to speak almost directly to the GPU in the language understood by the GPU. This is unlike most 3D applications created with frameworks like RealityKit where the programmer doesn’t speak to GPU directly and functions on much higher and more convenient levels of abstraction, among notions such as lights, 3D objects, animations, or scenes. On the contrary, direct GPU access is not needed by most apps, but it may unleash new opportunities:

• Fancy new ways of rendering can be created.
• Otherwise impossible advanced graphical effects can be created.
• GPUs can be used to perform computations (instead of rendering graphics).

More details:

• Metal Performance Shaders optimize graphics and compute performance with kernels that are fine-tuned for the unique characteristics of each Metal GPU family.
• MetalFX allows you to boost your Metal app’s performance by upscaling lower-resolution content to save GPU time.
• Separate IOSurface framework lets you share hardware-accelerated buffer data (framebuffers and textures) across multiple processes in order to manage image memory more efficiently.
• It's possible to use OpenGL ES, but it’s included for compatibility reasons and sits on top of Metal.

Technologies: Metal, MetalFX, MetalKit, Metal Performance Shaders, IOSurface, OpenGL ES.

6. User Controls and Sensors

Apple Vision Pro apps are controlled in a new human-centric approach that’s (as typical for Apple) not based on bringing new stuff to the table, but removing annoying stuff (like pads) and polishing to perfection the interaction with natural human features, which in this case are:

1. Eye gaze for pointing, and:
2. Hand gestures for action.

For privacy reasons, you will eye tracking isn't available for developers. Although SwiftUI visual elements react to eye focus, the application is not informed about this fact and cannot record eye movement in any way.

Hand gestures do not require the user to move hands like a dancer, they are very small gestures requiring very low energy. Gestures supported by the system are: tap, long-press, double tap, pinch and hold, pinch and drag, zoom, rotate.

App can ask for user permission to track the hand skeleton and recognize any fancy custom hand gestures.

Voice commands may be used instead of hand gestures for convenience. This is especially important for scenarios where the user needs to do something else with the hands in the physical world (e.g., cooking).

For text input, virtual keyboard or dictation can be used.

Users can connect additional input devices via Bluetooth: keyboards, mice, game controllers.

Frameworks that are available for developers for compatibility with iOS and iPad OS but obviously do not return any interesting data due to hardware differences:

• Core Motion for processing accelerometer, gyroscope, pedometer, and environment-related events.
• SensorKit for retrieving data and derived metrics from sensors on the device, or paired Apple Watch.

7. Real World Sensing, Mapping, and Recognition

ARKit has been developed for many years and receives huge investments. So far we were able to use it on phones and tablets, now we can experience how it performs on its target device. Here are the features:

• Meshing the environment: scanning the shapes around the user.
• Recognizing the types of the shapes: furniture, appliance, wall, or floor.
• Detecting flat surfaces that the app can easily search for and use: tables, floors, walls, doors.
• Integrate 3D object’s physics with the sensed surroundings (e.g., collisions of the virtual with the real).
• Recognize and store specific points in the real world (like a place on a real table where a virtual item was put) across app sessions.
• Recognize positions of given images in the real world (the developer has to pre-register the images).

Technologies: ARKit

8. Support for Group Experiences

Yes, you don’t need to look for and integrate any external technology in order to build basic group experiences, as visionOS by default supports:

• Defining app-specific activities your users can share and experience together by using your app, and then:
• Sharing these experiences, technically saying sharing the state of the app with a group of users within a Facetime call — it enables consuming content together, collaboration, or gaming.
• For games, enable players to interact with friends, compare leaderboard ranks, earn achievements, and participate in multiplayer games.

Technologies: CallKit, Message UI

9. Calls and Messaging

VisionOS also offers app developers some access to functions related to calls and messaging:

• Display the system-calling UI for your app’s VoIP services and coordinate your calling services with other apps and the system.
• Create a user interface for composing email and text messages, so users can edit and send messages without leaving your app.

Obviously cellular telephony is not available.

Technologies: CallKit, Message UI

10. Audio, Video, Streaming and Image Processing

General features available to visionOS developers:

• Playing of local as well as streamed content.
• Watching videos together with a group of people across devices and platforms, synchronized with a FaceTime call.

Apple Vision Pro features an advanced spatial audio system, with dual driver audio pods, which enables many options for developers:

• Support for spatial audio: setting different sound stages for example for inline or full-window playback; audio experience adequate to the content of the actual scene.
• PHASE — audio reacts to cues in the environment. For example, the sound will be different if there is an obstruction and an object generating the sound. This lets you create dynamic audio experiences in your game or app.
• Record audio, process audio, convert formats.
• Code your own tailor-made sophisticated audio manipulation and processing methods and create unit extensions that generate or modify audio in a different host app.
• Support for specialized data types to interact with audio streams, complex buffers, and audiovisual timestamps.

The Vision Pro headset is a robust spatial computer, with strong video features:

• 3D video with spatial audio playback.
• Play a video within a custom frame within the app, with customizable controls and playback progress monitoring.
• Display of subtitles and closed captions. Picture-in-picture is not available. Chapter navigation is in question.
• Video processing, including manipulation of individual frames, using a pipeline-based API.
• Coordinate the presentation of closed-captioned data for your app’s media files.
• Work directly with hardware-accelerated video encoding and decoding capabilities.

Image display and support:

• Reproduce colors accurately across a range of input, output, and display devices.
• Read and write most image file formats, and access an image’s metadata.

Additional interesting features:

• In-app editing of the timeline of played media and exporting edited content.
• Built-in or custom filters to process still and video images.
• Support of essential data types for audio and video.
• Enable users to configure HomePod speakers to stream music directly from your media service.
• Find and play Media Player assets: songs, audio podcasts, audio books, and more from within your app.

Things that are not available to the programmer:

• Image and video capture.
• Detailed control of camera operating parameters.
• Displaying custom destinations to stream media in the system route picker.

Technologies: Audio Toolbox, Audio Unit, AVFAudio, AVFoundation, AVKit, AVRouting, ColorSync, Core Audio, Core Audio Types, Core Image, Core Media, Core Video, CoreAudioKit, Image I/O, ImageCaptureCore, Media Accessibility, PHASE, Video Toolbox, Media Player, Media Setup

11. Artificial Intelligence and Machine Learning

As AI gets more and more important, it’s not a surprise that Apple’s mixed reality headset provides a large set of tools to build and use artificial intelligence within apps for an even better spatial computing experience.

When using machine learning models, you can:

• Integrate machine learning models into your app.
• Create and train machine learning models for use in your app, on the device! There are also functions to accelerate training and validation of neural networks across the CPU and one or more GPUs.
• Direct access to hardware (GPUs) to integrate or implement your own ML training mechanisms.

Besides handling AI on the device (which makes it available everywhere and with low latency), you can use networking to integrate with any server-side AI.

There is also a handful of pre-built AI services that programmers can use out of the box:

• Analyze natural language text and deduce its language-specific metadata.
• Classify various sounds by analyzing audio files or streams. Identify specific sounds, such as laughter or applause.
• Apply computer vision algorithms to perform a variety of tasks on input images and video. The Vision framework performs face and face landmark detection, text detection, barcode recognition, image registration, and general feature tracking. Vision also allows the use of custom Core ML models for tasks like classification or object detection. Warning: data scanners in Vision Kit do nothing!
• Perform speech recognition on live or prerecorded audio, and receive transcriptions, alternative interpretations, and confidence levels of the results.
• Artificial intelligence for gaming.

Technologies: Core ML, Create ML, Create ML Components, ML Compute, TabularData, Metal Performance Shaders, Metal Performance Shaders Graph, Natural Language, Sound Analysis, Vision, VisionKit, Speech, GameplayKit

12. User Notifications

Vision Pro supports several user notification technologies that we know from iPhone and iPad apps. The functionalities are:

• Pushing user-facing notifications to the user’s device from a server, or generating them locally from the app.
• Responding to push notifications related to the app’s complications, file providers, and VoIP.
• Customizing the interface that displays local and remote notifications.
• Support for sending and receiving Darwin notifications.

Technologies: User Notifications, User Notifications UI, PushKit, Darwin Notify

13. Integration with Other Apps and System Functions

Other device apps and services available programmatically to your visionOS app are:

• Contacts — Access the user's contacts, and format and localize contact information. Display information about users’ contacts in a graphical interface.
• Calendar — Create, view, and edit calendar and reminder events. Display an interface for viewing, selecting, and editing calendar events and reminders.
• Apple Music — integration.
• Photos app — Work with image and video assets, including those from iCloud Photos and Live Photos.
• Shazam — Find information about a specific audio recording when a segment of it’s part of captured sound in the Shazam catalog or your custom catalog.
• Siri — Empower users to interact with their devices through voice, intelligent suggestions, and personalized workflows.
• Weather — Deliver weather conditions and alerts to your users.
• Video Subscriber Account — Support TV provider and Apple TV app functionality. Video Subscriber Account provides APIs to help you create apps that require secure communication with a TV provider’s authentication service. The framework also informs the Apple TV app about whether your user has a subscription and the details of that subscription.
• MapKit — Display map or satellite imagery within your app, call out points of interest, and determine placemark information for map coordinates. Unfortunately all location services except the standard service are not available on Vision Pro.
• PDFKit — Display and manipulate PDF documents in your apps.

Apps and data that is not available:

• HealthKit
• SMS and Call Reporting — Create extensions to manage and report unwanted SMS messages and spam calls.

The Mechanisms that allow your app and data to integrate with other apps and be discovered:

• App Intents — Extend your app’s custom functionality to support system-level services like Siri, Spotlight, the Shortcuts app, and the Action button.
• Spotlight — Index your app so users can search the content from Spotlight and Safari.
• Transferable — Declare a transfer representation for your model types to participate in system sharing and data transfer operations.
• Extensions — Create executable bundles to extend the functionality of other apps, with the possibility of presenting a user interface (this feature may be limited and requires examination).
• File Provider — An extension other apps use to access files and folders managed by your app and synced with a remote storage. Possibility to add actions to the document browser's context menu.
• Link Presentation — Fetch, provide, and present rich links in your app.
• Uniform Type Identifiers — Provide uniform type identifiers that describe file types for storage or transfer.
• QuickLook — Create previews of files to use inside your app, or perform simple edits on previews.
• QuickLook Thumbnailing — Generate thumbnails for common file types and add a Thumbnail Extension to your app to enable others to create thumbnails of your custom files.
• DataDetection — Access and utilize common types of data that the data detection system matches.

System functions that can do work for your app in the background:

• Background Assets — Schedule background downloads of large assets after app installation, when the app updates, and periodically while the app remains on-device.
• Background Tasks — Request the system to launch your app in the background to run tasks.
• Core Services — Access and manage key operating system services, such as launch and identity services.
• os — Coordinate the scheduling and synchronization of your app’s tasks, and log information to the console to diagnose issues.

Technologies: Contacts, Contacts UI, EventKit, EventKit UI, MusicKit, PhotoKit, ShazamKit, App Intents, Background Assets, Background Tasks, SiriKit, SMS and Call Reporting, WeatherKit, Video Subscriber Account, Core Spotlight, Transferable, Extension Foundation, ExtensionKit, File Provider, File Provider UI, Link Presentation, Uniform Type Identifiers, QuickLook, QuickLook Thumbnailing, DataDetection, Core Services, os, HealthKit, Core Location, MapKit, PDFKit

14. Attribution of App-Download Campaigns

There is a way to programmatically attribute app-download campaigns that originate from: App Store, Apple News, or Stocks.

Technologies: AdServices

Subscriptions and Apple Pay

• PassKit (Apple Pay and Wallet) — Process Apple Pay payments in your app, and create and distribute passes for the Wallet app.
• StoreKit — Support in-app purchases and interactions with the App Store.
• StoreKit Test — Create and automate tests in Xcode for your app’s subscription and in-app purchase transactions, and SKAdNetwork implementations.

15. Authentication, Security, Cryptography

Apple Vision Pro introduces Optic ID: Iris scanning authentication. This is like Touch ID/Face ID in other Apple devices to be used instead of passwords for unlocking the device and for purchases. This feature is not directly available to the programmer, but impacts how other features (like Apple Pay) work for the user.

List of Apple Vision Pro programmatically available features related to authentication, security and crypto:

• App Tracking Transparency — Request user authorization to access app-related data for tracking the user or the device.
• Authentication Services — Make it easy for users to log into apps and services.
• Local Authentication — Authenticate users biometrically or with a passphrase they already know.
• Local Authentication Embedded UI — Present a standard local authentication view icon in a custom authentication view.
• CryptoTokenKit — Access security tokens and the cryptographic assets they store.
• DeviceCheck — Reduce fraudulent use of your services by managing device state and asserting app integrity.
• GSS — Conduct secure, authenticated network transactions.
• Security — Secure the data your app manages, and control access to your app.

16. Subscriptions and Apple Pay

• PassKit (Apple Pay and Wallet) — Process Apple Pay payments in your app, and create and distribute passes for the Wallet app.
• StoreKit — Support in-app purchases and interactions with the App Store.
• StoreKit Test — Create and automate tests in Xcode for your app’s subscription and in-app purchase transactions, and SKAdNetwork implementations.

17. Computation and Compression

• Accelerate — Make large-scale mathematical computations and image calculations, optimized for high performance and low energy consumption.
• Dispatch — Execute code concurrently on multicore hardware by submitting work to dispatch queues managed by the system.
• Apple Archive — Perform multithreaded lossless compression of directories, files, and data.
• Compression — Leverage common compression algorithms for lossless data compression.

18. Screen Recording and Streaming

ReplayKit is a framework originally created to record or stream video from the screen of the laptop. Surprisingly, ReplayKit is available on visionOS, however it is not clear whether it works or how it works (send me a note if you know!). What is known is that video capture from the cameras is not available.

19. Networking

• CFNetwork — Access network services and handle changes in network configurations. Build on abstractions of network protocols to simplify tasks such as working with BSD sockets, administering HTTP and FTP servers, and managing Bonjour services.
• dnssd — Discover, publish, and resolve network services on a local area or wide area network.
• System Configuration — Allow applications to access a device’s network configuration settings. Determine the reachability of the device, such as whether Wi-Fi or cell connectivity are active.
• Network — Create network connections to send and receive data using transport and security protocols.
• Network Extension — Customize and extend core networking features.

20. External Accessories Connectivity, Including Bluetooth

• Core Bluetooth — Communicate with Bluetooth low energy and BR/EDR (“Classic”) Devices.
• DeviceDiscoveryExtension — Stream media to a third-party device that a user selects in a system menu.
• External Accessory — Communicate with accessories that connect to a device with the Apple Lightning connector, or with Bluetooth wireless technology.
• Game Controller — Support hardware game controllers in your game.
• IOKit — Access hardware devices and drivers from your apps and services (may be limited).
• Multipeer Connectivity — Support peer-to-peer connectivity and the discovery of nearby devices.

21. Data and Storage — Local and in the Cloud

• CloudKit — Store structured app and user data in iCloud containers that all users of your app can share.
• Core Data — Persist or cache data on a single device, or sync data to multiple devices with CloudKit.
• SwiftData — Write your model code declaratively to add managed persistence and automatic iCloud sync.
• System — Perform low-level file operations using type-safe APIs.

22. MIDI for Music Makers

Good news for Music Makers! Core MIDI allows communication with MIDI devices such as hardware keyboards and synthesizers. This should work via Bluetooth. From documentation it looks that it is available, however please check it out (or ask us to check it out) before committing to the investment.

23. Building for Developers

If your product is intended for software developers, you may be interested in the following features of visionOS:

• DeveloperToolsSupport — Expose custom views and modifiers in the Xcode library.
• MetricKit — Aggregate and analyze per-device reports on exception and crash diagnostics and on power and performance metrics.
• OSLog — A unified logging system for the reading of historical data.

24. Javascript and Web

You can use WebKit and Safari Services to integrate web content seamlessly into your app, and customize content interactions to meet your app’s needs. You can also evaluate Javascript within your app with JavaScriptCore.

Technologies: WebKit, JavaScriptCore, Safari Services

25. Intelligent Home

• HomeKit — Configure, control, and communicate with home automation accessories.
• Matter — Communicate with and control smart home devices from a variety of manufacturers.
• MatterSupport — Coordinate and control compatible smart home accessories.
• ThreadNetwork — Share credentials and choose a designated network for each home. Thread is a low-power wireless mesh networking protocol based on the universally supported Internet Protocol (IP).

26. Accessibility

Apple treats accessibility very seriously and so they have provided an extensive list of features. Some of them may be used in innovative ways for purposes other than accessibility. Most important examples are:

• VoiceOver
• Switch Control
• Dwell Control
• Guided Access
• Head Pointer

Apple Vision Pro — Creating New Opportunities

Now, as you grasped the basics as well as some details, here are some proposals on what to do next in your exploration of Apple's new spatial operating system:

If you're toying with an app concept but are in doubt about its practicality, don't hesitate to reach out. We're here to swiftly address any technical concerns you might have. For broader perspective and inspiration, scour the internet. You'll find a plethora of examples showcasing how platform components amplify value across varied business scenarios, potentially offering fresh insights into your own project. For those keen on a deep dive into app design and development, Apple's dedicated visionOS page is your go-to resource.

Read more about the tech stack for Apple Vision Pro applications or the cost of building a spatial application for Vision Pro.

Lastly, if you're contemplating assembling an in-house delivery team, let us assist. We're poised to guide you on team structure and essential skills. While we're gearing up to share a comprehensive blog post on this topic, our real-time support remains at your disposal in the interim.