Meta Quest has launched the Passthrough Camera API for developers, although integration into the Meta Horizon Store is currently prohibited. Announced at Meta Connect 2024, the API is now available for testing on Quest 3 and Quest 3S, allowing developers to create applications for platforms like SideQuest. This API enables access to the headset’s front cameras for various applications, though performance is limited by the XR2 Gen 2 chip. Future access to the Meta Horizon Store is anticipated after the experimental phase.
The eagerly awaited Passthrough Camera API from Meta Quest has officially opened its doors to developers, inviting them to explore and innovate with new functionalities. However, it is important to note that integrating this API into applications designed for the Meta Horizon Store is not yet permissible.
Release and Development Insights
This exciting feature was unveiled during Meta Connect 2024 last September, with an initial promise of a rollout by year’s end. Although that timeline was missed, the experimental version is now available for the Quest 3 and Quest 3S. Developers are encouraged to test it out and publish applications (APKs) on other platforms such as SideQuest. However, the API is still restricted from being used in applications on the official Meta Horizon Store.
Meta has previously adopted a similar strategy for launching new APIs: providing a testing phase followed by wider accessibility in subsequent months. It’s worth mentioning that select studios and developers were granted early access to the API. Additionally, Meta plans to host a session with Niantic, Creature, and Resolution at the upcoming GDC (Game Developers Conference) to delve deeper into this topic.
Understanding Passthrough Camera Functionality
When users launch an application utilizing the Passthrough Camera API, the application requests permission to access the headset’s cameras, similar to how apps ask for microphone access on smartphones. Upon approval, the application can utilize the Quest’s front color cameras along with valuable data (such as intrinsic lens specifications and headset positioning). This capability allows for the implementation of custom computer vision models or the use of cloud-based AI services.
Potential applications for this technology include:
- Scanning and tracking QR codes.
- Detecting board game layouts for integrating virtual characters.
- Recognizing physical objects in professional settings (like assembly instructions and logistics).
- Incorporating computer vision into cloud-hosted AI models.
The primary limitations stem from the XR2 Gen 2 chip’s processing capabilities, which must manage real-time computer vision tasks, as well as the budget available for using remotely hosted models (Cloud).
Technical Performance and Considerations
The Passthrough video stream offers a resolution of 1280×960 per camera at 30 frames per second, with latency reported between 40 to 60 milliseconds. This makes the solution less suitable for tracking fast-moving objects (like custom controllers) or for recognizing tiny details or small text.
From a software standpoint, there isn’t a distinct Meta Quest Camera Passthrough API or a dedicated OpenXR extension. Developers must simply request the “Headset Cameras” permission from Horizon OS and utilize the Camera2 API from Android to access both the video feed and headset position (with headset pose information accessible via OpenXR).
For those working with Unity, access to these cameras can be achieved through the engine’s WebCamTexture API, which is also used for mobile and PC webcams. However, a significant limitation is that WebCamTexture supports only one camera at a time, rather than both simultaneously.
Final Thoughts
The introduction of the Passthrough Camera API for Quest 3 and Quest 3S marks a significant step forward in mixed reality technology, providing direct access to front cameras and opening the door to enhanced experiences in object recognition, tracking, and environmental interaction. While developers can currently experiment and share applications via SideQuest or other alternative platforms, the use of this API within the Meta Horizon Store remains restricted for the time being. Notably, as Android XR plans to enable this access upon its launch, it likely influences Meta’s current stance on the matter.
As a result, it is highly probable that Meta will eventually permit API usage in the Store after the completion of the experimental phase. Developers are encouraged to take advantage of this period to familiarize themselves with the technology and to create innovative experiences for an upcoming public release.