Skip to main content

Depth Module

Request depth frames on SPECS to understand scene geometry, measure distances, and place content against real-world surfaces.

  • Snapchat: not supported
  • SPECS: fully supported
  • Camera Kit Web: not supported
  • Camera Kit Mobile: not supported
  • SnapchatNot supported
  • SPECSFully supported
  • Camera Kit WebNot supported
  • Camera Kit MobileNot supported

Platform details

  • Snapchat — Not supported.

  • SPECS — Fully supported.

  • Camera Kit Web — Not supported.

  • Camera Kit Mobile — Not supported.

On SPECS, the DepthModule gives a Lens depth frames: images whose pixels encode the distance to the real-world scene, in centimeters. Use them to understand the user's surroundings, measure distances, and place content against real surfaces.

The Depth Module requires camera access, and a depth frame contains user information. On SPECS, a Lens that combines camera access with internet access is publishable through Transparent Permission: the Lens asks for permission at launch and the LED blinks while it captures. Users who turn on Extended Permissions skip the permission prompt.

Lens Studio Preview doesn't stream depth: onNewFrame never fires in Preview. Test depth code on SPECS (Send a Lens to SPECS).

Request and receive depth frames on SPECS

The DepthModule creates a DepthFrameSession, which delivers a steady stream of DepthFrameData. Frames arrive about five times a second, a rate independent of the color camera frame rate and the render rate.

Start and stop a depth frame session

To receive depth frames, create a DepthFrameSession and add a callback to its onNewFrame event, which passes a DepthFrameData for each frame. The session starts computing depth when you call DepthFrameSession.start; onNewFrame fires only after that. DepthFrameSession.stop stops depth computation for all of the session's callbacks.

Create the session in OnStartEvent, as in this example: createDepthFrameSession can't be called inside onAwake.

@component
export class DepthFrameRequest extends BaseScriptComponent {
private depthModule: DepthModule = require('LensStudio:DepthModule');
private session: DepthFrameSession;
private registration: EventRegistration;

onAwake() {
this.createEvent('OnStartEvent').bind(() => {
this.session = this.depthModule.createDepthFrameSession();

this.registration = this.session.onNewFrame.add(
(depthFrameData: DepthFrameData) => {
// Process the depth frame data.
}
);

// Start depth estimation to receive depth frame data via the onNewFrame event.
this.session.start();
});

this.createEvent('OnDestroyEvent').bind(() => {
this.session.onNewFrame.remove(this.registration);

// Stop depth estimation for this session.
this.session.stop();
});
}
}

Read the depth camera's parameters, pose, and timestamp

Each DepthFrameData carries:

  • depthFrame: the depth values as a linear Float32Array in centimeters, one per pixel, row by row.
  • deviceCamera: the DeviceCamera that describes how the depth camera maps 3D space to the 2D frame: its resolution, focalLength, and principalPoint in pixels, and its pose, the offset between the device's reference point and the camera.
  • toWorldTrackingOriginFromDeviceRef: the pose of the device reference point relative to the frame of reference of the tracked device position (the world tracking origin).
  • timestampSeconds: the start of the frame's exposure, in seconds. Use it to pair a depth frame with the matching color frame from the CameraModule.
this.session.onNewFrame.add((depthFrameData: DepthFrameData) => {
const depthDeviceCamera = depthFrameData.deviceCamera;
print(depthDeviceCamera.resolution);
print(depthDeviceCamera.focalLength);
print(depthDeviceCamera.principalPoint);
print(depthDeviceCamera.pose); // The relative offset between a reference point on the device and the camera

print(depthFrameData.toWorldTrackingOriginFromDeviceRef); // The pose of the device reference relative to the frame of reference of the tracked device position.

print(depthFrameData.timestampSeconds); // The timestamp of the depth frame can be used to find the matching color frame used for estimating depth.
});

DeviceCamera also has project and unproject methods, which convert a 3D point in device reference space to a normalized 2D point in the frame (origin at the top left), and back.

Convert a depth pixel to a world position

This script samples the depth at one pixel of each depth frame, back-projects it to the device's reference space with unproject, and transforms it to world space with toWorldTrackingOriginFromDeviceRef.

  1. In the Asset Browser panel, select + > Scripting, then create a TypeScript Component or a JavaScript Component.

    The Asset Browser + menu open on Scripting, with the TypeScript Component and JavaScript Component entries
  2. Paste the following code into the file.

  3. Attach the script to an object in the scene.

  4. Send the Lens to SPECS. For each depth frame, the script prints the world position of the sampled pixel.

@component
export class DepthModuleExample extends BaseScriptComponent {
private depthModule: DepthModule = require('LensStudio:DepthModule');
private session: DepthFrameSession;
private registration: EventRegistration;

onAwake() {
this.createEvent('OnStartEvent').bind(() => {
this.session = this.depthModule.createDepthFrameSession();

this.registration = this.session.onNewFrame.add(
(depthFrameData: DepthFrameData) => {
const depthDeviceCamera = depthFrameData.deviceCamera;

// Sample depth for specific pixel
const pixelCoordX = 112;
const pixelCoordY = 80;
const depthFrameArrayIdx = Math.floor(
pixelCoordX + pixelCoordY * depthDeviceCamera.resolution.x
);
const depthValue = depthFrameData.depthFrame[depthFrameArrayIdx];

// Back-project depth pixel to device reference space
const normalizedCoord = new vec2(
pixelCoordX / depthDeviceCamera.resolution.x,
pixelCoordY / depthDeviceCamera.resolution.y
);
const point3dInDeviceRef = depthDeviceCamera.unproject(
normalizedCoord,
depthValue
);

// Transform point in device reference space to world tracking origin space
const worldFromDeviceRef =
depthFrameData.toWorldTrackingOriginFromDeviceRef;
const point3dInWorld =
worldFromDeviceRef.multiplyPoint(point3dInDeviceRef);
print(point3dInWorld);
}
);

this.session.start();
});

this.createEvent('OnDestroyEvent').bind(() => {
this.session.onNewFrame.remove(this.registration);
this.session.stop();
});
}
}