Skip to main content

Camera Module

Request and receive camera frames from SPECS to build AR experiences that interact with the user's real-world environment.

  • Snapchat: not supported
  • SPECS: fully supported
  • Camera Kit Web: not supported
  • Camera Kit Mobile: not supported
  • SnapchatNot supported
  • SPECSFully supported
  • Camera Kit WebNot supported
  • Camera Kit MobileNot supported

Platform details

  • Snapchat — Not supported.

  • SPECS — Fully supported.

  • Camera Kit Web — Not supported.

  • Camera Kit Mobile — Not supported.

Overview

CameraModule gives a Lens access to the cameras SPECS makes available to Lenses:

  • Discover the available cameras and the streams each one supports.
  • Stream camera frames with a StreamingSession.
  • Capture high-resolution stills with a CaptureSession, with an optional crop for each shot.

Each frame provides its pixels as a texture and as raw bytes, with a timestamp and, when the camera reports them, exposure values.

In Lens Studio Preview, getCameras() returns an empty array, so test camera code on SPECS. On device, getCameras(), getStreams(), createStreamingSession() and createCaptureSession() can throw an Error when the camera can't be accessed or camera discovery fails; wrap them in try/catch.

Camera frames are sensitive user data. A Lens that sends frames to a backend (for example, Base64-encoding imageData for an ML or LLM service) combines camera access with internet access, which requires Transparent Permission to be publishable.

Discover Cameras and Streams

getCameras() returns the cameras available to the Lens. Each CameraInfo reports a type (Rgb or Grayscale), a location (Left or Right), a facing (World or User) and a human-readable name. camera.getStreams() returns that camera's stream capabilities, sorted largest-first by pixel area. Each StreamInfo has a fixed width and height, an imageFormat (Jpeg, Yuv420 or Y8), a quality tier (Performance, Balanced or Enhanced), a maxFrameRate and an operation.

Each capability works with one kind of session, reported by StreamInfo.operation:

  • CameraModule.Operation.Streaming: use with a StreamingSession.
  • CameraModule.Operation.Capture: use with a CaptureSession.

Choose by operation first, then filter by format, size or quality. Don't infer the operation from format or resolution: the same size and format can be offered once for each operation. description is a human-readable label for logs. Its format can change and it isn't unique, so don't parse it or use it as an identifier.

const cameraModule: CameraModule = require('LensStudio:CameraModule');
let cameras: CameraModule.CameraInfo[] = [];
try {
cameras = cameraModule.getCameras();
} catch (e) {
print('Camera unavailable: ' + e);
}
const rgb = cameras.find((c) => c.type === CameraModule.CameraType.Rgb);
const streams = rgb ? rgb.getStreams() : [];
const preview = streams.find(
(s) => s.operation === CameraModule.Operation.Streaming
); // largest streaming capability
const photo = streams.find(
(s) =>
s.operation === CameraModule.Operation.Capture &&
s.imageFormat === CameraModule.ImageFormat.Jpeg
); // largest JPEG still

On current SPECS software, getCameras() returns two world-facing cameras with these capabilities:

CameraSizeFormatQualityOperation
Right (RGB)4032×3024Yuv420EnhancedCapture
Right (RGB)4032×3024JpegEnhancedCapture
Right (RGB)2016×1512Yuv420EnhancedStreaming
Right (RGB)2016×1512JpegBalancedCapture
Right (RGB)1008×756Yuv420PerformanceStreaming
Right (RGB)1008×756JpegPerformanceCapture
Left (grayscale)1124×1364Y8PerformanceStreaming

Every capability reports a maxFrameRate of 30. Select capabilities at runtime: the list can differ on other devices and software versions.

Stream Frames

A StreamingSession delivers frames through onNewFrame until you stop it. Sessions are restartable: stop() releases the camera and start() can be called again.

Set StreamConfig.minFrameRate to the camera rate your Lens needs; the default, 0, requests the capability's maxFrameRate. The rate is rounded up to a whole number of frames per second. Assigning a negative value throws, and start() throws when the rounded rate exceeds maxFrameRate. The camera can run faster than the rate you request, and delivery to your Lens can be lower than the camera rate when its per-frame work is slow.

This example shows the smallest RGB stream on an Image component:

@component
export class CameraStreaming extends BaseScriptComponent {
@input
image: Image;

private cameraModule: CameraModule = require('LensStudio:CameraModule');
private session: CameraModule.StreamingSession;

onAwake() {
const rgb = this.cameraModule
.getCameras()
.find((c) => c.type === CameraModule.CameraType.Rgb);
// Smallest streaming capability: full frame rate at the lowest cost.
const stream = rgb
? rgb
.getStreams()
.filter((s) => s.operation === CameraModule.Operation.Streaming)
.pop()
: undefined;
if (!stream) {
print('No streaming camera available');
return;
}

this.session = this.cameraModule.createStreamingSession();
this.session.onNewFrame.add((frame) => {
if (frame.texture) {
this.image.mainPass.baseTex = frame.texture;
}
});
this.session.onStatusChanged.add((status) => {
if (status === CameraModule.StreamStatus.Disconnected) {
print('Camera disconnected; create a new session to continue');
}
});

const config = new CameraModule.StreamConfig();
config.stream = stream;
config.minFrameRate = 15;
const result = this.session.start(config);
if (result !== CameraModule.Result.Success) {
print('Streaming failed to start: ' + result);
}

this.createEvent('OnDestroyEvent').bind(() => this.session.stop());
}
}

Auto-Exposure Region

ConfigKey.AutoExposureRegion steers automatic exposure toward a region of the frame; it doesn't crop. Set it before start() with config.setConfig(key, values), or on a running stream with session.setConfig(key, values). The values are [left, right, top, bottom] in normalized 0.0–1.0 image coordinates, origin top-left, with left < right and top < bottom. An optional fifth value is a weight from 0 to 1 (default 1).

A session has one region, and the rest of the frame can then be brighter or darker. The region can stay in effect after the session stops: set [0, 1, 0, 1] before stop() to meter the whole frame again. It works on the 1008×756 RGB and grayscale streams, and has no effect on the 2016×1512 RGB stream or on capture sessions.

Manual exposure isn't supported: cameras always run automatic exposure. Read the values in use from each frame with frame.getMetadata(CameraModule.MetadataKey.Iso) and MetadataKey.ExposureTimeNs.

Capture Stills

A CaptureSession prepares the still pipeline in start() and returns one Frame for each capture(). Start the session when your Lens loads rather than at shutter press: the first capture also pays for warm-up. One capture can be in flight at a time.

To save power, the system ends a capture session that goes 60 seconds without a capture. onStatusChanged reports Disconnected, and later capture() calls reject. Each capture() restarts the 60-second timer. After Disconnected, create a new session to capture again.

@component
export class CameraCapture extends BaseScriptComponent {
@input
image: Image;

private cameraModule: CameraModule = require('LensStudio:CameraModule');
private capture: CameraModule.CaptureSession;
private ready = false;
private busy = false;

onAwake() {
const rgb = this.cameraModule
.getCameras()
.find((c) => c.type === CameraModule.CameraType.Rgb);
const stream = rgb
? rgb
.getStreams()
.find(
(s) =>
s.operation === CameraModule.Operation.Capture &&
s.imageFormat === CameraModule.ImageFormat.Jpeg
)
: undefined;
if (!stream) {
print('No still capture available');
return;
}

this.capture = this.cameraModule.createCaptureSession();
this.capture.onStatusChanged.add((status) => {
if (status === CameraModule.StreamStatus.Disconnected) {
this.ready = false; // create a new session to continue
}
});

const config = new CameraModule.StreamConfig();
config.stream = stream;
this.ready = this.capture.start(config) === CameraModule.Result.Success;

// Capture two seconds after start; call takePhoto() from your own trigger instead.
const delay = this.createEvent('DelayedCallbackEvent');
delay.bind(() => this.takePhoto());
delay.reset(2);

this.createEvent('OnDestroyEvent').bind(() => this.capture.stop());
}

async takePhoto() {
if (!this.ready || this.busy) {
return;
}
this.busy = true;
try {
const overrides = new CameraModule.StreamConfigOverrides();
overrides.crop = Rect.create(-0.5, 0.5, -0.5, 0.5); // centered 50% crop for this shot
const frame = await this.capture.capture(overrides);
if (frame.texture) {
this.image.mainPass.baseTex = frame.texture;
}
} catch (e) {
print('Capture failed: ' + e);
} finally {
this.busy = false;
}
}
}

capture() rejects when the session isn't started, another capture is in flight, the crop isn't finite, the session stops or disconnects before the frame arrives, the system rejects the request, or no frame arrives within 30 seconds. A frame whose image couldn't be read still resolves, with null texture and imageData, so check before use.

Still Size

A capture session returns stills at its capability's size unless you set StreamConfig.outputWidth and outputHeight. Any positive, even size up to the capability's width and height works, for example a 756×1008 portrait still from the 4032×3024 capability. A size with another aspect ratio than the capability is a centered crop of the camera's view, so the image is never stretched.

config.stream = photo; // a Capture capability, for example 4032x3024 Jpeg
config.outputWidth = 756;
config.outputHeight = 1008;

Assigning zero, a negative or an odd value throws. start() throws when only one of the two is set, when the size is larger than the capability, or when either is set on a StreamingSession: streams always have their capability's size. Set both to null to use the capability's size again.

Crop

StreamConfig.crop sets a default crop for every capture of the session; StreamConfigOverrides.crop overrides it for one shot. Crops are Rect values in normalized image space: [-1, 1] on each axis, with +1 at the top and right of the image, created with Rect.create(left, right, bottom, top). The cropped region is scaled to the still's size: outputWidth×outputHeight when set, otherwise the capability's size. Give the crop the same aspect ratio as the still. Streaming sessions ignore crops.

RegionRect
Full imageRect.create(-1, 1, -1, 1)
Centered 50%Rect.create(-0.5, 0.5, -0.5, 0.5)
Top-left quadrantRect.create(-1, 0, 0, 1)
Bottom-right quadrantRect.create(0, 1, -1, 0)

Crop and AutoExposureRegion use different coordinate spaces: crop is [-1, 1] normalized image space with y up, AutoExposureRegion is 0..1 image coordinates with y down.

Noise Reduction

Set ConfigKey.SnapshotMfnrEnable with config.setConfig() before start(): [1] turns on multi-frame noise reduction, [0] turns it off. Captures take longer with it. It applies to capture sessions only: a streaming session returns Result.InvalidConfig.

Work with Frames

  • frame.imageData is the raw image, tightly packed:

    • Jpeg: the JPEG file.
    • Yuv420: the Y plane (width * height bytes), then half-resolution chroma interleaved V then U (NV21, width * height / 2 bytes).
    • Y8: one byte per pixel.

    The array shares the frame's buffer; treat it as read-only.

  • frame.texture is an RGBA texture of the image; grayscale frames have R = G = B. Each session reuses one Texture: reading texture on a newer frame, including the next capture, replaces the image in the Texture you got from an earlier frame, and a later session of the same size can reuse it after stop(). To keep an image, keep its imageData, or copy the texture (see below). After a session stops, reading texture on a frame you kept decodes the image again into a new texture, so read it while the session runs.

  • Reading texture converts the frame to RGBA and uploads it to the GPU, which costs the most at 2016×1512. Read it only on the frames you display or process. Reading it on occasional frames costs more per read, because the conversion then runs when you read it.

  • Streaming frames are short-lived. A session keeps the pixels of its two most recent frames only. On older frames texture returns null, and so does imageData unless your Lens still holds the array it returned earlier. Read what you need inside onNewFrame. An imageData array you have already read stays valid and keeps that frame's bytes until it's garbage collected, so read imageData only on the frames you process. Captured frames keep their pixels for as long as you reference them.

  • frame.getMetadata(key) returns TimestampNs (CLOCK_MONOTONIC, always present) and Iso, ExposureTimeNs and FrameRate when the camera reports them. It returns null when a value is unavailable.

Read Texture Pixels

Assigning a frame texture to a material (image.mainPass.baseTex = frame.texture) works inside onNewFrame, and Base64.encodeTextureAsync works on a captured frame right after await capture(). To read pixels on the CPU, read frame.texture inside onNewFrame, keep that Texture, and copy it with ProceduralTextureProvider.createFromTexture in an UpdateEvent: inside onNewFrame or right after await capture(), createFromTexture throws an error. Hand the texture to your update handler instead:

@component
export class CameraPixels extends BaseScriptComponent {
private cameraModule: CameraModule = require('LensStudio:CameraModule');
private session: CameraModule.StreamingSession;
private pending: Texture | null = null;
private pixel = new Uint8Array(4);

onAwake() {
const rgb = this.cameraModule
.getCameras()
.find((c) => c.type === CameraModule.CameraType.Rgb);
const stream = rgb
? rgb
.getStreams()
.filter((s) => s.operation === CameraModule.Operation.Streaming)
.pop()
: undefined;
if (!stream) {
print('No streaming camera available');
return;
}

this.session = this.cameraModule.createStreamingSession();
this.session.onNewFrame.add((frame) => {
this.pending = frame.texture; // keep a reference only
});
const config = new CameraModule.StreamConfig();
config.stream = stream;
const result = this.session.start(config);
if (result !== CameraModule.Result.Success) {
print('Streaming failed to start: ' + result);
return;
}

this.createEvent('UpdateEvent').bind(() => {
if (this.pending) {
const copy = ProceduralTextureProvider.createFromTexture(this.pending);
this.pending = null;
const provider = copy.control as ProceduralTextureProvider;
provider.getPixels(0, 0, 1, 1, this.pixel); // RGBA of the pixel at (0, 0)
}
});
this.createEvent('OnDestroyEvent').bind(() => this.session.stop());
}
}

Results and Status

start(), setConfig() and stop() return a CameraModule.Result. Invalid values of minFrameRate, outputWidth and outputHeight throw instead, where you assign them or from start().

ResultMeaning
SuccessThe call succeeded.
NotInitializedThe camera isn't available to the session (rare).
InvalidArgumentMissing stream, an invalid setConfig() value, or a non-finite crop.
InvalidConfigThe capability's operation doesn't match the session or it can't be opened, or a key isn't allowed here.
AlreadyStartedstart() on a running session.
NotStartedstop() or setConfig() before start().
SessionStoppedThe session was disconnected; create a new one.
ErrorGenericThe camera couldn't be reached or opened, a setConfig() change couldn't be applied, or a session limit was reached.

onStatusChanged reports:

  • Interrupted: frames pause, for example while the camera switches modes because another session started, or while the camera recovers from an error. The session stays started. The camera can drop a capture in flight: its Promise rejects about 30 seconds after capture(). Capture again after Resumed.
  • Resumed: frames flow again.
  • Disconnected: the system ended the session: camera access was revoked, the system stopped the stream (for example, another stream switched the camera to a mode without this stream's size), the camera system restarted, or a capture session went 60 seconds without a capture. The session is finished; create a new one.

Sessions stop automatically when the Lens closes.

Known Limitations

  • Still capture from the grayscale camera isn't supported; use the RGB camera.
  • A Lens can run up to three streaming sessions across all its CameraModules, at most one at 2016×1512, or one streaming session and one capture session. start() past a limit returns Result.ErrorGeneric; stop a session to free its slot.
  • Starting a capture session while an RGB stream runs briefly interrupts the stream (Interrupted, then Resumed).
  • Session frames don't provide the camera's intrinsic and extrinsic parameters yet; see Projecting 3D Points.
  • Full-resolution images are large: a 4032×3024 frame's texture holds 48.8 MB of RGBA, and its Yuv420 imageData 18.3 MB. A captured frame keeps its pixels while you reference it, so drop captures you're done with and keep few full-resolution images at once. The script engine frees a dropped capture only when it next collects garbage. Until then, many 4032×3024 Yuv420 captures can add up to hundreds of MB. For repeated full-size stills, use the much smaller Jpeg. Under system memory pressure, SPECS can close a Lens that uses a lot of memory.

Projecting 3D Points

Session frames don't provide the camera's intrinsic and extrinsic parameters yet. To project between 3D space and the camera image, use requestCamera for the camera texture and DeviceCamera for the camera's parameters.

Camera Texture

Create a CameraRequest and set its cameraId: Default_Color for the RGB camera or Left_Grayscale for the grayscale camera. SPECS doesn't support Left_Color, and Right_Color returns the same RGB camera as Default_Color. imageSmallerDimension sets the size in pixels of the smaller dimension to return. requestCamera returns a Texture that updates with each frame. The texture contains the opaque handle that refers to the underlying camera data, and can be used as an input into an MLComponent or uploaded to a remote service via RemoteMediaModule. To be notified of each new frame, use its CameraTextureProvider, accessed via cameraTexture.control, and its onNewFrame event:

@component
export class CameraFrameExample extends BaseScriptComponent {
private cameraModule: CameraModule = require('LensStudio:CameraModule');

onAwake() {
this.createEvent('OnStartEvent').bind(() => {
const cameraRequest = CameraModule.createCameraRequest();
cameraRequest.cameraId = CameraModule.CameraId.Default_Color;
const cameraTexture = this.cameraModule.requestCamera(cameraRequest);

const onNewFrame = (cameraTexture.control as CameraTextureProvider)
.onNewFrame;
const registration = onNewFrame.add((frame: CameraFrame) => {
// Process the frame
});
this.createEvent('OnDestroyEvent').bind(() => {
onNewFrame.remove(registration);
});
});
}
}

Don't call createCameraRequest() in onAwake; the examples call it from OnStartEvent.

Camera Information

DeviceCamera, from the DeviceInfoSystem, describes how the camera maps 3D space to the 2D image: its focal length, principal point, resolution and pose. Its project and unproject methods convert 3D points to 2D image points and back.

@component
export class CameraInfoExample extends BaseScriptComponent {
onAwake() {
this.createEvent('OnStartEvent').bind(() => {
// Select the camera
const camera = global.deviceInfoSystem.getTrackingCameraForId(
CameraModule.CameraId.Default_Color
);

// Retrieve camera properties
const focalLength = camera.focalLength;
const principalPoint = camera.principalPoint;
const resolution = camera.resolution;
const pose = camera.pose;
});
}
}

Deprecated APIs

These APIs are deprecated and left out of the Lens Studio type definitions. Use the replacements in new Lenses.

DeprecatedUse instead
requestImageA CaptureSession. requestImage doesn't work on SPECS.
getSupportedStreams() and StreamFiltergetCameras() and camera.getStreams()
StreamInfo.type, location and frameRatestream.camera.type, stream.camera.location and stream.maxFrameRate
ConfigKey.AeRoi, [left, top, right, bottom]ConfigKey.AutoExposureRegion, [left, right, top, bottom]
ConfigKey.Iso and ConfigKey.ExposureTimeNsNone: cameras always run automatic exposure. Read the values in use with MetadataKey.Iso and MetadataKey.ExposureTimeNs.