Camera Module
Request and receive camera frames from SPECS to build AR experiences that interact with the user's real-world environment.
- Snapchat: not supported
- SPECS: fully supported
- Camera Kit Web: not supported
- Camera Kit Mobile: not supported
- SnapchatNot supported
- SPECSFully supported
- Camera Kit WebNot supported
- Camera Kit MobileNot supported
Overview
CameraModule gives a Lens access to the cameras SPECS makes available to Lenses:
- Discover the available cameras and the streams each one supports.
- Stream camera frames with a
StreamingSession. - Capture high-resolution stills with a
CaptureSession, with an optional crop for each shot.
Each frame provides its pixels as a texture and as raw bytes, with a timestamp and, when the camera reports them, exposure values.
In Lens Studio Preview, getCameras() returns an empty array, so test camera code on SPECS. On device, getCameras(), getStreams(), createStreamingSession() and createCaptureSession() can throw an Error when the camera can't be accessed or camera discovery fails; wrap them in try/catch.
Camera frames are sensitive user data. A Lens that sends frames to a backend (for example, Base64-encoding imageData for an ML or LLM service) combines camera access with internet access, which requires Transparent Permission to be publishable.
Discover Cameras and Streams
getCameras() returns the cameras available to the Lens. Each CameraInfo reports a type (Rgb or Grayscale), a location (Left or Right), a facing (World or User) and a human-readable name. camera.getStreams() returns that camera's stream capabilities, sorted largest-first by pixel area. Each StreamInfo has a fixed width and height, an imageFormat (Jpeg, Yuv420 or Y8), a quality tier (Performance, Balanced or Enhanced), a maxFrameRate and an operation.
Each capability works with one kind of session, reported by StreamInfo.operation:
CameraModule.Operation.Streaming: use with aStreamingSession.CameraModule.Operation.Capture: use with aCaptureSession.
Choose by operation first, then filter by format, size or quality. Don't infer the operation from format or resolution: the same size and format can be offered once for each operation. description is a human-readable label for logs. Its format can change and it isn't unique, so don't parse it or use it as an identifier.
- TypeScript
- JavaScript
const cameraModule: CameraModule = require('LensStudio:CameraModule');
let cameras: CameraModule.CameraInfo[] = [];
try {
cameras = cameraModule.getCameras();
} catch (e) {
print('Camera unavailable: ' + e);
}
const rgb = cameras.find((c) => c.type === CameraModule.CameraType.Rgb);
const streams = rgb ? rgb.getStreams() : [];
const preview = streams.find(
(s) => s.operation === CameraModule.Operation.Streaming
); // largest streaming capability
const photo = streams.find(
(s) =>
s.operation === CameraModule.Operation.Capture &&
s.imageFormat === CameraModule.ImageFormat.Jpeg
); // largest JPEG still
const cameraModule = require('LensStudio:CameraModule');
let cameras = [];
try {
cameras = cameraModule.getCameras();
} catch (e) {
print('Camera unavailable: ' + e);
}
const rgb = cameras.find(function (c) {
return c.type === CameraModule.CameraType.Rgb;
});
const streams = rgb ? rgb.getStreams() : [];
const preview = streams.find(function (s) {
return s.operation === CameraModule.Operation.Streaming;
}); // largest streaming capability
const photo = streams.find(function (s) {
return (
s.operation === CameraModule.Operation.Capture &&
s.imageFormat === CameraModule.ImageFormat.Jpeg
);
}); // largest JPEG still
On current SPECS software, getCameras() returns two world-facing cameras with these capabilities:
| Camera | Size | Format | Quality | Operation |
|---|---|---|---|---|
| Right (RGB) | 4032×3024 | Yuv420 | Enhanced | Capture |
| Right (RGB) | 4032×3024 | Jpeg | Enhanced | Capture |
| Right (RGB) | 2016×1512 | Yuv420 | Enhanced | Streaming |
| Right (RGB) | 2016×1512 | Jpeg | Balanced | Capture |
| Right (RGB) | 1008×756 | Yuv420 | Performance | Streaming |
| Right (RGB) | 1008×756 | Jpeg | Performance | Capture |
| Left (grayscale) | 1124×1364 | Y8 | Performance | Streaming |
Every capability reports a maxFrameRate of 30. Select capabilities at runtime: the list can differ on other devices and software versions.
Stream Frames
A StreamingSession delivers frames through onNewFrame until you stop it. Sessions are restartable: stop() releases the camera and start() can be called again.
Set StreamConfig.minFrameRate to the camera rate your Lens needs; the default, 0, requests the capability's maxFrameRate. The rate is rounded up to a whole number of frames per second. Assigning a negative value throws, and start() throws when the rounded rate exceeds maxFrameRate. The camera can run faster than the rate you request, and delivery to your Lens can be lower than the camera rate when its per-frame work is slow.
This example shows the smallest RGB stream on an Image component:
- TypeScript
- JavaScript
@component
export class CameraStreaming extends BaseScriptComponent {
@input
image: Image;
private cameraModule: CameraModule = require('LensStudio:CameraModule');
private session: CameraModule.StreamingSession;
onAwake() {
const rgb = this.cameraModule
.getCameras()
.find((c) => c.type === CameraModule.CameraType.Rgb);
// Smallest streaming capability: full frame rate at the lowest cost.
const stream = rgb
? rgb
.getStreams()
.filter((s) => s.operation === CameraModule.Operation.Streaming)
.pop()
: undefined;
if (!stream) {
print('No streaming camera available');
return;
}
this.session = this.cameraModule.createStreamingSession();
this.session.onNewFrame.add((frame) => {
if (frame.texture) {
this.image.mainPass.baseTex = frame.texture;
}
});
this.session.onStatusChanged.add((status) => {
if (status === CameraModule.StreamStatus.Disconnected) {
print('Camera disconnected; create a new session to continue');
}
});
const config = new CameraModule.StreamConfig();
config.stream = stream;
config.minFrameRate = 15;
const result = this.session.start(config);
if (result !== CameraModule.Result.Success) {
print('Streaming failed to start: ' + result);
}
this.createEvent('OnDestroyEvent').bind(() => this.session.stop());
}
}
//@input Component.Image image
const cameraModule = require('LensStudio:CameraModule');
const rgb = cameraModule.getCameras().find(function (c) {
return c.type === CameraModule.CameraType.Rgb;
});
// Smallest streaming capability: full frame rate at the lowest cost.
const stream = rgb
? rgb
.getStreams()
.filter(function (s) {
return s.operation === CameraModule.Operation.Streaming;
})
.pop()
: undefined;
if (stream) {
const session = cameraModule.createStreamingSession();
session.onNewFrame.add(function (frame) {
if (script.image && frame.texture) {
script.image.mainPass.baseTex = frame.texture;
}
});
session.onStatusChanged.add(function (status) {
if (status === CameraModule.StreamStatus.Disconnected) {
print('Camera disconnected; create a new session to continue');
}
});
const config = new CameraModule.StreamConfig();
config.stream = stream;
config.minFrameRate = 15;
const result = session.start(config);
if (result !== CameraModule.Result.Success) {
print('Streaming failed to start: ' + result);
}
script.createEvent('OnDestroyEvent').bind(function () {
session.stop();
});
} else {
print('No streaming camera available');
}
Auto-Exposure Region
ConfigKey.AutoExposureRegion steers automatic exposure toward a region of the frame; it doesn't crop. Set it before start() with config.setConfig(key, values), or on a running stream with session.setConfig(key, values). The values are [left, right, top, bottom] in normalized 0.0–1.0 image coordinates, origin top-left, with left < right and top < bottom. An optional fifth value is a weight from 0 to 1 (default 1).
A session has one region, and the rest of the frame can then be brighter or darker. The region can stay in effect after the session stops: set [0, 1, 0, 1] before stop() to meter the whole frame again. It works on the 1008×756 RGB and grayscale streams, and has no effect on the 2016×1512 RGB stream or on capture sessions.
Manual exposure isn't supported: cameras always run automatic exposure. Read the values in use from each frame with frame.getMetadata(CameraModule.MetadataKey.Iso) and MetadataKey.ExposureTimeNs.
Capture Stills
A CaptureSession prepares the still pipeline in start() and returns one Frame for each capture(). Start the session when your Lens loads rather than at shutter press: the first capture also pays for warm-up. One capture can be in flight at a time.
To save power, the system ends a capture session that goes 60 seconds without a capture. onStatusChanged reports Disconnected, and later capture() calls reject. Each capture() restarts the 60-second timer. After Disconnected, create a new session to capture again.
- TypeScript
- JavaScript
@component
export class CameraCapture extends BaseScriptComponent {
@input
image: Image;
private cameraModule: CameraModule = require('LensStudio:CameraModule');
private capture: CameraModule.CaptureSession;
private ready = false;
private busy = false;
onAwake() {
const rgb = this.cameraModule
.getCameras()
.find((c) => c.type === CameraModule.CameraType.Rgb);
const stream = rgb
? rgb
.getStreams()
.find(
(s) =>
s.operation === CameraModule.Operation.Capture &&
s.imageFormat === CameraModule.ImageFormat.Jpeg
)
: undefined;
if (!stream) {
print('No still capture available');
return;
}
this.capture = this.cameraModule.createCaptureSession();
this.capture.onStatusChanged.add((status) => {
if (status === CameraModule.StreamStatus.Disconnected) {
this.ready = false; // create a new session to continue
}
});
const config = new CameraModule.StreamConfig();
config.stream = stream;
this.ready = this.capture.start(config) === CameraModule.Result.Success;
// Capture two seconds after start; call takePhoto() from your own trigger instead.
const delay = this.createEvent('DelayedCallbackEvent');
delay.bind(() => this.takePhoto());
delay.reset(2);
this.createEvent('OnDestroyEvent').bind(() => this.capture.stop());
}
async takePhoto() {
if (!this.ready || this.busy) {
return;
}
this.busy = true;
try {
const overrides = new CameraModule.StreamConfigOverrides();
overrides.crop = Rect.create(-0.5, 0.5, -0.5, 0.5); // centered 50% crop for this shot
const frame = await this.capture.capture(overrides);
if (frame.texture) {
this.image.mainPass.baseTex = frame.texture;
}
} catch (e) {
print('Capture failed: ' + e);
} finally {
this.busy = false;
}
}
}
//@input Component.Image image
const cameraModule = require('LensStudio:CameraModule');
const rgb = cameraModule.getCameras().find(function (c) {
return c.type === CameraModule.CameraType.Rgb;
});
const stream = rgb
? rgb.getStreams().find(function (s) {
return (
s.operation === CameraModule.Operation.Capture &&
s.imageFormat === CameraModule.ImageFormat.Jpeg
);
})
: undefined;
let ready = false;
let busy = false;
const capture = cameraModule.createCaptureSession();
capture.onStatusChanged.add(function (status) {
if (status === CameraModule.StreamStatus.Disconnected) {
ready = false; // create a new session to continue
}
});
if (stream) {
const config = new CameraModule.StreamConfig();
config.stream = stream;
ready = capture.start(config) === CameraModule.Result.Success;
} else {
print('No still capture available');
}
async function takePhoto() {
if (!ready || busy) {
return;
}
busy = true;
try {
const overrides = new CameraModule.StreamConfigOverrides();
overrides.crop = Rect.create(-0.5, 0.5, -0.5, 0.5); // centered 50% crop for this shot
const frame = await capture.capture(overrides);
if (script.image && frame.texture) {
script.image.mainPass.baseTex = frame.texture;
}
} catch (e) {
print('Capture failed: ' + e);
} finally {
busy = false;
}
}
// Capture two seconds after start; call takePhoto() from your own trigger instead.
const delay = script.createEvent('DelayedCallbackEvent');
delay.bind(takePhoto);
delay.reset(2);
script.createEvent('OnDestroyEvent').bind(function () {
capture.stop();
});
capture() rejects when the session isn't started, another capture is in flight, the crop isn't finite, the session stops or disconnects before the frame arrives, the system rejects the request, or no frame arrives within 30 seconds. A frame whose image couldn't be read still resolves, with null texture and imageData, so check before use.
Still Size
A capture session returns stills at its capability's size unless you set StreamConfig.outputWidth and outputHeight. Any positive, even size up to the capability's width and height works, for example a 756×1008 portrait still from the 4032×3024 capability. A size with another aspect ratio than the capability is a centered crop of the camera's view, so the image is never stretched.
config.stream = photo; // a Capture capability, for example 4032x3024 Jpeg
config.outputWidth = 756;
config.outputHeight = 1008;
Assigning zero, a negative or an odd value throws. start() throws when only one of the two is set, when the size is larger than the capability, or when either is set on a StreamingSession: streams always have their capability's size. Set both to null to use the capability's size again.
Crop
StreamConfig.crop sets a default crop for every capture of the session; StreamConfigOverrides.crop overrides it for one shot. Crops are Rect values in normalized image space: [-1, 1] on each axis, with +1 at the top and right of the image, created with Rect.create(left, right, bottom, top). The cropped region is scaled to the still's size: outputWidth×outputHeight when set, otherwise the capability's size. Give the crop the same aspect ratio as the still. Streaming sessions ignore crops.
| Region | Rect |
|---|---|
| Full image | Rect.create(-1, 1, -1, 1) |
| Centered 50% | Rect.create(-0.5, 0.5, -0.5, 0.5) |
| Top-left quadrant | Rect.create(-1, 0, 0, 1) |
| Bottom-right quadrant | Rect.create(0, 1, -1, 0) |
Crop and AutoExposureRegion use different coordinate spaces: crop is [-1, 1] normalized image space with y up, AutoExposureRegion is 0..1 image coordinates with y down.
Noise Reduction
Set ConfigKey.SnapshotMfnrEnable with config.setConfig() before start(): [1] turns on multi-frame noise reduction, [0] turns it off. Captures take longer with it. It applies to capture sessions only: a streaming session returns Result.InvalidConfig.
Work with Frames
-
frame.imageDatais the raw image, tightly packed:Jpeg: the JPEG file.Yuv420: the Y plane (width * heightbytes), then half-resolution chroma interleaved V then U (NV21,width * height / 2bytes).Y8: one byte per pixel.
The array shares the frame's buffer; treat it as read-only.
-
frame.textureis an RGBA texture of the image; grayscale frames have R = G = B. Each session reuses one Texture: readingtextureon a newer frame, including the next capture, replaces the image in the Texture you got from an earlier frame, and a later session of the same size can reuse it afterstop(). To keep an image, keep itsimageData, or copy the texture (see below). After a session stops, readingtextureon a frame you kept decodes the image again into a new texture, so read it while the session runs. -
Reading
textureconverts the frame to RGBA and uploads it to the GPU, which costs the most at 2016×1512. Read it only on the frames you display or process. Reading it on occasional frames costs more per read, because the conversion then runs when you read it. -
Streaming frames are short-lived. A session keeps the pixels of its two most recent frames only. On older frames
texturereturns null, and so doesimageDataunless your Lens still holds the array it returned earlier. Read what you need insideonNewFrame. AnimageDataarray you have already read stays valid and keeps that frame's bytes until it's garbage collected, so readimageDataonly on the frames you process. Captured frames keep their pixels for as long as you reference them. -
frame.getMetadata(key)returnsTimestampNs(CLOCK_MONOTONIC, always present) andIso,ExposureTimeNsandFrameRatewhen the camera reports them. It returnsnullwhen a value is unavailable.
Read Texture Pixels
Assigning a frame texture to a material (image.mainPass.baseTex = frame.texture) works inside onNewFrame, and Base64.encodeTextureAsync works on a captured frame right after await capture(). To read pixels on the CPU, read frame.texture inside onNewFrame, keep that Texture, and copy it with ProceduralTextureProvider.createFromTexture in an UpdateEvent: inside onNewFrame or right after await capture(), createFromTexture throws an error. Hand the texture to your update handler instead:
- TypeScript
- JavaScript
@component
export class CameraPixels extends BaseScriptComponent {
private cameraModule: CameraModule = require('LensStudio:CameraModule');
private session: CameraModule.StreamingSession;
private pending: Texture | null = null;
private pixel = new Uint8Array(4);
onAwake() {
const rgb = this.cameraModule
.getCameras()
.find((c) => c.type === CameraModule.CameraType.Rgb);
const stream = rgb
? rgb
.getStreams()
.filter((s) => s.operation === CameraModule.Operation.Streaming)
.pop()
: undefined;
if (!stream) {
print('No streaming camera available');
return;
}
this.session = this.cameraModule.createStreamingSession();
this.session.onNewFrame.add((frame) => {
this.pending = frame.texture; // keep a reference only
});
const config = new CameraModule.StreamConfig();
config.stream = stream;
const result = this.session.start(config);
if (result !== CameraModule.Result.Success) {
print('Streaming failed to start: ' + result);
return;
}
this.createEvent('UpdateEvent').bind(() => {
if (this.pending) {
const copy = ProceduralTextureProvider.createFromTexture(this.pending);
this.pending = null;
const provider = copy.control as ProceduralTextureProvider;
provider.getPixels(0, 0, 1, 1, this.pixel); // RGBA of the pixel at (0, 0)
}
});
this.createEvent('OnDestroyEvent').bind(() => this.session.stop());
}
}
const cameraModule = require('LensStudio:CameraModule');
const rgb = cameraModule.getCameras().find(function (c) {
return c.type === CameraModule.CameraType.Rgb;
});
const stream = rgb
? rgb
.getStreams()
.filter(function (s) {
return s.operation === CameraModule.Operation.Streaming;
})
.pop()
: undefined;
if (stream) {
const session = cameraModule.createStreamingSession();
const pixel = new Uint8Array(4);
let pending = null;
session.onNewFrame.add(function (frame) {
pending = frame.texture; // keep a reference only
});
const config = new CameraModule.StreamConfig();
config.stream = stream;
const result = session.start(config);
if (result !== CameraModule.Result.Success) {
print('Streaming failed to start: ' + result);
}
script.createEvent('UpdateEvent').bind(function () {
if (pending) {
const copy = ProceduralTextureProvider.createFromTexture(pending);
pending = null;
copy.control.getPixels(0, 0, 1, 1, pixel); // RGBA of the pixel at (0, 0)
}
});
script.createEvent('OnDestroyEvent').bind(function () {
session.stop();
});
} else {
print('No streaming camera available');
}
Results and Status
start(), setConfig() and stop() return a CameraModule.Result. Invalid values of minFrameRate, outputWidth and outputHeight throw instead, where you assign them or from start().
| Result | Meaning |
|---|---|
Success | The call succeeded. |
NotInitialized | The camera isn't available to the session (rare). |
InvalidArgument | Missing stream, an invalid setConfig() value, or a non-finite crop. |
InvalidConfig | The capability's operation doesn't match the session or it can't be opened, or a key isn't allowed here. |
AlreadyStarted | start() on a running session. |
NotStarted | stop() or setConfig() before start(). |
SessionStopped | The session was disconnected; create a new one. |
ErrorGeneric | The camera couldn't be reached or opened, a setConfig() change couldn't be applied, or a session limit was reached. |
onStatusChanged reports:
Interrupted: frames pause, for example while the camera switches modes because another session started, or while the camera recovers from an error. The session stays started. The camera can drop a capture in flight: its Promise rejects about 30 seconds aftercapture(). Capture again afterResumed.Resumed: frames flow again.Disconnected: the system ended the session: camera access was revoked, the system stopped the stream (for example, another stream switched the camera to a mode without this stream's size), the camera system restarted, or a capture session went 60 seconds without a capture. The session is finished; create a new one.
Sessions stop automatically when the Lens closes.
Known Limitations
- Still capture from the grayscale camera isn't supported; use the RGB camera.
- A Lens can run up to three streaming sessions across all its CameraModules, at most one at 2016×1512, or one streaming session and one capture session.
start()past a limit returnsResult.ErrorGeneric; stop a session to free its slot. - Starting a capture session while an RGB stream runs briefly interrupts the stream (
Interrupted, thenResumed). - Session frames don't provide the camera's intrinsic and extrinsic parameters yet; see Projecting 3D Points.
- Full-resolution images are large: a 4032×3024 frame's
textureholds 48.8 MB of RGBA, and itsYuv420imageData18.3 MB. A captured frame keeps its pixels while you reference it, so drop captures you're done with and keep few full-resolution images at once. The script engine frees a dropped capture only when it next collects garbage. Until then, many 4032×3024Yuv420captures can add up to hundreds of MB. For repeated full-size stills, use the much smallerJpeg. Under system memory pressure, SPECS can close a Lens that uses a lot of memory.
Projecting 3D Points
Session frames don't provide the camera's intrinsic and extrinsic parameters yet. To project between 3D space and the camera image, use requestCamera for the camera texture and DeviceCamera for the camera's parameters.
Camera Texture
Create a CameraRequest and set its cameraId: Default_Color for the RGB camera or Left_Grayscale for the grayscale camera. SPECS doesn't support Left_Color, and Right_Color returns the same RGB camera as Default_Color. imageSmallerDimension sets the size in pixels of the smaller dimension to return. requestCamera returns a Texture that updates with each frame. The texture contains the opaque handle that refers to the underlying camera data, and can be used as an input into an MLComponent or uploaded to a remote service via RemoteMediaModule. To be notified of each new frame, use its CameraTextureProvider, accessed via cameraTexture.control, and its onNewFrame event:
- TypeScript
- JavaScript
@component
export class CameraFrameExample extends BaseScriptComponent {
private cameraModule: CameraModule = require('LensStudio:CameraModule');
onAwake() {
this.createEvent('OnStartEvent').bind(() => {
const cameraRequest = CameraModule.createCameraRequest();
cameraRequest.cameraId = CameraModule.CameraId.Default_Color;
const cameraTexture = this.cameraModule.requestCamera(cameraRequest);
const onNewFrame = (cameraTexture.control as CameraTextureProvider)
.onNewFrame;
const registration = onNewFrame.add((frame: CameraFrame) => {
// Process the frame
});
this.createEvent('OnDestroyEvent').bind(() => {
onNewFrame.remove(registration);
});
});
}
}
const cameraModule = require('LensStudio:CameraModule');
script.createEvent('OnStartEvent').bind(function () {
const cameraRequest = CameraModule.createCameraRequest();
cameraRequest.cameraId = CameraModule.CameraId.Default_Color;
const cameraTexture = cameraModule.requestCamera(cameraRequest);
const onNewFrame = cameraTexture.control.onNewFrame;
const registration = onNewFrame.add(function (frame) {
// Process the frame
});
script.createEvent('OnDestroyEvent').bind(function () {
onNewFrame.remove(registration);
});
});
Don't call createCameraRequest() in onAwake; the examples call it from OnStartEvent.
Camera Information
DeviceCamera, from the DeviceInfoSystem, describes how the camera maps 3D space to the 2D image: its focal length, principal point, resolution and pose. Its project and unproject methods convert 3D points to 2D image points and back.
- TypeScript
- JavaScript
@component
export class CameraInfoExample extends BaseScriptComponent {
onAwake() {
this.createEvent('OnStartEvent').bind(() => {
// Select the camera
const camera = global.deviceInfoSystem.getTrackingCameraForId(
CameraModule.CameraId.Default_Color
);
// Retrieve camera properties
const focalLength = camera.focalLength;
const principalPoint = camera.principalPoint;
const resolution = camera.resolution;
const pose = camera.pose;
});
}
}
// Select the camera
script.createEvent('OnStartEvent').bind(function () {
const camera = global.deviceInfoSystem.getTrackingCameraForId(
CameraModule.CameraId.Default_Color
);
// Retrieve camera properties
const focalLength = camera.focalLength;
const principalPoint = camera.principalPoint;
const resolution = camera.resolution;
const pose = camera.pose;
});
Deprecated APIs
These APIs are deprecated and left out of the Lens Studio type definitions. Use the replacements in new Lenses.
| Deprecated | Use instead |
|---|---|
requestImage | A CaptureSession. requestImage doesn't work on SPECS. |
getSupportedStreams() and StreamFilter | getCameras() and camera.getStreams() |
StreamInfo.type, location and frameRate | stream.camera.type, stream.camera.location and stream.maxFrameRate |
ConfigKey.AeRoi, [left, top, right, bottom] | ConfigKey.AutoExposureRegion, [left, right, top, bottom] |
ConfigKey.Iso and ConfigKey.ExposureTimeNs | None: cameras always run automatic exposure. Read the values in use with MetadataKey.Iso and MetadataKey.ExposureTimeNs. |