Media
Capture images, video, and microphone audio on SPECS, upload them to Snap Cloud Storage, or stream them live over Realtime to a web viewer.
- Snapchat: not supported
- SPECS: fully supported
- Camera Kit Web: not supported
- Camera Kit Mobile: not supported
- SnapchatNot supported
- SPECSFully supported
- Camera Kit WebNot supported
- Camera Kit MobileNot supported
The Media Suite in the Snap Cloud Examples package captures camera frames, video, and microphone audio from a SPECS Lens. It either uploads them to Snap Cloud Storage or streams them live over Realtime to a web page. This page explains each example and its core code.
Before you start
- Set up Snap Cloud: see Get Started. The code on this page adds members to the
SnapCloudStarterscript from Create the client and sign in and uses itsclient,uidanddelaymembers. - To run the package's own examples, install the Snap Cloud Examples package and place its prefab: see Examples.
- Create a Storage bucket for the uploads, for example
specs-bucket, and add policies that let authenticated users upload: see Storage.
The Media Suite panel switches between five sections with its side bar: Image, Video, Audio, Composite, and Sharing. All of its scripts read the project from the prefab's shared SnapCloudRequirements component, and all of them capture through one shared CameraService (in [CompositeImage]), so they don't make competing camera requests. Replace the prefab's placeholder bucket and channel names before you test.
A Lens that sends camera frames or microphone audio to a backend combines that access with internet access, which requires Transparent Permission to be publishable.
Capture a still image and upload it to Storage
The Image example (ImageCaptureUploader) captures a high-resolution still from the camera, encodes it as JPEG with Base64, and uploads the bytes to a Storage bucket. Instead of the raw camera image, it can capture a composite texture: the camera with the rendered AR content.
To set it up, select the ImageCaptureUploader component and set Storage Bucket. A capture button (a UI Kit RectangleButton) is optional.
The example requests the still with CameraModule's requestImage. The Camera Module guide documents a CaptureSession for high-resolution stills on SPECS.
Add these members to the SnapCloudStarter script and call captureAndUpload() from a button tap:
@input storageBucket: string = 'specs-bucket';
private cameraModule: CameraModule = require('LensStudio:CameraModule');
// Called from a button tap or programmatically
async captureAndUpload() {
if (!this.uid) {
print('Not authenticated');
return;
}
// Request a high-resolution still image from the camera
const imageRequest = CameraModule.createImageRequest();
// The package sets the camera ID, which ImageRequest doesn't declare:
// Default_Color in Lens Studio, Right_Color on the device
(imageRequest as any).cameraId = global.deviceInfoSystem.isEditor()
? CameraModule.CameraId.Default_Color
: CameraModule.CameraId.Right_Color;
const imageFrame = await this.cameraModule.requestImage(imageRequest);
const texture = imageFrame.texture;
if (!texture) {
print('No texture from camera');
return;
}
print('Captured: ' + texture.getWidth() + 'x' + texture.getHeight());
// Encode to JPEG, then decode the Base64 string to bytes
Base64.encodeTextureAsync(
texture,
async (base64String) => {
const bytes = Base64.decode(base64String);
await this.uploadToStorage(bytes);
},
() => print('Base64.encodeTextureAsync failed: texture not loaded'),
CompressionQuality.HighQuality,
EncodingType.Jpg
);
}
async uploadToStorage(imageBytes: Uint8Array) {
const fileName = 'captures/' + this.uid + '_' + Date.now() + '.jpg';
const { data, error } = await this.client.storage
.from(this.storageBucket)
.upload(fileName, imageBytes, {
contentType: 'image/jpeg',
upsert: true,
});
if (error) {
print('Upload failed: ' + error.message);
} else {
print('Uploaded: ' + data.path);
}
}
- Stills versus video.
requestImagereturns a single higher-resolution frame, not a frame from the live stream. For video, read frames from the camera texture'sonNewFrameevent instead (next section). - Composite texture. Set
useCompositeTextureto true and assign a Render Target texture to capture the final view with AR content instead of the raw camera image. Wait at least two frames after enabling it before encoding: a render target may have no pixel data on the first frame. - Folders per user. Prefix uploaded file names with the user ID (
captures/<uid>_<timestamp>.jpg) to keep files organized and scope storage policies to each user's folder.
Record video frames and upload them to Storage
The Video example (VideoCaptureUploader) records JPEG frames from the camera at a frame rate you set and keeps them in memory while recording, to avoid I/O stutter. When the user stops recording, it uploads every frame to Storage under frames/<session>/. To turn the frames into a video file, see Stitch uploaded frames and audio into a video.
To set it up, use the same bucket as the Image example, set Storage Bucket, and assign a record button (start and stop).
Add these members to the SnapCloudStarter script. Call setUpCamera() from onSignedIn(), and startRecording() and stopRecording() from your button.
@input storageBucket: string = 'specs-bucket';
@input captureFrameRate: number = 15; // target FPS (0 = max)
@input maxDurationSeconds: number = 30;
private isRecording: boolean = false;
private sessionId: string = '';
private capturedFrames: { bytes: Uint8Array; index: number }[] = [];
private cameraTexture: Texture;
private lastFrameTime: number = 0;
private frameInterval: number = 0;
setUpCamera() {
this.frameInterval =
this.captureFrameRate > 0 ? 1 / this.captureFrameRate : 0;
const cameraModule = require('LensStudio:CameraModule') as CameraModule;
const request = CameraModule.createCameraRequest();
request.cameraId = global.deviceInfoSystem.isEditor()
? CameraModule.CameraId.Default_Color
: CameraModule.CameraId.Right_Color;
this.cameraTexture = cameraModule.requestCamera(request);
const provider = this.cameraTexture.control as CameraTextureProvider;
// Capture frames from onNewFrame for accurate timing
provider.onNewFrame.add(() => {
if (!this.isRecording) return;
const now = getTime();
if (
this.frameInterval > 0 &&
now - this.lastFrameTime < this.frameInterval
)
return;
this.lastFrameTime = now;
this.captureFrame();
});
}
startRecording() {
if (this.isRecording) return;
this.capturedFrames = [];
this.sessionId = 'vid_' + Date.now();
this.isRecording = true;
print('Recording started: ' + this.sessionId);
// Stop automatically after the maximum duration
const stopEvent = this.createEvent('DelayedCallbackEvent');
stopEvent.bind(() => {
if (this.isRecording) this.stopRecording();
});
stopEvent.reset(this.maxDurationSeconds);
}
stopRecording() {
if (!this.isRecording) return;
this.isRecording = false;
print('Recording stopped. Frames: ' + this.capturedFrames.length);
this.uploadSession();
}
// Stores JPEG bytes in memory: no uploads during recording
captureFrame() {
const frameIndex = this.capturedFrames.length;
Base64.encodeTextureAsync(
this.cameraTexture,
(b64) => {
this.capturedFrames.push({ bytes: Base64.decode(b64), index: frameIndex });
},
() => print('Frame encoding failed'),
CompressionQuality.LowQuality,
EncodingType.Jpg
);
}
// Uploads every frame after recording stops
async uploadSession() {
if (this.capturedFrames.length === 0) return;
print('Uploading ' + this.capturedFrames.length + ' frames...');
for (const frame of this.capturedFrames) {
const path = `frames/${this.sessionId}/frame_${String(frame.index).padStart(5, '0')}.jpg`;
const { error } = await this.client.storage
.from(this.storageBucket)
.upload(path, frame.bytes, { contentType: 'image/jpeg', upsert: true });
if (error) print('Frame upload error: ' + error.message);
}
print('Session uploaded: ' + this.sessionId);
this.capturedFrames = [];
}
- Capture to memory, upload after. Uploading during recording competes with capture for I/O and drops frames. The script keeps every frame as a
Uint8Arrayin an array and starts uploading only after the user stops recording. - Use
onNewFrame, notUpdateEvent. The camera texture provider'sonNewFrameevent fires at the camera's real frame rate, when new pixels arrive. UpdateEvent fires at the render rate and can capture the same camera frame twice.
Record microphone audio and upload it as WAV
The Audio example (AudioCaptureUploader) reads the microphone through MicrophoneAudioProvider, collects audio frames while recording, and after the user stops recording, converts them to a 16-bit PCM WAV file and uploads it to Storage under audio-recordings/.
- In the Asset Browser, add a Microphone Audio asset, and assign it to the component's Microphone Asset input.
- Set Storage Bucket on the
AudioCaptureUploadercomponent. - Lens Studio declares the microphone permission for you. To check it, open Project Settings and select Platform Settings > Lens Permissions.
Add these members to the SnapCloudStarter script. Call setUpMicrophone() from onSignedIn(), and startRecording() and stopRecording() from your button.
@input storageBucket: string = 'specs-bucket';
@input storageFolder: string = 'audio-recordings';
@input microphoneAsset: AudioTrackAsset;
@input sampleRate: number = 16000; // 16 kHz is recommended for voice
private micControl: MicrophoneAudioProvider;
private isRecording: boolean = false;
private sessionId: string = '';
private recordedFrames: Float32Array[] = [];
private updateEvent: UpdateEvent;
setUpMicrophone() {
this.micControl = this.microphoneAsset.control as MicrophoneAudioProvider;
this.micControl.sampleRate = this.sampleRate;
// The device may not support the requested rate: use the rate it reports
this.sampleRate = this.micControl.sampleRate;
// Collect audio frames every update while recording
this.updateEvent = this.createEvent('UpdateEvent');
this.updateEvent.bind(() => {
if (!this.isRecording) return;
const frame = new Float32Array(this.micControl.maxFrameSize);
const shape = this.micControl.getAudioFrame(frame);
if (shape.x > 0) {
this.recordedFrames.push(frame.subarray(0, shape.x));
}
});
this.updateEvent.enabled = false;
}
startRecording() {
if (this.isRecording) return;
this.recordedFrames = [];
this.sessionId = 'audio_' + Date.now();
this.isRecording = true;
this.micControl.start();
this.updateEvent.enabled = true;
print('Audio recording started');
}
stopRecording() {
if (!this.isRecording) return;
this.isRecording = false;
this.micControl.stop();
this.updateEvent.enabled = false;
print('Audio recording stopped. Frames: ' + this.recordedFrames.length);
this.convertAndUpload();
}
async convertAndUpload() {
if (this.recordedFrames.length === 0) return;
// Build a WAV file from the recorded frames
const wavBytes = this.buildWav(this.recordedFrames, this.sampleRate);
const path = `${this.storageFolder}/${this.sessionId}.wav`;
const { data, error } = await this.client.storage
.from(this.storageBucket)
.upload(path, wavBytes, { contentType: 'audio/wav', upsert: true });
if (error) {
print('Audio upload failed: ' + error.message);
} else {
print('Audio uploaded: ' + data.path);
}
this.recordedFrames = [];
}
buildWav(frames: Float32Array[], sampleRate: number): Uint8Array {
// Flatten all frames and convert Float32 to Int16
const totalSamples = frames.reduce((sum, f) => sum + f.length, 0);
const pcm16 = new Int16Array(totalSamples);
let offset = 0;
for (const frame of frames) {
for (let i = 0; i < frame.length; i++) {
pcm16[offset++] = Math.max(-32768, Math.min(32767, frame[i] * 32767));
}
}
// Write the WAV header, then the PCM data
const dataBytes = totalSamples * 2;
const buffer = new ArrayBuffer(44 + dataBytes);
const view = new DataView(buffer);
const writeStr = (o: number, s: string) => {
for (let i = 0; i < s.length; i++) view.setUint8(o + i, s.charCodeAt(i));
};
writeStr(0, 'RIFF');
view.setUint32(4, 36 + dataBytes, true);
writeStr(8, 'WAVE');
writeStr(12, 'fmt ');
view.setUint32(16, 16, true);
view.setUint16(20, 1, true); // PCM
view.setUint16(22, 1, true); // mono
view.setUint32(24, sampleRate, true);
view.setUint32(28, sampleRate * 2, true);
view.setUint16(32, 2, true);
view.setUint16(34, 16, true);
writeStr(36, 'data');
view.setUint32(40, dataBytes, true);
for (let i = 0; i < pcm16.length; i++) {
view.setInt16(44 + i * 2, pcm16[i], true);
}
return new Uint8Array(buffer);
}
- Choose the sample rate for the content. Use 16,000 Hz for voice. Higher rates (44.1 kHz, 48 kHz) make files much larger without making speech clearer. If playback sounds high-pitched, the device didn't support the requested rate: fall back to 16 kHz, and write the rate the microphone reports into the WAV header, as the script does.
- Convert after recording. The script converts Float32 to 16-bit PCM and writes the WAV header after recording stops. Doing that work during recording would miss frames and lower the audio quality.
Stream live camera frames to a web viewer
The Video streaming example (VideoStreamingController) streams live JPEG frames from the SPECS camera to any web page subscribed to the same Realtime channel. Unlike the Video upload example, nothing is stored: frames travel over the channel as they're captured. It streams either the raw camera image or a composite (camera with AR content), with a buffer delay for composite mode.
- Select the
VideoStreamingControllercomponent. - Assign the
CameraServiceand a stream button. - Set Streaming Channel Name, for example
live-video-stream. The web viewer must use the same name. - Keep Stream Quality at 15 and Resolution Scale at 0.3: at those values frames stay under the 250 KB Realtime message limit.
- Open the web viewer in a browser, enter your project URL, anon key, and the same channel name, then connect.
Add these members to the SnapCloudStarter script, call setUpVideoStream() from onSignedIn(), and call startStreaming() and stopStreaming() from your button. CameraService comes from the Snap Cloud Examples package: import { CameraService } from 'SnapCloudExamples.lspkg/CompositeCameraWork/Scripts/CameraService'. Add RealtimeChannel to the starter's import from SupabaseClient.lspkg/supabase-snapcloud.
@input streamingChannelName: string = 'live-video-stream';
// Keep LOW to stay under the 250 KB Realtime message limit
@input @widget(new SliderWidget(1, 100, 1)) streamQuality: number = 15;
@input @widget(new SliderWidget(1, 30, 1)) streamFPS: number = 30;
@input @widget(new SliderWidget(0.1, 1.0, 0.1)) resolutionScale: number = 0.3;
@input cameraService: CameraService;
@input useCompositeTexture: boolean = false;
@input @allowUndefined compositeTexture: Texture;
private videoChannel: RealtimeChannel;
private isStreaming: boolean = false;
private frameCount: number = 0;
private streamSessionId: string = '';
private streamCameraTexture: Texture;
private streamProvider: CameraTextureProvider;
private frameRegistration: EventRegistration;
private lastStreamTime: number = 0;
setUpVideoStream() {
// Use the shared CameraService texture: no competing camera requests
this.streamCameraTexture = this.cameraService.cameraTexture;
this.streamProvider = this.cameraService.cameraTextureProvider;
this.videoChannel = this.client.channel(this.streamingChannelName, {
config: { broadcast: { self: false } },
});
this.videoChannel.subscribe((status) => {
if (status === 'SUBSCRIBED') print('Streaming channel ready');
});
}
startStreaming() {
if (this.isStreaming) return;
this.isStreaming = true;
this.streamSessionId = 'stream_' + Date.now();
this.frameCount = 0;
const frameInterval = 1000 / this.streamFPS;
// Camera mode: use onNewFrame for accurate timing
this.frameRegistration = this.streamProvider.onNewFrame.add(() => {
if (!this.isStreaming) return;
const now = Date.now();
if (now - this.lastStreamTime < frameInterval) return;
this.lastStreamTime = now;
this.streamFrame();
});
}
async streamFrame() {
const texture =
this.useCompositeTexture && this.compositeTexture
? this.compositeTexture
: this.streamCameraTexture;
if (!texture || texture.getWidth() === 0) return;
this.frameCount++;
const base64 = await new Promise<string>((resolve, reject) => {
const quality =
this.streamQuality > 50
? CompressionQuality.IntermediateQuality
: CompressionQuality.LowQuality;
Base64.encodeTextureAsync(texture, resolve, reject, quality, EncodingType.Jpg);
});
const frameData = base64 + '|||FRAME_END|||';
// Warn when a frame nears the 250 KB Realtime limit
if (frameData.length > 250000) {
print('WARNING: frame exceeds 250 KB, reduce quality or resolution');
}
this.videoChannel.send({
type: 'broadcast',
event: 'video-frame',
payload: {
sessionId: this.streamSessionId,
frameNumber: this.frameCount,
frameData,
metadata: { fps: this.streamFPS, quality: this.streamQuality },
},
});
}
stopStreaming() {
this.isStreaming = false;
if (this.frameRegistration) {
this.streamProvider.onNewFrame.remove(this.frameRegistration);
this.frameRegistration = null;
}
this.videoChannel.send({
type: 'broadcast',
event: 'stream-ended',
payload: {
sessionId: this.streamSessionId,
totalFrames: this.frameCount,
},
});
}
Video stream web viewer
The companion page subscribes to the same channel and draws each incoming JPEG frame on a <canvas>, using requestAnimationFrame for smooth playback. The package ships the full page as ExternalServicesExamples/media-example-web-viewers/video-stream-viewer.html; this is its connection logic:
<!-- video-stream-viewer.html: key connection logic -->
<script src="https://unpkg.com/@supabase/supabase-js@2"></script>
<canvas id="streamCanvas" width="640" height="480"></canvas>
<script>
const client = window.supabase.createClient(SUPABASE_URL, SUPABASE_ANON_KEY, {
realtime: { heartbeatIntervalMs: 2500 },
});
const channel = client.channel('live-video-stream', {
config: { broadcast: { self: false } },
});
const canvas = document.getElementById('streamCanvas');
const ctx = canvas.getContext('2d');
const frameBuffer = [];
channel
.on('broadcast', { event: 'video-frame' }, (msg) => {
const { frameData, frameNumber } = msg.payload;
// Strip the frame marker and decode the JPEG
const base64 = frameData.replace('|||FRAME_END|||', '');
const img = new Image();
img.onload = () => frameBuffer.push(img);
img.src = 'data:image/jpeg;base64,' + base64;
})
.on('broadcast', { event: 'stream-ended' }, () => {
console.log('Stream ended');
})
.subscribe((status) => {
if (status === 'SUBSCRIBED')
console.log('Connected, waiting for stream');
});
// Draw frames from the buffer with requestAnimationFrame
function render() {
if (frameBuffer.length > 0) {
const frame = frameBuffer.shift();
canvas.width = frame.width;
canvas.height = frame.height;
ctx.drawImage(frame, 0, 0);
}
requestAnimationFrame(render);
}
render();
</script>
- Stay under the 250 KB Realtime limit. Realtime allows at most 250 KB per message. At a stream quality of 15 and a resolution scale of 0.3, an encoded frame is typically 30–80 KB. Raising quality or resolution quickly pushes frames over the limit, and they're dropped without an error.
- Composite mode needs a buffer delay. When streaming the composite texture, the render target may have no valid pixels for the first few frames. The package's
VideoStreamingControllerfills a frame queue forcompositeBufferDelayseconds (default 5) before it starts sending. - Use
onNewFrame, notUpdateEvent, in camera mode.onNewFramefires at the camera's real frame rate and guarantees new pixels.UpdateEventcan fire between camera frames and send duplicate or black frames.
Stream live microphone audio to a web listener
The Audio streaming example (AudioStreamingController) records microphone audio in 500 ms chunks, converts each chunk to WAV, Base64-encodes it, and broadcasts it on a Realtime channel. A companion web page decodes each chunk and plays them in sequence with the Web Audio API, for near-real-time audio.
- Add a Microphone Audio asset and assign it to the component's Microphone Asset input.
- Set Streaming Channel Name, for example
live-audio-stream. - Lens Studio declares the microphone permission for you. To check it, open Project Settings and select Platform Settings > Lens Permissions.
- Open the web listener (
ExternalServicesExamples/media-example-web-viewers/audio-stream-listener.html) in a browser and connect to the same channel.
Add these members to the SnapCloudStarter script, call setUpAudioStream() from onSignedIn(), and call startAudioStreaming() and stopAudioStreaming() from your button. Add RealtimeChannel to the starter's import.
@input audioChannelName: string = 'live-audio-stream';
@input microphoneAsset: AudioTrackAsset;
@input @widget(new SliderWidget(8000, 48000, 1000)) sampleRate: number = 16000;
@input @widget(new SliderWidget(100, 1000, 100)) chunkSizeMs: number = 500;
private audioChannel: RealtimeChannel;
private micControl: MicrophoneAudioProvider;
private isAudioStreaming: boolean = false;
private chunkCount: number = 0;
private audioBuffer: Float32Array[] = [];
private audioUpdateEvent: UpdateEvent;
private streamingInterval: DelayedCallbackEvent;
private audioSessionId: string = '';
setUpAudioStream() {
this.micControl = this.microphoneAsset.control as MicrophoneAudioProvider;
this.micControl.sampleRate = this.sampleRate;
// Collect raw audio frames every update while streaming
this.audioUpdateEvent = this.createEvent('UpdateEvent');
this.audioUpdateEvent.bind(() => {
if (!this.isAudioStreaming) return;
const frame = new Float32Array(this.micControl.maxFrameSize);
const shape = this.micControl.getAudioFrame(frame);
if (shape.x > 0) {
this.audioBuffer.push(frame.subarray(0, shape.x));
}
});
this.audioUpdateEvent.enabled = false;
this.audioChannel = this.client.channel(this.audioChannelName, {
config: { broadcast: { self: false } },
});
this.audioChannel.subscribe((status) => {
if (status === 'SUBSCRIBED') print('Audio streaming channel ready');
});
}
startAudioStreaming() {
if (this.isAudioStreaming) return;
this.isAudioStreaming = true;
this.audioSessionId = 'audio_' + Date.now();
this.chunkCount = 0;
this.audioBuffer = [];
this.micControl.start();
this.audioUpdateEvent.enabled = true;
// Send a WAV chunk every chunkSizeMs milliseconds
const sendChunk = () => {
if (!this.isAudioStreaming) return;
if (this.audioBuffer.length > 0) {
const chunk = this.buildWavChunk(this.audioBuffer);
this.audioBuffer = [];
this.chunkCount++;
this.audioChannel.send({
type: 'broadcast',
event: 'audio-chunk',
payload: {
sessionId: this.audioSessionId,
chunkNumber: this.chunkCount,
data: chunk, // Base64-encoded WAV
metadata: {
sampleRate: this.sampleRate,
format: 'wav',
channels: 1,
},
},
});
}
this.streamingInterval = this.createEvent('DelayedCallbackEvent');
this.streamingInterval.bind(sendChunk);
this.streamingInterval.reset(this.chunkSizeMs / 1000);
};
sendChunk();
}
// Float32 frames to 16-bit PCM WAV to Base64
buildWavChunk(frames: Float32Array[]): string {
let total = 0;
for (const f of frames) total += f.length;
const pcm = new Int16Array(total);
let i = 0;
for (const f of frames) {
for (let s = 0; s < f.length; s++) {
pcm[i++] = Math.max(-32768, Math.min(32767, f[s] * 32767));
}
}
const dataSize = pcm.length * 2;
const buf = new ArrayBuffer(44 + dataSize);
const v = new DataView(buf);
const str = (o: number, s: string) => {
for (let j = 0; j < s.length; j++) v.setUint8(o + j, s.charCodeAt(j));
};
str(0, 'RIFF');
v.setUint32(4, 36 + dataSize, true);
str(8, 'WAVE');
str(12, 'fmt ');
v.setUint32(16, 16, true);
v.setUint16(20, 1, true);
v.setUint16(22, 1, true);
v.setUint32(24, this.sampleRate, true);
v.setUint32(28, this.sampleRate * 2, true);
v.setUint16(32, 2, true);
v.setUint16(34, 16, true);
str(36, 'data');
v.setUint32(40, dataSize, true);
for (let j = 0; j < pcm.length; j++) v.setInt16(44 + j * 2, pcm[j], true);
return Base64.encode(new Uint8Array(buf));
}
stopAudioStreaming() {
this.isAudioStreaming = false;
this.micControl.stop();
this.audioUpdateEvent.enabled = false;
}
Audio stream web listener
The companion page uses the Web Audio API to decode and play each incoming WAV chunk:
<!-- audio-stream-listener.html: key playback logic -->
<script src="https://unpkg.com/@supabase/supabase-js@2"></script>
<script>
const audioCtx = new AudioContext();
const client = window.supabase.createClient(SUPABASE_URL, SUPABASE_ANON_KEY, {
realtime: { heartbeatIntervalMs: 2500 },
});
const channel = client.channel('live-audio-stream', {
config: { broadcast: { self: false } },
});
channel
.on('broadcast', { event: 'audio-chunk' }, async (msg) => {
const { data, metadata } = msg.payload;
// Decode Base64 WAV to ArrayBuffer to AudioBuffer, then play
const binary = atob(data);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i);
try {
const audioBuffer = await audioCtx.decodeAudioData(bytes.buffer);
const source = audioCtx.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioCtx.destination);
source.start();
} catch (err) {
console.error('Audio decode error:', err);
}
})
.subscribe((status) => {
if (status === 'SUBSCRIBED') console.log('Connected, waiting for audio');
});
</script>
- Read back the sample rate. The device may not support the requested sample rate. Read
micControl.sampleRateafter setting it and use that value in the WAV header; otherwise playback is pitch-shifted. - Choose the chunk size. Smaller chunks (100–200 ms) lower latency but add Realtime overhead. Larger chunks (500–1000 ms) are more efficient but add noticeable delay. The default, 500 ms, suits most cases.
- Resume the AudioContext on a user gesture. Browsers block audio until the user interacts with the page. Create the
AudioContextin a click handler, or callaudioCtx.resume()in a button callback. - Match the listener's status events. The shipped listener logs stream start and end for
stream-startedandstream-ended, while the Lens sendsaudio-stream-startedandaudio-stream-ended. Only those log lines are affected;audio-chunkmatches.
Stream video and audio together
The Composite streaming example (CompositeStreamingController) combines the video and audio streams in one synchronized session on one Realtime channel, for example composite-live-stream. Video frames go out as composite-video-frame events and audio chunks as composite-audio-chunk events. Every payload carries the same sessionId, so the web viewer can line up audio chunks with video frames. The video side can stream the composite texture (camera with AR content), and the audio side sends 500 ms chunks.
- Select the
CompositeStreamingControllercomponent. - Assign the
CameraService, a Microphone Asset, and, for composite mode, a Render Target texture as the composite texture. - Set Streaming Channel Name, for example
composite-live-stream. - Open the composite viewer (
ExternalServicesExamples/media-example-web-viewers/composite-stream-viewer.html) in a browser, enter your project URL and anon key, and connect to the same channel.
The core pattern: one channel, two event types, one session ID. Add these members to the SnapCloudStarter script, call setUpCompositeStream() from onSignedIn(), and feed sendVideoFrame() and sendAudioChunk() from the video and audio pipelines in the two previous sections.
@input compositeChannelName: string = 'composite-live-stream';
private compositeChannel: RealtimeChannel;
private compositeSessionId: string = '';
setUpCompositeStream() {
// Every payload carries the same session ID so the viewer can sync them
this.compositeSessionId = 'composite_' + Date.now();
this.compositeChannel = this.client.channel(this.compositeChannelName, {
config: { broadcast: { self: false } },
});
this.compositeChannel.subscribe();
}
sendVideoFrame(base64Frame: string, frameNumber: number) {
this.compositeChannel.send({
type: 'broadcast',
event: 'composite-video-frame',
payload: {
sessionId: this.compositeSessionId,
frameNumber,
frameData: base64Frame + '|||FRAME_END|||',
},
});
}
sendAudioChunk(wavBase64: string, chunkNumber: number) {
this.compositeChannel.send({
type: 'broadcast',
event: 'composite-audio-chunk',
payload: {
sessionId: this.compositeSessionId,
chunkNumber,
data: wavBase64,
metadata: { format: 'wav', channels: 1 },
},
});
}
Composite stream web viewer
<!-- composite-stream-viewer.html: key sync logic -->
<script src="https://unpkg.com/@supabase/supabase-js@2"></script>
<canvas id="videoCanvas"></canvas>
<script>
const audioCtx = new AudioContext();
const client = window.supabase.createClient(SUPABASE_URL, SUPABASE_ANON_KEY, {
realtime: { heartbeatIntervalMs: 2500 },
});
const channel = client.channel('composite-live-stream');
const canvas = document.getElementById('videoCanvas');
const ctx = canvas.getContext('2d');
const videoBuffer = [];
channel
.on('broadcast', { event: 'composite-video-frame' }, (msg) => {
const base64 = msg.payload.frameData.replace('|||FRAME_END|||', '');
const img = new Image();
img.onload = () => videoBuffer.push(img);
img.src = 'data:image/jpeg;base64,' + base64;
})
.on('broadcast', { event: 'composite-audio-chunk' }, async (msg) => {
const bytes = Uint8Array.from(atob(msg.payload.data), (c) =>
c.charCodeAt(0)
);
const audioBuffer = await audioCtx.decodeAudioData(bytes.buffer);
const source = audioCtx.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioCtx.destination);
source.start();
})
.subscribe();
// Draw video frames
(function render() {
if (videoBuffer.length > 0) {
const frame = videoBuffer.shift();
canvas.width = frame.width;
canvas.height = frame.height;
ctx.drawImage(frame, 0, 0);
}
requestAnimationFrame(render);
})();
</script>
- Give the GPU time in composite mode. The GPU needs several frames to render AR content into the render target. Without a buffer delay, the first frames are black or show only the camera image.
- One session ID for both streams. Because every video and audio payload carries the same
sessionId, the web viewer can align the two streams without a separate sync protocol. Each message stays well under the 250 KB limit on its own.
Stitch uploaded frames and audio into a video
The Composite capture example (CompositeCaptureUploader) uploads video frames, WAV audio, and session metadata to Storage, then calls the trigger-composite-stitch edge function with the session details. The edge function doesn't run FFmpeg itself: it forwards the request to a separate Node.js service with FFmpeg, which reads the frames and audio from Storage, merges them into an MP4 video, and writes it back to Storage. The Sharing section of the Media Suite uses the same CompositeCaptureUploader and adds a caption and a shareToSpotlight flag to the request, for sharing the finished video to Spotlight; what happens with them is up to your stitching service.
The package ships both parts as sources to deploy yourself:
ExternalServicesExamples/media-example-server-composite-stitcher-edge-function/supabase-edge-function.txt: thetrigger-composite-stitchedge function, which forwards to<STITCHER_SERVICE_URL>/stitchand returns its JSON.ExternalServicesExamples/media-example-server-composite-stitcher/: the Node.js and FFmpeg stitching service.
Host the stitching service on infrastructure your organization approves, where the edge function's STITCHER_SERVICE_URL can reach it. Keep the service role key on the server side, in the edge function's or the service's environment. The end-to-end walkthrough ships as ExternalServicesExamples/_guides/complete-setup-guide.md.
Related
- Storage: create the bucket and policies the uploads use.
- Realtime: channels, broadcast, and subscriptions.
- Camera Module: camera streams and high-resolution stills on SPECS.
- Limits: the 250 KB Realtime message limit and storage limits.
- Examples: the whole Snap Cloud Examples package.