ASR Module
Add speech-to-text to SPECS Lenses with the ASR Module: 40+ languages, mixed-language input, translation, content types, and accuracy modes.
- Snapchat: not supported
- SPECS: fully supported
- Camera Kit Web: not supported
- Camera Kit Mobile: not supported
- SnapchatNot supported
- SPECSFully supported
- Camera Kit WebNot supported
- Camera Kit MobileNot supported
The AsrModule (automatic speech recognition) adds speech-to-text to SPECS Lenses. It supports 40+ languages, including mixed language input, and delivers high accuracy across various accents and non-native speakers. This page is for developers who want a SPECS Lens to turn what the wearer says into text.
When to use the ASR Module
Use the ASR Module for dictation, transcription, and chat input, such as sending what the wearer says to an AI model as a question. When the wearer speaks more than one language, the ASR Module handles the mix automatically.
The ASR Module isn't reliable for voice commands or keyword spotting. Partial transcripts keep changing, final transcripts arrive late, short command words are often transcribed wrongly, and homophones ("two" and "to") trigger false matches. For commands, use a pinch, a button, or gaze instead. The ASR Module replaces the VoiceML Module for transcription on SPECS; voice-command features of VoiceML have no ASR equivalent.
Requirements for the ASR Module
- Lens Studio v5.9.0 or later.
- Snap OS v5.61 or later.
- An internet connection: transcription streams over the internet. Without a connection, the
session reports
AsrModule.AsrStatusCode.NoInternet.
The ASR Module works in the Lens Studio Preview and on SPECS. The language,
secondaryLanguages, and contentType options need Lens Scripting Version 384 or later.
The ASR Module and the microphone
Using the ASR Module turns on the Voice audio input profile, which applies Echo Cancellation and Bystander Speech Rejection to the microphone. See Audio on SPECS.
For permissions, the ASR Module counts as a Microphone sensitive sensor. A Lens that combines the ASR Module with internet access, for example through a Web Socket, falls under Transparent Permission.
Add the ASR Module to a script
-
Create a new TypeScript or JavaScript file in the Asset Browser.
-
Drag the file into the scene.
-
Open the script for editing.
-
Add the following line to include the AsrModule:
const asrModule: AsrModule = require('LensStudio:AsrModule');
Handle ASR transcription updates and errors
The ASR Module reports transcription updates and errors through callbacks. A transcription update
carries a TranscriptionUpdateEvent with the text and
whether it isFinal. An error carries an AsrStatusCode:
InternalError, Unauthenticated, or NoInternet (the enum also has Success).
const onTranscriptionUpdate = (
eventArgs: AsrModule.TranscriptionUpdateEvent
) => {
print(
`onTranscriptionUpdateCallback text=${eventArgs.text}, isFinal=${eventArgs.isFinal}`
);
};
const onTranscriptionError = (errorCode: AsrModule.AsrStatusCode) => {
print(`onTranscriptionErrorCallback errorCode: ${errorCode}`);
switch (errorCode) {
case AsrModule.AsrStatusCode.InternalError:
print('stopTranscribing: Internal Error');
break;
case AsrModule.AsrStatusCode.Unauthenticated:
print('stopTranscribing: Unauthenticated');
break;
case AsrModule.AsrStatusCode.NoInternet:
print('stopTranscribing: No Internet');
break;
}
};
Set the ASR transcription options
The AsrTranscriptionOptions object lets you configure the ASR session. To create options:
const options = AsrModule.AsrTranscriptionOptions.create();
options.silenceUntilTerminationMs = 1000;
options.mode = AsrModule.AsrMode.HighAccuracy;
options.onTranscriptionUpdateEvent.add((eventArgs) =>
onTranscriptionUpdate(eventArgs)
);
options.onTranscriptionErrorEvent.add((errorCode) =>
onTranscriptionError(errorCode)
);
| Option | Default | What it does |
|---|---|---|
silenceUntilTerminationMs | 750 | Milliseconds of detected silence after which the current sentence or fragment is marked as final. |
mode | AsrModule.AsrMode.HighAccuracy | The operation mode of the session: HighAccuracy, Balanced, or HighSpeed. |
language | "auto" | The language of the transcription in ISO 639 format. "auto" detects the spoken language. |
secondaryLanguages | [] | Other languages, in ISO 639 format, that the speaker is expected to switch to. |
contentType | AsrModule.AsrContentType.Text | The kind of content the speech carries, which decides how the transcription is formatted. |
onTranscriptionUpdateEvent | The event triggered by transcription updates. | |
onTranscriptionErrorEvent | The event triggered by transcription errors. |
Once a transcription is finalized (isFinal is true), a new transcription begins within the same
session. This process continues until you stop the session.
The mode option trades speed for accuracy:
HighAccuracy: higher accuracy at the cost of speed.Balanced: a balance between accuracy and speed.HighSpeed: faster transcription at the cost of accuracy.
Set the ASR language and secondary languages
By default, the ASR Module detects the spoken language. To name the language instead, set
language to an ISO 639 code. If the speaker is expected to switch to other languages, list them
in secondaryLanguages: this nudges the model toward those languages and can improve accuracy when
several are spoken.
// The speaker mostly talks in English, but may switch to Italian or Spanish.
options.language = 'en';
options.secondaryLanguages = ['it', 'es'];
Translate speech with the ASR Module
To translate speech, set contentType to AsrModule.AsrContentType.Translation and language to
the language to translate into. With Translation, language is the target language and "auto"
isn't a valid value.
// Translate whatever is spoken into Spanish.
const translationOptions = AsrModule.AsrTranscriptionOptions.create();
translationOptions.contentType = AsrModule.AsrContentType.Translation;
translationOptions.language = 'es';
Format ASR transcriptions with a content type
The contentType option tells the ASR Module what the speech is expected to carry, so the
transcription is processed and formatted to match, for example to fill in a text field of that
type.
| Content type | Transcription |
|---|---|
Text (default) | General speech, transcribed as text. |
Translation | Speech translated into language. |
Search | A short query, such as a search term: lowercase, with no punctuation. |
Url | A URL. |
Email | An email address. |
Numeric | A number. |
Phone | A phone number. |
// Format the transcription as a URL.
const urlOptions = AsrModule.AsrTranscriptionOptions.create();
urlOptions.contentType = AsrModule.AsrContentType.Url;
Start and stop ASR transcribing
Start the session from an OnStartEvent handler, not from onAwake. To start transcribing, pass
your options object to startTranscribing:
asrModule.startTranscribing(options);
If you call startTranscribing again while a session is active, the ASR Module cancels the current
session and starts a new one.
To stop, call stopTranscribing. It discards the current session
and returns a Promise that resolves once transcribing has stopped:
asrModule.stopTranscribing().then(() => {
print('Transcribing stopped');
});
Full ASR Module script
This script starts an ASR session when the scene starts and stops it when its Scene Object is destroyed.
- TypeScript
- JavaScript
@component
export class AsrExample extends BaseScriptComponent {
private asrModule: AsrModule = require('LensStudio:AsrModule');
onAwake(): void {
// Start the session once the scene has started, not in onAwake.
this.createEvent('OnStartEvent').bind(() => this.startSession());
// Stop the session when the script's object is destroyed.
this.createEvent('OnDestroyEvent').bind(() => this.stopSession());
}
private onTranscriptionUpdate(eventArgs: AsrModule.TranscriptionUpdateEvent) {
print(
`onTranscriptionUpdateCallback text=${eventArgs.text}, isFinal=${eventArgs.isFinal}`
);
}
private onTranscriptionError(errorCode: AsrModule.AsrStatusCode) {
print(`onTranscriptionErrorCallback errorCode: ${errorCode}`);
switch (errorCode) {
case AsrModule.AsrStatusCode.InternalError:
print('stopTranscribing: Internal Error');
break;
case AsrModule.AsrStatusCode.Unauthenticated:
print('stopTranscribing: Unauthenticated');
break;
case AsrModule.AsrStatusCode.NoInternet:
print('stopTranscribing: No Internet');
break;
}
}
private startSession(): void {
const options = AsrModule.AsrTranscriptionOptions.create();
options.silenceUntilTerminationMs = 1000;
options.mode = AsrModule.AsrMode.HighAccuracy;
options.onTranscriptionUpdateEvent.add((eventArgs) =>
this.onTranscriptionUpdate(eventArgs)
);
options.onTranscriptionErrorEvent.add((errorCode) =>
this.onTranscriptionError(errorCode)
);
this.asrModule.startTranscribing(options);
}
private stopSession(): void {
this.asrModule.stopTranscribing().then(() => {
print('Transcribing stopped');
});
}
}
const asrModule = require('LensStudio:AsrModule');
function onTranscriptionError(errorCode) {
print('onTranscriptionErrorCallback errorCode: ' + errorCode);
switch (errorCode) {
case AsrModule.AsrStatusCode.InternalError:
print('stopTranscribing: Internal Error');
break;
case AsrModule.AsrStatusCode.Unauthenticated:
print('stopTranscribing: Unauthenticated');
break;
case AsrModule.AsrStatusCode.NoInternet:
print('stopTranscribing: No Internet');
break;
}
}
function onTranscriptionUpdate(eventArgs) {
const text = eventArgs.text;
const isFinal = eventArgs.isFinal;
print('onTranscriptionUpdateCallback text=' + text + ', isFinal=' + isFinal);
}
function startSession() {
const options = AsrModule.AsrTranscriptionOptions.create();
options.silenceUntilTerminationMs = 1000;
options.mode = AsrModule.AsrMode.HighAccuracy;
options.onTranscriptionUpdateEvent.add(onTranscriptionUpdate);
options.onTranscriptionErrorEvent.add(onTranscriptionError);
// Start session
asrModule.startTranscribing(options);
}
function stopSession() {
asrModule.stopTranscribing().then(function () {
print('stopTranscribing successfully');
});
}
// Start the session once the scene has started, and stop it when the script's object is destroyed.
script.createEvent('OnStartEvent').bind(startSession);
script.createEvent('OnDestroyEvent').bind(stopSession);
Related
- Audio on SPECS: the Voice audio input profile that the ASR Module turns on.
- Transparent Permission: publishing a Lens that combines the ASR Module with internet access.
- Migrating a Lens to SPECS 27: moving VoiceML transcription to the ASR Module.
- AsrModule: the API reference.