Skip to main content

SnapML on SPECS

Train a custom object detection model, quantize it to W8A8 and convert it to Qualcomm DLC, then run it with SnapML on SPECS.

  • Snapchat: not supported
  • SPECS: partially supported
  • Camera Kit Web: not supported
  • Camera Kit Mobile: not supported
1 platform note
  • SnapchatNot supported
  • SPECSPartially supported
    • Models run on device only after W8A8 quantization and conversion to Qualcomm DLC, and DLC models don't run in Lens Studio Preview
  • Camera Kit WebNot supported
  • Camera Kit MobileNot supported

Platform details

  • Snapchat — Not supported.

  • SPECS — Partially supported. Models run on device only after W8A8 quantization and conversion to Qualcomm DLC, and DLC models don't run in Lens Studio Preview

  • Camera Kit Web — Not supported.

  • Camera Kit Mobile — Not supported.

SnapML runs machine learning models in SPECS Lenses, for computer vision, object detection, and more. This guide walks through a generalized workflow to train a custom object detection model with Paperspace and Roboflow, prepare it for SPECS, and use it in a Lens. It builds on the workflow used for Snapchat Lenses, described in SnapML Overview.

What a SnapML model needs on SPECS 27

On SPECS 27, a SnapML model must be:

  • Quantized to W8A8: 8-bit weights and 8-bit activations.
  • Converted to Qualcomm's DLC format, with Qualcomm's AI tooling. The final conversion to DLC currently relies on internal Snap tooling: hand your quantized TFLite model to the SPECS Developer Team, who convert it for you.

The conversion route is: Float32 ONNX → TFLite → quantized TFLite (W8A8) → DLC. DLC models currently run only on SPECS, not in Lens Studio, so test them on the device.

For a complete example with Lens Studio scripts, see the SnapML Starter sample in the Spectacles (2024) samples repository on GitHub. It ships an ONNX model (Monitor.onnx) made for Spectacles (2024); on SPECS 27, replace it with your DLC model.

How the SnapML workflow for SPECS fits together

This workflow adapts the Multi-Object Detection template from the SnapML templates repository, using cloud-based development environments and specialized tools for managing datasets and training. It has these phases:

  1. Setup: prepare your development environment in Paperspace.
  2. Data preparation: manage and export your dataset with Roboflow.
  3. Training: train a custom YOLOv7 model on your dataset.
  4. Export: export the trained model to ONNX for SnapML.
  5. Quantization: convert the ONNX model to TFLite and quantize it to W8A8.
  6. Conversion: have the quantized model converted to DLC.
  7. Deployment: import the DLC model into Lens Studio and test it on SPECS.

This guide uses Paperspace and Roboflow for instruction only: Snap doesn't endorse these services, and any alternatives that provide similar functionality work.

Model training workflow

At a glance, the steps are: set up Paperspace → prepare the dataset → set up YOLOv7 → train the model → export to ONNX → quantize to W8A8 TFLite → convert to DLC → import into Lens Studio.

Quick start for experienced ML developers

If you're familiar with machine learning workflows, here's the condensed version of the steps:

# 1. Set up environment in Paperspace with PyTorch template
git clone https://github.com/Snapchat/snapml-templates.git
git clone https://github.com/hartwoolery/yolov7
cd yolov7
pip install -r requirements.txt

# 2. Download dataset from Roboflow in YOLOv7 format
pip install --upgrade roboflow
# Use Roboflow API to download your dataset

# 3. Download pre-trained weights
wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt

# 4. Train model
python train.py --data your-dataset/data.yaml --cfg cfg/training/yolov7-tiny.yaml \
--weights yolov7-tiny.pt --img 224 224 --batch-size 64 --epochs 200 \
--name detection --device 0 --hyp data/hyp.scratch.tiny.yaml

# 5. Export to ONNX for SnapML
python export.py --weights runs/train/detection/weights/best.pt --grid \
--simplify --export-snapml --img-size 224 224 --max-wh 224

# 6. Convert the ONNX model to TensorFlow, then quantize it to a W8A8 TFLite model
onnx2tf -i runs/train/detection/weights/best.onnx -o saved_model/
# See "Quantize the model to W8A8 TFLite" for the quantization script

# 7. Hand the W8A8 TFLite model to the SPECS Developer Team for conversion to DLC

# 8. Import the DLC model into Lens Studio and set up class labels
# See the detailed steps that follow

The rest of this guide walks through each step in detail, with the concepts behind it and troubleshooting tips.

Accounts you need before you start

  • A Paperspace account, or any alternative cloud-based development environment.
  • A Roboflow account for managing and annotating datasets, or any alternative dataset management tool.

Why this workflow?

Traditional ML workflowSnapML workflow
Complex environment setup on a local machineCloud-based development with pre-configured templates
Manual dataset preparation and augmentationAutomated dataset management with Roboflow
Full model training from scratchTransfer learning with pre-trained weights
Custom conversion steps for each deviceExport to ONNX, then quantize to W8A8 TFLite and convert to DLC
Custom integration with AR platformsIntegration with Lens Studio sample scripts

Steps to train and prepare a SnapML model for SPECS

1. Set up a Paperspace environment

Paperspace provides a cloud-based environment with the GPU power that training machine learning models needs.

  1. Sign up for an account. When you sign up, select Gradient as your product (not Core): it provides a fully managed JupyterLab notebook interface with GitHub integration.

  2. Select a template. For YOLOv7 PyTorch notebooks, choose the PyTorch 1.12 template, which includes most of the dependencies you need.

  3. Choose the hardware. Select a GPU option for faster training: free CPU options are available, but first come, first served. The P4000 or higher GPU options are recommended.

  4. Clone the SnapML templates repository:

    git clone https://github.com/Snapchat/snapml-templates.git
  5. Go to the Multi-Object Detection folder and open its notebook file (.ipynb). The notebook contains the code for the whole training workflow.

When your Paperspace machine stops, your work environment is wiped, and you need to re-run all the notebook steps when you restart. Save important outputs, such as trained models and ONNX files, to persistent storage or download them to your computer.

  • Restart kernel: Refreshes only your Python environment without losing installed packages or files

  • Restart space: Completely resets your environment to its default state

After doing a %pip install in your notebook, typically you only need to restart the kernel using the JupyterLab interface.

2. Prepare your dataset with Roboflow

Roboflow helps you prepare, annotate, and export datasets for machine learning models: it labels, augments, and exports your dataset in the format your framework needs.

  1. Create or sign in to your Roboflow account at Roboflow. You can sign up with GitHub, Google, or email.

  2. Prepare your dataset, in one of two ways:

    • Create your own dataset: upload your images to Roboflow, label the objects with Roboflow's annotation tools, and apply augmentation to expand the dataset.
    • Use an existing dataset: browse Roboflow Universe for public datasets, search for ones that match what you want to detect, and fork them to your workspace.
  3. Export your dataset in YOLOv7 format: on your project page in Roboflow, select Download this Dataset, select the YOLO v7 PyTorch format, and copy the code snippet it provides into your notebook. It looks like this:

# Install Roboflow package
pip install --upgrade roboflow

from roboflow import Roboflow
rf = Roboflow(api_key="YOUR_API_KEY_HERE")

# Point at your workspace
project = rf.workspace("workspace-name").project("project-name")

# Pick the version
version = project.version(1)

# Download in the format you need
dataset = version.download("yolov7")

# Save the dataset location
DATASET_DIR = dataset.location
print(DATASET_DIR)

If the dataset isn't public:

  1. In the Roboflow web app, go to Universe and find your dataset.
  2. Select Fork to copy it to your workspace.
  3. Create a version, following the guided steps.
  4. Get your API key from Settings > Private Key.

For optimal results when training object detection models:

  • Include at least 100 images per class for basic detection

  • Ensure diversity in lighting conditions, angles, and backgrounds

  • Include images with multiple objects and partial occlusions

  • Use Roboflow's augmentation features to expand your dataset:

    • Rotation
    • Brightness adjustments
    • Noise addition
    • Mosaic (combining images)

3. Set up YOLOv7 for training

YOLOv7 is an object detection algorithm known for its speed and accuracy, which suits mobile and wearable devices like SPECS. For SnapML, this guide uses a modified version that supports the export format SnapML needs.

  1. Clone the modified YOLOv7 repository. This fork of YOLOv7 adds the --export-snapml export option on its main branch:

    cd ~
    git clone https://github.com/hartwoolery/yolov7
    cd yolov7
    pip install -r requirements.txt

    The fork changes how the model is exported so that it works with SnapML: the export leaves out the NMS layer, so that NMS can run in a Lens script instead.

  2. Set the path to your dataset directory, so that the training script can find the dataset you downloaded from Roboflow:

    # Set the version number to match your downloaded dataset
    VERSION_NUMBER = 1

    # Build the path to your dataset
    DATASET_DIR = f"/root/yolov7/your-dataset-name-{VERSION_NUMBER}"
    print("Loading from", DATASET_DIR)
  3. Download the pre-trained weights, which speed up training through transfer learning:

    cd ~/yolov7
    wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt
    YOLOv7-tiny benefitsFull YOLOv7 benefits
    Smaller model size (about 5–6 MB)Higher detection accuracy
    Faster inferenceBetter performance with small objects
    Lower memory requirementsMore robust in challenging environments
    Optimal for mobile and wearable devicesBetter for complex scenes

Pre-trained weights are parameter values saved after a model has been trained on a large dataset like COCO (Common Objects in Context), which contains over 330,000 images with 80 object categories. Using these weights for transfer learning offers several advantages:

  • Faster convergence: Training starts from an optimized state rather than random initialization
  • Better performance: The model leverages patterns and features already learned from a larger dataset
  • Less data needed: You can achieve good results with a smaller custom dataset
  • Reduced overfitting: The model has already learned generalizable features

Transfer learning works by:

  1. Starting with weights pre-trained on a large dataset
  2. Replacing the final classification layer with your custom classes
  3. Fine-tuning the entire network or just later layers with your data

For wearable devices like SPECS, the smaller yolov7-tiny variant is recommended for real-time inference.

YOLOv7 offers several advantages for SPECS:

FeatureYOLOv7Older YOLO VersionsOther Models (SSD, R-CNN)
Speed★★★★★★★★★☆★★★☆☆
Accuracy★★★★☆★★★☆☆★★★★★
Size★★★★☆★★★★★★★☆☆☆
SnapML Compatibility★★★★★★★★☆☆★★★☆☆

YOLOv7 strikes an optimal balance between accuracy and performance for AR applications.

4. Train your model

Train your custom model on your dataset, starting from the pre-trained weights. Training is the most computationally intensive part of the process: it takes from minutes to hours, depending on your dataset size and hardware.

  1. Optional: set up TensorBoard to visualize training metrics in real time, so you can monitor progress and spot issues early:

    # In a notebook environment:
    # %reload_ext tensorboard
    # %tensorboard --logdir runs/train

    # In a terminal:
    tensorboard --logdir runs/train
  2. Configure and start training. This is the core training command, with its key parameters explained:

    import os
    os.environ['WANDB_MODE'] = 'disabled' # Disable Weights & Biases logging

    # Key training parameters
    EPOCH = 200 # Total number of training epochs
    BATCH_SIZE = 64 # Number of images processed at once
    IMG_SIZE = 224 # Input image resolution (height, width)
    JOB_NAME = 'detection' # Name for this training run

    # Change directory to yolov7 folder
    # In terminal: cd ~/yolov7

    # Run the training command
    python train.py \
    --data your-dataset-name-1/data.yaml \ # Path to dataset configuration
    --cfg cfg/training/yolov7-tiny.yaml \ # Model architecture
    --weights yolov7-tiny.pt \ # Pre-trained weights
    --img 224 224 \ # Input resolution
    --batch-size 64 \ # Batch size
    --epochs 200 \ # Number of epochs
    --name detection \ # Run name
    --device 0 \ # GPU device (0 = first GPU)
    --hyp data/hyp.scratch.tiny.yaml # Hyperparameters
    • Start small: begin with fewer epochs (for example, 50) to test your setup.
    • Adjust the batch size: reduce it if you run into memory errors.
    • Image size: 224×224 balances detection quality and speed on a device.
    • Monitor early: check TensorBoard after about 10 epochs to make sure training is progressing.
  3. Read the training output. During training, the console shows lines like this:

    Epoch gpu_mem box obj cls total labels img_size
    0/199 0.367G 0.07126 0.3772 0 0.4485 188 224
    • Epoch: Current training epoch / total epochs

    • gpu_mem: GPU memory usage

    • box: Bounding box regression loss (lower = better localization)

    • obj: Objectness loss (detecting if there's any object)

    • cls: Classification loss (which class the object belongs to)

    • total: Sum of all losses (main training metric to reduce)

    • labels: Number of ground truth objects in the batch

    • img_size: Input image resolution

    Look for decreasing loss values as training progresses. Plateauing loss indicates the model may have converged.

  4. Test your trained model on sample images:

    # Run detection on test images
    python detect.py \
    --img-size $IMG_SIZE \
    --weights runs/train/detection/weights/best.pt \
    --conf 0.5 \ # Confidence threshold
    --source your-dataset-name-1/test/images # Test images
  5. Display the detection results to evaluate your model's performance:

    # In notebook environments: %matplotlib inline

    from glob import glob
    from PIL import Image
    from IPython.display import display

    # Find your latest detect output folder
    exp_dirs = sorted(glob('runs/detect/exp*'))
    latest = exp_dirs[-1]

    # Display detection results
    paths = glob(f'{latest}/*.*')
    img = Image.open(paths[0])
    display(img)

When reviewing your model results, consider:

  • False Positives: Detections where no object exists

  • False Negatives: Missed objects that should be detected

  • Localization Accuracy: How precisely the bounding boxes fit around objects

  • Classification Accuracy: Whether objects are assigned the correct class

If performance is poor, consider:

  • Adding more training data
  • Increasing training epochs
  • Adjusting model hyperparameters
  • Using data augmentation to increase dataset diversity

5. Export the model to ONNX for SnapML

Export the trained model to ONNX (Open Neural Network Exchange), an open standard for representing machine learning models that works across frameworks. The ONNX model is the starting point for the W8A8 TFLite model SPECS needs.

  1. Install the dependencies for ONNX export:

    # Install dependencies for ONNX export
    pip install "protobuf<4.21.3"
    pip install "onnx>=1.9.0"
    pip install onnx-graphsurgeon
    pip install --user "onnx-simplifier>=0.3.6"
  2. Run the export script with these flags:

    # Run export script for SnapML compatibility
    python export.py \
    --weights ./runs/train/detection/weights/best.pt \ # Path to trained weights
    --grid \ # Export detection grid
    --simplify \ # Simplify ONNX graph
    --export-snapml \ # Special flag for SnapML compatibility
    --img-size 224 224 \ # Input dimensions (use your IMG_SIZE)
    --max-wh 224 # Maximum width/height (use your IMG_SIZE)

    The --export-snapml flag matters: it leaves box decoding and NMS (Non-Maximum Suppression) out of the ONNX graph. A Lens script runs them instead (in the SnapML Starter sample, MLController.js and YOLODetectionProcessor.ts do).

  3. Generate download links for the exported model:

    from IPython.display import FileLink, FileLinks

    # Link to the ONNX file
    display(FileLink('runs/train/detection/weights/best.onnx'))

    # Or browse the entire weights folder
    display(FileLinks('runs/train/detection/weights'))
  4. Generate the class labels, in class index order, for the Lens Studio scripts:

    import yaml

    with open("your-dataset-name-1/data.yaml", "r") as stream:
    data = yaml.safe_load(stream)

    for i, label in enumerate(data["names"].values() if isinstance(data["names"], dict) else data["names"]):
    print(f'"{i}":"{label}",')

    Example output:

    "0":"chair",
    "1":"monitor",
    "2":"tv",

    Keep this list: in Lens Studio, you enter the labels in this order, so that each class index gets its label (see Import and test your model in Lens Studio).

The exported ONNX model has several key attributes that make it suitable for SnapML:

  1. Simplified Graph: The --simplify flag optimizes the model by removing redundant operations

  2. Fixed Input Size: The input dimensions are fixed during export, matching what you'll use in Lens Studio

  3. No NMS: Non-Maximum Suppression is excluded, so a Lens script can run it with full control

  4. Quantization: Not applied at export. SPECS needs a W8A8-quantized model, which the next step produces

6. Quantize the model to W8A8 TFLite

These steps convert the trained float32 model into a W8A8-quantized TFLite model, ready for the conversion to DLC. You need TensorFlow and onnx2tf installed.

  1. Convert the ONNX (float32) model to a TensorFlow SavedModel with onnx2tf, which handles the NCHW→NHWC layout flip TFLite needs:

    onnx2tf -i model.onnx -o saved_model/

    The alternative, onnx-tf convert, is older and keeps NCHW, which causes more layout problems.

  2. Convert the SavedModel to a float32 TFLite model. This baseline confirms the graph converts before you add quantization:

    converter = tf.lite.TFLiteConverter.from_saved_model("saved_model/")
    open("model_fp32.tflite", "wb").write(converter.convert())
  3. Build a representative dataset to calibrate the activation ranges: a generator that yields about 100–500 real inputs, shaped [1, H, W, C] and preprocessed exactly as in training and inference:

    def rep_data():
    for img in calib_images[:200]:
    yield [preprocess(img).astype("float32")] # NHWC, same normalization as training
  4. Convert with W8A8 full-integer quantization, continuing with the converter from step 2:

    converter.optimizations = [tf.lite.Optimize.DEFAULT]
    converter.representative_dataset = rep_data
    # Forces int8; errors if any op can't quantize
    converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
    # Keeps I/O integer end-to-end
    converter.inference_input_type = tf.int8
    converter.inference_output_type = tf.int8
    open("model_w8a8.tflite", "wb").write(converter.convert())
  5. Verify the result: load it with tf.lite.Interpreter, check that get_input_details() reports the int8 data type with quantization parameters, and compare its outputs with the float32 baseline to confirm the accuracy drop is acceptable.

  • Weights-only ("dynamic range") quantization: if you set optimizations but provide no representative dataset, you get int8 weights with float activations (W8Afloat), not W8A8.
  • Layout: TFLite prefers NHWC. onnx2tf handles this cleanly; the onnx-tf route often adds unnecessary transpose ops or layout mismatches.

7. Convert the quantized model to DLC

The final conversion, from the W8A8 TFLite model to Qualcomm's DLC format, currently relies on internal Snap tooling. Hand your model_w8a8.tflite to the SPECS Developer Team, who convert it to DLC for you.

8. Use your model in Lens Studio

Import the DLC model into Lens Studio for use in your SPECS Lens.

  1. Open the SnapML Starter sample, or create a new Lens Studio project.

  2. Consider the input range your model accepts. Exported models typically expect inputs from 0 to 1, while the texture input of the ML Component in Lens Studio takes values from 0 to 255. When you bring the model into Lens Studio, set each input channel's Scale to 0.0039215 (1/255) and its Bias to 0:

    ChannelScaleBias
    Channel 00.00392150.00
    Channel 10.00392150.00
    Channel 20.00392150.00

    For more on importing models, see the ML Model Import section of ML Component Overview.

  3. Replace the default model: in the Asset Browser, right-click the default ML model, select Replace From File…, and select your DLC model.

  4. Set up your class labels: in the Inspector, enter your labels, in class index order, in the detection script's class list (the classSettings input of the sample's MLController.js, or the classLabels input of MLSpatializer).

  5. Configure the model parameters: match the input dimensions to your training settings, set the confidence (score) threshold to suit your model, and try different NMS (IoU) threshold values.

  6. Send your Lens to SPECS to test it, because DLC models don't run in Lens Studio Preview (see Send a Lens to SPECS). Evaluate the real-world performance, adjust, and optimize if detection is too slow.

To optimize your ML model's performance on SPECS:

  1. Reduce inference frequency if full frame rate isn't necessary

  2. Apply region of interest (ROI) to limit processing to specific areas

  3. Implement caching for stable detections

  4. Limit tracking to a reasonable number of objects (e.g., 3-5 most confident detections)

  5. Apply temporal smoothing to reduce jitter in detections

Advanced SnapML topics

Understanding Non-Maximum Suppression (NMS)

The --export-snapml export flag changes how Non-Maximum Suppression (NMS) is handled:

NMS is not a trainable convolutional layer but a deterministic post-processing step applied to raw box predictions during inference. Its purpose is to:

  1. Collect all predicted boxes whose confidence (objectness × class score) exceeds your threshold
  2. Sort those boxes by confidence (highest to lowest)
  3. Keep only the best boxes by removing highly overlapping detections of the same object

When exporting a YOLO model for SnapML, we exclude NMS from the graph and run it in a Lens script. This gives more control over the detection process. Lens scripts can also call the native TensorMath.applyNMS.

Optimize a model for SPECS

When you create models for SPECS, consider these strategies:

  • Use smaller variants, like YOLOv7-tiny, for real-time performance.
  • Reduce the input image size to balance detection quality and speed.
  • Limit the number of classes to make the model more efficient.
  • Quantize to W8A8, which SPECS requires.
  • Test thoroughly on SPECS to make sure inference runs in real time.

Troubleshooting SnapML training and export

Common issues and solutions

  • Restarting the kernel: Clears Python state without affecting installed packages or data files. Use after installing libraries with %pip install.

  • Restarting the space: Performs a full reset of your environment, including all files and installations. Only use when something at the system level has gone wrong.

If you encounter GPU memory errors during training:

  • Reduce the batch size
  • Use a smaller model variant
  • Reduce image resolution
  • Free up memory between operations with torch.cuda.empty_cache()

Common export issues:

  • Ensure all required packages are installed at the correct versions

  • When using --simplify, ensure you have onnx-simplifier installed

  • For protobuf-related errors, try installing a specific version with pip install protobuf==3.20.3

Frequently asked questions about SnapML on SPECS

For optimal performance on SPECS, prioritize models with:

  • Small size: Under 10MB for better loading times and memory usage

  • Low computational requirements: Models with fewer operations per inference

  • W8A8 quantization: 8-bit weights and activations, which SPECS requires and which reduces the memory footprint

  • Simple architectures: Fewer layers and simpler operations are more efficient

YOLOv7-tiny and MobileNet-based architectures are particularly well-suited for SPECS.

SnapML can technically detect as many objects as are present in the frame, but for performance reasons on SPECS:

  • Recommended: Limit detection to 3-5 objects simultaneously

  • Maximum: Up to 10 objects can be detected with adequate performance

  • Performance impact: More tracked objects require more processing power

You can control this by adjusting the NMS threshold and maximum objects parameters in your Lens Studio implementation.

To improve detection accuracy:

  1. Increase training data: More diverse images of your target objects

  2. Better annotations: More precise bounding boxes

  3. Data augmentation: Apply transforms to increase dataset variability

  4. Longer training: Try increasing the number of epochs

  5. Model fine-tuning: Adjust hyperparameters like learning rate

  6. Post-processing: Implement temporal smoothing and confidence thresholds

Yes. While this guide focuses on object detection, SnapML supports various model types:

  • Image classification: Identify what's in an image

  • Semantic segmentation: Identify each pixel's category

  • Style transfer: Apply artistic styles to images

  • Pose estimation: Track human body positions

The export process differs slightly, but the Paperspace and Lens Studio workflow stays similar, and the model still goes through W8A8 quantization and DLC conversion.

Resources for training SnapML models

What the training workflow gives you

Machine learning opens up new possibilities for AR interaction in SPECS experiences. By following this workflow, you've trained a custom object detection model and prepared it for SPECS with SnapML.

Cloud-based training environments like Paperspace, dataset management tools like Roboflow, and the export, quantization, and conversion steps for SnapML make it possible to build sophisticated machine learning features without extensive ML expertise.

As you continue developing for SPECS, experiment with different model architectures, datasets, and detection parameters to make your AR experiences more engaging and responsive.

Use a SnapML model in a SPECS Lens

Once you have your DLC model, the next step is to use it in Lens Studio for SPECS. This section covers the workflow from importing the model to writing custom logic for your AR experience. To follow along, open the SnapML Starter sample.

Scripts in the SnapML Starter sample

Diagram: a detection box in screen space at the camera frustum is projected along rays, at the detected distance, to a detection box on the real-world scene

The SnapML Starter sample's scripts work together in three groups:

  1. Core detection components

    • MLSpatializer: the main entry point, which connects ML detection with 3D space.
    • YOLODetectionProcessor: turns the raw model outputs into usable detections.
    • DetectionContainer and DebugVisualizer: show a detection's category, confidence, and distance, and visualize points for debugging.
  2. Camera and spatial awareness

    • PinholeCapture: interfaces with the device camera.
    • PinholeCameraModel: the mathematical model for 2D to 3D conversions.
  3. Positioning and transformation

    • MatchTransform and MatchTransformLocal: position objects relative to others.
    • InBetween: calculates positions between two objects.
    • SmartTether: positions an object based on the user's movement.

The following sections show how these components work together in Lens Studio.

Import and test your model in Lens Studio

  1. Import your model:

    1. Open the SnapML Starter sample in Lens Studio.
    2. In the Asset Browser, select + > Import Files, then select your DLC model.
    3. Replace the default model in the project with the model you imported.

    If your model exceeds the Lens size limit, reduce its size before conversion, for example with a smaller model variant or a smaller input size.

  2. Configure the detection script. Select the object with the MLSpatializer script in the Scene Hierarchy, and in the Inspector set model to your imported model and inputTexture to the device camera texture. Adjust scoreThreshold to your model's confidence output and iouThreshold for Non-Maximum Suppression. These are the inputs the sample declares:

    // From MLSpatializer.ts in the SnapML Starter sample (some hints shortened)
    @input
    @hint("ONNX model asset")
    model: MLAsset

    @input
    @hint("Input texture for the model (Device Camera Texture)")
    inputTexture: Texture

    @input
    @hint("Maximum number of objects to detect and place")
    @widget(new SliderWidget(1, 20, 1))
    maxDetectionCount: number = 5

    @input
    @hint("Score threshold for detections (0-1) - Lower values make it more sensitive to detections")
    @widget(new SliderWidget(0, 1, 0.01))
    scoreThreshold: number = 0.2

    @input
    @hint("IOU threshold for non-maximum suppression (0-1)")
    @widget(new SliderWidget(0, 1, 0.01))
    iouThreshold: number = 0.5

    @input
    @hint("Class labels")
    classLabels: string[] = ["Chair", "Table", "Sofa"]
  3. Test that your model detects objects before you add spatial features. The sample's MLController.js script triggers its onDetectionsUpdated event with each frame's detections. Each detection has a label, a score, an index, and a bbox (x, y, width, height in screen space, x and y at the box's center). From another script:

    // @input Component.ScriptComponent mlController

    script.mlController.onDetectionsUpdated.add(function (detections) {
    print('Detected ' + detections.length + ' objects');

    for (let i = 0; i < detections.length; i++) {
    const detection = detections[i];
    print(
    'Class: ' +
    detection.label +
    ', Score: ' +
    detection.score.toFixed(2) +
    ', Box: [' +
    detection.bbox.join(', ') +
    ']'
    );
    }
    });
  4. Check your class labels: the labels you entered (classLabels on MLSpatializer, or classSettings on MLController.js) must be in class index order, as generated in Export the model to ONNX for SnapML. A class without a label is reported as class_<index>.

Common issues when importing models and their solutions:

  • Model Not Loading: Verify that the model is the DLC conversion of a model exported with the --export-snapml flag

  • No Detections in Preview: DLC models run only on SPECS; send the Lens to SPECS to test

  • Detection Failures: Check that your input image size matches what the model expects

  • Poor Accuracy: Adjust the scoreThreshold to better match your model's confidence output

  • Memory Issues: If experiencing crashes, try using a smaller model like YOLOv7-tiny

Map 2D detections to 3D space with the MLSpatializer script

Spatialization maps 2D image detections to 3D world space, so AR content can interact with real-world objects.

  1. The role of MLSpatializer. MLSpatializer is the central script that runs the detection and spatialization pipeline. On each update, it decodes the model outputs into detections with YOLODetectionProcessor, filters them, and triggers its onDetectionsUpdated event:

    // From MLSpatializer.ts in the SnapML Starter sample
    // Process YOLO outputs
    const detections = this.yoloProcessor.parseYolo7Outputs(this.outputs)

    // Filter detections based on center threshold
    const filteredDetections = this.filterDetectionsByCenter(detections)

    this.onRunningFinished(filteredDetections)
  2. From 2D to 3D: PinholeCapture and PinholeCameraModel. These components translate 2D image coordinates to 3D world space. PinholeCameraModel has projectToUV, projectToPixel, unprojectFromUV, unprojectFromPixel, resize, and crop. PinholeCapture uses it to turn a point in the camera capture, at a given depth, into a world position:

    // From PinholeCapture.ts in the SnapML Starter sample
    captureToWorldTransform(captureUV: vec2, depth: number): vec3 {
    if (!this.isReady() || !this.viewToWorld) {
    this.logMessage("Error: PinholeCapture not ready for captureToWorldTransform")
    return new vec3(0, 0, 0)
    }

    const capturePos = this.cameraModel.unprojectFromUV(captureUV, depth)
    const viewPos = this.cameraDevice.pose.multiplyPoint(capturePos)
    const worldPos = this.viewToWorld.multiplyPoint(viewPos)
    return worldPos
    }
  3. YOLODetectionProcessor: making sense of model outputs. This processor converts the model's raw output tensors into structured detections, then applies Non-Maximum Suppression. Its entry point is:

    // From YOLODetectionProcessor.ts in the SnapML Starter sample
    public parseYolo7Outputs(outputs: OutputPlaceholder[]): Detection[]

Coordinate systems to keep apart:

  • Image space: 2D coordinates (x, y), where (0,0) is the top-left of the image.
  • Screen space: 2D normalized coordinates, where (0,0) is the bottom-left and (1,1) the top-right.
  • World space: 3D coordinates (x, y, z) in the AR scene.

The conversion from 2D to 3D relies on these key mathematical concepts:

  1. Pinhole Camera Model: Represents how 3D points are projected onto a 2D image plane
  2. Ray Casting: Projects a ray from the camera through the 2D detection point
  3. Intersection Testing: Determines where this ray intersects with surfaces in the real world
  4. Transformation Matrices: Converts between different coordinate systems

The PinholeCameraModel class encapsulates these calculations and exposes them through helper methods.

Smooth detections for a stable user experience

Raw ML detections can be jittery and unstable. Smoothing stabilizes where objects are placed, which makes AR experiences feel more natural. These examples illustrate the techniques; they aren't the sample's exact code.

  1. Motion smoothing with MatchTransform. MatchTransform interpolates between positions, rotations, and scales:

    // Example usage of MatchTransform
    @component
    export class MatchTransformTS extends BaseScriptComponent {
    @input
    target!: SceneObject; // Object to match

    @input
    usePositionLerp: boolean = true; // Enable smooth transitions

    @input
    positionLerpSpeed: number = 1.0; // Lower = smoother but slower

    // Implementation
    update() {
    if (this.usePositionLerp) {
    // Smooth position transition with lerp
    const currentPos = this.getSceneObject()
    .getTransform()
    .getLocalPosition();
    const targetPos = this.target.getTransform().getLocalPosition();

    // Calculate interpolated position
    const newPos = vec3.lerp(
    currentPos,
    targetPos,
    Math.min(1.0, this.positionLerpSpeed * getDeltaTime())
    );

    this.getSceneObject().getTransform().setLocalPosition(newPos);
    }
    }
    }
  2. Temporal filtering. Filtering positions over time reduces jitter. For example, smooth more for small movements than for large ones:

    // Excerpt from a temporal filtering implementation
    updateObjectPosition(index: number, newPosition: vec3, detection: Detection): void {
    // If we have a previous position, apply temporal smoothing
    if (this.previousPositions[index]) {
    // Calculate how much change is acceptable
    const moveThreshold = this.calculateMoveThreshold(detection);

    // If movement is small enough, apply smoothing
    if (this.getDistance(newPosition, this.previousPositions[index]) < moveThreshold) {
    // Apply heavier smoothing when movement is small
    newPosition = vec3.lerp(
    this.previousPositions[index],
    newPosition,
    0.3 // Lower value = more smoothing
    );
    } else {
    // Less smoothing for large movements
    newPosition = vec3.lerp(
    this.previousPositions[index],
    newPosition,
    0.7
    );
    }
    }

    // Update the position and store for next frame
    this.setObjectPosition(index, newPosition);
    this.previousPositions[index] = newPosition;
    }
  3. Position stabilization with InBetween. InBetween calculates intermediate positions between objects:

    // Using InBetween for stable positioning
    @component
    export class InBetween extends BaseScriptComponent {
    @input
    target1!: SceneObject; // First reference object

    @input
    target2!: SceneObject; // Second reference object

    @input
    positionBlend: number = 0.5; // Weight between targets (0.5 = halfway)

    update() {
    if (!this.target1 || !this.target2) return;

    // Get positions
    const pos1 = this.target1.getTransform().getWorldPosition();
    const pos2 = this.target2.getTransform().getWorldPosition();

    // Calculate weighted position
    const blendedPosition = vec3.lerp(pos1, pos2, this.positionBlend);

    // Apply position
    this.getSceneObject().getTransform().setWorldPosition(blendedPosition);
    }
    }
  • Adaptive smoothing: smooth small movements more and large ones less.
  • Confidence-based smoothing: the higher a detection's confidence, the less smoothing it needs.
  • Distance-based parameters: adjust smoothing by the object's distance from the camera.
  • Frame history: use several previous frames, not only the most recent one.
  • MatchTransform: Best for general-purpose movement smoothing
  • MatchTransformLocal: When you need smoothing in local space rather than world space
  • InBetween: For positioning objects relative to two reference points
  • SmartTether: When you want content to follow the user with intelligent repositioning

Each component has specific strengths that make it suitable for different scenarios.

Build custom logic on SnapML detections

Once detection and spatialization work, build the logic for your AR experience. These examples illustrate patterns; the helper functions they call (showChairCustomization and the like) stand for your own code.

  1. Place content relative to the user with SmartTether. SmartTether keeps content positioned relative to the user:

    // SmartTether implementation
    @component
    export class SmartTether extends BaseScriptComponent {
    @input
    verticalDistanceFromTarget: number = 0.1; // Vertical threshold

    @input
    horizontalDistanceFromTarget: number = 0.1; // Horizontal threshold

    update() {
    // Get current positions
    const targetPos = this.target.getTransform().getWorldPosition();
    const objectPos = this.getSceneObject()
    .getTransform()
    .getWorldPosition();

    // Check if object needs repositioning
    if (this.needsRepositioning(targetPos, objectPos)) {
    // Calculate new position based on target's view
    const newPosition = this.calculateOptimalPosition(targetPos);

    // Apply new position
    this.getSceneObject().getTransform().setWorldPosition(newPosition);
    }
    }

    // Determines if the object is too far from ideal position
    needsRepositioning(targetPos: vec3, objectPos: vec3): boolean {
    // Implementation details...
    }

    // Calculates the best position relative to target
    calculateOptimalPosition(targetPos: vec3): vec3 {
    // Implementation details...
    }
    }
  2. React when the user selects a detected object. On SPECS, users select objects through Interaction Kit. Give the object you place at a detection an Interactable and a collider, then handle its onTriggerEnd event:

    import { Interactable } from 'InteractionKit.lspkg/Components/Interaction/Interactable/Interactable';

    // Example of custom interaction with a detected object
    function setupObjectInteractions(detectionObject: SceneObject, label: string) {
    const interactable = detectionObject.getComponent(
    Interactable.getTypeName()
    ) as Interactable;

    // Bind an action to the user selecting the object
    interactable.onTriggerEnd.add(() => {
    print('User selected ' + label);

    // Custom logic based on detected object
    switch (label) {
    case 'chair':
    showChairCustomization();
    break;
    case 'tv':
    playVideoOnTV();
    break;
    // Add more interactions for other classes
    }
    });

    // Add visual feedback for interactive objects
    addVisualFeedback(detectionObject);
    }
  3. Adapt content to what the model detects:

    // Dynamic content system
    function updateSceneBasedOnDetections(detections) {
    // Track what types of objects are detected
    const detectedClasses = {};

    detections.forEach((detection) => {
    detectedClasses[detection.label] = true;
    });

    // Customize experience based on detected objects
    if (detectedClasses['chair'] && detectedClasses['table']) {
    // Show furniture arrangement UI
    showFurnitureArrangementUI();
    } else if (detectedClasses['tv']) {
    // Show media controls
    showMediaControls();
    } else {
    // Default state when no relevant objects detected
    showSearchingUI();
    }
    }
  4. Analyze spatial relationships between detected objects. This example assumes you stored each detection's world position (for example, from captureToWorldTransform) as worldPosition:

    // Analyzing spatial relationships
    function analyzeObjectRelationships(detections) {
    if (detections.length < 2) return;

    // Compare position of each pair of objects
    for (let i = 0; i < detections.length; i++) {
    for (let j = i + 1; j < detections.length; j++) {
    const pos1 = detections[i].worldPosition;
    const pos2 = detections[j].worldPosition;

    // Calculate distance between objects
    const distance = pos1.distance(pos2);

    // Check if objects are close to each other
    if (distance < 100) {
    // 1 meter threshold (world units are centimeters)
    print(detections[i].label + ' is near ' + detections[j].label);

    // Custom logic for nearby objects
    handleNearbyObjects(detections[i], detections[j]);
    }
    }
    }
    }

When you implement custom logic:

  • Batch processing: process detections in batches rather than one by one.
  • Throttle updates: don't update UI or logic every frame unless you need to.
  • Level of detail: simplify processing for distant or less important objects.
  • Asynchronous operations: move heavy computations off the main thread when possible.

For more sophisticated applications:

  1. State Machines: Use state management to control the flow of your experience
  2. Object Persistence: Implement tracking IDs to maintain object identity across frames
  3. Prediction: Estimate future positions for smoother tracking during occlusions
  4. Multi-Object Relationships: Build logic around how multiple detected objects relate to each other
  5. Environment Understanding: Combine ML detection with SPECS environment features for enhanced context

These patterns help create more robust and responsive AR experiences.

The complete SnapML workflow for SPECS

  1. Environment setup
  2. Dataset preparation
  3. Model training
  4. ONNX export
  5. W8A8 quantization and DLC conversion
  6. Model import
  7. Spatialization
  8. Custom logic

This guide has walked through the whole SnapML workflow for SPECS, from environment setup and model training to the Lens Studio scripts and custom interaction logic. With both the ML pipeline and the script architecture in hand, you can build AR experiences that interact with real-world objects.

Resources for SnapML on SPECS

What you can build next with SnapML

Machine learning opens up new possibilities for AR interaction in SPECS experiences. By following this workflow, you've trained a custom object detection model, prepared it for SPECS with SnapML, and seen the scripts for spatialization, smoothing, and custom logic.

Cloud-based training environments, dataset management tools, and the script architecture in Lens Studio make it possible to build sophisticated machine learning features without extensive ML expertise.

As you continue developing for SPECS, experiment with different model architectures, datasets, detection parameters, and interaction patterns to make your AR experiences more engaging and responsive.