Skip to main content

Interaction System

How Interaction Kit for SPECS turns hand and mouse input into hover, trigger, and drag events on Interactables, and how to handle those events in a script.

  • Snapchat: not supported
  • SPECS: fully supported
  • Camera Kit Web: not supported
  • Camera Kit Mobile: not supported
  • SnapchatNot supported
  • SPECSFully supported
  • Camera Kit WebNot supported
  • Camera Kit MobileNot supported

Platform details

  • Snapchat — Not supported.

  • SPECS — Fully supported.

  • Camera Kit Web — Not supported.

  • Camera Kit Mobile — Not supported.

Interaction Kit has a multi-modal interaction system for interactive AR content on SPECS. It has three core parts: interactors, which represent input methods such as the user's hands; Interactables, the objects in the scene that users can interact with; and the InteractionManager, which controls the lifecycle of each interaction. This page explains how they work together and how a script responds to an interaction.

How interactors, Interactables, and the InteractionManager work together

Each input source has its own interactor: a HandInteractor for each of the user's hands, and, in the Lens Studio Preview panel, a MouseInteractor that stands in for a hand. An interactor reads its input to target, trigger, and drag Interactables in the scene.

An Interactable makes a scene object with a collider respond to interactors. Its state reflects the interaction, such as hovered, triggered, or no longer triggered, and it sends an event for each change. Your scripts add callbacks to those events, such as onTriggerEnd, to run Lens-specific actions when a user selects an object.

The InteractionManager oversees every interaction and dispatches each event with a bubble-up and trickle-down approach: the ancestor Interactables of the targeted Interactable also receive the event.

InteractableManipulation lets users move, scale, and rotate an Interactable with an interactor, for movable elements such as menus.

For how these parts connect to Lens Studio's tracking modules, see Architecture.

Interaction Kit input types

The interactor's inputType tells a script which input caused an event. The values of InteractorInputType (InteractionKit.lspkg/Core/Interactor/Interactor) are bit flags:

ValueInput
LeftHand, RightHand (together BothHands)The user's hands, through a HandInteractor.
MouseThe mouse in the Lens Studio Preview panel, through the MouseInteractor.
MobileA phone used as a controller, through a MobileInteractor. The InteractionKit prefab has none, and on SPECS 27 a phone is no longer available as a motion controller.
BtController, CustomControllerInput types for other controllers.
AllEvery input type, for queries such as getInteractorsByType.

Interaction Kit targeting modes: indirect, direct pinch, and poke

Hand tracking lets users interact with AR content through natural gestures. An Interactable's Targeting Mode sets which of them can reach it:

Targeting ModeHow the user interacts
DirectPinches the Interactable directly, with the hand touching it.
IndirectTargets it from a distance with a ray, then pinches.
Direct/Indirect (default)Either of the two.
PokePokes it with the index finger.
AllDirect, Indirect, and Poke.

In scripts, targetingMode takes the values 1 (Direct), 2 (Indirect), 3 (Direct/Indirect), 4 (Poke), and 7 (All).

Interactable events

Each event getter on an Interactable returns an event you subscribe to with .add(callback); .add returns a function that removes the callback. Hover and trigger callbacks receive an InteractorEvent, with the interactor that caused it, the target Interactable, and the interactable the event was invoked on (an ancestor receives events for its descendants).

EventWhen it fires
onHoverEnterThe Interactable becomes hovered.
onHoverUpdateEach frame an interactor keeps hovering it.
onHoverExitIt is no longer hovered.
onTriggerStartIt becomes triggered (pinched, poked, or clicked).
onTriggerUpdateEach frame an interactor keeps triggering it.
onTriggerEndThe trigger ends while the interactor still hovers it.
onTriggerEndOutsideThe trigger ends while the interactor no longer hovers it.
onTriggerCanceledThe interactor that was triggering it is lost, for example when a hand loses tracking.
onDragStart, onDragUpdate, onDragEndAn interactor that triggered it moves past the drag threshold, keeps moving, and releases. Callbacks receive a DragInteractorEvent.
onSecondaryTriggerStart, onSecondaryTriggerUpdate, onSecondaryTriggerEnd, onSecondaryTriggerEndOutside, onSecondaryTriggerCanceledThe user holds a trigger. The hold starts only if the targeted Interactable itself has an onSecondaryTriggerStart callback.

The onInteractor… variants (onInteractorHoverEnter, onInteractorHoverExit, onInteractorTriggerStart, onInteractorTriggerEnd, onInteractorTriggerEndOutside, and the onInteractorSecondaryTrigger… events) fire for each interactor that starts or stops hovering or triggering the Interactable, while the events without the prefix fire when the Interactable's own state changes.

Interactable inputs in the Inspector

InputDefaultWhat it does
Targeting ModeDirect/IndirectWhich gestures can target the Interactable. See Interaction Kit targeting modes.
Targeting VisualCursorWhat shows while the Interactable is targeted: None, Cursor, or Ray. An enclosing InteractionPlane overrides it unless Ignore Interaction Plane is on.
Ignore Interaction PlaneOffIgnores a parent InteractionPlane, for an Interactable parented under one for organization but not placed in it.
Include Colliders In PhysicsOffLets the Interactable's colliders also take part in Lens physics: physics bodies, Physics raycasts, and overlap and collision events.
Keep Hover On TriggerOffKeeps the cursor locked on the Interactable, and its active visual state, while it is triggered, even after the interactor moves off it.
Enable Instant DragOffStarts a drag as soon as the trigger starts, rather than after the interactor's drag threshold.
Is ScrollableOffLets interactions with this Interactable scroll a parent scroll view whose content extends beyond its visible bounds.
Allow Multiple InteractorsOnLets several interactors control the Interactable at once, such as both hands manipulating it. Off, only one interactor type can interact with it at a time.
Enable Poke DirectionalityOffRestricts which directions a poke can come from, on the X, Y, and Z axes, to prevent accidental pokes.
Use Filtered PinchOffUses filtered pinch events, which are more stable while the hand moves fast but can add latency. Turn it on for interactions with large hand movement, such as dragging and scrolling.

From a script, targetingPriority (TargetingPriority.Normal or TargetingPriority.High) makes an Interactable preferred over closer, lower-priority ones.

How InteractableManipulation changes its Interactable

InteractableManipulation needs an Interactable on the same scene object. Without one it logs InteractableManipulation requires an interactable to function. when the Lens starts, and the object doesn't move. On that Interactable, InteractableManipulation:

  • Removes Poke from Targeting Mode, because poke targeting is incompatible with manipulation. An Interactable set to Poke alone then can't be grabbed, so keep manipulated objects on Direct/Indirect, and put poke-only controls on a separate Interactable without manipulation.
  • Turns on Use Filtered Pinch, which can delay the start of a grab by a few frames.

If an object shows hover and trigger feedback but doesn't move under the Preview mouse or a distant ray, turn on Enable Instant Drag on its Interactable.

Interaction plane

An InteractionPlane defines a rectangular zone in front of 2D UI with many buttons close together, where hand interactors switch to near-field targeting. Add one per UI; a scene object can hold only one InteractionPlane. Its inputs are Plane Size (the zone's X and Y size), Offset (its position relative to the scene object), and Targeting Visual, which overrides the Targeting Visual of the Interactables under it unless they set Ignore Interaction Plane.

From far to close, the InteractionPlane's zones are:

ZoneDefault depthTargeting
Far fieldBeyond the near-field depthStandard ray targeting.
Near fieldWithin 17 cm (nearFieldDepth)Near-field targeting.
Physical zoneWithin 3 cm (physicalZoneDepth)Physical targeting, for poke and direct pinch.
Behind zoneUp to 15 cm behind the plane (behindZoneDepth)Tolerance for a finger that pushes through.

The exit thresholds are larger than the entry thresholds, so targeting doesn't flicker at a zone boundary.

Respond to an Interactable event in a script

This script prints which input triggered the Interactable on its scene object, and where. It assumes the scene object already has an Interactable and a collider. It subscribes in OnStartEvent, after Interaction Kit's components have initialized.

import { Interactable } from 'InteractionKit.lspkg/Components/Interaction/Interactable/Interactable';
import { InteractorEvent } from 'InteractionKit.lspkg/Core/Interactor/InteractorEvent';
import { IK } from 'InteractionKit.lspkg/IK';

@component
export class ExampleInteractionScript extends BaseScriptComponent {
onAwake() {
this.createEvent('OnStartEvent').bind(() => {
this.onStart();
});
}

onStart() {
const interactionManager = IK.InteractionManager;

// Get the Interactable on this SceneObject.
let interactable: Interactable | null = this.sceneObject.getComponent(
Interactable.getTypeName()
) as Interactable;

// You can also get it through the InteractionManager.
interactable = interactionManager.getInteractableBySceneObject(
this.sceneObject
);
if (!interactable) {
print('Add an Interactable and a collider to this SceneObject.');
return;
}

// Run this callback each time an interactor starts triggering the Interactable.
interactable.onInteractorTriggerStart.add((event: InteractorEvent) => {
const hit = event.interactor.targetHitInfo?.hit;
print(
`The Interactable has been triggered by an Interactor with input type: ${event.interactor.inputType} at position: ${hit ? hit.position : 'unknown'}`
);
});
}
}

In JavaScript, IK.InteractionConfiguration.requireType returns the type name of an Interaction Kit component for getComponent. It accepts Billboard, HandInteractor, HandVisual, Headlock, Interactable, InteractableManipulation, and MouseInteractor, and throws for any other name.

Find Interactables and interactors from a script

IK.InteractionManager also answers these queries:

MethodReturns
getInteractableBySceneObject(sceneObject)The Interactable on a scene object, or null.
getInteractableByCollider(collider)The Interactable a ColliderComponent belongs to, or null.
getInteractablesThatTarget(targetingMode)The Interactables whose Targeting Mode includes a TargetingMode.
getInteractorsByType(inputType)The interactors of an InteractorInputType. See Hand Tracking.
  • Hand Tracking: hand data and HandInteractor in scripts.
  • Cursor: the cursor and ray that show what an interactor targets.
  • Helpers: ready-made audio and visual feedback for Interactable events.
  • Architecture: the order in which the InteractionManager sends events.