Interaction System
How Interaction Kit for SPECS turns hand and mouse input into hover, trigger, and drag events on Interactables, and how to handle those events in a script.
- Snapchat: not supported
- SPECS: fully supported
- Camera Kit Web: not supported
- Camera Kit Mobile: not supported
- SnapchatNot supported
- SPECSFully supported
- Camera Kit WebNot supported
- Camera Kit MobileNot supported
Interaction Kit has a multi-modal interaction system for interactive AR content on SPECS. It has three core parts: interactors, which represent input methods such as the user's hands; Interactables, the objects in the scene that users can interact with; and the InteractionManager, which controls the lifecycle of each interaction. This page explains how they work together and how a script responds to an interaction.
How interactors, Interactables, and the InteractionManager work together
Each input source has its own interactor: a HandInteractor for each of the user's hands, and, in the Lens Studio Preview panel, a MouseInteractor that stands in for a hand. An interactor reads its input to target, trigger, and drag Interactables in the scene.
An Interactable makes a scene object with a collider respond to interactors. Its state reflects the interaction, such as hovered, triggered, or no longer triggered, and it sends an event for each change. Your scripts add callbacks to those events, such as onTriggerEnd, to run Lens-specific actions when a user selects an object.
The InteractionManager oversees every interaction and dispatches each event with a bubble-up and trickle-down approach: the ancestor Interactables of the targeted Interactable also receive the event.
InteractableManipulation lets users move, scale, and rotate an Interactable with an interactor, for movable elements such as menus.
For how these parts connect to Lens Studio's tracking modules, see Architecture.
Interaction Kit input types
The interactor's inputType tells a script which input caused an event. The values of
InteractorInputType (InteractionKit.lspkg/Core/Interactor/Interactor) are bit flags:
| Value | Input |
|---|---|
LeftHand, RightHand (together BothHands) | The user's hands, through a HandInteractor. |
Mouse | The mouse in the Lens Studio Preview panel, through the MouseInteractor. |
Mobile | A phone used as a controller, through a MobileInteractor. The InteractionKit prefab has none, and on SPECS 27 a phone is no longer available as a motion controller. |
BtController, CustomController | Input types for other controllers. |
All | Every input type, for queries such as getInteractorsByType. |
Interaction Kit targeting modes: indirect, direct pinch, and poke
Hand tracking lets users interact with AR content through natural gestures. An Interactable's Targeting Mode sets which of them can reach it:
| Targeting Mode | How the user interacts |
|---|---|
| Direct | Pinches the Interactable directly, with the hand touching it. |
| Indirect | Targets it from a distance with a ray, then pinches. |
| Direct/Indirect (default) | Either of the two. |
| Poke | Pokes it with the index finger. |
| All | Direct, Indirect, and Poke. |
In scripts, targetingMode takes the values 1 (Direct), 2 (Indirect), 3 (Direct/Indirect), 4
(Poke), and 7 (All).
Interactable events
Each event getter on an Interactable returns an event you subscribe to with .add(callback);
.add returns a function that removes the callback. Hover and trigger callbacks receive an
InteractorEvent, with the interactor that caused it, the target Interactable, and the
interactable the event was invoked on (an ancestor receives events for its descendants).
| Event | When it fires |
|---|---|
onHoverEnter | The Interactable becomes hovered. |
onHoverUpdate | Each frame an interactor keeps hovering it. |
onHoverExit | It is no longer hovered. |
onTriggerStart | It becomes triggered (pinched, poked, or clicked). |
onTriggerUpdate | Each frame an interactor keeps triggering it. |
onTriggerEnd | The trigger ends while the interactor still hovers it. |
onTriggerEndOutside | The trigger ends while the interactor no longer hovers it. |
onTriggerCanceled | The interactor that was triggering it is lost, for example when a hand loses tracking. |
onDragStart, onDragUpdate, onDragEnd | An interactor that triggered it moves past the drag threshold, keeps moving, and releases. Callbacks receive a DragInteractorEvent. |
onSecondaryTriggerStart, onSecondaryTriggerUpdate, onSecondaryTriggerEnd, onSecondaryTriggerEndOutside, onSecondaryTriggerCanceled | The user holds a trigger. The hold starts only if the targeted Interactable itself has an onSecondaryTriggerStart callback. |
The onInteractor… variants (onInteractorHoverEnter, onInteractorHoverExit,
onInteractorTriggerStart, onInteractorTriggerEnd, onInteractorTriggerEndOutside, and the
onInteractorSecondaryTrigger… events) fire for each interactor that starts or stops hovering
or triggering the Interactable, while the events without the prefix fire when the Interactable's
own state changes.
Interactable inputs in the Inspector
| Input | Default | What it does |
|---|---|---|
| Targeting Mode | Direct/Indirect | Which gestures can target the Interactable. See Interaction Kit targeting modes. |
| Targeting Visual | Cursor | What shows while the Interactable is targeted: None, Cursor, or Ray. An enclosing InteractionPlane overrides it unless Ignore Interaction Plane is on. |
| Ignore Interaction Plane | Off | Ignores a parent InteractionPlane, for an Interactable parented under one for organization but not placed in it. |
| Include Colliders In Physics | Off | Lets the Interactable's colliders also take part in Lens physics: physics bodies, Physics raycasts, and overlap and collision events. |
| Keep Hover On Trigger | Off | Keeps the cursor locked on the Interactable, and its active visual state, while it is triggered, even after the interactor moves off it. |
| Enable Instant Drag | Off | Starts a drag as soon as the trigger starts, rather than after the interactor's drag threshold. |
| Is Scrollable | Off | Lets interactions with this Interactable scroll a parent scroll view whose content extends beyond its visible bounds. |
| Allow Multiple Interactors | On | Lets several interactors control the Interactable at once, such as both hands manipulating it. Off, only one interactor type can interact with it at a time. |
| Enable Poke Directionality | Off | Restricts which directions a poke can come from, on the X, Y, and Z axes, to prevent accidental pokes. |
| Use Filtered Pinch | Off | Uses filtered pinch events, which are more stable while the hand moves fast but can add latency. Turn it on for interactions with large hand movement, such as dragging and scrolling. |
From a script, targetingPriority (TargetingPriority.Normal or TargetingPriority.High) makes
an Interactable preferred over closer, lower-priority ones.
How InteractableManipulation changes its Interactable
InteractableManipulation needs an Interactable on the same scene object. Without one it logs
InteractableManipulation requires an interactable to function. when the Lens starts, and the
object doesn't move. On that Interactable, InteractableManipulation:
- Removes Poke from Targeting Mode, because poke targeting is incompatible with manipulation. An Interactable set to Poke alone then can't be grabbed, so keep manipulated objects on Direct/Indirect, and put poke-only controls on a separate Interactable without manipulation.
- Turns on Use Filtered Pinch, which can delay the start of a grab by a few frames.
If an object shows hover and trigger feedback but doesn't move under the Preview mouse or a distant ray, turn on Enable Instant Drag on its Interactable.
Interaction plane
An InteractionPlane defines a rectangular zone in front of 2D UI with many buttons close together, where hand interactors switch to near-field targeting. Add one per UI; a scene object can hold only one InteractionPlane. Its inputs are Plane Size (the zone's X and Y size), Offset (its position relative to the scene object), and Targeting Visual, which overrides the Targeting Visual of the Interactables under it unless they set Ignore Interaction Plane.
From far to close, the InteractionPlane's zones are:
| Zone | Default depth | Targeting |
|---|---|---|
| Far field | Beyond the near-field depth | Standard ray targeting. |
| Near field | Within 17 cm (nearFieldDepth) | Near-field targeting. |
| Physical zone | Within 3 cm (physicalZoneDepth) | Physical targeting, for poke and direct pinch. |
| Behind zone | Up to 15 cm behind the plane (behindZoneDepth) | Tolerance for a finger that pushes through. |
The exit thresholds are larger than the entry thresholds, so targeting doesn't flicker at a zone boundary.
Respond to an Interactable event in a script
This script prints which input triggered the Interactable on its scene object, and where. It
assumes the scene object already has an Interactable and a collider. It subscribes in
OnStartEvent, after Interaction Kit's components have initialized.
- TypeScript
- JavaScript
import { Interactable } from 'InteractionKit.lspkg/Components/Interaction/Interactable/Interactable';
import { InteractorEvent } from 'InteractionKit.lspkg/Core/Interactor/InteractorEvent';
import { IK } from 'InteractionKit.lspkg/IK';
@component
export class ExampleInteractionScript extends BaseScriptComponent {
onAwake() {
this.createEvent('OnStartEvent').bind(() => {
this.onStart();
});
}
onStart() {
const interactionManager = IK.InteractionManager;
// Get the Interactable on this SceneObject.
let interactable: Interactable | null = this.sceneObject.getComponent(
Interactable.getTypeName()
) as Interactable;
// You can also get it through the InteractionManager.
interactable = interactionManager.getInteractableBySceneObject(
this.sceneObject
);
if (!interactable) {
print('Add an Interactable and a collider to this SceneObject.');
return;
}
// Run this callback each time an interactor starts triggering the Interactable.
interactable.onInteractorTriggerStart.add((event: InteractorEvent) => {
const hit = event.interactor.targetHitInfo?.hit;
print(
`The Interactable has been triggered by an Interactor with input type: ${event.interactor.inputType} at position: ${hit ? hit.position : 'unknown'}`
);
});
}
}
const IK = require('InteractionKit.lspkg/IK').IK;
const interactionManager = IK.InteractionManager;
const interactionConfiguration = IK.InteractionConfiguration;
function onAwake() {
// Wait for other components to initialize by deferring to OnStartEvent.
script.createEvent('OnStartEvent').bind(() => {
onStart();
});
}
function onStart() {
// Get the Interactable on this SceneObject.
var interactableTypename =
interactionConfiguration.requireType('Interactable');
var interactable = script.sceneObject.getComponent(interactableTypename);
// You can also get it through the InteractionManager.
interactable = interactionManager.getInteractableBySceneObject(
script.sceneObject
);
if (!interactable) {
print('Add an Interactable and a collider to this SceneObject.');
return;
}
// Run this callback each time an interactor starts triggering the Interactable.
interactable.onInteractorTriggerStart.add((event) => {
var hitInfo = event.interactor.targetHitInfo;
print(
`The Interactable has been triggered by an Interactor with input type: ${event.interactor.inputType} at position: ${hitInfo ? hitInfo.hit.position : 'unknown'}`
);
});
}
onAwake();
In JavaScript, IK.InteractionConfiguration.requireType returns the type name of an Interaction
Kit component for getComponent. It accepts Billboard, HandInteractor, HandVisual,
Headlock, Interactable, InteractableManipulation, and MouseInteractor, and throws for any
other name.
Find Interactables and interactors from a script
IK.InteractionManager also answers these queries:
| Method | Returns |
|---|---|
getInteractableBySceneObject(sceneObject) | The Interactable on a scene object, or null. |
getInteractableByCollider(collider) | The Interactable a ColliderComponent belongs to, or null. |
getInteractablesThatTarget(targetingMode) | The Interactables whose Targeting Mode includes a TargetingMode. |
getInteractorsByType(inputType) | The interactors of an InteractorInputType. See Hand Tracking. |
Related
- Hand Tracking: hand data and HandInteractor in scripts.
- Cursor: the cursor and ray that show what an interactor targets.
- Helpers: ready-made audio and visual feedback for Interactable events.
- Architecture: the order in which the InteractionManager sends events.