Pick Event

In 3D applications, it is often necessary to click on objects in the scene. The engine supports ray box picking and frame buffer picking.

The supported pick events are:

Name	Explanation
PICK_OVER	Triggered once when the touch point enters the collision body range
PICK_OUT	Triggered once when the touch point leaves the collision body range
PICK_CLICK	Triggered once when the touch point is pressed and released in the collision body range
PICK_MOVE	Triggered when the touch point moves within the collision body range
PICK_UP	Triggered once when the touch point is released within the collision body range
PICK_DOWN	Triggered once when the touch point is pressed within the collision body range

Pick Detection

The engine will catch all mouse events, performs hit detection on all clickable Object3D objects in the scene, and triggers the corresponding events. Users can add a ColliderComponent to mark objects as clickable and listen for the corresponding PointerEvent3D events.

The engine provides a unified encapsulation for two types of hit detection methods, which can be switched through simple configurations.

//Pick and pick type need to be configured before the engine starts
Engine3D.setting.pick.enable = true;
// Bound: ray box picking, pixel: frame buffer picking
Engine3D.setting.pick.mode = `bound`; // or 'pixel'

await Engine3D.init()
// Picking detection depends on the Collider component
let obj = Object3D();
obj.addComponent(ColliderComponent);

// Add a PickEvent event listener to the node, where the corresponding event can be obtained in the callback function
obj.addEventListener(PointerEvent3D.PICK_CLICK, onPick, this);

// Or listen to all object click events through view.pickFire
view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, onPick, this);

//Get event information in the callback function
function onPick(e: PointerEvent3D) {
    e.target // the clicked Object
    e.data.worldPos // clicked position in world coordinate
    e.data.worldNormal // clicked normal in world coordinate
    ...
}

Ray Box Picking

Ray box picking (bound mode) is a commonly used CPU-based picking method. It needs to calculate the intersection of the ColliderComponent's ColliderShape and the mouse ray. It performs well in scenes with few objects, but has poor accuracy because the bounding box often cannot accurately represent the true shape of the object.

Currenly, the basic ColliderShape provided by engine includes BoxColliderShape, SphereColliderShape and CapsuleColliderShape. You can also construct a MeshColliderShape based on the shape of the object's own Mesh.

import {Object3D, Collider, BoxColliderShape, Vector3} from '@orillusion/core';

let box = new Object3D();
let mr = box.addComponent(MeshRenderer);
// Set the box geometry
mr.geometry = new BoxGeometry(1,1,1);
// Add collision box detection
let collider = box.addComponent(ColliderComponent);
// For the bound mode, the style and size of the collision box need to be set manually
// The picking accuracy depends on the match between box.geometry and collider.shape
collider.shape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(1, 1, 1));

• The box on the left uses BoxColliderShape with the same shape for detection, which has better accuracy.
• The sphere on the middle also uses BoxColliderShape, but the clickable area is larger than the actual model, resulting in lower accuracy.
• The sphere on the right uses MeshColliderShape, which cloud perfectly conform to all vertices of the model, offering the highest precision, but it consumes more performance for collision detection, therefore it is not recommended for complex objects.

WebGPU is not supported in your browser
Please upgrade to latest Chrome/Edge

<

import { Engine3D, Scene3D, Vector3, Object3D, AtmosphericComponent, Camera3D, View3D, LitMaterial, MeshRenderer, BoxColliderShape, ColliderComponent, BoxGeometry, Color, PointerEvent3D, SphereGeometry, DirectLight, BoundingBox, MeshColliderShape } from '@orillusion/core';
import { Graphic3D } from '@orillusion/graphic';

class TouchDemo {
    scene: Scene3D;
    cameraObj: Object3D;
    camera: Camera3D;
    graphic3D: Graphic3D;

    constructor() {}

    async run() {
        console.log('start demo');
        // enable pick and use bound mode
        Engine3D.setting.pick.enable = true;
        Engine3D.setting.pick.mode = `bound`;

        await Engine3D.init();

        this.scene = new Scene3D();
        this.scene.addComponent(AtmosphericComponent);
        this.cameraObj = new Object3D();
        this.camera = this.cameraObj.addComponent(Camera3D);
        this.scene.addChild(this.cameraObj);
        this.camera.lookAt(new Vector3(0, 0, 10), new Vector3(0, 0, 0));
        this.camera.perspective(60, Engine3D.aspect, 1, 10000.0);

        // add a base light
        let lightObj = new Object3D();
        lightObj.addComponent(DirectLight);
        this.scene.addChild(lightObj);

        let box = this.createBox(-4, 0, 0);
        let sphere = this.createSphere(0, 0, 0);
        let sphere2 = this.createSphere2(4, 0, 0);

        this.graphic3D = new Graphic3D();
        this.scene.addChild(this.graphic3D);
        this.graphic3D.drawBoundingBox(box.instanceID, box.bound as BoundingBox, Color.COLOR_GREEN);
        this.graphic3D.drawBoundingBox(sphere.instanceID, sphere.bound as BoundingBox, Color.COLOR_GREEN);
        this.graphic3D.drawMeshWireframe(sphere2.instanceID, new SphereGeometry(1.01, 8, 8), sphere2.transform, Color.COLOR_GREEN)
        
        let view = new View3D();
        view.scene = this.scene;
        view.camera = this.camera;
        // start render
        Engine3D.startRenderView(view);

        // listen all pick_click events
        view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, this.onPick, this);
    }

    createBox(x: number, y: number, z: number) {
        let boxObj = new Object3D();
        boxObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: BoxColliderShape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(size, size, size));
        // add a box collider
        let collider = boxObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = boxObj.addComponent(MeshRenderer);
        mr.geometry = new BoxGeometry(size, size, size);
        mr.material = new LitMaterial();
        this.scene.addChild(boxObj);
        return boxObj;
    }

    createSphere(x: number, y: number, z: number) {
        let sphereObj = new Object3D();
        sphereObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: BoxColliderShape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(size, size, size));
        // add a box collider
        let collider = sphereObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = sphereObj.addComponent(MeshRenderer);
        mr.geometry = new SphereGeometry(size / 2, 8, 8);
        mr.material = new LitMaterial();
        this.scene.addChild(sphereObj);
        return sphereObj;
    }

    createSphere2(x: number, y: number, z: number) {
        let sphereObj = new Object3D();
        sphereObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: MeshColliderShape = new MeshColliderShape()
        
        // add a box collider
        let collider = sphereObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = sphereObj.addComponent(MeshRenderer);
        mr.geometry = shape.mesh = new SphereGeometry(size / 2, 8, 8);
        mr.material = new LitMaterial();
        this.scene.addChild(sphereObj);
        return sphereObj;
    }

    onPick(e: PointerEvent3D) {
        console.log('onClick:', e);
        let mr: MeshRenderer = e.target.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }
}
new TouchDemo().run();

Frame Buffer Picking

Unlike the bound mode, Frame Buffer Picking (pixel mode) utilizes the pixel detection of the GPU, which consumes almost no CPU performance and can ignore the number and complexity of interactive objects in the scene, supporting all touch events. When the shape of the scene model is complex or there are a large number of objects, we recommend using the pixel mode for picking detection.

WebGPU is not supported in your browser
Please upgrade to latest Chrome/Edge

<

import { AtmosphericComponent, BoxColliderShape, Camera3D, CameraUtil, ColliderComponent, Color, View3D, DirectLight, Engine3D, LitMaterial, HoverCameraController, KelvinUtil, MeshRenderer, Object3D, PointerEvent3D, Scene3D, SphereGeometry, Vector3 } from '@orillusion/core';

class Sample_MousePick {
    lightObj: Object3D;
    cameraObj: Camera3D;
    scene: Scene3D;
    hover: HoverCameraController;

    constructor() {}

    async run() {
        // enable pick and use pixel mode
        Engine3D.setting.pick.enable = true;
        Engine3D.setting.pick.mode = `pixel`;

        await Engine3D.init({});

        this.scene = new Scene3D();
        this.scene.addComponent(AtmosphericComponent);
        let camera = CameraUtil.createCamera3DObject(this.scene);
        camera.perspective(60, Engine3D.aspect, 1, 5000.0);

        this.hover = camera.object3D.addComponent(HoverCameraController);
        this.hover.setCamera(-30, -15, 120);

        let wukong = await Engine3D.res.loadGltf('https://cdn.orillusion.com/gltfs/wukong/wukong.gltf');
        wukong.transform.y = 30;
        wukong.transform.scaleX = 20;
        wukong.transform.scaleY = 20;
        wukong.transform.scaleZ = 20;
        wukong.forChild((node) => {
            if (node.hasComponent(MeshRenderer)) {
                node.addComponent(ColliderComponent);
            }
        });
        this.scene.addChild(wukong);

        this.initPickObject(this.scene);

        let view = new View3D();
        view.scene = this.scene;
        view.camera = camera;
        // start render
        Engine3D.startRenderView(view);

        // listen all mouse events
        view.pickFire.addEventListener(PointerEvent3D.PICK_UP, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_DOWN, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_OVER, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_OUT, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_MOVE, this.onPick, this);
    }

    private initPickObject(scene: Scene3D): void {
        /******** light *******/
        {
            this.lightObj = new Object3D();
            this.lightObj.rotationX = 125;
            this.lightObj.rotationY = 0;
            this.lightObj.rotationZ = 40;
            let lc = this.lightObj.addComponent(DirectLight);
            lc.lightColor = KelvinUtil.color_temperature_to_rgb(5355);
            lc.castShadow = true;
            lc.intensity = 5;
            scene.addChild(this.lightObj);
        }

        let size: number = 9;
        let shape = new BoxColliderShape();
        shape.setFromCenterAndSize(new Vector3(), new Vector3(size, size, size));

        let geometry = new SphereGeometry(size / 2, 20, 20);
        for (let i = 0; i < 10; i++) {
            let obj = new Object3D();
            obj.name = 'sphere ' + i;
            scene.addChild(obj);
            obj.x = (i - 5) * 10;

            let mat = new LitMaterial();
            mat.emissiveMap = Engine3D.res.grayTexture;
            mat.emissiveIntensity = 0.0;

            let renderer = obj.addComponent(MeshRenderer);
            renderer.geometry = geometry;
            renderer.material = mat;
            obj.addComponent(ColliderComponent);
        }
    }

    private onPick(e: PointerEvent3D) {
        console.log(e.type, e.target.name, e.data);
        if(e.type !== 'onPickMove'){
            let obj = e.target as Object3D;
            let mr = obj.getComponent(MeshRenderer);
            mr.material.baseColor = Color.random();
        }
    }
}
new Sample_MousePick().run();