Overview

MultiSense is a perception framework that enables multiple sensing and understanding modules to inter-operate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker and FAAST which can be used with a webcam or Kinect. The Toolkit provides an example of how to use the MultiSense framework (also known as multimodal framework, developed by Multicomp Lab).

MultiSense uses these technologies:

Each module is implemented within the MultiSense framework and runs in separate threads in a synchronized manner. The output from the MultiSense framework is PML (the pml xml schema can be found at PML.xsd; please cite the IVA paper referenced here if you wish to use this PML) sent via vrPerception messages (through VHMsg).

The Toolkit includes two main components that use MultiSense:

Users

MultiSense supports multiple modules, including a vision module (for tracking face features like smile, gaze, attention, activity, etc) and a speech recognition module. To use MultiSense, run the MultiSense application. It will broadcast vrPerception messages (based on PML) generated by each module, which can be received by any other module or external component.

Using MultiSense

Two components are required to make full use of MultiSense: the MultiSense Application and the Perception Application.

Running MultiSense Application

To run the MultiSense Application, either:

By default, only the Gavam and CLM facetrackers are turned on (for the Toolkit; otherwise only the Gavam is turned on).

Notes:

Running the Perception Application (PerceptionTest)

To run the PerceptionTest application, either:

Message API

Sends:

Receives:

Known Issues

FAQ

See the Main FAQ. Please use the Google Groups emailing list for unlisted questions.