MultiSense is a perception framework that enables multiple sensing and understanding modules to inter-operate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker and FAAST which can be used with a webcam or Kinect. The Toolkit provides an example of how to use the MultiSense framework (also known as multimodal framework, developed by Multicomp Lab).

MultiSense uses these technologies:

Each module is implemented within the MultiSense framework and runs in separate threads in a synchronized manner. The output from the MultiSense framework is PML (the pml xml schema can be found at PML.xsd; please cite the IVA paper referenced here if you wish to use this PML) sent via vrPerception messages (through VHMsg).

The Toolkit includes two main components that use MultiSense:


MultiSense supports multiple modules, including a vision module (for tracking face features like smile, gaze, attention, activity, etc) and a speech recognition module. To use MultiSense, run the MultiSense application. It will broadcast vrPerception messages (based on PML) generated by each module, which can be received by any other module or external component.

Using MultiSense

Two components are required to make full use of MultiSense: the MultiSense Application and the Perception Application.

Running MultiSense Application

To run the MultiSense Application, either:

By default, only the Gavam and CLM facetrackers are turned on.  If you wish to change the configuration for running other modules:

Running the Perception Application (PerceptionTest)

To run the PerceptionTest application, either:

Message API



Known Issues


See the Main FAQ. Please use the Google Groups emailing list for unlisted questions.