Page tree


MultiSense is a perception framework that enables multiple sensing and understanding modules to inter-operate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker and FAAST which can be used with a webcam or Kinect. The Toolkit provides an example of how to use the MultiSense framework (also known as multimodal framework, developed by Multicomp Lab).

MultiSense uses these technologies:

Each module is implemented within the MultiSense framework and runs in separate threads in a synchronized manner. The output from the MultiSense framework is PML (the pml xml schema can be found at PML.xsd; please cite the IVA paper referenced here if you wish to use this PML) sent via vrPerception messages (through VHMsg).

The Toolkit includes two main components that use MultiSense:

  • ssi_vhmsger (\core\multisense\ssi_vhmsger\)
    This is the core application that uses MultiSense and it only works if there is a webcam installed on the system. This application integrates many different MultiSense components like vision, audio and analysis into a single application that outputs PML.
  • PerceptionTest (\core\multisense\PerceptionApplication)
    This application is a middle-man between the ssi_vhmsger and any other application which plans to use the PML for specific capabilities. For example, in the Toolkit, this application takes the PML as input (via the vrPerception message) and outputs messages to Smartbody to mimic the head orientation of the detected user.


MultiSense supports multiple modules, including a vision module (for tracking face features like smile, gaze, attention, activity, etc) and a speech recognition module. To use MultiSense, run the MultiSense application. It will broadcast vrPerception messages (based on PML) generated by each module, which can be received by any other module or external component.

Using MultiSense

Two components are required to make full use of MultiSense: the MultiSense Application and the Perception Application.

Running MultiSense Application

To run the MultiSense Application, either:

  • Run the application from the Launcher (go to the 'Perception Components' tab)
  • Navigate to \bin\multisense\ssi_vhmsger\bin folder and run "run.bat".

By default, only the Gavam and CLM facetrackers are turned on.  If you wish to change the configuration for running other modules:

  • Edit \bin\multisense\ssi_vhmsger\bin\config.ini and change the "useXXXX" parameter to "true" to make a module work.
  • If the dependencies for the module are not setup correctly, the ssi_vhmsger will crash.
  • If running FAAST, make sure the "useKinect" as well as "useFAAST" parameters are both "true" in the config.ini.
  • A webcam must be connected to the computer.
  • If the application fails to start, one of the dependencies might be missing.
  • ssi_vhmsger will not exit when the "Kill" button from the launcher is clicked. Also, if it's killed from the launcher, the ssi tool will turn yellow.

Running the Perception Application (PerceptionTest)

To run the PerceptionTest application, either:

  • Run it from the Launcher (go to the 'Perception Components' tab)
  • Navigate to \bin\multisense\PerceptionTest\ and run "PerceptionTest.exe".

Message API


  • vrComponent


  • vrPerception
  • vrPerceptionProtocol
  • vrAllCall
  • vrKillComponent (does not react to it yet)

Known Issues

  • The MultiSense Application currently does not listen to the vrKillComponent message. The application must be manually closed; attempts to kill it from the Launcher will cause it to highlight yellow and not close.
  • If the application does not appear to start up, make sure your camera is working properly. Also, if there is camera.option file present at the location "\bin\multisense\ssi_vhmsger\bin", delete it and try to run the application again. If there is a problem with running MultiSense.
  • If a big red X appears on the screen after GAVAM is enable, it means that GAVAM is not able to detect the face properly. The application needs to be restarted; currently, there is no other way to reset. The CLM tracker can be restarted, though, by right clicking the output window.


See the Main FAQ. Please use the Google Groups emailing list for unlisted questions.

  • No labels