Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents


This page explains MultiSense is a perception framework that enables multiple sensing and understanding modules to inter-operate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker and FAAST which can be used with a webcam or Kinect. The Toolkit provides an example of how to use the MultiSense framework framework (also known as multimodal framework, developed by Multicomp Lab) focusing on .

MultiSense uses these technologies:






module is implemented within the MultiSense framework and runs in separate threads in a synchronized manner. The output from the MultiSense framework is PML (the pml xml schema can be found at PML.xsd


; please cite the IVA paper referenced here if you wish to use this PML) sent via vrPerception messages (through VHMsg).

The Toolkit includes two main components that use MultiSense:

  • ssi_vhmsger (\core\multisense\ssi_vhmsger\)
    This is the core application which is will only work if you have a web cam installed with that uses MultiSense and it only works if there is a webcam installed on the system. This application integrates all the many different MultiSense components like vision, audio and packages analysis into a single application whose output is a that outputs PML.
  • PerceptionTest (\core\multisense\PerceptionApplication)
    This application is a middle-man between the ssi_vhmsger and any other application which plans to use the PML for specific capabilities. For example, in the toolkitToolkit, this application takes input the PML as input (via the vrPerception messagesmessage) and outputs messages to the smartbody Smartbody to mimic the head orientation of the detected user.


MultiSense currently supports a lot of modules like vision multiple modules, including a vision module (for tracking face features , smileslike smile, gaze, attention, activity, gestures, among others), speech. To run the MultiSense, the user needs to etc) and a speech recognition module. To use MultiSense, run the MultiSense application which basically sends out the . It will broadcast vrPerception messages (based on pmlPML) generated by each module, which can be subscribed received by any module for specific dataother module or external component.


Using MultiSense

There are two main steps involved in running MultiSense -


Two components are required to make full use of MultiSense: the MultiSense Application and the Perception Application.

Running MultiSense Application

To run the MultiSense moduleApplication, either:

  • Run the application from the Launcher (go to the 'Perception Components' tab)
  • Navigate to \bin\multisense\ssi_vhmsger\bin folder and run "run.bat".Or, after you run the launcher from \run-launcher.bat, click on the launch for MultiSense.

By default, only the Gavam and CLM facetracker facetrackers are turned on (for the toolkit; otherwise only the Gavam is turned on). If .  If you wish to change the configuration for running other modules, you can navigate to :

  • Edit \bin\multisense\ssi_vhmsger\bin\config.ini and change the "useXXXX" parameter to "true" to make a module work.


  • If the dependencies for the module are not setup correctly, the ssi_vhmsger will crash.


  • If running FAAST, make sure the "useKinect" as well as "useFAAST" parameters are both "true" in the config.ini.


  • A webcam must be connected to the computer.
  • If the application fails to start, one of the dependencies might be missing.


  • ssi_vhmsger will not


  • exit when


  • the "Kill" button from the launcher is clicked. Also, if


  • it's killed from the launcher, the ssi tool will turn yellow.


Running the Perception


Application (PerceptionTest)

To run the PerceptionTest application, either:

  • Run it from the Launcher (go to the 'Perception Components' tab)
  • Navigate to \bin\multisense\PerceptionTest\ and run "PerceptionTest.exe".
  • Or, after you run the launcher from \run-launcher.bat, click on the launch for Perception Application.

Please contact either Unknown User (suri) or Unknown User (stratou) if you have any issues in running the application


Setting up the MultiSense

To setup the MultiSense, the user will need to checkout the repository from -

After the user downloads the framework, they will need to build module which allows to send and receive VHMsgs (located in \tools\ssi_vhmsger; The solution file is located at \tools\ssi_vhmsger\build\ssi_vhmsger.sln). Make "ssi_vhmsger_test as the "Startup project" . The user will also need to set the working directory as "..\..\..\..\bin". Once the project builds successfully, the user will need to have a web cam connected to the computer to send vrPerception messages using activeMQ (note that this module has a dependency on activeMQ and it will need to be installed. The latest installer for apache-activemq can be found at \\vhbuild\SASO-Installs).

You can check out the detailed description of how to setup the project at the following location -

VrPerception messages with SSI-ICT framework

As an illustration of integration, the please check out the following link (integrates the toolkit with MultiSense)

VHToolkit with Perception

If there is a problem with installing MultiSense, please contact Unknown User (suri) or Unknown User (stratou).

Message API



Message API


  • vrComponent


  • vrPerception
  • vrPerceptionProtocol
  • vrAllCall
  • vrKillComponent (does not react to it yet)

Known Issues

  • The MultiSense Application currently does not listen to the vrKillComponent message which means if the user tries . The application must be manually closed; attempts to kill it from the launcher, it will turn Launcher will cause it to highlight yellow and not close. The user has to manually close the application.
  • If the application does not seem appear to start up, make sure the your camera is working properly in your computer. Also, if there is camera.option file present at the location "\bin\multisense\ssi_vhmsger\bin", delete it and try to run the application again. If there is a problem with running MultiSense, please contact us at
  • If the user enables Gavam and sees a big red X appears on the screen after GAVAM is enable, it only means that Gavam GAVAM is not able to detect the face properly. CurrentlyThe application needs to be restarted; currently, there is no other way to reset Reset (other than restarting the application). The user can restart CLM tracker can be restarted, though, by right clicking the output window.


See the Main FAQ for frequently asked questions regarding the installer. Please use the Google Groups emailing list for unlisted questions.