Page History

Overview

This page explains how to use the MultiSense framework (also known as multimodal framework, developed by Multicomp Lab) focusing on CLM (developed by Jason Saragih et .al) / Gavam (developed by Louis-Philippe Morency's Multicomp Lab) face-tracker, Kinect, FAAST (developed by Mixed Reality Group) which is based on the ssi lib (developed by Johannes Wagner, University of Augsburg). Each of these modules is implemented within the MultiSense framework and runs separate threads in a synchronized manner. The output from the MultiSense framework is PML (the pml xml schema can be found at PML.xsd) which is currently passed on using vrPerception message (based on VHMsg API).

...

MultiSense currently supports a lot of modules like vision (for tracking face features, smiles, gaze, attention, activity, gestures, among others), speech. To run the MultiSense, the user needs to run the ssi_vhmsger MultiSense application which basically sends out the vrPerception messages (based on pml) which can be subscribed by any module for specific data.

Running

...

MultiSense

There are two main steps involved in running MultiSense -

Running

...

MultiSense Application

To run the ssi_vhmsger MultiSense module, either

• Navigate to \bin\multisense\ssi_vhmsger\bin folder and run "run.bat".
• Or, after you run the launcher from \run-launcher.bat, click on the launch for MultiSense.

...

vrPerceptionProtocol

Known Issues

• The MultiSense Application currently does not listen to vrKillComponent message which means if the user tries to kill it from the launcher, it will turn yellow and not close. The user has to manually close the application.
• If the application does not seem to start up, make sure the camera is working properly in your computer. Also, if there is camera.option file present at the location "\bin\multisense\ssi_vhmsger\bin", delete it and try to run the application again. If there is a problem with running MultiSense, please contact us at vh-support@ict.usc.edu
• If the user enables Gavam and sees a big red X on the screen, it only means that Gavam is not able to detect the face properly. Currently, there is no way to reset Reset (other than restarting the application). The user can restart CLM tracker though by right clicking the output window.

FAQ

See Main FAQ for frequently asked questions regarding the installer. Please use the Google Groups emailing list for unlisted questions.