Page History

...

• Jul 11 2012 - Released a minor update to the toolkit, fixing some usability and stability issues.
• May 31 2012 - Released an exciting new version of the toolkit offering the MultiSense framework, the Rapport research platform and the SBMonitor tool. MultiSense is a perception framework that enables multiple sensing and understanding modules to interoperate inter-operate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker andFAAST which you can use with webcam or Kinect. The Rapport agent is a “virtual human listener” providing nonverbal feedback based on human nonverbal and verbal input. It has been used in a variety of international studies related to establishing rapport between real and virtual humans. Finally, the SBMonitor is a stand-alone tool for easy debugging of SmartBody applications, including testing available (facial) animations, gazes and more complex BML commands.
• Mar 2 2012 - Released a minor update of the toolkit updating the Unity version to 3.5 as well as providing incremental changes to the Unity/SmartBody debug tools in the Unity Editor (VH menu in Unity).
• Dec 22 2011 - Happy holidays!  Released a new version of  the toolkit which includes the ability to interrupt Brad, improved support for higher resolutions, and a fix for text-to-speech not working properly.
• Aug 10 2011 - Released a new version of the toolkit, which offers support for the free version of Unity 3D; users may now create scenes for Brad.  Download Unity here: http://www.unity3d.com.  For instructions on how to use the Unity 3D Editor with the toolkit, see the vhtoolkitUnity section. In addition, the user interaction has been improved; Unity now launches in full-screen automatically and users get visual feedback when talking to Brad. To directly talk to Brad, first make sure you have a microphone plugged in, wait for Brad to finish his introduction, and close the tips window. Now click and hold the left mouse button while asking Brad a question; release the mouse button when you are done talking. The recognized result will be displayed above Brad in white font (toggle on/off with the O key), and Brad will answer your question. It is advised to update Java and ActiveMQ, which are provided with the 3rd party installer versions.

...

Our technology emphasizes natural language interaction, nonverbal behavior, and visual recognition and is broken up into the following main modules:

• AcquireSpeech: A tool to send audio, or text, to speech recognizers and to relay the information to the entire system.  The toolkit uses PocketSphinx as a 3^rd party speech recognition solution.
  
• MultiSense: A perception framework that enables multiple sensing and understanding modules to inter-operate simultaneously, broadcasting data through Perception Markup Language.  It currently contains GAVAM, CLM FaceTracker, and FAAST which work with a webcam or Kinect.
• Non-Player Character Editor (NPCEditor):  A suite of tools which work together to create appropriate dialogue responses to users’ inputs.  A text classifier selects responses based on cross-language relevance models; the authoring interface relates questions and answers; and a simple dialogue manager controls aspects of output behavior.
• Nonverbal Behavior Generator (NVBG): A rule-based behavior planner that infers communicative functions from the surface text and selects appropriate behaviors that augment and complement the characters’ dialogue.
• Rapport 1.0: An agent that provides nonverbal feedback based on human nonverbal and verbal input.  Rapport has been used in a variety of international studies related to establishing rapport between real and virtual humans.
• SmartBody (SB):  A modular, controller-based character animation system that uses the Behavior Markup Language.
• Speech Client (AcquireSpeech): A tool to send audio, or text, to speech recognizers and to relay the information to the entire system.  The toolkit uses PocketSphinx as a 3^rd party speech recognition solution.
• Watson: A real-time visual feedback recognition library for interactive interfaces which uses the images from either a monocular or stereo camera to recognize eye and head gazes and gestures.

...

In addition, many groups outside of ICT use some of the toolkit components, most notably SmartBody and Watson. :

• • University of Reykjavik
  • German Research Center for Artificial Intelligence
  • ArticuLab at Northwestern University
  • Telecom Paris Tech
  • Affective Computing Research group at MIT Media Lab
  • ICSI/UCB Vision Group at UC Berkeley
  • Human-Centered, Intelligent, Human-Computer Interaction group at Imperial College
  • Worcester Polytechnic Institute
  • Microsoft Research
  • Relational Agents group at Northeastern University
  • Component Analysis Lab at Carnegie Mellon University

...