Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This Confluence site contains the main documentation portion for the ICT Virtual Human Toolkit.

The main Toolkit site can be accessed here.

Table of Contents
maxLevel2

Introduction

Welcome to the ICT Virtual Human Toolkit Website.

The ICT Virtual Human Toolkit is a collection of modules, tools and libraries that allow users, authors and developers to create their own virtual humans. This software is being developed at the University of Southern California Institute for Creative Technologies and can be licensed without cost for academic research purposes.

Download Virtual Human Toolkit

News

See Release Notes for details.

  • Jul 11 2012 - Released a minor update to the toolkit, fixing some usability and stability issues.
  • May 31 2012 - Released an exciting new version of the toolkit offering the MultiSense framework, the Rapport research platform and the SBMonitor tool. MultiSense is a perception framework that enables multiple sensing and understanding modules to interoperate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker and FAAST which you can use with webcam or Kinect. The Rapport agent is a “virtual human listener” providing nonverbal feedback based on human nonverbal and verbal input. It has been used in a variety of international studies related to establishing rapport between real and virtual humans. Finally, the SBMonitor is a stand-alone tool for easy debugging of SmartBody applications, including testing available (facial) animations, gazes and more complex BML commands.
  • Mar 2 2012 - Released a minor update of the toolkit updating the Unity version to 3.5 as well as providing incremental changes to the Unity/SmartBody debug tools in the Unity Editor (VH menu in Unity).
  • Dec 22 2011 - Happy holidays!  Released a new version of  the toolkit which includes the ability to interrupt Brad, improved support for higher resolutions, and a fix for text-to-speech not working properly.
  • Aug 10 2011 - Released a new version of the toolkit, which offers support for the free version of Unity 3D; users may now create scenes for Brad.  Download Unity here: http://www.unity3d.com.  For instructions on how to use the Unity 3D Editor with the toolkit, see the vhtoolkitUnity section. In addition, the user interaction has been improved; Unity now launches in full-screen automatically and users get visual feedback when talking to Brad. To directly talk to Brad, first make sure you have a microphone plugged in, wait for Brad to finish his introduction, and close the tips window. Now click and hold the left mouse button while asking Brad a question; release the mouse button when you are done talking. The recognized result will be displayed above Brad in white font (toggle on/off with the O key), and Brad will answer your question. It is advised to update Java and ActiveMQ, which are provided with the 3rd party installer versions.

News Archive

Toolkit Overview

Goal

The University of Southern California Institute for Creative Technologies (ICT) has created the Virtual Human Toolkit with the goal of reducing some of the complexity inherent in creating virtual humans. Our toolkit is an ever-growing collection of innovative technologies, fueled by basic research performed at the ICT and its partners. The toolkit provides a solid technical foundation and modularity that allows a relatively easy way of mixing and matching toolkit technology with a research project's proprietary or 3rd-party software. Through this toolkit, ICT hopes to provide the virtual humans research community with a widely accepted platform on which new technologies can be built.

What is it

The ICT Virtual Human Toolkit is a collection of modules, tools and libraries that supports the creation of virtual human conversational characters. At the core of the toolkit lies innovative, research-driven technologies which are combined with other software components in order to provide a complete embodied conversational agent. Since all ICT virtual human software is built on top of a common framework, as part of a modular architecture, researchers using the toolkit can do any of the following:

  • utilize the toolkit and all of its components as is;
  • utilize certain components while replacing others with non-toolkit components;
  • utilize certain components in other existing systems.

The technology emphasizes natural language interaction, nonverbal behavior and visual recognition. The main modules are:

  • Nonplayer Character Editor (NPCEditor), a package for creating dialogue responses to inputs for one or more characters. It contains a text classifier based on cross-language relevance models that selects a character's response based on the user's text input, as well as an authoring interface to input and relate questions and answers, and a simple dialogue manager to control aspects of output behavior.
  • Nonverbal Behavior Generator (NVBG), a rule-based behavior planner that generates behaviors by inferring communicative functions from a surface text and selects behaviors to augment and complement the expression of those functions.
  • SmartBody (SB), a character animation platform that provides locomotion, steering, object manipulation, lip syncing, gazing and nonverbal behavior in real time, using the Behavior Markup Language.
  • MultiSense, 
  • Rapport 1.0, 
  • Watson, a real-time visual feedback recognition library for interactive interfaces that can recognize head gaze, head gestures, eye gaze and eye gestures using the images of a monocular or stereo camera.
  • AcquireSpeech (Speech Client), a tool that can send audio to one or more speech recognizers and relay the information to the rest of the system. It also allows text to be typed into the system, simulating speech input. The toolkit uses PocketSphinx as a 3rd party speech recognition solution.

The target platform for the overall toolkit is Microsoft Windows, although some components are multi-platform.

What is it not

Although the toolkit supports virtual humans development, some components are prototypes rather than state-of-the-art technologies. The Components section lists several potential alternatives for some components.

The toolkit does not contain many of the current basic research technologies at ICT, such as the reasoning SASO agents. Most of the toolkit technology, however, is the result of basic research, which is continually evaluated for potential use in future releases.

Currently, we are not at liberty to publicly distribute any project-specific data. However, interested parties are encouraged to contact us directly. In addition, we are considering creating a forum where users can share their creations.

Who can use it

The toolkit has two target audiences:

  • Users are people who use the provided technology as is, usually either running a component or using it to create new content. Users can configure and run systems and are expected to have basic computers skills and some minor scripting skills.
  • Developers are software engineers / programmers who can build and modify the code and create new capabilities for the system, either by modifying or extending existing code or by creating new modules that interface with the rest of the system. 

All toolkit software can be used without cost for academic research purposes provided all associated licenses are being honored. If you are using the toolkit or any of its components for published research, please cite us appropriately, as per clause 3 of the license. See the Papers section for more details. Please contact us if you are interested in a commercial license.

Where To Get It

Please see the Support section for instructions on how to obtain the ICT Virtual Human Toolkit. The Getting Started section below will guide you through the first steps of using the software.

License and Disclaimers

The complete License Agreement and supporting documentation can be read in the License section. The License Agreement states, but is not limited to:

  • The toolkit and any of its components can only be used for academic research and US Government purposes.
  • If you are using the toolkit or any of its components for published research, cite us appropriately. See Papers for details.
  • Toolkit users are required to honor all licenses of components and supporting software as defined in Exhibit A of the License Agreement.

Please contact us if you are interested in a commercial license.

Please be aware that the toolkit consists of research software for which documentation and support is limited. However, both the software as well as the accompanying documentation are actively being developed and updated.

Current Toolkit Users

There are many [[Projects|ICT projects]] that use a subset of the technology provided with the toolkit. Below is a list of some examples:
* [[Projects#SASO|SASO]]
* [[Projects#Virtual Patient|Virtual Patient]]
* [[Projects#Sergeant Star|Sergeant Star]]
* [[Projects#Gunslinger|Gunslinger]]
* Elect BiLat
* [http://www.mos.org/interfaces/ InterFaces Project] (with Boston Museum of Science)
* Tactical Questioning

In addition, many groups outside of ICT use some of the toolkit components, most notably [[SmartBody]] and [[Watson]]:
* University of Reykjavik
* German Research Center for Artificial Intelligence
* ArticuLab at Northwestern University
* Telecom Paris Tech
* Affective Computing Research group at MIT Media Lab
* ICSI/UCB Vision Group at UC Berkeley
* Human-Centered, Intelligent, Human-Computer Interaction group at Imperial College
* Worcester Polytechnic Institute
* Microsoft Research
* Relational Agents group at Northeastern University
* Component Analysis Lab at Carnegie Mellon University

 

Recently Updated