Please see the Support section on how to obtain the toolkit.
Please see the tutorial on how to install the toolkit.
Please see the tutorial on how to run the example.
Some components are open source, while others are binary only. We might release the source code of these in the future, though. SmartBody has its own SourceForge page which you can access here.
The toolkit as a whole is targeted for the Microsoft Windows platform, including XP and Windows 7, both 32 and 64 bit. Some components are multi-platform, most notably the NPCEditor which drives the characters, and the Launcher and Logger. Some other components could be ported to support other platforms, like SmartBody and Ogre.
With the toolkit we are aiming to provide a technical platform allowing researchers to more quickly create their own virtual humans. Whether project specific agents will be made available to the public will be determined on a case by case basis, but will usually fall outside the scope of the toolkit.
Not all 3rd party software come with their own installers. Some might just copy over files to the correct location.
If you installed the MS SAPI voices and you are running Windows XP 64 bit, some changes in the registry are required to be able to use the installed voices. You can do this by double clicking "Microsoft-Voices.reg" in "C:\vhtoolkit\scripts". You can check to see if the changes worked by going to the Speech settings in Windows (Start -> Control Panel -> Speech). Under "Voice Selection" you should have several options, including "Microsoft Mike".
It might be that the executable used to start the Launcher can not be found. Please make sure that javaw.exe is part of your path. The location is usually "C:\Program Files\Java\jre1.6.0_10\bin" or "C:\Program Files (x86)\Java\jre1.6.0_10\bin". The path is defined in your Environment Variables; right-click on My Computer, go to the Advanced tab and click Environment Variables.
Orange means that the Launcher has tried to launch a component, but hasn't gotten a confirmation message back indicating it is up and running. This is likely due to not running the ActiveMQ service. Please make sure it is running (Right click My Computer, select Manage, Services and Applications, Services). If it is not running, turn it on (the Play button at the top). If it is not present, please install using the toolkit installer with 3rd party software.
Is SmartBody running? You can see this by looking at the SmartBody row in the Launcher. If it's green, it is running, if it is orange, its status is unknown (usually, the Launcher tried running the component, but did not get confirmation it is actually up and running), and if the color is neutral it means it is not running. Make sure the renderer is up and running before you start SmartBody. Please also make sure the renderer is completely up and running before you start SmartBody. If you use the Run All button, you might have to kill SmartBody and launch it manually.
See if Brad is moving his lips. If so, make sure your volume is not all the way down, that it is not muted, and that you know where sound should come from (i.e., desktop speakers, or a head phone that has just been plugged in). If Brad is not moving his lips, make sure the Text To Speech module is running. If it is, look in its console window for any error messages that might give you an idea of what is going wrong. The TTS executable is called "MsSpeechRelay.exe".
It might not have the correct answer for you, especially if you have tried asking the same question a couple of times. There might also be a delay in some of the modules. Try waiting a minute after trying again. If there is no response at all, make sure all modules are still green in the Launcher. If a module that is checked is neutral, try launching it. If one of the checked rows is orange, the module might have crashed. Find the associated window or process and kill it manually, then restart that module. If all checked rows are green, try to restart SmartBody, the NVBG and the NPCEditor. If this doesn't help, kill all modules and restart them. As a last resort, try rebooting your computer and start all modules.
If you are running Gamebryo, by default dynamic shadows are turned on. Modern machines shouldn't have any problems with that, but machines with older video cards, or laptops with integrated video cards, might have difficulty performing at a decent framerate. We are currently working on reducing the needed power to render the scene. In the mean time, you can disable dynamic shadows with the '-' (minus) key.
There could be a number or reasons why this is happening:
If you have installed previous versions of the toolkit and used it before on your system, then the NPCEditor would most likely have created user setting files on your machine. Once in a while, a change from our side to NPCEditor's settings will need to be propogated to the users, which will require them to delete these user files on their machines so that they don't interfere/override the new settings.
The NPCEditor creates .plist files which it stores in the user directory for Windows. These files contain NPCEditor specific information.
So if you need to delete these user settings from your machine, follow the below steps.
Depending on which OS you are using, these files will be present in the mentioned folders
Windows 7:
If you are using Windows 7, then you should look under
Your Primary Drive:\Users_Your user name_\AppData\Roaming\NPCEditor\people
You should see files with the extension ".plist". These are the local files created by the NPCEditor on your machine to save user settings. Delete these files. If these files don't exist, then it shouldn't matter.
Windows XP:
Under Windows XP, These files should be present under
Your Primary Drive:\Documents and Settings_Your user name_\Application Data\NPCEditor\people
These are the local files created by the NPCEditor on your machine to save user settings. Delete these files. If these files don't exist, then it shouldn't matter.
Once you delete these files, then the NPCEditor will, by default, use the .plist files that the installer has copied within the toolkit folder, with the new settings.
Note: The AppData/ApplicationData folder under both OS's are hidden folders. So you will not be able to see them unless you make sure that hidden folders are visible in Windows Explorer. In order to do this, open any Explorer window and in the dropdown menu on top, go to 'Tools->Folder Options'. Go to the 'View' tab. Under 'Advanced settings', check the button that says "Show hidden files, folders, and drives".
This will display hidden folders in Windows Explorer.
Use the ~ key to bring up the console.
Commands:
Ogre is only provided with the toolkit to show a source code proof of concept integration with SmartBody. It is a very bare bones implementation. The main renderer for the toolkt is Unity.
You can use the A, S, W and D keys to move the camera. Hit the J key to toggle using mouse look (you can move the mouse to look around).
Hit the J key, or use ALT + TAB to switch to another window.
You can either use the Kill button on the Launcher, or hit the Q key on your keyboard, while in the Ogre window.
The voice ID of a character is defined in the SmartBody sequence file. For the Brad character this is 'BradVoice', as defined in \data\sbm-toolkit\scripts\init-toolkit-characters-Gamebryo.seq:
set character brad voice remote BradVoice
This line tells SmartBody that it should send a request to a TTS engine for any speech (as opposed to using prerecorded speech), and that the voice ID to be used in that request is 'BradVoice'.
There usually is a distinction between the actual TTS engine, and a small relay program that sits in between SmartBody and the TTS engine. By default, the toolkit uses the MSSpeechRelay module for that. MSSpeechRelay offers the functionality of mapping the requested voice ID to an internal voice ID. In the case of the Brad character, it maps the generic 'BradVoice' to the actual Windows voice 'Microsoft Mike', as defined in \tools\launch-scripts\run-toolkit-tts.bat:
start msspeechrelay -m BradVoice "Microsoft Mike"
By default, MSSpeechRelay will try to use the defined voice, but it will fall back to the Windows default voice if the former can not be found.
If you want to change the voice, you can map 'BradVoice' to a different, already installed voice in 'run-toolkit-tts.bat'. Alternatively, you can map the voice to an unknown voice with the result that the default Windows voice will be used. You can change the default Windows voice by going to the Windows Control Panel (click Start) and selecting Speech. In the Text To Speech tab, select a voice. You can only select voices that you have installed.
See the components section for an overview of voices that are compatible with the toolkit.
You can push the Start button on the first tab. This will bring the user to the Recorder tab and will start a new session, in which all speech input will be saved on disk. The toolkit only comes with PocketSphinx, which has not been optimized yet for use with the toolkit.
The AcquireSpeech client starts on the first tab in which you can modify some of the default settings. The Recorder and Player tabs allow users to type in their utterances. The Player tab also loads a script with some example questions which can be used by double clicking on it.
In the Advanced menu, click Show Console option (CTRL + T).
In the Advanced menu, disable Enable vrAllCall (CTRL + SHIFT + T).
Unsaved changes are automatically saved to a Last Known Config profile, which will be loaded by default on start up. If you want to save changes under a specific name, for instance to save several often used configurations, you can do that in the Profiles menu, Save As New Profile (CTRL + SHIFT + S). This will bring up a new window in which you can enter your profile name.
If supported, right click on any component name and select the appropriate Switch To option. Note that this feature assumes a symmetrical installation of both a build version and a SVN (version control) version of your project. This is currently not supported by the toolkit.
The Logger window is minimized on start-up. On the task bar, you can usually find it next to the Launcher.
By default, the Launcher sends out status request messages to all modules and tools (vrAllCall). These then respond with a confirmation message, indicating they are online. This enables the Launcher to show which components are still online. You can turn this behavior off in the Advanced menu of the Launcher (Enable vrAllCall option). Note that the Launcher will then not correctly indicate component status anymore. For more information on this particular message protocol, see Messages.
It might be that the executable used to start the Launcher can not be found. Please make sure that javaw.exe is part of your path. The location is usually "C:\Program Files\Java\jre1.6.0_10\bin" or "C:\Program Files (x86)\Java\jre1.6.0_10\bin". The path is defined in your Environment Variables; right-click on My Computer, go to the Advanced tab and click Environment Variables.
We have noticed this as problem on a nVidia GeForce 7300 GT with dynamic shadows enabled. It is likely due to a driver problem. Unfortunately, we cannot support individual graphics cards at the moment. You can disable dynamic shadows (the '-', or minus key), or upgrade to a new video card.
There is a problem using Gamebryo with the Apple Cinema Display USB hub. This matter is being investigated further. For now, please allow plenty of time for Gamebryo to load, or pull out the USB cable going from the monitor to the computer.
The toolkit should be referred to as either 'Virtual Human Toolkit' or 'toolkit'. Please do not use 'VH toolkit', 'Virtual Humans Toolkit' (notice the plural humans), 'vhtoolkit' or 'VHT'. If you want to use some abbreviation, just use 'toolkit'. 'vhtoolkit' is exclusively used as an abbreviation in code, URL's, email addresses, etc., not in documentation. For more terms see the Glossary below.
General purpose messaging system, used by the toolkit components.
[AcquireSpeech, one of the toolkit modules. Can also be used for typing in questions or for selecting utterances from a predefined script.
Creates characters.
A component is either a module, tool, or library that's part of the toolkit. It is used to refer to a general toolkit element, rather than more specific elements.
Modifies the toolkit technology in some way, or uses the technology in other systems.
See Institute for Creative Technologies.
Plain awesomeness.
A software component of the toolkit. Support modules and/or tools and is usually hidden from users and authors.
A runtime toolkit component that is part of the toolkit architecture. Modules are essential for running a virtual character, as opposed to tools or libraries which just play a support role.
See Nonverbal Behavior Generator.
Generates nonverbal behavior, based on textual input. One of the toolkit modules. Often referred to with acronym NVBG. Developed at the Institute for Creative Technologies.
Statistical text classifier that takes text input and selects text output. One of the toolkit modules. Serves as a character's brain and source for natural language input and output. Developed at the Institute for Creative Technologies.
Data file used by the NPCEditor. Although a general XML format, in the toolkit the plist is often used when referring to a character's knowledge.
Stands for SmartBody Module. See SmartBody.
Procedural animation system. One of the toolkit modules. Sometimes referred to as SBM. Developed at the Institute for Creative Technologies.
Module that generates audio based on textual input. Provides a character's voice.
A toolkit component that allows users, authors and developers supportive functions.
See Text-To-Speech.
See University of Southern California.
University with which the Institute for Creative Technologies is affiliated.
Runs the toolkit.
Usually an acronym used to refer to the ICT Virtual Humans group.
Collection of modules, tools and libraries that allow users to create their own virtual humans. This software is being developed at the University of Southern California Institute for Creative Technologies and is freely available for the research community. Sometimes referred to as VHT in text, always at 'vhtoolkit' in code, urls, etc.
Real-time visual feedback recognition library for interactive interfaces that can recognize head gaze, head gestures, eye gaze and eye gestures using the images of a monocular or stereo camera. One of the toolkit modules.