Page History

Overview

Unity3D vhToolkitUnity is the Unity rendered version of the Virtual Human Toolkit.  This project integrates several research systems together in order for the user to interact with a virtual human character through usage of text to speech and language understanding. Some of the systems that are integrated into his project are the Smartbody animation system, text to speech engines, and AcquireSpeech.  Unity is a 3rd party proprietary rendering engine and is provided as binary-only with the Virtual Human Toolkit. You can download the free version of Unity here. For changing art assets , you would need to to purchase Unity Pro. vhtoolkitUnity was created using the vhAssets package as a starting point.

The Toolkit here. The toolkit also contains an open source renderer, renderer Ogre, which  which only serves as a proof-of-concept and is not fully supported.

Users

Launching Unity

In the Launcher, click the Launch button in the Renderer row (click the Advanced button if you don't see any rows). Be sure to have the Engine parameter set to 'Unity'. A small option screen pops up allowing you to select several graphical settings. The most important ones are:

...

Once you're happy with your settings, click 'Play!'. Unity should now continue loading.

Navigating

...

Unity

You can use the W,A,S and D keys to move the camera around, and Q and E to move it up and down. Hit J to toggle mouse look; this allows you to use the mouse to change the direction the camera is facing.

Interacting with Brad Using Text

...

Toggle the text box with L. Use the arrow keys to see suggested questions, or type in your own questions. Hit Say to ask Brad the question.

Interacting with Brad Using Speech

...

You can talk to Brad using a mouse and microphone when Unity is in full screen mode. Use the M key to turn this ability on and off. When the ability is on, click and hold the left mouse button while you are asking your question, and release when you're done talking.

Keyboard Commands

• W,A,S,D - camera movement. Q & E - camera up/down
• J - mouse visibility toggle - mouse look mode
• L - toggles the fake recognizer text box
• O - toggles the user's recognized text
• M - (fullscreen window only) - toggles speech recognition mode. When on, click and hold and talk in the mic. Release to quit talking.
• X - reset camera
• Z - show debug statistics
• I - Toggles sbm character subtitles
• P - Toggle entire GUI
• Alt-enter - toggle windowed / fullscreen
• Escape - quit

Command Console

Use the ~ key to bring up the console.

See Using the Console

Developers

Load the VHToolkit Level in Unity

1. Open Unity
2. File->Open Project (this may already be done for you if this is the first time you've launched Unity)
3. On the Open Project tab, click the Open Other button.
4. Navigate to the VHToolkit Unity Project here: <VHToolkit>\core\vhtoolkitUnity
5. Click Select Folder
6. In the Project column, double click on vhtoolkitScene.
7. Click Play

Create your own Project using the VHToolkit Unity Package

1. Open Unity
2. File->New Project
3. Copy the vhAssetsPackage.unitypackage file to this new project. You'll find it here: <VHToolkit>\bin\vhtoolkitUnity\vhAssetsPackage.unitypackage. Copy this file to <your project>\Assets
4. In Unity, in the Project column, double click on the vhAssetsPackage file
5. You now have a Unity project containing all of the VHToolkit code needed to create your own virtual human.

Create a Virtual Human using the VHToolkit Unity Package

These steps assume you have virtual human art assets.

These instructions are for both Unity Pro and Unity Free. When there are specific instructions for a particular version, they have been marked as such.

There are differences because Unity Pro uses an .dll version of smartbody, and runs within the process. For Unity Free, Smartbody is launched as a separate process and communicates to Unity via network traffic.

Steps:

...

1. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\characters\Brad.fbx
2. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Face_2048.jpg
3. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Hand.jpg
4. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Pants.jpg
5. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Shirt.tif

...

1. In Unity, in the Project pane, click to open the Resources folder, then click to open the Materials folder.
2. Click on Brad-Brad_SkinRig_Brad_Hands
3. In the Inspector, in the Texture2D square, click the Select.
4. In the dialog that pops up, select the Brad_Hand texture
5. Repeat these steps for the Head, Pants, and Shirt materials.
6. To test to see if the character is imported correctly, select Brad in the Project column, and drag it to the Scene view. You should see the character, fully textured. Click delete to remove him from the scene when done.

...

1. Select the Brad object in the Project View
2. On the menu bar, click VH->Prefabs->Create Smartbody Prefab(s) from Selected

...

...

1. (Unity Pro) In the Hierarchy, select SmartbodyManager
2. (Unity Free) - In the Hierarchy, select SmartbodyManagerBonebus.
3. Change Position Scale to 0.01 since Brad's data is in centimeters. This is unnecessary if all your data is in meters.
4. (Unity Pro) In the Project column, go to Create C Sharp script
5. (Unity Pro) Select NewBehaviourScript and click Edit in the Inspector
6. (Unity Pro) Add the following code to the Start() function:

`

static bool once = false;

SmartbodyManager.Get().SetMediaPath("Assets"); // https://jira.ict.usc.edu/browse/VH-235

FreeMouseLook camera = (FreeMouseLook)Component.FindObjectOfType(typeof(FreeMouseLook));

`

1. 1. (Unity Pro) Go to GameObject->Create Empty.
   2. (Unity Pro) Drag NewBehaviourScript to GameObject in the Hierarchy
   3. (Unity Pro) Copy sbm-common folder to get Smartbody assets. Copy <VHToolkit>\data\sbm-common to <your project>\Assets\sbm-common
   4. <todo> Add simple lighting
2. Now we are ready to test
   1. In Unity, hit Play button
   2. (Unity Free) Launch Smartbody as a separate process. Run <VHToolkit>\tools\launch-scripts\run-toolkit-sbm.bat
   3. Brad should appear and animating with his idle animation.
   4. Try and play an animation
   5. Open the Unity debug console by hitting tilde (~)
   6. Type the following command: vhmsg sbm bml char brad anim HandsAtSide_Arms_Chop

Message API

Receiving:

• vrAllCall
• vrKillComponent
• PlaySound

Sending:

• vrComponent
• vrProcEnd

Using the Console

Hit the ~ key to bring up the console. Here you can see some debug information. Type 'help' for all available commands.

...

• q - quit
• play_intro - Play intro sequence
• vhmsg ... - Send vhmsg out to system. 'vhmsg sbm ...' for sending a smartbody command
• setresolution x y - Set resolution to 'x' x 'y'. Example 'setresolution 1024 768'
• toggle_fullscreen - toggle windowed / fullscreen

Message API

Receives:

• vrAllCall
• vrKillComponentMessages
• PlaySound

Sends:

• vrComponent
• vrProcEnd

Known Issues

• ...

FAQ

See Main FAQ for frequently asked questions regarding the installer. Please use the Google Groups emailing list for unlisted questions.