vhtoolkitUnity

Overview

vhToolkitUnity is the Unity3D rendered version of the Virtual Human Toolkit.  This project integrates several research systems together in order for the user to interact with a virtual human character through usage of text to speech and language understanding. Some of the systems that are integrated into his project are the Smartbody animation system, text to speech engines, and AcquireSpeech .  Unity is a 3rd party proprietary rendering engine and is provided as binary-only with the Virtual Human Toolkit. You can download the free version of Unity here. For changing art assets you need to purchase Unity Pro. vhtoolkitUnity was created using the vhAssets package as a starting point.

The Toolkit also contains an open source renderer Ogre which only serves as a proof-of-concept and is not fully supported.

Users

Launching Unity

In the Launcher, click the Launch button in the Renderer row (click the Advanced button if you don't see any rows). Be sure to have the Engine parameter set to 'Unity'. A small option screen pops up allowing you to select several graphical settings. The most important ones are:

• Screen resolution. The higher the resolution, the better the graphics look, but the slower they are.
• Graphics quality. The higher the quality, the better the graphics look, but the slower they are.
• Windowed. Check if you want the renderer to be windowed, uncheck if you want it to launch full screen.

Once you're happy with your settings, click 'Play!'. Unity should now continue loading.

Navigating Unity

You can use the W,A,S and D keys to move the camera, and Q and E to move it up and down. Hit J to toggle mouse look; this allows you to use the mouse to change the direction the camera is facing.

Interacting with Brad Using Text

Toggle the text box with L. Use the arrow keys to see suggested questions, or type in your own questions. Hit Say to ask Brad the question.

Interacting with Brad Using Speech

You can talk to Brad using a mouse and microphone when Unity is in full screen mode. Use the M key to turn this ability on and off. When the ability is on, click and hold the left mouse button while you are asking your question, and release when you're done talking.

Keyboard Commands

• W,A,S,D - camera movement. Q & E - camera up/down
• J - mouse visibility toggle - mouse look mode
• L - toggles the fake recognizer text box
• O - toggles the user's recognized text
• M - toggles speech recognition mode. When on, click and hold and talk in the mic. Release to quit talking.
• X - reset camera
• Z - show debug statistics
• I - Toggles sbm character subtitles
• P - Toggle entire GUI
• Alt-enter - toggle windowed / fullscreen
• Escape - quit

Accessing the Command Console

Use the ~ key to bring up the console.

See Using the Console

Developers

Loading VHToolkit Scene

1. Open Unity
2. File->Open Project (this may already be done for you if this is the first time you've launched Unity)
3. On the Open Project tab, click the Open Other button.
4. Navigate to the VHToolkit Unity Project here: <VHToolkit>\core\vhtoolkitUnity
5. Click Select Folder
6. In the Project column, double click on vhtoolkitScene.
7. Click Play

Creating your own Project using the VHToolkit Unity Package

1. Open Unity
2. File->New Project
3. Copy the vhAssetsPackage.unitypackage file to this new project. You'll find it here: <VHToolkit>\bin\vhtoolkitUnity\vhAssetsPackage.unitypackage. Copy this file to <your project>\Assets
4. In Unity, in the 'Project' column, double click on the vhAssetsPackage file
5. You now have a Unity project containing all of the VHToolkit code needed to create your own virtual human.

Creating a Virtual Human using the VHToolkit Unity Package (simple)

1. Open Unity
2. Copy bin/vhtoolkitUnity/vhAssetsPackage.unityPackage and bin/vhtoolkitUnity/vhAssetsTestScenePackage.unityPackage into <your project>\Assets
3. Double click vhAssetsPackage in the project view and import
4. Double click vhAssetsTestScenePackage in the project view and import
5. Open vhAssetsTextScene

Creating a Virtual Human using the VHToolkit Unity Package (detailed)

These steps assume you have virtual human art assets.

These instructions are for both Unity Pro and Unity Free. When there are specific instructions for a particular version, they have been marked as such.

There are differences because Unity Pro uses an .dll version of smartbody, and runs within the process. For Unity Free, Smartbody is launched as a separate process and communicates to Unity via network traffic.

Steps:

1. Follow the steps above to get the VHToolkit Unity Package in your project
2. Copy your art assets to your Unity Project. You can use the assets in VHToolkit if you wish. Copy these files to <your project>\Assets\Resources:
   
   1. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\characters\Brad.fbx
   2. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Face_2048.jpg
   3. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Hand.jpg
   4. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Pants.jpg
   5. <VHToolkit>\core\vhtoolkitUnity\Assets\Resources\art\textures\Brad_Shirt.tif
3. You now have to match up the Materials with the textures:
   
   1. In Unity, in the Project pane, click to open the Resources folder, then click to open the Materials folder.
   2. Click on Brad-Brad_SkinRig_Brad_Hands
   3. In the Inspector, in the Texture2D square, click the Select.
   4. In the dialog that pops up, select the Brad_Hand texture
   5. Repeat these steps for the Head, Pants, and Shirt materials.
   6. To test to see if the character is imported correctly, select Brad in the Project column, and drag it to the Scene view. You should see the character, fully textured. Click delete to remove him from the scene when done.
4. You now have to create a Prefab for the character.
   
   1. In the Project column, click Create -> Prefab
   2. Drag 'New Prefab' to the Resources folder
   3. Click on 'New Prefab' to rename it to 'BradPrefab'
   4. Drag 'Brad' onto 'BradPrefab' to create a prefab of Brad
   5. Still in the Project column, open up vhAssets\sbm
   6. Drag UnitySmartbodyCharacter to BradPrefab
   7. Create a SoundNode for the character:
      
      1. Go to GameObject->Create Empty
      2. In the Hierarchy, rename GameObject to SoundNode
      3. With SoundNode selected, go to Component->Audio->Audio Source
      4. In the Project column, drag BradPrefab to the Hierarchy
      5. Drag SoundNode to BradPrefab in the Hierarchy. A dialog will pop up saying that this will lose the prefab connection. Click Continue.
      6. In the Hierarchy, drag BradPrefab onto the BradPrefab in the Project column.
      7. Remove BradPrefab from Hierarchy
      8. <todo> <set the position of the SoundNode to the character's mouth>
   8. Select BradPrefab. In the Inspector, change 'Bone Parent Name' to CharacterRoot1/AccumulationRoot since that's how the hierarchy is set up in our current character. That points to the 'top' of the skeleton hierarchy.
   9. Still in the Inspector, select the checkbox 'Is Face Bone Driven'. Turn that on.
5. Now you need to add the components to the scene.
   
   1. In the Project column, open up vhAssets\Resources
   2. Drag VHBehaviourManager to the Hierarchy column
   3. Drag DebugConsole to the Hierarchy column https://jira.ict.usc.edu/browse/VH-232
   4. Drag Fps Counter to the Hierarchy column https://jira.ict.usc.edu/browse/VH-231
   5. Drag FreeMouseLookCamera to the Hierarchy column. In the Hierarchy column, select 'Main Camera' and delete it.
   6. Drag VHMsgManager to the Hierarchy column
   7. (Unity Pro) Drag SmartbodyManager to the Hierarchy column.
   8. (Unity Free) Drag SmartbodyManagerBonebus to the Hierarchy column.
6. Now you need to configure and initialize SmartbodyManager
   
   1. (Unity Pro) In the Hierarchy, select SmartbodyManager
   2. (Unity Free) - In the Hierarchy, select SmartbodyManagerBonebus.
   3. Change Position Scale to 0.01 since Brad's data is in centimeters. This is unnecessary if all your data is in meters.
   4. (Unity Pro) In the Project column, go to Create C Sharp script
   5. (Unity Pro) Select NewBehaviourScript and click Edit in the Inspector
   6. (Unity Pro) Add the following code to the Start() function:

`

static bool once = false;

SmartbodyManager.Get().SetMediaPath("Assets"); // https://jira.ict.usc.edu/browse/VH-235

FreeMouseLook camera = (FreeMouseLook)Component.FindObjectOfType(typeof(FreeMouseLook));

`

1. 1. (Unity Pro) Go to GameObject->Create Empty.
   2. (Unity Pro) Drag NewBehaviourScript to GameObject in the Hierarchy
   3. (Unity Pro) Copy sbm-common folder to get Smartbody assets. Copy <VHToolkit>\data\sbm-common to <your project>\Assets\sbm-common
   4. <todo> Add simple lighting
2. Now we are ready to test
   1. In Unity, hit Play button
   2. (Unity Free) Launch Smartbody as a separate process. Run <VHToolkit>\tools\launch-scripts\run-toolkit-sbm.bat
   3. Brad should appear and animating with his idle animation.
   4. Try and play an animation
   5. Open the Unity debug console by hitting tilde (~)
   6. Type the following command: vhmsg sbm bml char brad anim HandsAtSide_Arms_Chop

Using the Console

Hit the ~ key to bring up the console. Here you can see some debug information. Type 'help' for all available commands.

Overview of the main commands:

• q - quit
• play_intro - Play intro sequence
• vhmsg ... - Send vhmsg out to system. 'vhmsg sbm ...' for sending a smartbody command
• setresolution x y - Set resolution to 'x' x 'y'. Example 'setresolution 1024 768'
• toggle_fullscreen - toggle windowed / fullscreen

Message API

Receives:

• vrAllCall
• vrKillComponent
• PlaySound

Sends:

• vrComponent
• vrProcEnd

Known Issues

• ...

FAQ

See Main FAQ for frequently asked questions regarding the installer. Please use the Google Groups emailing list for unlisted questions.