The Character Customizer is a tool that allows for the quick setup of a single-character scene and a set of lines for the character to act out. It offers control over camera angles, backgrounds, voices and facial animation. The goal is to be able to render out movies of the character. Because of this special purpose we opted to create a separate tool. Depending on user feedback we may eventually fold in some of this functionality into VHBuilder.
The Toolkit is a collection of various components, so there is some manual work required to set everything up for creating movies using the Character Customizer. In general you will need to have the following elements up and running:
Unity (the renderer / game engine)
Character Customizer tool
Supporting VH modules
Instructions for setting up Unity:
Run the Launcher by double clicking 'run-launcher.bat' in the root of your installation directly ('c:\vhtoolkit' by default).
Click on the Advanced button in the lower right corner. This will expand the view to show all components.
In the Renderer row, change some of the start-up parameters:Launch Unity by clicking on the button with the green up arrow icon. Unity may take a while to load and will initially take over the screen before going back to window mode.
Once Unity comes up, select the window and hit the C key on your keyboard to bring up the debug menu (you may need to click away the Tips window, depending on your earlier start-up settings). In the debug menu, select 'Load Customizer'. This will load the scene specifically made for the Character Customizer.
Instructions for setting up the Character Customizer:
In the Launcher, click on the button with the green up arrow in the Character Customizer row in the Tools section.
Instructions for setting up the supporting VH modules:
In the Character Customizer window, go to the System menu and select Launch All.
The Character Customizer uses a Unity movie plug-in which in turn uses the video codecs present on your machine. You can try out different ones by selecting them from the Codecs int the Options menu of the Character Customizer. MJPEG may be available and is pretty good. Alternatively, you can install other codecs. The Lagarith video codec is a free lossless codec with good performance. It does produce quite large files, which the Character Customizer automatically converts to a more manageable format. You can find this codec here http://lags.leetcode.net/codec.html
Once you have the codec installed, you can select the codec from the Character Customizer by going to Options->Codec in the menu. Your movies will now be recorded using this codec. If a specified codec cannot be found, uncompressed video will be produced. (note that you can see a warning of this in the Unity console window when you select a non-existing codec; hit the ~ key in the Unity window to bring up the console).
For more detailed information on how to use the recording plugin for Unity, please refer to http://www.renderheads.com/portfolio/UnityAVProMovieCapture/docs/UnityAVProMovieCapture.pdf
Most settings are saved per utterance. You can select multiple utterances and change the setting for all selected ones at once. You can select multiple lines by:
Holding the mouse button and drag up and down over the lines.
Adding individual lines to your selection by holding the CTRL button and clicking lines with the mouse.
Adding a range of lines by selecting the first line with the mouse button, then select the last line while holding the Shift key.
You can select all lines by clicking a single line and then use CTRL + A.
You can add new lines by typing them into the text box on the utterance list and hitting enter.
For each line you there are three ways of getting the character to speak the line:
Use text-to-speech. This is the default and you don't need to do anything for this. You can however change the voice from the Voice pull down menu in the Settings tab, and clicking the Set button
Record your own voice. Select the appropriate line. Hold down the Record button while speaking the line. Make sure to:
Either double click the line or click the Play button in the Line panel.
There are several ways to change the background:
Select a pre-existing option from the Backdrop pull down menu at the top and click Set
Click Browse to choose one of your own pictures; these ideally are very wide in order to fill the entire backdrop without too much stretching; the aspect ratio is roughly 5:2
Click Color to open a color picker; this adds a hue to the existing picture. You can for instance use the existing Gradient01 background and select any color you want. Note that the default color selections are very bold; it is advised to click Define Custom Colors in the picker to select a more subtle one. This hue will also work on photo's.
You can select a pre-defined camera from the Camera list in the Settings pane and click Set. Note that you can also directly move the camera around in Unity with the W, A, S & D keys. Any changes won't be saved, however.
By default the character looks at the Medium Center camera. To change the target, select one from the Gaze Target pull down menu and click Set. Alternatively, you can turn on Always Gaze At Camera in the Options menu.
The Nonverbal Behavior Generator analyses the text a character says and dynamialy generates a behavior schedule for SmartBody to execute. You can turn off individual behaviors with the check boxes in the Settings pane. Furthermore, you can toggle whether the character will dislay ilde fidget behaviors (looking around, stretching. etc.) in the Options menu.
Make sure you have good codec installed first, like Lagarith, see above. After that, you can hit the Rec button in the Video panel at any time. Clicking it the first time will start the recording, clicking it a second time will stop the recording. These recorded movies by default are very large, so by default they are converted to a more manageable MP4 format. You can turn this behavior on and off in the Options menu.
You can also render all lines in a row as a batch from the Options menu. Note that this may take some time and that the subtitles may not fully line up. Also make sure that the first line is a text-to-speech line, as otherwise a bug will result in the same audio clip being used for all lines.
Currently this plugin only supports recording audio from a single Windows audio device. There is a “trick” you can use though. In Windows 7 (and perhaps Vista) you may be able to set your microphone to play through the speakers by going to: Control Panel > Sound > Recording > Select your microphone > right click >Properties > Listen > check "Listen to this device". You should then hear your microphone recording through your speakers. It’s recommended to use headphones during recording to prevent feedback from the speakers into the microphone.
In order for you to be able to see the 'Listen to this device' option, please make sure that you have installed the correct drivers for your soundcard.
renderer destroy Brad
renderer destroy Rachel
renderer create Brad Brad (on creating Brad)
renderer background file Gradient01_UscShield.png
Change Background Color
renderer color 0 255 255
Add audio file
vhtspeechrecorder wizard_text Hello, I am a virtual human.
vhtspeechrecorder processaudiofile C:\Build4272-08-06-2013-ci\vhtoolkit\core\vhtoolkitUnity\Assets\StreamingAssets\Sounds\brad_askmeabout.wav Hello, I am a virtual human.
VHBuilder audiofile_created utterance_20130906121840_Iamavirtualhuman.
nvbg_set_option disable_nvbg false
Flow of messages sent in order to record speech and generate audio file and bml:
120528396 acquireSpeech startSession
120528396 acquireSpeech set mic_utteranceFile.enabled true
120528396 nvbg_set_option disable_nvbg true
120528396 render_text_overlay disable
120528494 vhtspeechrecorder wizard_text I am having a great time
120528495 acquireSpeech startUtterance mic
120528503 vrSpeech start user0001 user
120528503 acquireSpeech startedListening mic 20130906115616,033 user0001 1378494328500
120528522 vrSpeech partial user0001 1 1.0 normal
120528665 vrSpeech partial user0001 2 1.0 normal WHAT
120528799 TO_ASR <start id="temp">
120528799 TO_ASR <stop id="temp">
120528800 TO_ASR <start id="temp">
120528800 TO_ASR <stop id="temp">
120528808 TO_ASR <start id="temp">
120528859 vrSpeech partial user0001 3 1.0 normal WHAT
120529103 vrSpeech partial user0001 4 1.0 normal DON'T
120529301 vrSpeech partial user0001 5 1.0 normal DON'T WHAT
120529496 vrSpeech partial user0001 6 1.0 normal DON'T WHAT IS
120529691 vrSpeech partial user0001 7 1.0 normal DON'T WHAT IS THE
120529706 acquireSpeech stopUtterance mic
120529707 acquireSpeech stopSession
120529707 acquireSpeech set mic_utteranceFile.enabled false
120529710 vrSpeech finished-speaking user0001
120529710 acquireSpeech stoppedListening mic 20130906115616,033 user0001 1378494329709
120529711 acquireSpeech stoppedSession null 20130906115616,033 1378494329711
120529885 vrSpeech partial user0001 8 1.0 normal DON'T WHAT IS THE
120529983 vrSpeech partial user0001 9 1.0 normal DON'T WHAT IS THE
120529995 vrSpeech interp user0001 1 1.0 normal DON'T WHAT IS THE
120529995 vrSpeech asr-complete user0001
120530048 TO_ASR <stop id="temp">
120535424 VHBuilder audiofile_created utterance_20130906120529_DON'TWHATISTHE
120535427 vhtspeechrecorder stop
120535427 render_text_overlay enable
120535427 NPCEditor <script target="user">document.getModel().getAnswers().getUtterances().get(document.getModel().getAnswers().findIndexOfWithID("utterance_20130206114108_IAMHAVINGAGREATTO")).setID("utterance_20130906120529_DON'TWHATISTHE");</script>
120535428 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/ExampleScenario_dialog.plist").toURI().toURL();document.saveToURLOfTypeForSaveOperation(url, document.getApplication().fileTypeForURL(url), com.leuski.af.Document.SaveOperation.kSaveAs);</script>
120535428 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/QNA.xml").toURI().toURL();document.writeToURL(url, new edu.usc.ict.npc.editor.io.ClassifierJAXBFileType());</script>
120535575 nvbg_set_option disable_nvbg false
120535575 acquireSpeech startSession
120535585 acquireSpeech startedSession null 20130906120535,575 1378494335585
120535675 acquireSpeech startUtterance mic
120535679 vrSpeech start user0002 user
120535679 acquireSpeech startedListening mic 20130906120535,575 user0002 1378494335678
120535717 vrSpeech partial user0002 1 1.0 normal
vrExpress Brad all 1623190679 <?xml version="1.0" encoding="UTF-8" standalone="no" ?><act><participant id="Brad" role="actor" /><fml><turn start="take" end="give" /><affect type="neutral" target="addressee"></affect><culture type="neutral"></culture><personality type="neutral"></personality></fml><bml><speech id="sp1" ref="utterance_20130906121840_Iamavirtualhuman." type="application/ssml+xml">Hello, I am a virtual human.</speech></bml></act>
renderer setcamera set Camera01_mediumCt
Gaze at camera
sbm python scene.getCharacter("Brad").setVoiceBackupCode("Festival_voice_cmu_us_jmk_arctic_clunits")
Set Facial Expression
sbm char * viseme au_1_left 0 0
sbm char * viseme au_1_right 0 0
sbm char * viseme au_2_left 0 0
sbm char * viseme au_2_right 0 0
sbm char * viseme au_4_left 0 0
sbm char * viseme au_4_right 0 0
sbm char * viseme au_5 0 0
sbm char * viseme au_6 0 0
sbm char * viseme au_7 0 0
sbm char * viseme au_10 0 0
sbm char * viseme au_12_left 0 0
sbm char * viseme au_12_right 0 0
sbm char * viseme au_26 0 0
sbm char * viseme au_112 0 0
sbm char * viseme au_130 0 0
sbm char * viseme au_124 0 0
sbm char * viseme au_129 0 0
sbm char * viseme au_136 0 0
sbm char * viseme au_103 0 0
sbm char * viseme au_102 0 0
sbm char * viseme au_101 0 0
sbm char * viseme au_100 0 0
sbm char * viseme au_126 0 0
sbm char * viseme au_127 0 0
sbm char * viseme au_131 0 0
sbm char * viseme au_132 0 0
sbm char * viseme au_133 0 0
sbm char * viseme au_134 0 0
sbm char * viseme au_112 1 0.2
Generate Gestures checkbox
nvbg_set_option Brad speaker_gesture false
nvbg_set_option Brad speaker_gesture true
Generate Facial expressions/head movement checkbox
nvbg_set_option Brad nvbg_POS_rules false
nvbg_set_option Brad nvbg_POS_rules true
sbm bml char Brad <saccade finish="true"/>
sbm bml char Brad <saccade mode="listen"/>
sbm bml char Brad <saccade mode="talk"/>
sbm bml char Brad <saccade mode="think"/>