Page History

Overview

The Non-Verbal-Behavior-Generator (NVBG), as the name suggests, is the module that generates behavior other than speech, such as gestures, facial expressions and gazes. The NVBG generates the afore mentioned behavior based on the speech input and other messages that it receives. These behaviors serve to augment and emphasize spoken communication. Based on who speaks and who listens, the NVBG can characterize the NPC as a listener, speaker or a bystander and generate appropriate behavior. These behaviors are configurable using xml files provided as input to NVBG. Using these xml files, we can specify which words or parts-of-speech trigger which animations. We can specify idle animations and idle gazes that get triggered when the character is idle for a specified amount of time.

...

The NVBG also allows us to configure facial expressions for a particular character using FACS units and then based on xml tags in the input messages, can trigger those facial animations.

Quick facts

• The NVBG is located at https://svn.ict.usc.edu/svn_vh/trunk/core/NVBG
• Main Developer: Shridhar Ravikumar

Snapshot of NVBG

       

Users

Currently the NVBG needs to be setup on a per character basis i.e. each character in the scene needs to have his/her own process of NVBG.The character itself needs to be configured using the various confuguration options that NVBG provides.

...

• The rule_input_[culture].xml file, that specifies which rules should be generated for parts of speech in the spoken sentence and also which animations map to which words if any. Assuming you have the rule-input-[char-name].xml file you need to pass it as an argument to NVBG when you run the executable so that it knows which file to use for rules.
• The .xslt transform files which process the the intermediate xml generated by the module and generate the final bml to be output. The defaults are available at http://svn.ict.usc.edu/svn_vh/trunk/data/nvbg-common/. The relevant files are NVBG_transform.xsl, NVBG_rules.xsl and NVBG_behavior_description.xsl
• The command line parameters to NVBG including the character-name, character-culture, idle posture and some other optional parameters
• OPTIONAL - we can also specify a saliency map and a facial-expressions configuration if needed. 
  
  1. The saliency map specifies which objects in the scene are of particular importance which allows NVBG to generate idle gazes appropriately.
  2. The facial-expressions configuration file allows us to specify which facs units should be triggered with a certain weight so that a facial expression is achieved. This facial expression can later be tagged in the input text as markup to trigger the facial expression.

Command line parameters

Required:

-agent_name: specifies the agent name that the current instance of NVBG corresponds to.

...

On providing the required command line arguments, it should work with the other VH components and in accordance with the rules specified for it.

Input Messages

NVBG mainly subscribes to vrExpress messages and a few other control messages which allow for setting some options. Below is a list of messages that NVBG subscribes to:

vrExpress

This message is sent to NVBG by the NLU, NPCEditor or similar module. This message can be used to convey information about speech data, posture, status change, emotion change, gaze data etc. as shown below.

Speech

The speech messages are characterized by the speech tag within them. They are interpreted and the corresponding output bml is generated with the speech time marks, animations, head-nods, facial-movements etc. These animations are generated based on the content of the speech tag and the fml tag in the input message.

 vrExpress "harmony" "ranger" "harmony221" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="harmony" role="actor"/>
<fml>
<intention>
<object name="A316">
<attribute name="addressee">ranger</attribute>
<attribute name="speech-act">
<object name="A317">
<attribute name="content">
<object name="V28">
<attribute name="modality">
<object name="V29">
<attribute name="conditional">should</attribute>
</object>
</attribute>
<attribute name="polarity">negative</attribute>
<attribute name="attribute">jobAttribute</attribute>
<attribute name="value">bartender-job</attribute>
<attribute name="object-id">utah</attribute>
<attribute name="type">state</attribute>
<attribute name="time">present</attribute>
</object>
</attribute>
<attribute name="motivation">
<object name="V27">
<attribute name="reason">become-sheriff-harmony</attribute>
<attribute name="goal">address-problem</attribute>
</object>
</attribute>
<attribute name="addressee">ranger</attribute>
<attribute name="action">assert</attribute>
<attribute name="actor">harmony</attribute>
</object>
</attribute>
</object>
</intention>
</fml>
<bml>
<speech id="sp1" type="application/ssml+xml">ranger utah cant be bartender if he becomes sheriff</speech>
</bml>
</act>"

vrExpress "utah" "ranger" "utah200" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="utah" role="actor"/>
<fml>
<intention>
<object name="A131">
<attribute name="addressee">ranger</attribute>
<attribute name="speech-act">
<object name="A133">
<attribute name="reason">become-sheriff</attribute>
<attribute name="content">
<object name="V6">
<attribute name="content">
<object name="V7">
<attribute name="q-slot">polarity</attribute>
<attribute name="type">question</attribute>
<attribute name="prop">
<object name="P24">
<attribute name="type">event</attribute>
<attribute name="location">town</attribute>
<attribute name="theme">sheriff-job</attribute>
<attribute name="event">providePublicServices</attribute>
<attribute name="agent">utah</attribute>
<attribute name="time">future</attribute>
</object>
</attribute>
</object>
</attribute>
<attribute name="type">csa</attribute>
<attribute name="action">info-req</attribute>
<attribute name="actor">ranger</attribute>
<attribute name="addressee">utah</attribute>
</object>
</attribute>
<attribute name="motivation">
<object name="V5">
<attribute name="reason">become-sheriff</attribute>
<attribute name="goal">avoid</attribute>
</object>
</attribute>
<attribute name="action">avoid</attribute>
<attribute name="type">backward</attribute>
<attribute name="addressee">ranger</attribute>
<attribute name="actor">utah</attribute>
</object>
</attribute>
</object>
</intention>
</fml>
<bml>
<speech id="sp1" type="application/ssml+xml">im not sure about that</speech>
</bml>
</act>"

Posture change

These messages are characterized by the <body posture=""> tag which allows NVBG to know that there has been a change in posture.

vrExpress "harmony" "None" "??" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="harmony" role="actor" />
<bml>
<body posture="HandsAtSide" />
</bml>
</act>"

Status / request

The idle_behavior and all_behavior attributes within the request tag allows NVBG to keep track of whether or not to generate the corresponding behavior.

vrExpress "harmony" "None" "??" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="harmony" role="actor" />
<fml>
<status type="present" />
<request type="idle_behavior" value="off" />
</fml>
</act>"

Gaze

These gaze tags, if present within the input message are transferred unaltered to the output message.

vrExpress "harmony" "ranger" "constant103" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="harmony" role="actor" />
<fml>
<gaze type="weak-focus" target="ranger" track="1" speed="normal" > "listen_to_speaker" </gaze>
</fml>
</act>"

vrExpress "utah" "ranger" "gaze199" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="utah" role="actor" />
<fml>
<gaze type="avert" target="ranger" track="eyes-offset" speed="slower" > "planning_speech_hold_turn" </gaze>
</fml>
</act>"

Emotion

The affect tag contains data about the emotional state the character is currently in. This can be used to affect output behavior.

vrExpress "harmony" "None" "schererharmony17" "<?xml version="1.0" encoding="UTF-8" standalone="no" ?><act>
<participant id="harmony" role="actor" />
<fml>
<affect type="Fear" STANCE="LEAKED" intensity="110.475"></affect>
</fml>
<bml> </bml>
</act>"

nvbg_set_option

These are control messages to set options for NVBG. They are as shown below:

nvbg_set_option [char-name] all_behavior true/false - sets/unsets flag that allows all behavior generated by NVBG.

nvbg_set_option [char-name] saliency_glance true/false - sets/unsets flag that allows saliency map generated gazes. These gazes are based on content in the speech tag and the information in the saliency map.

nvbg_set_option [char-name] saliency_idle_gaze true/false - sets/unsets flag that allows idle gazes generated by the saliency map. These idle gazes are based on the priority of pawns in the saliency map and are generated when the character is idle.

nvbg_set_option [char-name] speaker_gaze true/false - sets/unsets flag that allows for the character to look at the person he's speaking to.

nvbg_set_option [char-name] speaker_gesture true/false - sets/unsets flag that allows gestures to be generated when speaking.

nvbg_set_option [char-name] listener_gaze true/false - sets/unsets flag that allows the listener to gaze at the speaker when he speaks.

nvbg_set_option [char-name] nvbg_POS_rules true/false - sets/unsets flag that allows behavior to be generated based on parts of speech returned by the parser.

Output

vrSpeak

Below is an example of the vrSpeak message that is output from NVBG. It contains animation/head-nods/facial behavior based on the text that is to be spoken/heard.

...

The "priority" attribute allows NVBG to decide which behavior is to be culled if any of the behaviors overlap. This is done before the final message is sent out.

Developers

Functional Block Diagram

Configuring Rules

The rule-input-[culture].xml file contains the rules that map certain words or parts of speech to animations. You can also specify any idle animations that you want to trigger when the character is idle for a certain period of time.

...

****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

POS Transform rules

TO BE POPULATED

Saliency map

The saliency map specifies which objects/characters/pawns in the environment are important to the character and what priority they are. These priorities can vary based on what story-point we are at i.e. the priorities can change on loading a new story-point.

...

The above example contains only one story-point but in general it can contain many. NVBG can be notified of which storypoint it should load so that the appropriate priorities are assigned to the objects in the scene.

Facial-expression file

TO BE POPULATED

Known Issues

Either list of common known issues and/or link to all Jira tickets with that component name.

Related Components

FAQ

What is NVBG?

The Nonverbal Behavior Generator (NVBG) is a tool a that automates the selection and timing of nonverbal behavior for Embodied Conversational Agents (aka Virtual Humans). It uses a rule-based approach that generates behaviors given the information about the agent's cognitive processes but also by inferring communicative functions from a surface text analysis

...