Overview
The Rapport 1.0 system is a set of components to investigate how to build rapport with virtual humans by using audio and visual cues from the human participant. It has been used in a few user studies at ICT.
It consists of 5 components:
- core\rapport\audio-laun - This is an audio detection app which listens to audio on the microphone, looking for key signals, like loudness, backchannel feedback, etc. Whatever it detects, it sends out vhmsgs.
- core\rapport\gesturedetection - This is an app that communicates with Watson and based on the data received, detects if there's a head nod, head shake and other features. It sends out vhmsgs for features that it detects.
- core\Watson - This is the vision system that uses a webcam or a firewire stereo webcam
- core\rapport\response - This is an app that receives the vhmsgs from 'audio-laun' and 'gesturedetection' and based on rules in the given config file, sends out vhmsgs to control the behavior of the character.
- core\rapport\soundwoz - This is a helper app that you can use to test out 'response' to make sure it's working correctly and the rules are set up properly.
Publications:
Jonathan Gratch, Ning Wang, Jillian Gerten, Edward Fast and Robin Duffy. Creating Rapport with Virtual Agents. 7th International Conference on Intelligent Virtual Agents, Paris, France 2007
Users
To run Rapport:
- Make sure you have a microphone and webcam properly set up
- From the Launcher, run:
- Rapport Audio
- Rapport Gesture Detection
- Rapport Response
- Rapport Soundwoz
- Unity Renderer
- On the soundwoz, hit the 'Start Rapport' button
- You can now interact with Brad using the webcam and microphone:
- Try speaking a sentence, then pause, Brad might nod at you.
- Try speaking a sentence, then pitch your voice up at the end, as if you're asking a question, Brad might nod again
- Try speaking loudly at Brad, he should play an animation
- Try nodding at the webcam, brad should nod back at you.
- Try gazing off to the side, left/right/up/down, Brad might gaze that direction as well.
- If you are having trouble getting the components to detect your voice or head, you can use the buttons on the SoundWoz to trigger the same events.
Developers
Currently we are only distributing the compiled binaries for the Rapport 1.0 system. But there are configuration options.
For Response:
- Three configuration files are provided to demonstrate the response capabilities:
- core\rapport\response\conf\rapport.rsp (default)
- This configuration file is a demonstration of the capabilities. Most audio/visual cues have some sort of response from the virtual character
- core\rapport\response\conf\rapport_responsive.rsp
- This configuration file most closely matches the configuration file used in the rapport user studies. Character does head nods and posture shifts based on the audio/visual cues.
- core\rapport\response\conf\rapport_nonresponsive.rsp
- This configuration file has all behavior turned off (by setting all probabilities to 0). A configuration like this was used in the rapport user studies.
Known Issues
- vrProcEnd messages do not get sent out on exit. This causes components to remain orange in the launcher