[sldev] Gesture client code

Philippe Bossut (Merov Linden) merov at lindenlab.com
Tue May 26 00:16:23 PDT 2009


Hi all,

Gee! This is quite a thread! Thanks for all the suggestions, ideas and  
passion. As always, this community shows it's full of talented (and  
opiniated) people :)

Since I'll be the guy tagged with driving/mentoring the intern on this  
project on a day to day basis, I'm taking on summarizing and answering  
some of the questions that surfaced in that thread (vaguely in  
chronological order):

- Why OpenCV? As pointed out by folks, it may look like overkill for  
the goals currently stated but those are goals *a minima*. We sure  
hope to expand from this first experimentation since, it is indeed an  
experimental project. Jan Ciger confirmed though that, if the goal is  
to capture a wide set of face and body elements, OpenCV is a good  
starting point.

- Emulate mouse/joystick driver? Several folks mentioned implicitly  
the parallel between capturing elements and emulating yet another kind  
of input device. The problem with mice or joysticks is that they have  
only one or few effectors and that those do not map on avatars  
controls. With a camera, we certainly want more than few effectors.  
Eventually, we'll rapidly grow to the point we want VR like richness  
for that input. VRPN in that respect is an interesting idea (thanks  
Jan again). Besides, we're not currently trying to "drive" the  
interaction (i.e. building prims or triggering camera movements and  
avatar navigation) but rather *add* to the currently rather canned  
(and unused) avatar animations (aka gestures) so to see how much of  
the RL emotional state we can share with other residents. IOW, we want  
to improve and add to the current experience, not replace or use as an  
alternative to keyboard, mouse, etc...

- Gaze tracking: Often mentioned so that's certainly a misnomer in our  
case: we'd like to track "what the user is looking at" rather than  
"what the mouse is pointing to" (as it is today) but not to control  
the interaction as is the case with gaze tracking as understood in the  
field. Just share eye attention between residents. It'll be perfectly  
inconsequential then if the gaze of the user change to look outside  
the scene (or the screen), we'd just switch back to the current  
attitude and animation

- VRwear (mentioned by Aimee): In my former incarnation, I shared a  
panel on Orange Island with those guys discussing avatar animation and  
puppeteering. Their video is great but, last I tried (more than 6  
months ago) I couldn't get the demo to work at all (other  
experiences?). Note that apparently they've been modifying the viewer  
directly which is not a bad idea if you want to get results fast and  
put something in the hands of users (that's what I did with  
HandsFree3D). Developing a protocol (using RPC or UDP or the already  
existing VRPN) is more open and would actually allow those guys to  
focus on the image analysis and extraction rather than spending time  
tracking the various viewer releases. Although we would provide a  
first widely available incarnation of this, the architecture shouldn't  
prevent others (including commercial firms) to come up with their own  
wares (hardware and software) promising a better experience (may be  
specialized on some aspects: face, body, emotion, whatever...). There  
are a lot of players out there (GestureTek, Softkinetic, Mamoca to  
name a few) not to mention again the ones pointed to in that thread  
already.

- Puppeteering: Since it's mentioned in the thread, just reminding  
people that I've been interested in that since some time (http://www.handsfree3d.com/ 
) and provided a patch to fix the crashing puppeteering branch  
(VWR-7703). Let's just say that full body puppeteering is great,  
feasible in SL, but there are a lot of non-trivial problems, far  
trickier than driving a bunch of effectors for a game. Rigging (i.e.  
the mapping of user's movement to an avatar that is geometrically  
different) in particular is a pretty big (and pretty interesting)  
problem which is very much SL specific.

- What are "gestures"? We were really thinking about  
"animations" ("gestures" are they are called in SL), i.e. triggering  
those without requiring the user to enter keyboard actions. This  
indeed would not require any server change as pointed out by Melinda.  
Sorry for the confusion Jan.

Stopping there for now but we'll sure be coming with more info in the  
coming days.

Cheers,
- Merov


More information about the SLDev mailing list