[sldev] Gesture client code
Philippe Bossut (Merov Linden)
merov at lindenlab.com
Tue May 26 00:16:23 PDT 2009
Hi all,
Gee! This is quite a thread! Thanks for all the suggestions, ideas and
passion. As always, this community shows it's full of talented (and
opiniated) people :)
Since I'll be the guy tagged with driving/mentoring the intern on this
project on a day to day basis, I'm taking on summarizing and answering
some of the questions that surfaced in that thread (vaguely in
chronological order):
- Why OpenCV? As pointed out by folks, it may look like overkill for
the goals currently stated but those are goals *a minima*. We sure
hope to expand from this first experimentation since, it is indeed an
experimental project. Jan Ciger confirmed though that, if the goal is
to capture a wide set of face and body elements, OpenCV is a good
starting point.
- Emulate mouse/joystick driver? Several folks mentioned implicitly
the parallel between capturing elements and emulating yet another kind
of input device. The problem with mice or joysticks is that they have
only one or few effectors and that those do not map on avatars
controls. With a camera, we certainly want more than few effectors.
Eventually, we'll rapidly grow to the point we want VR like richness
for that input. VRPN in that respect is an interesting idea (thanks
Jan again). Besides, we're not currently trying to "drive" the
interaction (i.e. building prims or triggering camera movements and
avatar navigation) but rather *add* to the currently rather canned
(and unused) avatar animations (aka gestures) so to see how much of
the RL emotional state we can share with other residents. IOW, we want
to improve and add to the current experience, not replace or use as an
alternative to keyboard, mouse, etc...
- Gaze tracking: Often mentioned so that's certainly a misnomer in our
case: we'd like to track "what the user is looking at" rather than
"what the mouse is pointing to" (as it is today) but not to control
the interaction as is the case with gaze tracking as understood in the
field. Just share eye attention between residents. It'll be perfectly
inconsequential then if the gaze of the user change to look outside
the scene (or the screen), we'd just switch back to the current
attitude and animation
- VRwear (mentioned by Aimee): In my former incarnation, I shared a
panel on Orange Island with those guys discussing avatar animation and
puppeteering. Their video is great but, last I tried (more than 6
months ago) I couldn't get the demo to work at all (other
experiences?). Note that apparently they've been modifying the viewer
directly which is not a bad idea if you want to get results fast and
put something in the hands of users (that's what I did with
HandsFree3D). Developing a protocol (using RPC or UDP or the already
existing VRPN) is more open and would actually allow those guys to
focus on the image analysis and extraction rather than spending time
tracking the various viewer releases. Although we would provide a
first widely available incarnation of this, the architecture shouldn't
prevent others (including commercial firms) to come up with their own
wares (hardware and software) promising a better experience (may be
specialized on some aspects: face, body, emotion, whatever...). There
are a lot of players out there (GestureTek, Softkinetic, Mamoca to
name a few) not to mention again the ones pointed to in that thread
already.
- Puppeteering: Since it's mentioned in the thread, just reminding
people that I've been interested in that since some time (http://www.handsfree3d.com/
) and provided a patch to fix the crashing puppeteering branch
(VWR-7703). Let's just say that full body puppeteering is great,
feasible in SL, but there are a lot of non-trivial problems, far
trickier than driving a bunch of effectors for a game. Rigging (i.e.
the mapping of user's movement to an avatar that is geometrically
different) in particular is a pretty big (and pretty interesting)
problem which is very much SL specific.
- What are "gestures"? We were really thinking about
"animations" ("gestures" are they are called in SL), i.e. triggering
those without requiring the user to enter keyboard actions. This
indeed would not require any server change as pointed out by Melinda.
Sorry for the confusion Jan.
Stopping there for now but we'll sure be coming with more info in the
coming days.
Cheers,
- Merov
More information about the SLDev
mailing list