[sldev] Body motion and facial expression tracking, Microsoft did it

[Philippe Bossut (Merov Linden)]

Hi,

As Philip said, this is the 3DV camera (MS acquired 3DV end of 2008)  
and most of those demos have been seen before at Game Developer  
Conferences and others since a couple of years now. The best one I've  
seen (and experimented myself) was made by Softkinetic (http://www.softkinetic.net/ 
). Awesome feeling! But that needed pretty careful setting and  
calibration. Also note that the videos by MS are *not* demos (i.e.  
live effective code) but staged shooting.

One big issue so far with those systems (especially the 3DV camera  
based on TOF technology, something similar to Lidar) is the ambient  
noise you get in a real life busy setting. With Lidar in particular  
the presence of a reflective surface in the field of view screws up  
the reading big time. If your room is a little crowded, getting the  
background out is not always easy and mixing several people gets you  
into interesting tracking problems...

Plus, one very interesting and SL specific problem is "rigging": if  
you want the movements to be representing yourself (and not just  
knocking off stuff on screen), meaningful rigging is pretty hard as  
you need to track every joint (relying on FP/IK for untracked joint  
like in puppeteering gives weird results... I tried...) and you need  
to remap positions to keep the respective positions of end points  
correct (just using the quaternions for each joint for instance won't  
guarantee that your hands will end up on your head if you take that  
position in front of the camera, as small bone length differences  
between your skeleton and the avatar's skeleton will translate in  
different end positions). Keeping an expressive avatar is actually  
harder than just using your body as a joystick in a game (which  
expressivity is limited). This is a challenge they won't even try to  
stand up for I think (irrelevant for games) but that's very important  
for SL (assuming we want full body tracking one day).

Well, all that to say that, no, "they" haven't beaten us to it yet.

Cheers,
- Merov

On Jun 1, 2009, at 8:58 PM, Tigro Spottystripes wrote:

> my main use for it probably would be puppetering of the avatar, at  
> least
> above waist since I don't like standing up all that much, but I  
> imagine
> this can be expanded to many uses, like the example on one of the  
> vids,
> driving vehicles without any special controlers, and with a bit of
> cooperation of the coders, this could even help to  provide in-world
> animation RECORDING, you would be able to turn your own room into a
> mocap studio with just SL and a little gadget, and you would be able  
> to
> get live preview on the very avatar you intend to use the anim on (in
> the cases where a anim is meant for a specific avatar isntead of  
> being a
> generic anim)
>
>
> I hope when that thing hits the stores it will cost at most about as
> much as a Wiimote and got a easy way to interface with regular PCs
> (isn't the XBox basicly a PC with some purpose built hardware and it's
> own unique OS partially inspired by Windows and the DirectX tech?)
>
> Maya Remblai escreveu:
>> Eh. Yeah it's cool, yeah it's great for games made to use it. Me,
>> though? Ain't no way I could play Phantasy Star Universe, or Katamari
>> Damacy, or MOTHER3 with that. Besides, I'd collapse from exhaustion
>> after about ten seconds anyway. So I don't see the relevance to SL,  
>> I'd
>> never use it even if I could afford it. How would it handle flying
>> anyway? ;)
>>
>> Also:
>>
>> Thomas Grimshaw wrote:
>>
>>> But they did beat you to it...
>>>
>>> What's $100 when you're charging $200 a month for a quarter of a  
>>> server?
>>>
>>> ~T
>>>
>>>
>> Ooh, burn. ;P
>>
>> Maya
>> _______________________________________________
>> Policies and (un)subscribe information available here:
>> http://wiki.secondlife.com/wiki/SLDev
>> Please read the policies before posting to keep unmoderated posting  
>> privileges
>>
>>
> _______________________________________________
> Policies and (un)subscribe information available here:
> http://wiki.secondlife.com/wiki/SLDev
> Please read the policies before posting to keep unmoderated posting  
> privileges