[sldev] Body motion and facial expression tracking, Microsoft did it

[Jan Ciger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dahlia Trimble wrote:
> I dont think the math or the complexity of IK is beyond all of the
> scripters in SL, There are some talented scripters around who could
> easily solve the problem, some of them in SL may even be PhD level
> mathematicians ;) 

Right :) I didn't say there aren't any, but that they are tiny minority.
I do happen to have that PhD, but most people I know that are scripting
(and even very good scripters) don't. Of course, that doesn't mean
anything, but I somehow do not think that an average SL creator will be
able to use it meaningfully.

> Rather I think synchronization and lag would prevent a
> lot of real time on-the-fly animations. 

There is no synchronization at the moment. That is one issue I have
described on my reply to Argent. If we had that, I think we won't need
IK for the most part. It would be interesting to have it, but 99% of
things people would want it for could be done by pre-baked but
synchronized animations.

> I don't think that computing IK on the fly is all that expensive either,
> it may only be a few hundred floating point operations per keyframe for
> each skeleton, certainly not expensive for modern day hardware.

Eh, did you try? Do you realize that you are dealing with heavily
underconstrained problem that even with 15 joints (that is what SL
avatars have, if I recall right) will generate lots of solutions and you
need to weed bad ones out somehow. That takes time and computational power.

We were doing semi-real-time IK on decent machines with 70+ joints
(HAnim skeleton), but you could definitely feel the impact (character
"thinking" for a few seconds every time IK was triggered). And that was
only *one* guy. Now imagine a sim with 5 couples dancing (e.g.
ballroom), some people hugging or whatever - and you are calculating IK
for every one of them suddenly. The impact would be terrible.

You can see an example of the performance here:
http://vrlab.epfl.ch/Movies/Movies_index.html

Look at the "Counting Beers" video. The characters look ugly even
compared to SL, but that was back in 2005 and the design wasn't the
point of the video. The animations are all procedurally generated except
the few guys dancing and one guy drinking the beer. The walking is
procedural, the grasping is IK + special grasping code of my colleague.
There you can see clearly also the delays due to the IK solver - e.g.
when the two guys are exchanging the beer. And that was on a single,
fairly decent machine and heavily tweaked. Moreover, if I recall
correctly, it wasn't even using all the joints - I think we have
constrained it to upper body only for performance reasons. It won't be
too much faster today, because the issue are the algorithms, not the raw
number crunching power.

The website also shows work on motion re-targeting and adaptation using
IK which is state of the art. That should give you an idea what is
doable and at what cost - but many of the IK videos are/were produced
offline in Maya (calculated using the lab's code and rendered in Maya).
It would have been way too slow in real time.


> I *do* think that if real time on-the-fly animations were to be
> implemented in any kind of believable sense, there would need to be an
> improvement made in the speed in which the data could be distributed
> between clients. Somehow I don't see it as possible with the current
> load that many SL sims have to deal with already.

Yes, that is the synchronization issue. That would need to be sorted out
first, before any attempts at things like this.

Regards,

Jan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mandriva - http://enigmail.mozdev.org

iD8DBQFKKz7tn11XseNj94gRAp7BAJ99kNbZev+b6+BteuwVW5SDMEot8ACgvVJJ
5M6MycOEFfj5oHPB2SiVDDs=
=Mekj
-----END PGP SIGNATURE-----