[sldev] Body motion and facial expression tracking, Microsoft did it

[Jan Ciger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dahlia Trimble wrote:

> But that's not how SL works at all. The animated avatar is decoupled
> from the physical representation on the simulator. The physical "agent"
> is a simple shape and does not take the mass of the avatar appendages
> into account. It is also constrained stand upright. In most cases the
> beer would have no mass and would not even be "grabbed", but would be
> "attached" to the hand of the avatar skeleton. 

Right, I didn't say that is the way SL works or should work. However,
this is the way to go if you want to produce realistic looking results.
Otherwise your character will behave like a Wile E. Coyote braking at
the edge of a cliff using his toes. If a human did that, he would fall
over, basic physics. The engine we have used allowed to satisfy this
constraint as well, most commercial ones don't.

> From there the only IK
> solving would be moving the beer to the mouth and tilting the glass. A
> well designed procedural animation algorithm that took gravity into
> account might make changes to the entire skeleton to give the appearance
> of the avatar compensating for the changes in center of mass, 

That is something else. The IK solution I have described above is purely
kinematic (i.e. no physics involved). However, it has an extra
constraint to force the centre of gravity of the character to be between
the legs, so that the generated pose would be stable if you did the
physical simulation or tried to assume that pose yourself. It may force
the solver to extend one leg to counter-balance the character, but
again, only centre of gravity is calculated, not really any physical
simulation. Try to stand upright and then start leaning forward. At some
point you have to either make a step or extend your leg backwards to
stop falling over. That is what the solver was doing. It brings added
realism to the animations and rules out poses that could reach the goal,
but are not realistic for a normal human (standing on tiptoes at a 45
degree angle, for example).

> I think adding the ability to do *simple* tweaks to existing animations
> such as aligning avatars for a handshake might be helpful feature.

Yep, but you do not need IK for that. And IK will still not help you
with vastly different skeleton sizes - i.e. if my arm is too short, IK
cannot make it "longer" to reach you. Not to mention that there seems to
be the impression that "making tweaks to existing animations" is somehow
"simpler". It is not - you only load the solver with more constraints.

> Currently scripts have no access to any keyframe data, which makes sense
> since animations are not synchronized. 

Not sure how are these things actually related. However, even if you had
access to the raw keyframe data (joint orientations/positions), it
wouldn't help you much - both LSL and Mono are way too slow and limited
to do anything meaningful with them in real time.

> Scripts currently can change the
> position of an avatar but not rotate it. For something like a handshake,
> I think letting a script compute the avatar positions and rotations and
> then making adjustments to the hand positions could produce a higher
> quality handshake than can be done in SL today. Unfortunately the
> current system will not allow it.

You can always put two guys on aligned poseballs and it will work just
fine, as proved by all the people doing various naughty stuff in SL
every day. However, it will work only within the limits of the guys
having skeletons compatible with the animation (but IK will not really
help you there) and missing animation synchronization (where local,
client-only IK is a bad and performance expensive kludge, IMO).

To conclude this discussion on IK - if someone knows how to program in
C++ and wants to play with IK, have a look at HMS IKAN library:
http://cg.cis.upenn.edu/hms/software/ikan/ikan.html

It allows up to three joints (I think) to be simulated, e.g. an arm or
leg. We have used this library for initial development, it does work,
but you will quickly see the problems I was talking about in my previous
e-mails.

Regards,

Jan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mandriva - http://enigmail.mozdev.org

iD8DBQFKLBK4n11XseNj94gRAt0xAJ9PKwSXsqJmDwIpeVWU83IPFpoTIACg41+P
loliFqbxuaUJFrJM0pHqBqQ=
=7Hud
-----END PGP SIGNATURE-----