[sldev] Body motion and facial expression tracking, Microsoft did it

[Jan Ciger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Argent Stonecutter wrote:
> Of course, the actual modifications would be small angles applied to the
> joints. A few degrees at most. If the motion goes outside the range it
> just doesn't happen.

That would be an incredibly frustrating experience - it would work
sometimes and sometimes not, only because the av was a centimeter
farther than last time and the IK couldn't converge any more because of
your too tight constraints.

> 
>> That could lead to inconsistencies, like
>> "rubber-like" stretching limbs if the other joints are not moved
>> correctly as well.
> 
> How do you get stretching from an animation system that's only changing
> angles?

You don't. However you are forgetting that once you are interpolating
between the original pose and an IK generated pose, you are screwed in
this regard. The angles produced by blending may not be consistent in
this way, twisting and stretching the body out of shape. This is why
motion adaptation by IK is so difficult - these cases need to be
recognized and ruled out or fixed somehow. Just imagine the following
interpolation/blending - hand and lower arm from IK and the upper arm
from the original keyframe. Suddenly your elbow is stretched and
twisted, because the upper arm from the keyframe is completely elsewhere
than where the IK needs it.

> That's already an issue for multi-animation scripts. It's handled by
> eye. Does this combination look right to me?

What do you mean "handled by eye"? The animation engine does this
automatically - that is what the animation priorities in SL are for when
you are uploading animations. I am not talking about "baking animations"
into one final keyframe file in Max or Blender here, but about real-time
blending - a guy walking, turning his head and waving, while still
continuing to walk. Think of playing a gesture while walking in Sl.

> None of it would need to be on the server side. There's no
> synchronization between different screens in SL, even when effects
> associated with avatars are synchronized (eg, particle targets for leash
> effects) on each individual screen.

That is not relevant. The fact that there is no synchronization now
doesn't mean you won't need it if you want to achieve consistent
grasping that will look correct to everyone looking at it. If you do
everything client side, everyone will see different thing - with IK even
WILDLY different. A tiny accuracy difference introduces huge and
accumulating errors in IK solvers.

> In the absence of lag, we already have synchronization. In the presence
> of lag, synchronization fails.

Eh? What synchronization exactly? That you start the animations at the
same time? I am not sure that you know what synchronization is actually
about. That is *not* sufficient - the machines run at slightly different
speed, with different load and what not, you get the animations drifting
apart almost immediately. Lag or no lag, even if you started the
animations at perfect same moment.

Saying that we have "synchronization" in the absence of lag reminds me
of my former colleague who claimed that everything
synchronization-related will be solved if we can synchronize the random
number generators at the different computers (as if the RNGs were
somehow the cause of drift in the animations ...)

> But there's no synchronization between clients, just between avatars on
> each client.

There is no synchronization between any avatars neither - btw, client
and avatar, what do you mean? As if the avatar was not only on the
client or what ...

This is why you need proper animation synchronization first, before we
can think about "shaking hands", "hugging people" or whatever else. It
is doable, but with quite a bit of network overhead.

Regards,

Jan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mandriva - http://enigmail.mozdev.org

iD8DBQFKK+K8n11XseNj94gRAvfuAJ9MX6o49Nx1Su0gAAOb5tzMiWSFswCZARcz
ihow9lVKu3owoYvBjieDY+A=
=Pfk+
-----END PGP SIGNATURE-----