[sldev] Body motion and facial expression tracking, Microsoft did it

[Rob Lanphier]

Hi folks,

Let's wrap this conversation up, at least until someone takes the time
to document the salient, actionable points of this conversation here:
https://wiki.secondlife.com/wiki/Camera-based_Input

If you'd like to continue to discuss this, a good place to put your
thoughts is here:
https://wiki.secondlife.com/wiki/Talk:Camera-based_Input

Thanks,
Rob

On 06/08/2009 09:24 AM, Dahlia Trimble wrote:
> A few points about the Natal video that I find difficult to believe
> from a technological perspective:
>
> * the actor's arms are often occluded from the sensor by their bodies
> yet the avatars react as if the arm positions are tracked
>
> * a few of the actors are wearing flowing clothing that can occlude
> body positions such as loose open sweaters
>
> * at the very beginning of one of the videos there is some text along
> the bottom left: "Product vision: actual features and functionality
> may vary"
>
> Given that, I don't think a company like Microsoft would publicize
> non-existent technology unless they had some proof of concept working
> in their labs. 3D scene recognition via stereo cameras has been around
> for quite a few years now and I think the time is ripe for consumer
> products like this to come about. I somehow doubt that the first
> generation products will work as well as the video suggests.
>
> On Mon, Jun 8, 2009 at 9:08 AM, Ron Blechner <ron at involve3d.com
> <mailto:ron at involve3d.com>> wrote:
>
>     Oh, p.s.
>
>     I met up with someone at the CT Film Festival this weekend who
>     confirmed that the whole body interactive demos by Microsoft? STAGED.
>     Entirely a mock-up demo. The technology apparently works but they're
>     no where near actual product demo.
>
>     -Ron / Hiro
>
>     On Mon, Jun 8, 2009 at 11:31 AM, Ron Blechner<ron at involve3d.com
>     <mailto:ron at involve3d.com>> wrote:
>     > The problem with the handshake:
>     >
>     > In short - it's 2 seconds worth of interaction. That's a tiny amount
>     > of reward for a tremendous amount of work it would take to make it
>     > better. And, there are already handshake and bow animations - they
>     > don't line up with another avatar, but doing all the work to make
>     > handshakes doesn't even create a new function - it just polishes an
>     > existing one.
>     >
>     > Compare that with putting in effort to do better facial animations -
>     > that would affect 100% of the time of interaction between two
>     people.
>     > And consider that facial animation programming is *client-side* by
>     > nature, and doesn't require breaking apart and redoing the
>     avatar code
>     > in Second Life to the degree that handshakes / puppeteering does.
>     > That's a helluva lot more results for a helluva lot less effort.
>     >
>     > I'd also like to add that, as Lawson pointed out, there is cultural
>     > bias. According to Neilsen, there are 170 million Americans actively
>     > using social media. There are twice that in China alone. China
>     has 20%
>     > of the world's population. Are we here to make a Western
>     metaverse, or
>     > a global one based on ideals not bound by one country, or one
>     culture?
>     > The reason we handshake instead of bow is because the West has
>     > controlled the vast amounts of wealth in the world. Like it or not,
>     > the West is spending its Wealth and a tremendous amount is going to
>     > the East. In 20 years - we all may be bowing, like it or not.
>     >
>     > I don't want to ruffle too many features, but it's an important
>     > exercise to escape one's own assumptions and biases when debating
>     > functionality with a platform whose stated goal is to be "a new
>     > country", as Rosedale would say.
>     >
>     > And to bring this back - the larger issue is that people tend to get
>     > caught up in particular niceties like handshakes and miss
>     > bigger-picture items. What good are handshakes if the rest of the
>     > conversation is less meaningful because it lacks facial expression?
>     >
>     > I say, FORGET puppeteering. Forget handshakes. Let's focus our
>     efforts
>     > on something much easier to accomplish, and with far more impact on
>     > Second Life. Facial expressions from video.
>     >
>     > *runs off to blog these thoughts*
>     >
>     >
>     > --
>     > Ron Blechner
>     > Chief Technology Officer
>     > Involve, Inc
>     > www.involve3d.com <http://www.involve3d.com>
>     > SL: Hiro Pendragon
>     >
>     >
>     > p.s. PLUG! http://secondtense.blogspot.com
>     >
>     > On Fri, Jun 5, 2009 at 1:33 PM, Tateru
>     Nino<tateru.nino at gmail.com <mailto:tateru.nino at gmail.com>> wrote:
>     >> Maybe it isn't really about handshakes, and more about general
>     >> lining-up-of-body-parts between avatars? :)
>     >>
>     >> However, for most people in first-world western cultures, a
>     handshake is the
>     >> frequently sole form of socially allowable physical contact
>     between two
>     >> people who aren't intimates at some level. That makes it
>     strongly symbolic.
>     >>
>     >> For handshake you can substitute a few variations:
>     Knuckle-bumps, high-fives
>     >> and such, but they're all basically a handshake with different
>     emotional
>     >> flavoring.
>     >>
>     >> Ron Blechner wrote:
>     >>
>     >> Question:
>     >>
>     >> Why are handshakes so important that they are much more of a
>     topic of
>     >> discussion of implementation, against facial expressions?
>     >>
>     >> -Ron / Hiro
>     >>
>     >> On Fri, Jun 5, 2009 at 7:34 AM, Argent
>     >> Stonecutter<secret.argent at gmail.com
>     <mailto:secret.argent at gmail.com>> wrote:
>     >>
>     >>
>     >> On 2009-06-04, at 08:55, Jan Ciger wrote:
>     >>
>     >>
>     >> Argent, read my comment to Tigro's mail. It wouldn't work. At least
>     >> not
>     >> in a nice way. For reaching and grasping you need much more IK than
>     >> just
>     >> the three arm joints and then you are hitting a severely
>     >> under-constrained and computationally expensive problem.
>     >>
>     >>
>     >> That's why you don't try and solve it computationally. You don't
>     >> replace normal animation, you use this for minor adjustments to the
>     >> existing animation, and you limit the strength of the adjustment to
>     >> small angles and specific joints.
>     >>
>     >> So it's down to the person selecting the base animation and
>     providing
>     >> the strength and possibly range (either distance or angle).
>     >>
>     >>
>     >>
>     >> E.g. in one case I have seen the solver to keep the hands next
>     to the
>     >> avatar's waist but stick the waist forward to reach a goal.
>     >>
>     >>
>     >> Wouldn't happen, unless the person selected the waist as the joint
>     >> that would move, and unless the waist was already close to the
>     goal.
>     >>
>     >>
>     >>
>     >> IK is a nice tool, but extremely hard to use unless you have an
>     >> animator
>     >> guiding it.
>     >>
>     >>
>     >> Which is the point.
>     >>
>     >> _______________________________________________
>     >> Policies and (un)subscribe information available here:
>     >> http://wiki.secondlife.com/wiki/SLDev
>     >> Please read the policies before posting to keep unmoderated posting
>     >> privileges
>     >>
>     >>
>     >>
>     >>
>     >>
>     >> --
>     >> Tateru Nino
>     >> http://dwellonit.taterunino.net/
>     >>
>     >>
>     >
>
>
>
>     --
>     Ron Blechner
>     Chief Technology Officer
>     Involve, Inc
>     www.involve3d.com <http://www.involve3d.com>
>     _______________________________________________
>     Policies and (un)subscribe information available here:
>     http://wiki.secondlife.com/wiki/SLDev
>     Please read the policies before posting to keep unmoderated
>     posting privileges
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Policies and (un)subscribe information available here:
> http://wiki.secondlife.com/wiki/SLDev
> Please read the policies before posting to keep unmoderated posting privileges