[sldev] Why does the simulator do object culling for clients?

[Jake Simpson]

As Douglas has indicated, the subscribe / unsubscribe code in SL needs 
attention. Whats there works, but not in the most optimal fashion.

Right now, as has been mentioned, unsubscribe / subscribe is UDP and 
will probably need to remain that way since UDP as asynchronous and TCP 
is not - if object updates are coming via UDP (and they need to, trust 
me on this. Go dig out the TCP version of Doom and seeing what the does 
:) ) then the "Client, you need to know about this object" needs to be 
UDP to be sure it gets there first - at least attempting to, since UDP 
isn't guaranteed delivery.

Generally there are approaches that will work for this kind of thing - 
if the client receives an update for an object it doesn't know about 
yet, it can just ignore that update, unless it's enough to actually 
construct a complete object on the client side in which case it should 
do that. Generally though you don't get enough in an update packet since 
it's mostly deltas from an known state, not a complete state for an 
object, so construction is less likely an option.

There are mechanisms we can put into place whereby the sim sends a 
'subscribe' notice to the client, indicating a new object of interest 
has entered the list of objects for this particular client, and can do 
that via UDP. And it will continue to do that, sending down a complete 
state until such time as the client acks this and says "Yes, I know 
there's a new object here, thanks" which it can do simply by acking the 
packet it received with that new objects information with in it. Then 
the sim 'knows' the client has that object and it can start sending just 
updates instead. You do the same thing for unsubscribing - the sim 
continues to send updates for a given object, as well as a "you are done 
with this object" message, until the client acknowledges it got it, in 
which case the object is just removed on the sim side and not considered 
again until it comes back into view.

There are some other things we can do as well in terms of the object 
update packets - Object Update packets should be mostly just deltas from 
a known state that the sim and client agree on - e.g. the initial state 
that the sim sent down to the client, which the client acked. The sim 
now knows what the client has and will only send down information that 
has deviated from that state. If, every now and then, when you get a 
smaller packet constructed for sending down, you can insert a new 
'complete state' for an object to pad it out, which once acked brings 
the client and sim states up to one that hopefully should have less 
deviation between them because it's 'newer'.

Further to this there are other things to be tried, like not sending 
positional updates at all, but only sending velocity changes. If you 
send down a known state the client and sim have agreed on, then you 
don't need to send positional updates any more because the client and 
sim _should_ be in sync - all you need to do is send down velocity 
changes and the client and sim should remain in sync. Of course if 
packets are dropped, even though the velocity change is still being sent 
(and will be until the client acks it and the server says "Right, we are 
in sync right now") then this breaks some stuff since the client got the 
velocity change later than the sim started it, which means the dead 
reckoning position of the object is now wrong, but that's alleviated by 
the peridoric complete states being sent down. You can also send times 
on these packets and have the client re-interpret what the position 
should be given that the velocity change occured at time X (although 
this starts to involve some somewhat hairy code on the client side to 
make things match up). Simpler indeed, to just including a position when 
sending down a velocity change. You'd be surprised how effective that 
actually is - if you get to the point where so many packets are dropped 
that this become very apparent, then you are pretty much teetering on 
getting dumped anyway.

Most of the time good enough is surprisingly low in terms of real time 
correction and so on, because nothing particularly heavy is done client 
side that relies on this - it's all just visualization, rather than 
collision requirements and so on. The moment you start doing physics on 
a client, all this suddenly becomes far more important. But for now, 
it's less so.

Most of this is a known and solved problem; it's just time and resources 
that prevent us from looking at this immediately - that and the fact 
that while we see some artifacting now and again, it's not really 
something that destroys residents experience.

Note - this isn't necessarily how SL works right now, just an ideal of 
what it might be nice to get to.

In terms of why objects are subscribed and unsubscribed in the first 
place, there's also the factor of network bandwidth to consider 
alongside the issue of CPU usage on the Sim. If you don't remove those 
objects from the stream that are not considered within the clients 
perview (and not just frustrum perview as most FPS games have it - you 
need to be able to consider objects behind you to ensure you get sounds, 
particles that overlap your view and so on) then it doesn't take long 
before your stream is *full* of objects you can't see, don't care about 
and yet are getting updates for you don't need.

Moving some of this to the client is a definite possibility - however 
there are implications to that. It enables people to ask the sim for 
information about objects they can't see in an attempt to cheat at 
games, and generally see stuff they don't have the permissions to see, 
plus timing suddenly becomes that much more important.

Realistically the Sim has to be the final arbiter of what it sends for 
several reasons - network bandwidth (you may have extra bandwidth to 
receive more data, but the sim doesn't necessarily have the bandwidth to 
be sending it out, given it's sending out these streams of updates to 
gobs of clients, not just yours), clients seeing what they need to see 
and no more, and the sim owning what you can see.

Jake

John Hurliman wrote:
> Tateru Nino wrote:
>> John Hurliman wrote:
>>  
>>> Tateru Nino wrote:
>>>    
>>>> Dale Glass wrote:
>>>>  
>>>>      
>>>>> On Thu, Jul 19, 2007 at 01:38:02AM +1000, Tateru Nino wrote:
>>>>>             
>>>>>> Considering the amount of ghosting we generally see (due to lost
>>>>>> ObjectKill packets), I'm wondering if it isn't a good idea for the
>>>>>> viewer to be able to periodically poll 'stale' objects (ones that 
>>>>>> have
>>>>>> had no Update for a while) just to verify that they're still 
>>>>>> present.
>>>>>> Having avatars and objects that have long since gone hang around in
>>>>>> your
>>>>>> viewer for an hour or two - that can be a little odd, or even
>>>>>> unsettling
>>>>>> at times.
>>>>>>                     
>>>>> That sounds like a hack.
>>>>>
>>>>> ObjectKill is sent over UDP, I suppose? I haven't looked at 
>>>>> networking
>>>>> code much yet. But if so, polling isn't the right way to do it, the
>>>>> problem is that the message should be reliable and isn't. So move 
>>>>> it to
>>>>> TCP, ensuring it's always delivered. Then so long the sim and viewer
>>>>> aren't buggy that should be all that's needed.
>>>>>
>>>>> Imagine the amount of extra bandwidth needed for periodically 
>>>>> checking
>>>>> all the objects you can see if you set draw distance to 512M.
>>>>>
>>>>>               
>>>> Many of our most common complaints are caused by dropped messages that
>>>> should be reliable, but aren't. :)
>>>>         
>>> Isn't ObjectKill sent as "Reliable"? If there is a problem with the
>>> ACKing code that's a whole different story, and if the connection
>>> quality is poor enough that packets are being given up on after the
>>> minimum five resends or whatever it is then switching to TCP probably
>>> won't fix the connection.     
>> There seem to be some things going missing, and I've started to look a
>> little more closely at the messaging code to try to figure out why.
>> Symptoms suggest that messages are not turning up at all (avatar
>> standing up, avatar departing, object removed from simulation, etc) or
>> are having their parameters zeroed out or mangled somehow (avatar
>> position, appearance, attachment position).
>>
>> In recent days I've been starting to wonder if it is possible for the
>> parameters for one message to be incorrectly interpreted as being part
>> of an earlier message whose parameters may have been dropped in transit
>> somewhere, or if there is a data-path with degenerate encoding/decoding
>> behaviour.
>>
>> Time to UTSL, and see what can be seen.
>>   
>
> One thing we found while developing libsecondlife is that out of order 
> packets can sometimes produce the same behavior as a dropped packet. 
> For example, if someone stands up then sits down again quickly and 
> then sit down packet is handled before the stand up packet, the avatar 
> will appear to be standing over the new target. The only solution we 
> could think up was holding out of order packets in a processing queue 
> and replaying them, which didn't seem like an acceptable solution as 
> it would effectively lock up the network stream every time things were 
> handled out of order. Not saying that is what is going on there but 
> it's another angle to consider.
> _______________________________________________
> Click here to unsubscribe or manage your list subscription:
> /index.html