[sldev] message_template.msg ordering and packing : attentionLLfolks

[Mark Lentczner]

Whew - there's a lot piled into this thread.  So I'm going to respond  
in sections:


:: Order ::

Yes, blocks and variables are out of order with respect to the  
message_template.msg file.  We intend to change this in an upcoming  
release.

The ordering doesn't depend on the implementation of std::map at  
all.  std::map guarantees that items will be iterated in sorted key  
order.

In this case, the keys are pointers into the message string table,  
and hence are essentially the hash bucket of the strings.  So the  
order is strictly the bucket LLMessageStringTable hashes a string  
into.  None of that depends on STL code.  It does makes the order  
unstable in the sense that changing message_template.msg changes the  
hash of strings.

Indeed, both problems pointed out with the hash function are true.   
The hash works, they just make the hash function less optimal than it  
could be.


:: Changes in the Works ::

We will be making two major changes to the message system in the  
coming weeks:

A) The existing message transport will be altered to be essentially  
the same, but allow for extension and backward/forward  
compatibility.  This will be achieved by making four changes:

	1- Making the order of blocks and variables stable and well defined
	2- Removing the checksum check
	3- Fixing the message number for each message
	4- Only allowing compatible changes: message addition,
		message extension (adding a block), message deprecation

B) We are creating an LLSD based encoding and transport layer for  
messages.  Yes, it is another system - but it we expect it to be the  
one that lasts us into the future.  It is much more flexible, and  
easier to interoperate with other languages and systems.

Please note that the LLSD system does not message_template.msg at  
all.  New messages can (and will) be defined without adding them to  
message_template.msg.  The template will exist only for messages that  
continue to be binary encoded and sent via UDP.  We expect this to be  
very few messages.

You can read about this in the transcript (with pretty pictures) from  
my office hours of March 1st:
	https://wiki.secondlife.com/wiki/ZeroTranscript2007Mar01
	
	
:: Performance ::

So much about the message system is inefficient, that these problems  
don't even show up.  As an old mentor used to say: "A fart in a  
whirlwind."


:: History & "Is It A Bug?" ::

I don't know why it is the way it is.  I wasn't here when it was  
done.  Yes, it seems complicated, obtuse and slow for no reason.  No  
point in debating or defending it.  Reasons, if there ever were any,  
are lost in the mists of time...

Is it a bug or just a poor design or a lousy implementation?  I only  
have these kinds of discussions over beer.


:: Patching ::

The code involved has been extensively refactored internally.  Since  
we most of the way into equivalent work, a patch from the community  
isn't wouldn't help.  Our plan is to normalize the ordering to that  
of message_template.msg.  We probably won't fix the hashing problems,  
since they aren't worth the risk right now.


:: Thanks ::

I'll be very honest - this subtly of the binary message ordering was  
brought to our attention by this thread.  Thanks!


	- Zero
	

----
Mark Lentczner / Zero Linden
Director, Studio Icehouse
Linden Lab

markl at lindenlab.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://lists.secondlife.com/pipermail/sldev/attachments/20070316/72b3c552/PGP-0001.pgp