LLSD for Parameters (was re: [sldev] Plugin API)

[Tim Shephard]

Thanks for agreeing with me - yes LLFUNC is an awkard match for LLSD.
  But I am pretty sure you missed the thrust of my email, Ryan.

The API needs to be versioned and hopefully defined in a semi-formal
way.   I was suggesting LLSD would be an appropiate springboard for
that interface definition since it's already being used for Client <->
Server RPC.

It would also help us get from a to b quickly, which is what everyone
seems to want.

However, if you prefer to discuss something like MIDL, that would be
obviously superior.   I wasn't quite sure we wanted to tackle
something that significant at this point.

But if we're going to discuss MIDL, then there is a lot more we should
probably looking at as well..

Cheers,

Tim.

On 3/2/07, Ryan Williams <rdw at lindenlab.com> wrote:
> As a disclaimer, I really have no experience with plugin apis, I was
> just suggesting LLSD because it is the data interchange format of the
> future, useful for interoperating between parts that are loosely coupled.
>
> Tim Shephard wrote:
> > I was thinking on the email proposed by Ryan regarding LLSD and it has
> > some interesting implications.
> >
> > If we were to use LLSD for parameter passing, one useful area might be
> > (I stress the word, might, as I have significant concerns about this
> > approach) for registering services.
> >
> > For example, we might have UIServices, RenderingServices, and
> > MessageServices.
> >
> > Here is what a C api might roughly look like (so far, only 6 top level
> > functions):
> >
> > //Get a list of the services (Ie: UI Services, Message Services,
> > //Rendering Servers) this plugin wishes to subscribe to
> > // Included in the LLSD object will be function pointers to
> > // callbacks that the client will call into the plugin.
> > LLSD *SLP_GetServiceSubscriptions();
> >
> > //Tell the client it's trying to subscribe to a no longer supported
> > // service and version tuple.  This would be at a coarse grain level.
> > void SLP_InvalidateServiceSubscription(const char *ServiceLabel);
> >
> > //Initialization / Shutdown callbacks
> > //called when the client loads /unloads the plugin.
> > //XML Manifest file would be more suited
> > SLPError SLP_Load();
> > SLPError SLP_Unload();
> >
> > // called when the client starts up /shuts down
> > SLPError SLP_ClientStartup();
> > SLPError SLP_ClientShutDown();
> >
> >
> > Given that services from the client themselves (on the assumption that
> > it's a rapidly evolving client) will need to be versioned, I was
> > thinking we may want to use the LLSD object to register services with
> > the client.
>
> I really like LLSD and I think it is the perfect solution to a bunch of
> problems.  I don't think you can store function pointers in LLSD though,
> unless you want to get into some horrible horrible casting.  Hmmm... I
> don't know if plugin registration absolutely needs function pointers.
>
> Or you could use the LLSD for everything but the function pointers, and
> then also pass along a list of function pointers...  I dunno, just
> thinking out loud.
>
> > For example (this is just to give you an idea.  This would be highly
> > evolvable, which is the whole point of using LLSD objects):
> >
> > There would be a part of the plugin api:
> >
> > LLSD *SLP_GetServiceSubscriptions();
> >
> > SLP_GetServiceSubscriptions could return an
> >
> > Services Low    Trusted  Zerocoded
> > {
> > UIService Variable
> >       {
> >              ServiceData  Single
> >               { Version         S32}
> >               { ServiceLabel Variable 2}
> >               {  XMLSpec     Variable 2}   //UI Xml Spec
> >               {  MenuName  Variable 2}   //There could be a 'plugin' menu
> >               {  Show           LLFUNC}    //call back when show is
> > called, has a signature not documented here
> >               {  .. etc .. }
> >
> >               Events Variable
> >               {       EventType                       S8
> > }  //eg, setAction,
> >               {       EventFunc                       LLFUNC        }
> > //callback function
> >                    .. etc
> >       }
> >
> > MessageService Variable
> > // Dynamic structure for MessageService
> > // Probably something that specifies message types and
> > // associated callbacks
> >
> > RenderingService Variable
> > // etc
> > ..
> >
> > }
>
> So I think I created some confusion by conflating LLSD with the message
> system.  They have similar properties, but the fundamental difference is
> that LLSD is just a wire format combined with an in-memory api.  The
> message system is more specific and limited.  I believe the acronym
> stands for "Linden Lab Data System" in some sort of oddly-ordered
> universe.  We are in the process of replacing the on-the-wire message
> system format with an LLSD-based system (see Zero's blog post here:
> http://blog.secondlife.com/2006/12/21/a-big-change-youll-barely-notice/).
>
> So what you have up there looks like a message system message, not LLSD.
>  As LLSD you would notate it more like:
>
> {
> 'uiservices': {
> 'servicedata':{'version': <int>,
>                'servicelabel': <string>,
>                'xmlspec':      <string>,
>                'menuname':     <string>,
>                'show':         <some sort of function reference>,
>                ....etc....}
> 'events':[
> {'eventtype':<int>, 'eventfunc': <some sort of function reference>}]
> }
>
> This notation doesn't correspond to any actual parseable specification,
> because the actual LLSD on-the-wire formats are either annoyingly
> verbose or binary.  This json syntax is the closest human-readable
> syntax we use (and I'd be happy if we had an LLSD formatter/parser for
> it).  Nor can we enforce any sort of data structure using LLSD because
> it is dynamically-typed like Python.  However, this is actually a good
> thing because you can deal more gracefully with failures.
>
> Anyway, this little pedantic data system tale doesn't really change
> anything about your concept here.
>
> > This has four aspects, two I think is relatively straightforward, the
> > other two are concerns.
> >
> > The first aspect is versioning.  Right now the versioning in the LLSD
> > objects are done in different parts with different nomenclature.
> > Ideally, we could develop a more  consistent approach not just in
> > verbiage but also handling revision tracking of the message protocol
> > itself so that future tools for dealing with versioning on a finer
> > grained scale will be more successful.
>
> True.  One question is whether the LLSD objects themselves contain their
> own version number, or if we have a meta-object that describes the
> versions of all the objects it knows about.
>
> > The second issue is how flexible the LLSD protocol is.   Looking
> > through message_template, I couldn't find examples of realtively
> > complex hierarchical types.   It would be useful to urge LL to XMLify
> > this.
>
> Take a look in llsd.h.  It supports arbitrary hierarchies, and it
> serializes to XML.
>
> > The third aspect are function pointers as a LLFUNC type which is a
> > pointer.  This will have a different size on different platforms, and
> > isn't entirely in the spirit of the LLSD object which is mostly used
> > for RPC / serialization .. though there is some interesting discussion
> > there to be had for using the message spec for services in other
> > plugins which work over the wire.
>
> I guess I talked about this before.  I don't think we should try to
> wrangle LLSD to do something platform/implementation-specific like carry
> function pointers.  A side channel would probably be better.
>
> > The fourth aspect is related to the second in that we do not really
> > need to encode and we're using this message spec for communicating the
> > structure of the Service specifications, on the theory that they
> > change rapidly and we'll need dynamic typing.  Perhaps we're over
> > building if we do this.
> >
> > The alternative approach is to use LLSD like type C structs.
> >
> > I still think we need some kind of specification for what the structs
> > will look like
> > (we could use the message protocol) and we still need to allow for
> > dynamic typing because of changing features (adding and deprecating
> > data fields, LLFUNCS, etc), however, instead of returning LLSD class
> > objects, we'd return a C version which is similar but without the C++
> > features of the full on LLSD object.
> >
> > The advantage of that approach is that we'd be able to deal with C++
> > ABI issues and we'd be able to share concept, code, and grammar
> > between the two types, LLSD and CLLSD.   The Plugin SDK would have
> > helper funcs for creating these CLLSD objects.
>
> I know nothing about C/C++ interoperation, but I like the idea of C LLSD
>  bindings.  Part of the fun of C++ LLSD is how syntactically sugared it
> is, but it's certainly possible to make a nice C api.  Actually that
> would make a nice self-contained project if someone is interested in it.
>
> -RYaN
>
> > Anyhow, let me know your thoughts.
> >
> > For those who don't like the idea, alternative header files are your
> > best answer!  This idea should inspire better ideas only, not empty
> > critique :)
> >
> > For those who like the idea, suggestions on alternative nomenclature
> > and function grammar that clearly evokes purpose.  Ideas for parameter
> > signatures would be useful.   Expanding the API is fine, however,
> > keeping it as simple as possible .. ((especially the bits that are
> > likely to be stable) is critical for Plugin API success.
> >
> > Cheers,
> >
> > Tim.
>
>