[sldev] Why hasn't Linden Lab implemented WRITING to NOTECARD?

[kelly at lindenlab.com]

MAGIC!

I don't know the exact details about how the current version works so
*please* don't respond to my answer with the million ways it could be
optimized or what it will miss etc.  A lot of effort and revisions have
gone into this.  This is extremely roughly what it does, or did at one
point and covers the gist of how GC on assets might work.

* Step 1: Scan the contents of every asset, every simstate and every
residents inventory for anything that looks vaguely like a UUID and build
a list of those UUIDs.  This means the text of every script, the contents
of every notecard, the inventory of every item.
* Step 2: Delete anything not in that list, but created before the scan
started.
* Step 3: Go to step 1.

That is an extremely simplified version of course, and it gets more
complicated as you work to make it more robust (find all those ids), safer
(don't delete straight away) and faster.

 - Kelly

> slightly off-topic question, could you explain how exaclty the system
> confirms that some old UUID isn't inside someone's inventory or inside
> an object or even a notecard before deleting it please?
>
> kelly at lindenlab.com escreveu:
>> Dale is pretty darn close, so I'll only hit on the differences.  And
>> some
>> about other emails I have seen in this thread.  :)
>>
>> * We do garbage collection on the asset system.  We scan for references
>> to
>> other assets (and anything that looks like an asset ID etc) and move
>> unreferenced objects off to the side.  After some period of no requests
>> for those assets they are deleted.  This is relatively effective at our
>> current rate of asset creation.  The actual growth rate of the asset
>> server is well within reason at this point and is not unbounded. (I say
>> 'current' and 'at this point' because this hasn't always been the case
>> and
>> I've been involved in more than a couple projects to reduce the rate at
>> which we create new assets. :) )
>>
>> * A large part of the "new asset for every save" is pure legacy.  The
>> system started and was designed as "write once", and there are a lot of
>> optimizations you can make if you know a specific UUID will always point
>> to the exact same thing.  Now our system is large and complex.  We have
>> worked on rewritable assets for various reasons (usually attachment
>> related) but these projects are difficult and complicated and tend to
>> break things you wouldn't expect.
>>
>> * Content creators tend to think notecards because notecards are what we
>> have and what we can store manually entered data into and read from.
>> They
>> aren't optimal though even if we ignore the show stopping asset creation
>> rate in the current system.  Every time you read from a notecard the
>> simulator must fetch that notecard from the asset server and load it
>> into
>> memory.  Sure it keeps a LRU cache of cards around so it doesn't fetch
>> for
>> *every* read, but this is hardly the right way to go about script data
>> storage.  Also this data isn't exactly random access, I think reading a
>> notecard via lsl line by line is an O(n^2) operation.  And another point
>> -
>> how do you handle the inevitable race conditions as two scripts read and
>> write to the same asset?  SVC-1406 that Argent mentions is a good
>> example
>> of better out of the box thinking.
>>
>> * I *hate* the project name "memory limits".  While it is true that part
>> of this project will be limiting total sim-wide script memory available,
>> we are likely talking about levels that already significantly degrade
>> simulator performance.  As a content creator I am *really* looking
>> forward
>> to "memory limits" because with it we can introduce dynamic memory sizes
>> for individual scripts.  Forget that project that needs 10 scripts, half
>> of each of which is the communication glue for them to work together.
>> Instead have one script that can use the memory it needs.  I don't have
>> all the details on the final design, and I'm sure it will be adjusted
>> with
>> every round of statistics we collect, but you could look at how we
>> handle
>> URLs for HTTP-In as a starting point.
>>
>> In short, writable notecards just aren't going to happen.  It would be a
>> horrible hack anyway with crappy performance.  We have llHTTPRequest
>> already which is ideal for accessing and storing data on an external
>> host.
>>  I have even seen services specifically for LSL that will store a small
>> amount for free.  Perhaps http_request, maybe specifically because it
>> can
>> do obj->obj, will open new options in 1.27 (on aditi now!).  And so
>> probably will the "script memory limits" project.
>>
>>  - Kelly
>>
>>
>>
>>> Saving a notecard makes a new one. The old data hangs around in the
>>> asset server apparently forever, so if you have 16k of temporary data
>>> and you change 1 byte of it and save the change, you're burning 16 kb
>>> per write. Over time this could end up being gigabytes to terabytes of
>>> wasted space in the asset system across thousands of program write
>>> operations, and LL has to provide expensive RAID and do tape
>>> archiving, etc etc of all that..
>>>
>>> UUIDs are not reused when objects are modified because of some design
>>> concept regarding data-access efficiency and caching. Woohoo, don't
>>> you love clear and direct answers? I think this was discussed a long
>>> while back but I don't see anything in my SLDev keyword searches.
>>>
>>>
>>> Okay, um, if I recall right, saving UUID changes would make the cached
>>> asset state stale and then you need to add mechanisms to keep the
>>> cache up to date with the main db, or for the main db to notify cache
>>> siblings of UUID state changes.
>>>
>>> If you don't ever allow UUID changes then the cached state never needs
>>> to be checked and can always be handed out to clients at much greater
>>> speed than if state checks were needed, but this speed comes at the
>>> price of poor storage efficiency of not recovering space used by
>>> changed UUIDs that may never be accessed ever again.
>>>
>>>
>>> There's no way to assess whether a saved asset was just temporary data
>>> that will never be used again vs a UUID that just won't be used again
>>> for a very very long time. LL can prune "infrequently used" UUIDs out
>>> of the main db running on expensive 15,000 RPM SAS drives, and move
>>> the infrequent assets to slower less-expensive "nearline storage", but
>>> LL can't ever really delete anything since there's no way to know if
>>> it might be needed by a user, or really never again.
>>>
>>> The slack from unused objects that were temporary and will never be
>>> accessed, probably accounts for a certain sizable chunk of those
>>> terabytes of growing asset storage you sometimes hear about.
>>>
>>> - Dale Mahalko / Scalar Tardis
>>>
>>>
>>> On Wed, Jun 10, 2009 at 7:35 PM, Fire<fire at b3dmultitech.com> wrote:
>>>
>>>> just wondering (and I am sure this question has been asked before)
>>>> but why, oh why, hasn't LL implemented the feature of writing to
>>>> notecards?
>>>>
>>>> Wouldnt this solve a lot of our scripting nightmares?  Ie: List memory
>>>> limits
>>>> etc?
>>>>
>>> _______________________________________________
>>> Policies and (un)subscribe information available here:
>>> http://wiki.secondlife.com/wiki/SLDev
>>> Please read the policies before posting to keep unmoderated posting
>>> privileges
>>>
>>>
>>
>>
>> _______________________________________________
>> Policies and (un)subscribe information available here:
>> http://wiki.secondlife.com/wiki/SLDev
>> Please read the policies before posting to keep unmoderated posting
>> privileges
>>
>>
>