[sldev] Cache speed experiment & results...

[Dale Mahalko]

My only concern is that the GUID distribution is may be just merely
pseudo-random.  When this system was dreamed up at the start of SL,
did anyone actually do studies of the randomness of the distributions?

I do not know if LL is using a randomness algorithm rigorously tested
by professional mathematicians and statisticians, or if it just
something Cory Ondrejka threw together one day and called good.

It is possible that the GUID distribution is not as truly random as we
may think, and many GUIDs end up falling into certain branches, sort
of like how a 3D array of dots may appear random from one angle but
end up all aligning into an even grid when rotated properly. As far as
I know, no one has yet tried caching absolutely everything LL throws
at them and just never deleting anything, and running statistics on
that, so no one outside LL knows where the GUIDs tend to fall at this
point.

Your suggested method is four levels deep, and offers up to four
billion subdirectories for GUIDs to fall into, which seems granular
enough... but what if for some reason they all end up piling up into
just a few of these branches? Then you'll end up with 50,000+ GUIDs
sitting in Cache\00\41\31\d1\* and no way to split that any deeper.

For an "ultimate, never-delete" local cache spanning terabytes of
local storage, I am in favor of allowing for splitting all the way to
the "bottom" if necessary, up to 19 directories deep, to deal with
possible pseudo-random distribution problems. At this ridiculously
deep splitting, there can only ever be 256 GUIDs per directory, at the
bottom-most level.

%CACHEDIR%\textures\55\0e\84\00\55\0e\84\00\e2\9b\41\d4\a7\16\44\66\55\44\00\00



While some people will debate the value of this vs just requesting the
data from LL directly, I am also still looking for caching options for
people with extremely limited bandwidth. This could be laying the
foundation for the design of a local network proxy/cache server.

I am aware that public schools must share a minuscule network pipe
with hundreds of students, and the only way to effectively deal with
SL in a school environment is to have some sort of local building-wide
asset-cache server.

With regard to the ISP bandwidth limiting problem also being
discussed, imagine if an ISP could buy a "Second Life Asset Appliance"
similar to the current Google Search Appliance. They could effectively
cut user traffic to LL in half or less by caching all assets locally
and transparently proxying SL asset requests, and meanwhile be able to
offer a greater quality of service to all other users of their
network..

- Scalar Tardis / Dale Mahalko


On Wed, Jun 4, 2008 at 8:13 AM, Argent Stonecutter
<secret.argent at gmail.com> wrote:
> I didn't pull this scheme out of thin air. I actively use it.
>
> There's no searching or iterating by the client needed. No empty directories
> are created. When you create an object in the cache, you do something like
> this:
>
> sprintf(filename, "%2.2s/%2.2s/%2.2s/%2.2s/%s", uuid, uuid+2, uuid+4,
> uuid+6, uuid);
> if(!(fp = fopen(filename, "w"))) {
>    filename[2] = 0; success = mkdir(filename); filename[2] = '/';
>    if(success != ERR) { filename[4] = 0; success = mkdir(filename);
> filename[4] = '/'; }
>    if(success != ERR) { filename[6] = 0; success = mkdir(filename);
> filename[6] = '/'; }
>    if(success != ERR) { filename[8] = 0; success = mkdir(filename);
> filename[8] = '/'; }
>    if(success != ERR) { fp = fopen(filename, "w"); }
>    if(!fp) { fatal(filename, strerror(errno)); }
> }