[sldev] Cache speed experiment & results...

[Dale Mahalko]

For OS-filesystem caches permitting hundreds of thousands to millions
of cached files, perhaps a follow-behind cache optimizer process would
work better and provide more organizational efficiency.


To start a new cache, just make 256 subdirectories: Cache\00\* to Cache\ff\*

The client walks this tree when adding files to the cache and makes a
note of any directory which has exceeded some maximum performance
limit, say 10,000 GUIDs in a directory. When the user is Away, the
cache optimizer process goes to each of these flagged subdirectories,
and will split that directory into 256 subdirectories:

- Cache\4c\* was found to contain over 10,000 GUIDs, so make it
Cache\4c\00\* to Cache \4c\ff\* and move the files into their
respective subdirectories.

- At some later time, perhaps months later, Cache\4c\e3\* was found to
contain over 10,000 GUIDs so make it Cache\4c\e3\00\* to
Cache\4c\e3\ff\* and move the files into their respective
subdirectories

Once a directory level has been forked, the client walks as deep as
necessary whenever adding new GUIDs to the cache.


This would keep the number of subdirectories small and fast when there
are few files, but allows the cache to branch out into as many smaller
branches as necessary, to keep file counts per directory low and allow
fast file searching by limiting each subbranch to 256 subdirectories.

Since GUID assignments are apparently random, it would be a waste of
time to create deep subbranches before they are needed since many
branches would likely remain empty, and empty branches must be
iterated through when looking for a file.

- Scalar Tardis / Dale Mahalko


On Tue, Jun 3, 2008 at 10:03 PM, Argent Stonecutter
<secret.argent at gmail.com> wrote:
> Nobody is, so you store the texture for
>
> 550e8400-e29b-41d4-a716-446655440000
>
> In
>
> $CACHEDIR/textures/55/0e/84/00/550e8400-e29b-41d4-a716-446655440000 on
> linux, or
> %CACHEDIR%\textures\55\0e\84\00\550e8400-e29b-41d4-a716-446655440000 on
> Windows.