[sldev] Cache speed experiment & results...

[Matthew Underwood]

On Tue, Jun 3, 2008 at 8:45 PM, Dahlia Trimble <dahliatrimble at gmail.com> wrote:
> If the goal were to improve seek time in a cache, how about keeping the
> cache on one of those readyboost-capable usb flash drives? They can have
> seek times as low as 1 ms or better, although the transfer rate and write
> times arent all that fast.
>
> Then again it's probably moot if the bottleneck is the decoding :/
>
> On Tue, Jun 3, 2008 at 5:12 PM, Felix Duesenburg <kfa at gmx.net> wrote:
>>
>> Dale Mahalko wrote:
>>>
>>> ...
>>> Windows isn't good at dealing with directories containing tens of
>>> thousands of files. Ever tried opening a folder containing 20,000
>>> files? There is a delay of several seconds trying to do this in
>>> Windows. Also the hidden MFT and directory structures can become
>>> ridiculously fragmented, also leading to slowdowns.
>>>
>>
>> Reiser4 is excellent with exactly these requirements. It's not mature
>> enough to be considered reliable for critical systems, but for a cache? It
>> doesn't help that Reiser himself is removed from the development process,
>> but it's still there and the source available. Shouldn't that be possible to
>> be included as a virtual filesystem even under Windows?
>>
>> I also always thought it must be a good way to boost performance if the
>> cache lived on its own partition, like a Linux swap partition. For the Linux
>> version of SL this would even sound like the most natural way to do it.
>> Setting such a thing up on an existing Windows machine has a slight fear
>> factor attached and is not for everyone, but if a savvy user can handle it,
>> why not offer. I'm speculating, but the difference could be dramatic,
>> especially if we had a better file system than NTFS on that partition.
>>
>> >
>> > A proper local database engine would likely deal with these huge
>> > numbers of tiny files more efficiently than the local operating
>> > system.
>>
>> Yes, but the usual architecture with the database running as a localhost
>> server is not ideal for our situation since everything is squeezed back and
>> forth through the network system, plus SQL parsing, unnecessarily. With
>> MySQL you could build directly on top of the storage engine though, even
>> bypassing the parsing with direct API calls. It would just serve as a
>> virtual file system the same way as suggested above. I still believe that
>> Reiser4 would be even faster. And according to the description on its (now
>> defunct) website, they claimed reaching 94% package density with small
>> files.
>>
>> Felix

SQLite in a single user environment is incredibly fast so long as you
don't let the database grow too large.  Of course you have to figure
out how efficiently textures can be stored and retrieved in SQL
compared to the current caching system.

On a side note: Does anyone know where it defines the polygon meshes
for rendering cubes.  I really want to try forcing untwisted, un-holed
cubes down to 1-2 polygons per side and see if that will help out with
the frame rate.  I tried looking through the source but it is lacking
in [helpful] comments.