[sldev] Cache speed experiment & results...

Felix Duesenburg kfa at gmx.net
Tue Jun 3 17:12:00 PDT 2008


Dale Mahalko wrote:
> ...
> Windows isn't good at dealing with directories containing tens of
> thousands of files. Ever tried opening a folder containing 20,000
> files? There is a delay of several seconds trying to do this in
> Windows. Also the hidden MFT and directory structures can become
> ridiculously fragmented, also leading to slowdowns.
> 

Reiser4 is excellent with exactly these requirements. It's not mature 
enough to be considered reliable for critical systems, but for a cache? 
It doesn't help that Reiser himself is removed from the development 
process, but it's still there and the source available. Shouldn't that 
be possible to be included as a virtual filesystem even under Windows?

I also always thought it must be a good way to boost performance if the 
cache lived on its own partition, like a Linux swap partition. For the 
Linux version of SL this would even sound like the most natural way to 
do it. Setting such a thing up on an existing Windows machine has a 
slight fear factor attached and is not for everyone, but if a savvy user 
can handle it, why not offer. I'm speculating, but the difference could 
be dramatic, especially if we had a better file system than NTFS on that 
partition.

 >
 > A proper local database engine would likely deal with these huge
 > numbers of tiny files more efficiently than the local operating
 > system.

Yes, but the usual architecture with the database running as a localhost 
server is not ideal for our situation since everything is squeezed back 
and forth through the network system, plus SQL parsing, unnecessarily. 
With MySQL you could build directly on top of the storage engine though, 
even bypassing the parsing with direct API calls. It would just serve as 
a virtual file system the same way as suggested above. I still believe 
that Reiser4 would be even faster. And according to the description on 
its (now defunct) website, they claimed reaching 94% package density 
with small files.

Felix


More information about the SLDev mailing list