[sldev] Cache speed experiment & results...
Felix Duesenburg
kfa at gmx.net
Tue Jun 3 17:12:00 PDT 2008
Dale Mahalko wrote:
> ...
> Windows isn't good at dealing with directories containing tens of
> thousands of files. Ever tried opening a folder containing 20,000
> files? There is a delay of several seconds trying to do this in
> Windows. Also the hidden MFT and directory structures can become
> ridiculously fragmented, also leading to slowdowns.
>
Reiser4 is excellent with exactly these requirements. It's not mature
enough to be considered reliable for critical systems, but for a cache?
It doesn't help that Reiser himself is removed from the development
process, but it's still there and the source available. Shouldn't that
be possible to be included as a virtual filesystem even under Windows?
I also always thought it must be a good way to boost performance if the
cache lived on its own partition, like a Linux swap partition. For the
Linux version of SL this would even sound like the most natural way to
do it. Setting such a thing up on an existing Windows machine has a
slight fear factor attached and is not for everyone, but if a savvy user
can handle it, why not offer. I'm speculating, but the difference could
be dramatic, especially if we had a better file system than NTFS on that
partition.
>
> A proper local database engine would likely deal with these huge
> numbers of tiny files more efficiently than the local operating
> system.
Yes, but the usual architecture with the database running as a localhost
server is not ideal for our situation since everything is squeezed back
and forth through the network system, plus SQL parsing, unnecessarily.
With MySQL you could build directly on top of the storage engine though,
even bypassing the parsing with direct API calls. It would just serve as
a virtual file system the same way as suggested above. I still believe
that Reiser4 would be even faster. And according to the description on
its (now defunct) website, they claimed reaching 94% package density
with small files.
Felix
More information about the SLDev
mailing list