[sldev] Re: Reformatting Textures for the cache

[David Baker]

I couldn't disagree more.  In redesigning the cache system we must presume 
that the user we are writing for is running a hugely fragmented FAT32 drive 
and a file open in a 10,000 file directory will take 10,000 times longer 
than a file open in a 1 file directory (so maybe I exaggerate).

Sorry to burst your bubble, but MacOS, Linux, BSD and all the rest are tiny 
minority operating systems compared to Windows XP.  And will NTFS might be 
better than FAT32 for performance, Windows XP doesn't even encourage users 
to use it - FAT32 and NTFS are presented as equal options - so we have to 
presume that maybe 40-50% of SL users will be running FAT32.  Consequently 
any option that is non-optimal for FAT32 should be seen as a non-starter - 
which I suspect includes most direct file system options.

If we want to rely on file system functionality we are going to have to look 
at file-system-in-a-file solutions - perhaps using something third party 
rather than dusting off the VFS - but not relying on direct file system 
storage unless we can prove that that is the best option _on FAT32_.

David

> Message: 2
> Date: Mon, 26 Mar 2007 04:36:00 -0500
> From: Callum Lerwick <seg at haxxed.com>
> Subject: Re: [sldev] Re: Reformatting Textures for the cache
> To: sldev at lists.secondlife.com
> Message-ID: <1174901760.4609.21.camel at max.booze>
> Content-Type: text/plain; charset="us-ascii"
>
> On Mon, 2007-03-26 at 01:09 -0700, Kamilion wrote:
>> Has anyone considered perhaps using an uncompressed common container
>> format, like perhaps .tar? It should be fully platform independent,
>> and reduce the number of open file handles, and may improve I/O
>> performance by only having a group of hundreds of tar files instead of
>> potentially millions of independent texture files?
>
> Is it still not clear that reinventing the wheel here is not a
> productive idea? Because that's what you're doing. You're reinventing a
> filesystem on top of a filesystem. Hundreds of thousands of man hours
> have gone into refining and optimizing filesystem efficiency in Linux
> and BSD and *cough* other operating systems, based upon decades of
> accumulated research and experience in the field of computer science.
>
> Dealing with thousands (Come on now, my entire 95% full 250gb disk has
> about a million files on it *total*) of independent textures quickly and
> efficiently is exactly the kind of thing filesystems are *designed for*.
> (Well, except maybe FAT but anyone still using FAT deserves what they
> get...)