[sldev] Reformatting Textures for the cache

[Dale Glass]

В сообщении от 22 марта 2007 13:47 Tim Shephard написал(a):
> >  even sqlite is probably overkill.
>
> I suppose we could write a highly optimized lookup system from
> scratch, I doubt it'll buy us much though.
>
> But there might be high quality open source alternatives out
> there... any suggestions?

Filesystems have several problems:

1. Inefficient way of locating files on the disk. Say, ext2 in its 
original incarnation (and FAT too I think) scan the directory from 
top to bottom when looking for a file. This is bad when you have 
several thousand files in there.

Solution 1: /cache/6c/48/94/6c489466-3058-6475-6b1b-e5fc1d49f1f3.tga
Solution 2: Use a FS that handles this in a sane way, like ReiserFS 
(or I think NTFS)

2. Fragmentation

Solution 1: Run a defragmenter
Solution 2: Use a FS that doesn't fragment so much
Solution 3: Use Flash or RAM, where fragmentation is irrelevant

3. Overhead. Filesystems, as the rather general things they are, carry 
overhead that may be higher than that of a specialized solution.

Solution 1: Use the store-in-file method, mostly bypassing the FS
Solution 2: See below


My idea:

Keep the current system of storing images files on disk. Probably keep 
them compressed too.

Add an index. Index would be a list of key/values. Key is the image's 
key, of course, with the value being metadata. Metadata would 
include:

* Filesize
* MD5 sum
* Very rough texture approximation
* Very rough alpha layer approximation

My suggestion for the index would be using 
http://en.wikipedia.org/wiki/Berkeley_DB which is a very simple DB 
that maps keys to values, but provides the benefit of being ACID 
compliant, so it won't have integrity issues.

This index would be very small and compact, and have a fixed row size. 
It would be small enough to load into RAM entirely. By using the 
index it'd be possible to instantly check whether a file is in the 
cache, thus avoiding trying to open a file that may not exist.

Filesize and MD5 sum would be stored for integrity purposes, to make 
sure that the file on disk is complete and not corrupted (with may 
happen if the client crashes). This would ensure that the cache is 
always good, and avoids any need to clean the cache as 
troubleshooting.

Texture and alpha approximations would be used as a texture 
replacement while it's not loaded. This could be reduced to 2 bytes 
for each: simply average the whole image, and you get its overall 
color. Small, and much better than grey everywhere.


Additional idea: Add a position/texture database. A rough idea for 
this is: For each sim, take the region coordinates, round them to 
say, multiples of 16m, and store under that key a list of textures in 
that area. To avoid fragmentation issues, the amount of textures per 
slot could be limited to a fixed number.

Say:

(120,90,56) -> (112, 80, 48)

This will allow to get a rough list of textures used in an area. This 
would be used to start preloading textures before the grid transmits 
the actual data. Then when idle, the client can start decoding 
textures that are used nearby, but which you're not seeing yet, so 
that you could start flying forward and have things already decoded 
in advance.

How does that sound?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.secondlife.com/pipermail/sldev/attachments/20070322/446d0caa/attachment.pgp