[opensource-dev] J2C fast decoder

Tateru Nino tateru at taterunino.net
Mon Sep 13 00:24:08 PDT 2010

  Wouldn't we be making a network saving by omitting the discard levels 
entirely? Granted, I don't have hard data about that - would the base 
texture encoded in a lighter-weight format end up causing less data to 
traverse for a given texture in the long-run than the more-efficiently 
compressed j2c of the same texture including discard levels? My gut 
instinct says 'probably', but I can't prove that with data.

If it *does* then we would have a double-bonus of also saving on 
decoding time.

On 13/09/2010 4:38 PM, Dahlia Trimble wrote:
> Jpeg 2000 discard levels are also used for reducing the resolution of 
> textures for distant objects which reduces data download requirements. 
> Few other formats offer comparable compression ratios with the quality 
> that Jpeg 2000 offers. HTTP transfer doesn't magically make data 
> traverse the network faster; much of the reduced download time is due 
> to offloading the sim from the task of sending textures as they can 
> come from another server process (or even another physical server).
> On Sun, Sep 12, 2010 at 10:40 PM, Tateru Nino <tateru at taterunino.net 
> <mailto:tateru at taterunino.net>> wrote:
>     If we're using HTTP textures, is there actually any need for the
>     JPEG 2000 format? Since the transfer time of individual textures
>     is vastly reduced (from the first byte to the last byte) the
>     intermediate quality levels supported by jpg2k would seem to be
>     redundant. Indeed, you could argue that transferring the textures
>     in jpg2k format imposes a now-redundant workload on the
>     texture-pipeline, and that providing HTTP textures in a simpler
>     format that is more tractable to high-speed, low-cost decoding
>     would save a whole lot of problems.
>     Would it be a huge problem, for example, to transfer HTTP textures
>     as TGA or PNG and use one of the rather well-optimized decoder
>     libraries for those instead? It seems to me that it would be more
>     efficient both on the network and on the system - though at the
>     expense of conversion of all the textures at the store.
>     Just thinking out loud.
>     On 13/09/2010 1:58 PM, Sheet Spotter wrote:
>>     Some comments on SNOW-361 (Upgrade to OpenJPEG v2) suggested that
>>     changing to version 2 of OpenJPEG might improve performance,
>>     while other comments suggested it might not support progressive
>>     decoding.
>>     http://jira.secondlife.com/browse/SNOW-361
>>     Is an upgrade to OpenJPEG v2 under active development?
>>     Sheet Spotter
>>     ------------------------------------------------------------------------
>>     *From:* opensource-dev-bounces at lists.secondlife.com
>>     <mailto:opensource-dev-bounces at lists.secondlife.com>
>>     [mailto:opensource-dev-bounces at lists.secondlife.com] *On Behalf
>>     Of *Philippe (Merov) Bossut
>>     *Sent:* September 9, 2010 10:35 PM
>>     *To:* Nicky Fullton
>>     *Cc:* opensource-dev at lists.secondlife.com
>>     <mailto:opensource-dev at lists.secondlife.com>
>>     *Subject:* Re: [opensource-dev] J2C fast decoder
>>     Hi Nicky,
>>     As it happens, I've been working on instrumenting the code to add
>>     metric gathering for image decompression as part of the Snowstorm
>>     sprint.
>>     You may want to use my branch
>>     (https://bitbucket.org/merov_linden/viewer-development-vwr-22761)
>>     and create a baseline for openjpeg then run a test for Jasper.
>>     You'll have to sort out the failing cases certainly and just
>>     throw them so we compare what gets truly decompressed (though,
>>     clearly, working in all cases is pretty critical if we look at
>>     Jasper as an alternative).
>>     Here's what I got comparing KDU and OpenJpeg:
>>     Label     Metric                              KDU(B)   
>>      OJ2C(T)     Diff(T-B)     Percentage(100*T/B)
>>     ImageCompressionTester-1
>>          TotalBytesInDecompression    5048643    5003370   
>>     -45273        99.1
>>          TotalBytesOutDecompression 40415336  46592896    6177560   
>>     115.29
>>          TimeTimeDecompression        3.74           17.04         
>>     13.3            455.39
>>     ImageCompressionTester-2
>>          TotalBytesInDecompression    5000744    5000144     -600   
>>            99.99
>>          TotalBytesOutDecompression 46440040  44248324   -2191716   
>>     95.28
>>          TimeTimeDecompression        3.64           15.02          
>>     11.37         412.02
>>     For that test, I output data every time 5MB of compressed data
>>     have been processed. It's partial but shows that OpenJpeg is
>>     roughly 4 times slower than KDU (at least, the version we're
>>     using in the official viewer currently). Would be nice to have a
>>     similar set of numbers for Jasper before going too far down the
>>     implementation path.
>>     I wrote a short (and still incompleted) wiki to explain a bit how
>>     the metric gathering system works:
>>     - https://wiki.secondlife.com/wiki/Performance_Testers
>>     BTW, that's something we should be using more generally for other
>>     perf sensitive areas, especially when starting a perf improvement
>>     project.
>>     See http://jira.secondlife.com/browse/VWR-22761 for details.
>>     Cheers,
>>     - Merov
>>     On Fri, Sep 3, 2010 at 9:05 AM, Nicky Fullton
>>     <nickyd_sl at yahoo.com <mailto:nickyd_sl at yahoo.com>> wrote:
>>     Hello,
>>     >> i'm testing in RL office (not or a viewer) JasPer decoder for
>>     JPG2000
>>     >> images, after a short test with openjpeg2000 from EPFL we have
>>     tested
>>     >> last 3 days JasPer (only a POC apps to do some bench), we must
>>     do a lot
>>     >> of work too, but this is a lil question... anybody here around
>>     never
>>     >> tried it as alternative to OpenJPEG/KDU in a viewer?
>>     >I'm not aware of anyone publishing results for such a test, but
>>     if you
>>     >have the time it would be interesting reading.
>>     You might be interested in:
>>     http://bitbucket.org/NickyD/viewer-development/changeset/027bf44c5582
>>     I made a rather quick hack to try Jasper instead of OpenJpeg to
>>     decode
>>     images.
>>     The patch has some very rough edges. In fact is the decoding into the
>>     LLImageRaw buffer not correct.
>>     I did not fix this (yet) because the results so far are not very
>>     promising.
>>     Jasper can only decode around 20% of the jpeg, for the other 80%
>>     it will
>>     create an error and then my code falls back to OpenJpeg.
>>     This fallback makes the whole decoding rather slow, so it is hard
>>     to say
>>     if Jasper would really be any faster.
>>     Right now I am not sure if it would be reasonable to invest more time
>>     looking at Jasper. First the code would need to fixed upstream,
>>     so all
>>     images can be properly decoded. As this project looks rather
>>     dead, one
>>     with JPEG2000 knowledge might have to step up for this.
>>     On another note, you might like to try:
>>     http://bitbucket.org/NickyD/viewer-development/changeset/e4eff3e2af39
>>     This will at least skip the step of calling OpenJpeg in
>>      LImageJ2COJ::getMetadata (if possible, it will do sanity checks
>>     first).
>>     >Some things to keep in
>>     >mind. OpenJpeg has patches floating around on its ML against 1.3 that
>>     >reports have claimed up to 40% speed increase in places due to
>>     >unrolling the inner loops so finding them and testing would be good.
>>     I did not find any of those, but then again maybe I did not look hard
>>     enough.
>>     There is certainly some potential in OpenJpeg.
>>     There are some loops in t1_dec_sigpass and t1_dec_refpass that can be
>>     easily rewritten. But there is some pretty tricky stuff in
>>     t1_dec_clnpass
>>     that would need some cleaning and mqc decoder (mqc_decode) burns
>>     a lot
>>     of time. But that one is especially hairy as it has side effects
>>     on its
>>     input parameter.
>>     I am not sure if anyone without enough deep knowledge of OpenJpeg
>>     (and
>>     the dedication to recode a good part of it) would be able to improve
>>     much of it.
>>     Cheers,
>>       Nicky
>>     _______________________________________________
>>     Policies and (un)subscribe information available here:
>>     http://wiki.secondlife.com/wiki/OpenSource-Dev
>>     Please read the policies before posting to keep unmoderated
>>     posting privileges
>>     _______________________________________________
>>     Policies and (un)subscribe information available here:
>>     http://wiki.secondlife.com/wiki/OpenSource-Dev
>>     Please read the policies before posting to keep unmoderated posting privileges
>     -- 
>     Tateru Nino
>     Contributing Editorhttp://massively.com/
>     _______________________________________________
>     Policies and (un)subscribe information available here:
>     http://wiki.secondlife.com/wiki/OpenSource-Dev
>     Please read the policies before posting to keep unmoderated
>     posting privileges
> _______________________________________________
> Policies and (un)subscribe information available here:
> http://wiki.secondlife.com/wiki/OpenSource-Dev
> Please read the policies before posting to keep unmoderated posting privileges

Tateru Nino

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.secondlife.com/pipermail/opensource-dev/attachments/20100913/6ff2c3bd/attachment-0001.htm 

More information about the opensource-dev mailing list