[sldev] Something is up with crash rate

Rob Lanphier robla at lindenlab.com
Tue Aug 4 13:30:48 PDT 2009

Hi folks,

One of the things I've been meaning to fix is the crash report stats
that go out on the snowglobe-automails alias.  What's currently being
mailed to that mailing list is crash reports, which doesn't give the
full range of problems that exist.  Pulling the right number is
something that I've not found the time to really automate right.

In lieu of that I've started crunching the numbers for the crash rates
for Monday, and unfortunately, the news isn't great:

Crash rates per 100 hours:

Snowglobe Release       12.26
Snowglobe Release       11.25
Snowglobe Release       13.76
Snowglobe Test Build     5.82
Snowglobe Test Build     9.57

(lower is better)

The test builds have a fraction of the usage of the release builds, so
what we're seeing with the test builds may not be indicative of the true
stability of the build.  I've attached the raw spreadsheet I used to
derive the numbers.

The main column I'm using is the "last_exec_event" column, and treating
non-zero values as a crash.  Here's the decoder ring for that column:

#*** 0 = Previous session ended normally
#*** 1 = Froze
#*** 2 = Fatal Error
#*** 3 = Crash detected
#*** 4 = Froze after logout
#*** 5 = Crash after logout

Assuming I'm making the calculation right, we've still got some work to
do here.

Unfortunately, in the crash reports, there's no standout crashers for us
to find and kill.  All of the reports I see come from two agents, max. 
Here's a couple of crashes:

Crash #1
LLTextureFetchWorker::callbackCacheRead: ASSERT (imagesize >= 0)

    * [0] LLError::crashAndLoop
          o e:/w-sweeper/robla/linden/indra/llcommon/llerror.cpp:1214
    * [1] LLError::Log::flush
          o e:/w-sweeper/robla/linden/indra/llcommon/llerror.cpp:1138
    * [2] LLTextureFetchWorker::callbackCacheRead
          o e:/w-sweeper/robla/linden/indra/newview/lltexturefetch.cpp:1233
    * [3] LLTextureFetchWorker::CacheReadResponder::completed
          o e:/w-sweeper/robla/linden/indra/newview/lltexturefetch.cpp:79
    * [4] LLTextureCache::update
          o e:/w-sweeper/robla/linden/indra/newview/lltexturecache.cpp:791
    * [5] LLAppViewer::mainLoop
          o e:/w-sweeper/robla/linden/indra/newview/llappviewer.cpp:1007
    * [6] WinMain
          o e:/w-sweeper/robla/linden/indra/newview/llappviewerwin32.cpp:228
    * [7] __tmainCRTStartup
          o f:/sp/vctools/crt_bld/self_x86/crt/src/crtexe.c:589
    * [8] kernel32.dll
    * [9] ntdll.dll
    * [10] ntdll.dll

2 agents, 4 crashes.

Crash #2:
LLVertexBuffer::setupVertexBuffer: LLVertexBuffer::setupVertexBuffer
missing required components for supplied data mask.
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLVertexBuffer::setupVertexBuffer [snowgloberelease.exe
[3] LLVertexBuffer::setBuffer [snowgloberelease.exe llvertexbuffer.cpp]
[4] LLDrawPoolBump::pushBatch [snowgloberelease.exe lldrawpoolbump.cpp]
[5] LLDrawPoolBump::renderDeferred [snowgloberelease.exe lldrawpoolbump.cpp]
[6] LLPipeline::renderGeomDeferred [snowgloberelease.exe pipeline.cpp]
[7] display [snowgloberelease.exe llviewerdisplay.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]

1 agent, 3 crashes

Anyway, this is partial data, and I'm still sifting through the raw
data.  We'll have to see if the ratios hold, and if they do, figure out
how we're going to fix our new batch of problems.  Hopefully, there's
something wrong with how I'm crunching the numbers rather than with
Snowglobe 1.1.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: crashrates-2009-08-03.ods
Type: application/vnd.oasis.opendocument.spreadsheet
Size: 12126 bytes
Desc: not available
Url : http://lists.secondlife.com/pipermail/sldev/attachments/20090804/3033f31a/attachment.ods 

More information about the SLDev mailing list