[sldev] Something is up with crash rate

[Rob Lanphier]

Hi folks,

One of the things I've been meaning to fix is the crash report stats
that go out on the snowglobe-automails alias.  What's currently being
mailed to that mailing list is crash reports, which doesn't give the
full range of problems that exist.  Pulling the right number is
something that I've not found the time to really automate right.

In lieu of that I've started crunching the numbers for the crash rates
for Monday, and unfortunately, the news isn't great:

Crash rates per 100 hours:

Snowglobe Release 1.0.2.2451       12.26
Snowglobe Release 1.0.3.2537       11.25
Snowglobe Release 1.1.2.2584       13.76
Snowglobe Test Build 1.1.0.2558     5.82
Snowglobe Test Build 1.1.0.2565     9.57

(lower is better)

The test builds have a fraction of the usage of the release builds, so
what we're seeing with the test builds may not be indicative of the true
stability of the build.  I've attached the raw spreadsheet I used to
derive the numbers.

The main column I'm using is the "last_exec_event" column, and treating
non-zero values as a crash.  Here's the decoder ring for that column:

#*** 0 = Previous session ended normally
#*** 1 = Froze
#*** 2 = Fatal Error
#*** 3 = Crash detected
#*** 4 = Froze after logout
#*** 5 = Crash after logout


Assuming I'm making the calculation right, we've still got some work to
do here.

Unfortunately, in the crash reports, there's no standout crashers for us
to find and kill.  All of the reports I see come from two agents, max. 
Here's a couple of crashes:

Crash #1
LLTextureFetchWorker::callbackCacheRead: ASSERT (imagesize >= 0)

    * [0] LLError::crashAndLoop
          o e:/w-sweeper/robla/linden/indra/llcommon/llerror.cpp:1214
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/llcommon/llerror.cpp?view=annotate#l1214>
    * [1] LLError::Log::flush
          o e:/w-sweeper/robla/linden/indra/llcommon/llerror.cpp:1138
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/llcommon/llerror.cpp?view=annotate#l1138>
    * [2] LLTextureFetchWorker::callbackCacheRead
          o e:/w-sweeper/robla/linden/indra/newview/lltexturefetch.cpp:1233
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/newview/lltexturefetch.cpp?view=annotate#l1233>
    * [3] LLTextureFetchWorker::CacheReadResponder::completed
          o e:/w-sweeper/robla/linden/indra/newview/lltexturefetch.cpp:79
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/newview/lltexturefetch.cpp?view=annotate#l79>
    * [4] LLTextureCache::update
          o e:/w-sweeper/robla/linden/indra/newview/lltexturecache.cpp:791
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/newview/lltexturecache.cpp?view=annotate#l791>
    * [5] LLAppViewer::mainLoop
          o e:/w-sweeper/robla/linden/indra/newview/llappviewer.cpp:1007
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/newview/llappviewer.cpp?view=annotate#l1007>
    * [6] WinMain
          o e:/w-sweeper/robla/linden/indra/newview/llappviewerwin32.cpp:228
            <https://svn.lindenlab.com/viewvc/viewvc.cgi/linden/branches/viewer/viewer_1-23/indra/newview/llappviewerwin32.cpp?view=annotate#l228>
    * [7] __tmainCRTStartup
          o f:/sp/vctools/crt_bld/self_x86/crt/src/crtexe.c:589
    * [8] kernel32.dll
    * [9] ntdll.dll
    * [10] ntdll.dll

2 agents, 4 crashes.

Crash #2:
LLVertexBuffer::setupVertexBuffer: LLVertexBuffer::setupVertexBuffer
missing required components for supplied data mask.
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLVertexBuffer::setupVertexBuffer [snowgloberelease.exe
llvertexbuffer.cpp]
[3] LLVertexBuffer::setBuffer [snowgloberelease.exe llvertexbuffer.cpp]
[4] LLDrawPoolBump::pushBatch [snowgloberelease.exe lldrawpoolbump.cpp]
[5] LLDrawPoolBump::renderDeferred [snowgloberelease.exe lldrawpoolbump.cpp]
[6] LLPipeline::renderGeomDeferred [snowgloberelease.exe pipeline.cpp]
[7] display [snowgloberelease.exe llviewerdisplay.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]

1 agent, 3 crashes

Anyway, this is partial data, and I'm still sifting through the raw
data.  We'll have to see if the ratios hold, and if they do, figure out
how we're going to fix our new batch of problems.  Hopefully, there's
something wrong with how I'm crunching the numbers rather than with
Snowglobe 1.1.

Rob

-------------- next part --------------
A non-text attachment was scrubbed...
Name: crashrates-2009-08-03.ods
Type: application/vnd.oasis.opendocument.spreadsheet
Size: 12126 bytes
Desc: not available
Url : http://lists.secondlife.com/pipermail/sldev/attachments/20090804/3033f31a/attachment.ods