[sldev] Top Snowglobe crash reports

[Rob Lanphier]

Hi all,

Below are some call stacks for the top Snowglobe crash reports in
1.0.2.2451.  The headline is that SNOW-2 and SNOW-103 are our top
crashers by a longshot, but there's a few others mixed in there.

Dev updates on those two:
SNOW-2: Merov is working on this now.  We suspect that upgrading curl to
the latest version plus using the "ares" asynch resolver will fix our
crashres.  We had a real bailing wire/bubblegum approach to building
curl for Win32/Visual Studio, so much of the work right now is coming up
with a more sustainable way of maintaining our curl implementation.  We
made the pleasant discovery that Kitware had already been working on
CMake-ifying curl, so we're working with them to make sure we can build
something to spec (static library with ares enabled).

SNOW-103: Aleric has done a ton of really great legwork on this one, and
has a patch sitting in JIRA which changes the API to LLAPRFile.  One of
the developers here (Bao) has a different approach that he'd like to
take, which he should be posting to SNOW-103 later today.  That'll give
us a couple of approaches to choose from, so we should be able to nail
this one down pretty quickly.

Now for the call stacks.  I'm going to look into automating this email
so that at a minimum, I can generate it more quickly than I'm currently
capable of.  Let me know if I've ascribed the wrong issue number to the
stack trace in question:

#1
https://jira.secondlife.com/browse/SNOW-2 :
205 crashes from 114 agents:
[0] Curl_llist_insert_next [snowgloberelease.exe unknownfile]
[1] Curl_hash_add [snowgloberelease.exe unknownfile]
[2] Curl_cache_addr [snowgloberelease.exe unknownfile]
[3] curl_getdate [snowgloberelease.exe unknownfile]
[4] Curl_addrinfo4_callback [snowgloberelease.exe unknownfile]
[5] Curl_cookie_clearall [snowgloberelease.exe unknownfile]
[6] Unknown [msvcr80.dll ]
[7] Unknown [msvcr80.dll ]
[8] Unknown [ntdll.dll ]
[9] Unknown [ntdll.dll ]

#2
https://jira.secondlife.com/browse/SNOW-103 :
95 crashes from 62 agents:
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLTextureCache::openAndReadEntry [snowgloberelease.exe
lltexturecache.cpp]
[3] LLTextureCache::removeHeaderCacheEntry [snowgloberelease.exe
lltexturecache.cpp]
[4] LLTextureCache::removeFromCache [snowgloberelease.exe
lltexturecache.cpp]
[5] LLTextureFetchWorker::setupPacketData [snowgloberelease.exe
lltexturefetch.cpp]
[6] LLTextureFetch::sendRequestListToSimulators [snowgloberelease.exe
lltexturefetch.cpp]
[7] LLTextureFetch::update [snowgloberelease.exe lltexturefetch.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]
[12] Unknown [ntdll.dll ]
[13] Unknown [ntdll.dll ]


#3:
https://jira.secondlife.com/browse/SNOW-2 :
80 crashes from 55 agents
[0] Curl_llist_insert_next [snowgloberelease.exe unknownfile]
[1] Curl_hash_add [snowgloberelease.exe unknownfile]
[2] Curl_cache_addr [snowgloberelease.exe unknownfile]
[3] curl_getdate [snowgloberelease.exe unknownfile]
[4] Curl_addrinfo4_callback [snowgloberelease.exe unknownfile]
[5] Curl_cookie_clearall [snowgloberelease.exe unknownfile]
[6] Unknown [msvcr80.dll ]
[7] Unknown [msvcr80.dll ]
[8] Unknown [kernel32.dll ]
[9] Unknown [msvcr80.dll ]
[10] Unknown [msvcr80.dll ]
[11] Unknown [msvcr80.dll ]
[12] Unknown [msvcr80.dll ]
[13] Unknown [msvcr80.dll ]
[14] Unknown [msvcr80.dll ]
[15] Unknown [msvcr80.dll ]
[16] Unknown [msvcr80.dll ]
[17] Unknown [msvcr80.dll ]
[18] Unknown [msvcr80.dll ]
[19] Unknown [msvcr80.dll ]


#4
Not filed in JIRA or issue unknown:
60 crashes from 51 agents
[0] LLMutex::lock [snowgloberelease.exe llthread.cpp]
[1] LLTextureCache::readComplete [snowgloberelease.exe lltexturecache.cpp]
[2] LLTextureFetchWorker::finishWork [snowgloberelease.exe
lltexturefetch.cpp]
[3] LLQueuedThread::processNextRequest [snowgloberelease.exe
llqueuedthread.cpp]
[4] LLQueuedThread::run [snowgloberelease.exe llqueuedthread.cpp]
[5] LLThread::staticRun [snowgloberelease.exe llthread.cpp]
[6] apr_threadattr_guardsize_set [snowgloberelease.exe unknownfile]
[7] Unknown [msvcr80.dll ]

#5
https://jira.secondlife.com/browse/SNOW-103 :
55 crashes from 39 agents
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLTextureCache::openAndReadEntry [snowgloberelease.exe
lltexturecache.cpp]
[3] LLTextureCache::removeHeaderCacheEntry [snowgloberelease.exe
lltexturecache.cpp]
[4] LLTextureCache::removeFromCache [snowgloberelease.exe
lltexturecache.cpp]
[5] LLTextureFetchWorker::setupPacketData [snowgloberelease.exe
lltexturefetch.cpp]
[6] LLTextureFetch::sendRequestListToSimulators [snowgloberelease.exe
lltexturefetch.cpp]
[7] LLTextureFetch::update [snowgloberelease.exe lltexturefetch.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]

#6
Not filed in JIRA or issue unknown:
53 crashes from 49 agents
[0] LLMutex::lock [snowgloberelease.exe llthread.cpp]
[1] LLTextureCache::readComplete [snowgloberelease.exe lltexturecache.cpp]
[2] LLTextureFetchWorker::deleteOK [snowgloberelease.exe lltexturefetch.cpp]
[3] LLTextureFetch::update [snowgloberelease.exe lltexturefetch.cpp]
[4] LLAppViewer::cleanup [snowgloberelease.exe llappviewer.cpp]
[5] LLAppViewerWin32::cleanup [snowgloberelease.exe llappviewerwin32.cpp]
[6] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[7] Unknown [kernel32.dll ]

#7
Not filed in JIRA or issue unknown (related to SNOW-103?):
52 crashes from 35 agents
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLTextureCache::writeEntryAndClose [snowgloberelease.exe
lltexturecache.cpp]
[3] LLTextureCache::removeHeaderCacheEntry [snowgloberelease.exe
lltexturecache.cpp]
[4] LLTextureCache::removeFromCache [snowgloberelease.exe
lltexturecache.cpp]
[5] LLTextureFetchWorker::setupPacketData [snowgloberelease.exe
lltexturefetch.cpp]
[6] LLTextureFetch::sendRequestListToSimulators [snowgloberelease.exe
lltexturefetch.cpp]
[7] LLTextureFetch::update [snowgloberelease.exe lltexturefetch.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]
[12] Unknown [ntdll.dll ]
[13] Unknown [ntdll.dll ]


#8
Not filed in JIRA or issue unknown:
31 crashes from 21 agents
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[3] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[4] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[5] Unknown [kernel32.dll ]
[6] Unknown [ntdll.dll ]
[7] Unknown [ntdll.dll ]

#9
Not filed in JIRA or issue unknown:
31 crashes from 7 agents
[0] Unknown [nvoglnt.dll ]
[1] Unknown [nvoglnt.dll ]
[2] Unknown [nvoglnt.dll ]
[3] Unknown [nvoglnt.dll ]
[4] Unknown [nvoglnt.dll ]
[5] LLDrawPoolAvatar::render [snowgloberelease.exe lldrawpoolavatar.cpp]
[6] LLPipeline::renderGeom [snowgloberelease.exe pipeline.cpp]
[7] display [snowgloberelease.exe llviewerdisplay.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]

#10
Not filed in JIRA or issue unknown (looks pretty close to #8):
29 crashes from 19 agents
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[3] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[4] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[5] Unknown [kernel32.dll ]

#11:
Not filed in JIRA or issue unknown (looks pretty close to #7):
[0] LLError::crashAndLoop [snowgloberelease.exe llerror.cpp]
[1] LLError::Log::flush [snowgloberelease.exe llerror.cpp]
[2] LLTextureCache::writeEntryAndClose [snowgloberelease.exe
lltexturecache.cpp]
[3] LLTextureCache::removeHeaderCacheEntry [snowgloberelease.exe
lltexturecache.cpp]
[4] LLTextureCache::removeFromCache [snowgloberelease.exe
lltexturecache.cpp]
[5] LLTextureFetchWorker::setupPacketData [snowgloberelease.exe
lltexturefetch.cpp]
[6] LLTextureFetch::sendRequestListToSimulators [snowgloberelease.exe
lltexturefetch.cpp]
[7] LLTextureFetch::update [snowgloberelease.exe lltexturefetch.cpp]
[8] LLAppViewer::mainLoop [snowgloberelease.exe llappviewer.cpp]
[9] WinMain [snowgloberelease.exe llappviewerwin32.cpp]
[10] __tmainCRTStartup [snowgloberelease.exe crtexe.c]
[11] Unknown [kernel32.dll ]

#12:
Not filed in JIRA or issue unknown:
28 crashes from 16 agents
[0] Unknown [kernel32.dll ]
[1] Unknown [msvcr80.dll ]
[2] Unknown [msvcr80.dll ]
[3] LLImageBase::allocateData [snowgloberelease.exe llimage.cpp]
[4] LLImageRaw::allocateData [snowgloberelease.exe llimage.cpp]
[5] LLImageRaw::LLImageRaw [snowgloberelease.exe llimage.cpp]
[6] LLImageDecodeThread::ImageRequest::processRequest
[snowgloberelease.exe llimageworker.cpp]
[7] LLQueuedThread::processNextRequest [snowgloberelease.exe
llqueuedthread.cpp]
[8] LLQueuedThread::run [snowgloberelease.exe llqueuedthread.cpp]
[9] LLThread::staticRun [snowgloberelease.exe llthread.cpp]
[10] apr_threadattr_guardsize_set [snowgloberelease.exe unknownfile]
[11] Unknown [msvcr80.dll ]
[12] Unknown [ntdll.dll ]
[13] Unknown [ntdll.dll ]