[sldev] How to solve this bug? (OpenAL blocking)

[Rob Lanphier]

Hi folks,

It'd be really good to know if this problem is driver dependent, and I
think it's a fine suggestion to research it. I also think that we should
avoid being hasty about busting things off into independent threads.

I'm not entirely sure that the conclusion of such an investigation is
necessarily leave OpenAL on the main thread. Since shoddy drivers are a
perennial problem on all platforms, it seems prudent to isolate the pain
caused by those drivers if it's practical. Having a separate thread
isolates the consequences of a bad driver to the audio handling, which
seems like the ideal solution, since the end-user symptoms would
presumably change from "poor framerate" to "choppy audio", which makes
it easier to correctly diagnose the problem as an audio problem.

So, I'm not saying we should rush out and implement AL handling on a
separate thread, but I think it's worth keeping the idea on the table
after we have more information.

Rob

On 07/31/2009 05:51 AM, Tofu Linden wrote:
> If I understand your description correctly -
>
> I think the bug is that OpenAL should NOT be mixing sources that are
> not in a playing state.  That would be an AL bug (especially since
> sources are traditionally considered a 'cheap' resource in AL as
> long as they're not doing anything).
>
> Now, if we're leaving those sources in a playing state but just
> feeding silence into them - that would be a viewer bug.
>
> I don't think that moving AL calls out of the main thread is
> the correct solution - (most) AL calls are not expected to significantly
> block in the first place, and this sounds like it is AL's own
> calls waiting for locks on AL's own backend.  This would bite
> any AL-using app - this AL implementation should be fixed.
>
> I suggest investigating whether the viewer is doing the right thing
> (stopping non-playing sources) and if so, taking investigation over
> to the problematic AL implementation (openal-soft?).
>
> Thanks!
>
> Aleric Inglewood wrote:
>   
>> I think I finally figured out what caused the slow FPS (in my case).
>> I created a new jira for this because it is impossible to know
>> (from the data provided by others) if they are suffering from the
>> same problem, or that their problem is (totally) different.
>>
>> jira: http://jira.secondlife.com/browse/VWR-14914
>>
>> To summarize the problem in this mail:
>>
>> * Viewer often works OK (I have FPS up to 80 at time, when
>>   looking at nothing; or 30 when looking at -say- my house).
>>
>> * Sometimes the FPS drops dramatically. This can be anywhere
>>   between 0.3 and -say- 4 or 5, but I after my investigation
>>   my conclusion is that this bug/problem could in principle
>>   cause any delay (ie, a drop from 30 to 20, or 10).
>>
>> My conclusions so far are as follows:
>>
>> * The FPS drops because the main thread is waiting for a lock
>>   inside the libopenal library with increasing times up till
>>   an average of 0.2 seconds (sometimes less sometimes more),
>>   and several/many such calls are being made per frame.
>>   This happens, obviously, during calls to libopenal (in
>>   indra/llaudio/audioengine_openal.cpp).
>>
>> * The reason it cannot obtain the lock quickly, while normally
>>   it can, is because the lock is being help almost 100% of
>>   the time by the audio thread, and although the audio thread
>>   is not holding the lock for more than 400 microseconds at
>>   a time, it is *immediately* taking the lock again without
>>   calling a cancelation functions (or so is my guess), therefore
>>   not giving other threads the chance to 'take over'. At any
>>   rate it is clear that the audio thread had become so busy
>>   with "mixing" the audio channels that it stops waiting
>>   between releasing the lock and obtaining it again, making
>>   it very hard for the main thread to get the lock.
>>
>> To understand this in more detail, here is how the audio
>> thread works:
>>
>> * The audio thread runs entirely inside libopenal.
>>   It loops, normally waiting for the sound card to have
>>   digested the last sound batch, and when the buffer
>>   is empty enough (but hopefully not entirely empty yet)
>>   it calls aluMixData which locks the audio context object
>>   and starts adding up all audio channels.
>>
>> * Normally aluMixData is called 2757 times per second,
>>   this probably has to do with the number of samples
>>   per second in the audio used (so, it's stable).
>>
>> * Mixing channels takes about 10 microseconds per channel
>>   (on my machine, with a custom libopenal compiled without
>>   optimization; but the bug is the same thing with different
>>   figures, so lets just use this as example).
>>
>> From this we can conclude that the audio thread has no
>> time anymore to idle once it has to mix 1/2757 / 10*10^-6 =
>> 36 channels. The maximum number of channels is 31, but
>> for some reason, during the bug, the time to mix the channels
>> is more like 20 microseconds. I did not investigate why
>> that is the case.
>>
>> My conclusion is that it is "normal" for the audio thread
>> of have to mix 31 channels, which causes it to be so busy
>> that the main thread starts stalling, which you can see
>> in the frame console an increasing "audio" bar.
>>
>> At some point it can become increasingly difficult for
>> the main thread to get the lock, causing it to stall longer
>> and longer up till 0.5 seconds (0.2 average) several times
>> per frame, causing an FPS as low as 0.3.
>>
>> *************************************************************
>>
>> As a side note, once my viewer gets into this state I cannot
>> get out of it anymore without a relog. My hypothesis is that
>> this is caused by two factors:
>>
>> 1) The number of audio channels is never decreased(!), not
>>    even after a teleport.
>>
>> 2) Once in this busy state, mixing a channel takes on average
>>    20 microseconds, so that it gets "sucked" into the state
>>    and won't fall out of it anymore.
>>
>> The reason for point 1 is that an audio channel is
>> registered with libopenal when it is created, and never released
>> or removed anymore (imho, we're mixing silence in that case).
>>
>> If some sound source gets out of range so that it stops playing
>> the source is marked as "not playing" but libopenal is still
>> mixing it. When a new source comes into range, a not-playing
>> source is REUSED. So, you will ONLY get more than (say) 20
>> sources when you visit a sim with 20 sources that are simultaneously
>> active (at 20 it starts to visibly delay my FPS), but you can
>> never get rid of them anymore; not even if you return to a sim
>> with just 4 sound sources.
>>
>> *************************************************************
>>
>> The question is: how to fix this problem?
>>
>> There are a few options:
>>
>> * Remove channels from libopenal when they are non-playing.
>>
>> This wouldn't solve the problem, but at least you can recover
>> in that case by TP-ing or walking away from the many sound
>> sources.
>>
>> * Do not call libopenal functions from the main thread.
>>
>> This solution is my favourite, it would require a new thread
>> that handles the interface of LLAudioEngine: each of it's
>> virtual function will have to be changed to passing requests
>> to a queue of this new thread, which then will do the
>> actual call.
>>
>> I'm sure I won't have problems coding this, but I want to
>> ask the list for their opinion before I start to implement
>> a change of this magnitude :)
>> _______________________________________________
>> Policies and (un)subscribe information available here:
>> http://wiki.secondlife.com/wiki/SLDev
>> Please read the policies before posting to keep unmoderated posting privileges
>>     
> _______________________________________________
> Policies and (un)subscribe information available here:
> http://wiki.secondlife.com/wiki/SLDev
> Please read the policies before posting to keep unmoderated posting privileges
>