Holding the meeting in voice already excludes people. The last time I tried to enable voice for a meeting with a Linden I couldn't actually get it to work. The dot came up, and my computer was receiving data because his dot was showing activity, but I couldn't hear anything.