[sldev] [LSL] [POSTFIX PROBLEMS] Object Email Receive from external domain problems

[Kamilion]

Replies in-line.


On 6/11/07 at 10:22AM, Kelly Washington <kelly at lindenlab.com> wrote:
>
>  We are currently (this minute, really) investigating emails from outside to
> LSL being broken.

Yup, I just received a very kind IM from James Linden:
[10:09] James Linden: Yes, inbound email to scripts appears to be
broken.  It looks like our  mail queue got flooded by email from some
outside domain.  Operations is clearing the queue now.  Thanks for
pointing this out!  Don Linden is leading the recovery efforts.

This is why I love the SLDev ML -- Signal to noise is great, the
lindens actually listen because the list is low traffic, and the
people here are awesome!

Three cheers for Kelly, James, and Don!
And congrats, Soft!
>
>  Some more comments about communication between LSL and outside SL:
>
>   - All XML-RPC requests go through a central server.  That 'server' is
> really an apache host running a C++ cgi.  This cgi module looks up object
> presence info (what host it is on) based on the channel ID and routes the
> message accordingly.  This server is consistently overloaded although I
> believe we upgraded it, again, recently.  As a central point of failure it
> will never scale as well as needed, and should eventually be deprecated.

How hard would it be to deploy this CGI on more than one server?
Anotherwords, where is the bottleneck here, CPU, Memory, network, or
presance database?

Technically, you guys should be able to bring a second server online
called, for instance, xmlrpc2. New projects could use the second
server and ignore the primary; or a round robin could be set up to
load balance between them, but just the act of setting up a second
unloaded server could mitigate a lot of the problem; Theoretically it
should be simple to rig the simulator's LSLVM to remember which XMLRPC
server the incoming request came in from and reply via the same.

>
>   - All email from outside goes through a single server that maps to
> lsl.secondlife.com.  The emails are passed to a perl script that checks some
> edge cases and looks up presence info (what host the object is on) in
> backbone (the python service that hosts agent presence as well).  However,
> objects only have presence in backbone if they have called llGetNextEmail at
> least once; if there is no presence info for the object or the object is not
> where we think it is then it is put in the database and retrieved the next
> time the object calls llGetNextEmail.  However, like xml-rpc email from
> outside SL to LSL goes through a central server that will never scale as
> well as needed and will eventually need to be deprecated or changed.  By
> changed I mean for example requiring region information in addition to
> object ID in the email address used.  (Email from lsl->lsl cheats, is never
> really an email and avoids the problems mentioned above except the presence
> part.  If presence is broken then lsl->lsl email will fail unless the
> objects are on the same sim)

Hm, This looks like another set of systems that could use some redundancy.

James said the email server had been flooded. Are you guys using rate
control to process incoming mail only, or do you have additional
services deployed on that box like SpamAssassin or another Realtime
Blackhole List for ignoring evil domains?

Most proper domains have SPF set up, and if they don't, it's
relatively simple to do if you have access to your DNS provider.
(Personally, I use afraid.org 's free DNS, they rock!)

>
>  *NOTE: While I used the word 'deprecated' above we currently have no solid
> plans to do so, I am not saying they are already deprecated and we will work
> with and communicate with the community a LOT before we move ahead with any
> plans related to that.  Additionally we expect more communication
> alternatives to be in place before then.  We are still supporting these
> methods and as I mentioned in the first line we are currently investigating
> the current email issues.
>
>   - llHTTPRequest is the bomb.  Really.  It is lightweight and distributed.
> The entire action happens on the host the region with the object is on.
> There is no central server, no required presence registration.  Lots and
> lots of requests can be done at the same time with essentially no effect on
> the rest of the simulator process.   Yes there have been bugs before but
> most of those are now resolved.  It is much better to have an object ping a
> server every second with an llHTTPRequest than it is use either XML-RPC or
> email->lsl.

Yeah, llHTTPRequest has been my savior since it was introduced; it
makes coding PHP a breeze!

Here's a little trick for y'all out there in PHP land:

<?php 	session_name("SessID"); session_start(); // Session Startup & headers
	if($_GET['debug'] == TRUE) { $_SESSION['debug'] = TRUE; } else {
header("Content-type: text/plain"); } // If we're in debug mode, leave
HTML on.
	if($_GET['destroy'] == TRUE){if(isset($_COOKIE[session_name()]))
{setcookie(session_name(), session_id(), 1, '/');} }// Kill Client's
Cookie(s) if we're destroying the session.
	//echo session_name(), "=", session_id(); // Print the SessionID for
the script to use
?>
YOUR CODE GOES HERE
<?php // Session Destruction
	if($_GET['destroy'] == TRUE) {echo "\n<br>Destroying Session.<br>";
// Wipe & Destroy session if asked.
	$_SESSION = array(); session_destroy();} // Wipe our Session
Variables & Destroy our on-disk session file.
?>


>
>   - Future communication methods: Zero Linden has talked about LSL
> communication methods at his office hours and has some pretty interesting
> stuff on his wiki page about it:
> http://wiki.secondlife.com/wiki/User:Zero_Linden/Office_Hours/Discussion#Incoming_HTTP
>  No one, to my knowledge, is currently working on this.
That's sad to hear.
I also submitted a proposal for llIMObject not too long ago ;)


>
>   - Kelly
>
>  Chance Unknown wrote:
> email between objects is more robust and breaks with less frequency than the
> xmlrpc interface. but, when your sim is constipated on the email, trying to
> find someone to come fix it is a nightmare. and without support channels or
> access to lindens. you can feel basically screwed. HTTPRequest() is the way
> to go right now.

Yeah, Live Help is dead, support at secondlife.com is dead; now all we
have is a trouble ticket system and the MLs. Fortunately, we have a
couple lindens who have an ear to the ground! Kudos!

>
>

On 6/11/07, Jason Giglio <gigstaggart at gmail.com> wrote:
>
> Pear has a mail class that does all you want and more.  Why would anyone
> reinvent this wheel?
>
>
> http://email.about.com/od/emailprogrammingtips/qt/et073006.htm
>

I like a class I can just drop in; some people on hosting don't really
have access to PECL or PEAR.

Swiftmailer's features:
    * Persistent connectivity improves performance
    * Connection types selected by user - extendable
    * Complete header control with RFC 2822 requirements handled
    * Internationalization support (i18n)
    * Connection redundancy support
    * Load balancing and/or throttling support
    * SSL & TLS Support - for Gmail servers
    * Embedded images or other file types
    * Full MIME 1.0 library included (create multipart messages,
attachments etc)
    * Batch mail processing
    * Smart runtime caching (in small, self-maintained packets)
    * Send attachments of any size even with PHP's 8MB Memory Limit
    * Support for multiple attachments
    * Lossless protection against header injection (encode, don't strip)
    * Set message priority
    * Request read receipts
    * Pluggable SMTP authentication (LOGIN, PLAIN, MD5-CRAM, POP Before SMTP)
    * Anti-flooding support for servers with limits on emails-per-connection
    * Bandwidth monitor included
    * Extensive event-driven plugin support (easy to write)

Plus the developer runs forums here:
http://forums.devnetwork.net/viewforum.php?f=52

I left a message for him, and 15 minutes later I had a forum response from him.

And Swiftmailer's updated regularly -- the last time the pear class
had been updated was 2006-10-11, and PEAR very nicely tells me the
following about it:
Number of open bugs: 6 (84 total bugs)
Average age of open bugs: 197 days
Oldest open bug: 567 days

Browsing the Swiftmailer forums, I've found a couple bugs which were
solved in several hours.

Fanatical support is ALWAYS something I look for. But thanks, Jason!


-- Kami