[sldev] [LSL] [POSTFIX PROBLEMS] Object Email Receive from external domain problems

[Erik Hill]

Couldnt you do some load balancing on  the XML-RPC, multiple servers, even
round robin DNS could do better then a single server.

SL really needs a realtime incoming communication method.

My problem with useing HttpRequest is that its already being used in my
system .. just below 1 call per second on large scale deployments. Around 36
objects (max) making HttRequests to get data into the system per sim.

That dosent leave much time for any other httprequests, let alone others
that arnt from my system and that are in that sim and under the same user.

The consept im useing is more on the lines of a thin client.  Considering
LSL cant really becalled robust or fast. It would take me about 40 scripts
and several months to recreate what can be done in  a hour in most other
languages, let alone data storage.. I need to store several mb's of
information, and have it acessable fast - something thats not vary easy to
do with LSL.

So im porting every thing possable out of the server, and haveing only the
reactions in LSL.

Im hoping to also deploy something like this in upwards of 50 sims. The
larger the scale the more important it becomes to get rid of unnessary data
calls, it becomes necessary load on my server, and the outbound connection
of SL....

Stable inbound connection would decrease the amount of load id have to put
on my server, and decrease the httprequest calls that result in nothing more
then saying 'Do i have a message?'
When it should just simpily be able to drop a message to whatever object
needs to get a message, saves bandwidth and resources, no extra sql querys
to see if the message was gotten or if it was sent or if there is a message
waiting.









On 6/11/07, Kelly Washington <kelly at lindenlab.com> wrote:
>
>  We are currently (this minute, really) investigating emails from outside
> to LSL being broken.
>
> Some more comments about communication between LSL and outside SL:
>
>  - All XML-RPC requests go through a central server.  That 'server' is
> really an apache host running a C++ cgi.  This cgi module looks up object
> presence info (what host it is on) based on the channel ID and routes the
> message accordingly.  This server is consistently overloaded although I
> believe we upgraded it, again, recently.  As a central point of failure it
> will never scale as well as needed, and should eventually be deprecated.
>
>  - All email from outside goes through a single server that maps to
> lsl.secondlife.com.  The emails are passed to a perl script that checks
> some edge cases and looks up presence info (what host the object is on) in
> backbone (the python service that hosts agent presence as well).  However,
> objects only have presence in backbone if they have called llGetNextEmail at
> least once; if there is no presence info for the object or the object is not
> where we think it is then it is put in the database and retrieved the next
> time the object calls llGetNextEmail.  However, like xml-rpc email from
> outside SL to LSL goes through a central server that will never scale as
> well as needed and will eventually need to be deprecated or changed.  By
> changed I mean for example requiring region information in addition to
> object ID in the email address used.  (Email from lsl->lsl cheats, is never
> really an email and avoids the problems mentioned above except the presence
> part.  If presence is broken then lsl->lsl email will fail unless the
> objects are on the same sim)
>
> *NOTE: While I used the word 'deprecated' above we currently have no solid
> plans to do so, I am not saying they are already deprecated and we will work
> with and communicate with the community a LOT before we move ahead with any
> plans related to that.  Additionally we expect more communication
> alternatives to be in place before then.  We are still supporting these
> methods and as I mentioned in the first line we are currently investigating
> the current email issues.
>
>  - llHTTPRequest is the bomb.  Really.  It is lightweight and
> distributed.  The entire action happens on the host the region with the
> object is on.  There is no central server, no required presence
> registration.  Lots and lots of requests can be done at the same time with
> essentially no effect on the rest of the simulator process.   Yes there have
> been bugs before but most of those are now resolved.  It is much better to
> have an object ping a server every second with an llHTTPRequest than it is
> use either XML-RPC or email->lsl.
>
>  - Future communication methods: Zero Linden has talked about LSL
> communication methods at his office hours and has some pretty interesting
> stuff on his wiki page about it:
> http://wiki.secondlife.com/wiki/User:Zero_Linden/Office_Hours/Discussion#Incoming_HTTP
>
> No one, to my knowledge, is currently working on this.
>
>  - Kelly
>
> Chance Unknown wrote:
>
> email between objects is more robust and breaks with less frequency than
> the xmlrpc interface. but, when your sim is constipated on the email, trying
> to find someone to come fix it is a nightmare. and without support channels
> or access to lindens. you can feel basically screwed. HTTPRequest() is the
> way to go right now.
>
> On 6/11/07, Erik Hill <zortiger at gmail.com > wrote:
> >
> > This dosent sound vary good
> >
> >  XMLRPC is already vary screwed up.
> >
> > Ill be testing this within the next few days my self.
> >
> > I am currently working on an application that has tight integration
> > between a website and SL, that normally means some sort of communication.
> > While i plan to make use of email for a good minority of it, as its only
> > simi time sensitve, XMLPRC is a must for time sensitive data, such as
> > results when they interact with a web page, I need both up and working
> > correctly in order for my plans to completely work.
> >
> > I know from my current venders that rely on XMLRPC that its failor rate
> > is up in the 20-30% depending on the time of day.
> > From what i can figure out single threaded applications tell horror
> > stories about SL's XMLRPC functions, such as getting stuck infinity reading
> > an open port that never gets data.
> > From what iv seen with the XMLRPC ill have to build a threaded
> > application that opens up multiple attempts at the same time to get data in
> > to SL, then shuts down after a thread receives data back from the object.
> > Its the only way I can see to get data into SL at a realtime speed, and
> > realtime is necessary in some cases.
> >
> > But yea, im next to getting into the email stuff for my project, so
> > testing its delay is short list of things to do.
> >
> > On 6/11/07, Kamilion < kamilion at gmail.com> wrote:
> > >
> > > Holy crap. This is far worse than I expected.
> > > I didn't want to use XMLRPC because I figured that a single point of
> > > failure and lag of a couple minutes wasn't worth it.
> > >
> > > >From what I knew, postfix ran on every simulator; so it was more
> > > distributed and hopefully more robust.
> > >
> > > Now it looks like something is SERIOUSLY wrong in email land.
> > >
> > > Last I knew, Michael, James, and Charity Linden were messing with
> > > postfix back in april's blog post.
> > >
> > > Now -- What can we do about this? I'm glad I said something, otherwise
> > >
> > > this might have been a quiet annoyance.
> > >
> > > I know postfix can be blazingly fast, I used to use it on my own
> > > domain before moving over to Google Apps for your Domain, So there's
> > > obviously some kind of misconfiguration somewhere.
> > >
> > > If you feel like sharing; I'd like a copy of that PHP and LSL you use;
> > > because this is EXACTLY the sort of setup I'm trying to achieve (PHP
> > > Mail to LSL).
> > >
> > > So far, I've had to bash at PHP to even get it to talk to Google's
> > > AfyD server.
> > > Google uses SSL/TLS *ONLY* for SMTP auth, so php's mail() function
> > > doesn't work so well.
> > >
> > > After doing exhaustive research, I came up with the following options:
> > > http://www.bytewize.com/linux/BetterSMTPhelp.php
> > > BetterSMTP, a phpbb2.x mod to use the SquirrelMail SMTP classes.
> > > Looked to be broken and forum posts about it were awash with PLZ HELP.
> > > No good.
> > > http://www.vulgarisoip.com/?p=17
> > > PHPGmailer, a Cut & Paste job on PHPMailer to kitbash it into using
> > > TLSSTART. Also no good, PHPMailer isn't maintained anymore and this
> > > kitbash comes up with SSL errors on connection close.
> > >
> > > The winner:
> > > Still poking along, I ran across this bug report for Wordpress:
> > > http://trac.wordpress.org/ticket/4326    #4326 (WP can't use Google
> > > Apps For Your Domain email accounts.)
> > > Which linked me here:
> > > http://www.shiftthis.net/wordpress-swift-smtp-plugin/
> > > Which linked me here:
> > > http://www.swiftmailer.org/
> > > "Swift is a free feature-rich PHP Mailer -- Swift is a fully OOP
> > > library for sending e-mails from PHP websites and applications. It
> > > does not rely on PHP's native mail() function which is known for using
> > >
> > > high server resources when sending multiple emails. Instead, Swift
> > > communicates directly with an SMTP server or a MTA binary to send mail
> > > quickly and efficiently."
> > >
> > > Score!
> > > One of the features it lists: "SSL & TLS Support - for Gmail servers"
> > >
> > > So, here's how I hacked together a nice little test script:
> > >
> > > <?php
> > > require_once "swiftmailer/Swift.php";
> > > require_once "swiftmailer/Swift/Connection/SMTP.php";
> > >
> > > echo "Starting...\n<br>\n<br>\n";
> > >
> > > // Create a SMTP connection handler (Google Apps for your Domain)
> > > $smtp = new Swift_Connection_SMTP(" smtp.googlemail.com",
> > > Swift_Connection_SMTP::PORT_SECURE, Swift_Connection_SMTP::ENC_TLS);
> > > $smtp->setUsername("phpsmtp at sllabs.com");
> > > $smtp->setpassword("smtpmail");
> > >
> > > // Create the Swift object interface
> > > $swift =& new Swift($smtp);
> > >
> > > $to = "513d65af-98e3-f2ef-db09-9a112eca5d28 at lsl.secondlife.com ";
> > > $subject = "Testing my subject";
> > > $message = "A very long\n and drawn out\n message.\n\nRar!";
> > >
> > > //Create the message
> > > $swiftmessage =& new Swift_Message($subject, $message);
> > >
> > > //Now check if Swift actually sends it
> > > if ($swift->send($swiftmessage, $to, " phpsmtp at sllabs.com"))
> > > { echo "Sent"; } else { echo "Failed"; }
> > >
> > > ?>
> > >
> > >
> > > Here's a sample result:
> > >
> > > Delivered-To: kamilion at gmail.com
> > > Received: by 10.100.165.16 with SMTP id n16cs242953ane;
> > >         Sun, 10 Jun 2007 20:14:00 -0700 (PDT)
> > > Received: by 10.64.204.19 with SMTP id b19mr8395228qbg.1181531640038;
> > >         Sun, 10 Jun 2007 20:14:00 -0700 (PDT)
> > > Return-Path: <phpsmtp at sllabs.com >
> > > Received: from qb-out-0506.google.com (qb-out-0506.google.com [72.14.204.225
> > > ])
> > >         by mx.google.com with ESMTP id
> > > q14si5481907qbq.2007.06.10.20.13.59;
> > >         Sun, 10 Jun 2007 20:14:00 -0700 (PDT)
> > > Received-SPF: neutral (google.com : 72.14.204.225 is neither permitted
> > >
> > > nor denied by best guess record for domain of phpsmtp at sllabs.com)
> > > Received: by qb-out-0506.google.com with SMTP id q12so1717954qba
> > >         for <Kamilion at gmail.com>; Sun, 10 Jun 2007 20:13:59 -0700
> > > (PDT)
> > > Received: by 10.115.110.6 with SMTP id n6mr5060449wam.1181531638487 ;
> > >         Sun, 10 Jun 2007 20:13:58 -0700 (PDT)
> > > Return-Path: <phpsmtp at sllabs.com>
> > > Received: from ?216.218.196.73? ( [ 216.218.196.73 ])
> > >         by mx.google.com with ESMTP id
> > > n38sm6862970wag.2007.06.10.20.13.57
> > >         (version=SSLv3 cipher=OTHER);
> > >         Sun, 10 Jun 2007 20:13:58 -0700 (PDT)
> > > Return-Path: < phpsmtp at sllabs.com>
> > > To: Kamilion at gmail.com
> > > From: phpsmtp at sllabs.com
> > > Reply-To: phpsmtp at sllabs.com
> > > Subject: My subject
> > > Date: Sun, 10 Jun 2007 20:15:09 -0700
> > > X-LibVersion: 3.2.6
> > > MIME-Version: 1.0
> > > Content-Type: text/plain; charset=iso-8859-1; format=flowed
> > > Content-Transfer-Encoding: 8bit
> > > Message-ID: <20070611031509.22850.1036293226.swift at vendor.sllabs.com >
> > >
> > > My body
> > >
> > >
> > >
> > > So, anyway.
> > > External Domain Email's obviously broked somehow.
> > > Included some data that could be useful for others.
> > >
> > > Harold, I've got land in multiple mainland and private sims. If you
> > > feel like it, I can set up some additional prims to take multiple
> > > metrics, and see if there's a correlation.
> > >
> > > sllabs.com is run from a Gentoo VPS hosted by atarack.com in Hurricane
> > > Electric's Fremont-II datacenter, roughly 2ms from the main LL
> > > datacenter in SF.
> > >
> > > If you should decide to decline sharing your code, heck, I'll bash up
> > > a similar PHP&LSL script and share it in this thread, just because I
> > > wanna get to the bottom of this; and hopefully share as much data as I
> > >
> > > can with the lindens so they can fix it properly.
> > >
> > > -- Graham Cantin (Kamilion Schnook)
> > >
> > > (sorry for the longwinded mails, but this stuff's archived, so it may
> > > help out others in the future who are browsing the SLDev archives or
> > > the Google Coop search :)
> > >
> > >
> > >
> > > On 6/10/07, Harold Brown <labrat.hb at gmail.com> wrote:
> > > > Welcome to the hell of trying to use external -> internal E-Mail
> > > with SL.
> > > >
> > > > For a nice history of the reliability of external -> internal E-Mail
> > > from
> > > > May 2 -> current date,  you can view my test log at this URL:
> > > > http://www.rpgstats.com/SL/maillog.txt
> > > >
> > > > If you would like to review the history from February8 -> May2 it is
> > > > available here:
> > > > http://www.rpgstats.com/SL/maillog1.txt
> > > >
> > > > As of the time I'm replying there seems to be about 4 hour delay in
> > > e-mail
> > > > processing for SL...  There have been delays up to around 24 hours.
> > > >
> > > > For those interested in how the test works.
> > > >
> > > > I have an LSL script running on my sim that once every 30 minutes
> > > requests a
> > > > PHP script on my webserver to send an e-mail.  The PHP script
> > > generates a
> > > > random string for the subject and logs the time that it sends to the
> > >
> > > > maillog.txt file.  The LSL script checks every 5 seconds for a new
> > > E-Mail,
> > > > and upon receiving one it tells the PHP script on the server that it
> > > > recieved an E-Mail and what the subject was.  The PHP script then
> > > logs this
> > > > information as well.
> > > >
> > > > On 6/10/07, Kamilion <kamilion at gmail.com> wrote:
> > > > >
> > > > > I'm having some problems with objects and receiving email from
> > > outside
> > > > > of the secondlife.com domain -- They seem to be failing without an
> > > > > error or bounce message, and the object never appears to receive
> > > the
> > > > > email. I was hoping someone could look into this or point someone
> > > who
> > > > > can at it. I submitted a ticket in the new support system,
> > > TicketID
> > > > > 4051-4201708.
> > > > >
> > > > > Synopsis:
> > > > > Object Sending Email (Working)
> > > > > Object Receiving Email from another Object (Working)
> > > > > Object Receiving Email from Google Apps for your Domain hosted
> > > system:
> > > > > ( sllabs.com) (NOT WORKING)
> > > > > Object Receiving Email from Google Mail ( gmail.com) (NOT WORKING)
> > > _______________________________________________
> > > Click here to unsubscribe or manage your list subscription:
> > > /index.html
> > >
> >
> >
> > _______________________________________________
> > Click here to unsubscribe or manage your list subscription:
> > /index.html
> >
> >
> ------------------------------
>
> _______________________________________________
> Click here to unsubscribe or manage your list subscription:
>
> /index.html
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.secondlife.com/pipermail/sldev/attachments/20070611/bbdf13b9/attachment-0001.htm