Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are currently developing a new email deployment system. We have the system currently configured so that it reads a record from the database and loads the email content and deploys it to the target. Now we want to move this over to multiple threads. That is easily done, except we then hit the email providers returning SMTP codes referring to "Too many connections", or "Deferred connection". The solution to this is to have a thread open up a connection to the email provider and deploy n emails and then disconnect. We have currently configured the application so that it will support these session based email deployments.

The problem is this, the database table has multiple email addresses in and they aren't grouped by email provider because that will show favouritism. We need to be able to retrieve a set number of, i.e. Hotmail, emails (@hotmail.com, @hotmail.co.uk, @live.co.uk) so that we are reducing the number of connections to Hotmail and reducing the risks of getting the "Too many connections" error. We are at the point now where we have gone round and round in circles trying to get a solution, so I thought I'd throw it out there and see if anyone has any ideas?

EDIT

I would like to stress that this application is not used for spamming purposes.

share|improve this question
    
Let me see if I get this straight: you have a table of randomly mixed email addresses, from which you want to select emails in batches by email provider, where the email provider for each batch is consistent across the batch, but is random from one batch to the next? –  blueberryfields Oct 20 '11 at 16:19
    
Kind of. Because of the multi-threading it adds an extra complexity to the problem. What the service needs to do is build a "batch" of email addresses, grouped by email provider and send to a thread to deploy too. –  Neil Knight Oct 20 '11 at 16:22

3 Answers 3

up vote 1 down vote accepted

In the table that contains the messages that are queued, I'd also include a parsed out provider name (ie, 'hotmail.com', 'yahoo.com') and a flag indicating if the email address is from a 'bulk' provider (the exact meaning of which depends on whatever limits your'e seeing now - if there's a lot of email addresses from a specific provider, mark it as bulk). Have one or more processes/threades work against non-bulk provider emails, and one or more others dedicated to a single bulk provider each.

If you want to pull them back in random order, when you queue the message add a random number. Sort by that first when you pull out addresses for a particular provider. This way they go out in a different order each time.

share|improve this answer

In pseudo-code:

dictionary = Dictionary<Host, Queue<EmailMessage>>;
for each email message in original list:
   dictionary[host].Enqueue(message);

for (i in 1..number of threads):
    in a new thread:
        while (!queue.Empty):
            (host, messages) = dictionary.ReturnAndRemoveARandomItemAtomically();
            send the first X emails in messages to host
            if the message list isn't empty:
                dictionary[host] = messages
share|improve this answer

Divide and conquer is certainly the way to go. Setup an array of queues where each queue is serviced by one smtp connection to a single provider. Load the list of addresses for that provider into the specific queue and let it sned the messages one by one. If for a certain provider/queue the number of addresses is below a certain treshold you can merge them and have an extra connection (to the additional provider) service that specific queue.

share|improve this answer
    
This is kind of what we have at the moment. The problem is we are talking large volumes of emails, potentially a million. –  Neil Knight Oct 20 '11 at 16:34
    
So you are most likele to hit the physical connection limit of a singel machine. To scale this out over multiple machines you can create a service that accepts a range of providers it should email and let this service in turn run the above mentioned process autonomously. Upon completion it can asynchronously notify the coordinator of its completion. –  Carlo Kuip Oct 20 '11 at 17:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.