Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I've looked at other anti-spam algorithms, notably Reddit's, but they seem to be inadequate and naive: for example, only banning specific words (like "spam") that appear in the title of a post.

How can one approach this problem? Are there any tools that would help in this case, and for high-activity online community websites in general?

share|improve this question
    
reCAPTCHA is hardly perfect, and any sort of captcha that gives modern spambots much trouble will be very hard on a lot of humans. –  David Thornley Dec 29 '10 at 16:35
    
Mark them spam and they'll be filtered next time :) –  Ayush Goyal Dec 29 '10 at 17:03
    
Should I flag this as spam? Of course not - it's an excellent question - but it demonstrates one very effective technique of getting rid of it. –  Gary Rowe Dec 29 '10 at 20:13

3 Answers 3

Email systems use Bayesian filters to filter spam, and they perform their job very well in both my Yahoo and my Gmail accounts.

share|improve this answer
    
Here is a very simple bayesian class: mailsystem.codeplex.com/SourceControl/changeset/view/… –  user2567 Dec 29 '10 at 16:32
    
Of course, they need a training set, which means people assigned to decide what postings are spam and what aren't. –  David Thornley Dec 29 '10 at 16:35
    
They aren't perfect, though. How many e-mails have you had sent to your junk mail that you marked as "not spam" and sent back to your inbox? Probably several. If you set up a filter like this, you will get a lot of false positives. –  Corey Dec 29 '10 at 16:59
    
@Corey: In the past year, almost none, and I check my junk mail at least four or five times a week. You will get false positives, and the better the algorithm the worse each false positive will be, but not necessarily a lot of them. –  David Thornley Dec 29 '10 at 17:04
    
@David but like you said, it needs a training set. Your e-mail's perfect junk mail algorithm is that way probably because of years of training it. My Gmail account is the same way; maybe once a year I'll expect an email and it's in my junk folder. However, this would take a long time to implement on a website. –  Corey Dec 29 '10 at 17:09

Ultimately there are no 100% solutions.

The "best" approach is one that incorporates multiple methods such as:

  • Response Time: Check posting interval. Most bots will immediately post upon hitting a web page, so simply ignore these.
  • Honey Pot: Place fake input fields above the actual posting fields, and make them invisible. Some bots will simply post on the first set of input fields found. Ignore these.
  • Service: Use online anti-spam services such as Akismet.
  • IP Logging: Log IPs, and known Agent browsers.

When combined, these measures should drastically reduce spam, however they will never be 100% effective.

Fighting spam will never end.

share|improve this answer

An algorithm of anti-spam is only as good as its intelligence to distinguish between "probably spam" and "probably not spam." The best ones base themselves on statistics. What words and phrases were used in a large percentage of spam posts and very few non-spam posts. Everytime a post is labeled as spam, this algorithm ought to learn from it and incorporate it into its statistics of what is and isn't spam.

Anything else is simply taking stabs at words that may or may not be representative of spam. The more obvious words are evident, but it's easy enough to fool such algorithms, requiring that a list of words and expressions be constantly expanded on. I think a good place to start would be e-mail filters.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.