2

which is the best way to correlate and group similar articles?

I mean something like Google News, which groups under a single topic different articles from different sources.

I'm not interested in something much sophisticated, it is just important to be efficient in terms of speed and resources needed. I can live with a correlation only on short text like the titles of the articles, not the article itself.

I read a paper which talks about tuples of two or three words (best with three) to extract and use to produce a score to achieve the correlation, but it's way too much academic for me

Improve this question
1
  • 3
    If I had such an algorithm then I would probably write my own software around it and form a startup. :)
    – maple_shaft ♦
    Nov 22, 2011 at 12:12

3 Answers 3

Reset to default
2

You can use Lucene Term Vectors.

Here is a blog post explaining it in detail. http://sujitpal.blogspot.com/2011/10/computing-document-similarity-using.html

Lucene is an indexing library: http://lucene.apache.org/java/docs/index.html

Improve this answer
1
  • Sounds interesting, I'll take a deeper look into it. I wonder if I could do the same thing with Sphinx Search... the version 2 should have something like term vectors.
    – spider
    Nov 23, 2011 at 13:46
0

If you are using Solr then there is a MoreLikeThisHandler

And in case using Sunspot with it, then its even simpler

post = Post.first Sunspot.more_like_this(post, Post, Page) do fields :title, :body with(:updated_at).greater_than(1.month.ago) facet(:category_ids) end

Improve this answer
-1

LDA is surely a way to get your job done - however it is complicated and involves the article.

"correlation only on short text like the titles of the articles", the information correlated in titles are too little to construct any model

Improve this answer
2
  • 1
    Depends on the titles, I guess. Titles of academic papers are usually detailed enough to be useful that way, but other than that the titles will be much too idiosyncratic to be useful.
    – David Thornley
    Nov 22, 2011 at 14:29
  • The analysis only on the titles is not a requisite. I just thought it was less resource intensive.
    – spider
    Nov 23, 2011 at 13:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.