Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

which is the best way to correlate and group similar articles?

I mean something like Google News, which groups under a single topic different articles from different sources.

I'm not interested in something much sophisticated, it is just important to be efficient in terms of speed and resources needed. I can live with a correlation only on short text like the titles of the articles, not the article itself.

I read a paper which talks about tuples of two or three words (best with three) to extract and use to produce a score to achieve the correlation, but it's way too much academic for me

share|improve this question

closed as too broad by gnat, MichaelT, Bart van Ingen Schenau, Dan Pichelman, GlenH7 May 21 '14 at 13:40

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

If I had such an algorithm then I would probably write my own software around it and form a startup. :) – maple_shaft Nov 22 '11 at 12:12
up vote 2 down vote accepted

You can use Lucene Term Vectors.

Here is a blog post explaining it in detail.

Lucene is an indexing library:

share|improve this answer
Sounds interesting, I'll take a deeper look into it. I wonder if I could do the same thing with Sphinx Search... the version 2 should have something like term vectors. – spider Nov 23 '11 at 13:46

If you are using Solr then there is a MoreLikeThisHandler

And in case using Sunspot with it, then its even simpler

post = Post.first Sunspot.more_like_this(post, Post, Page) do fields :title, :body with(:updated_at).greater_than(1.month.ago) facet(:category_ids) end

share|improve this answer

LDA is surely a way to get your job done - however it is complicated and involves the article.

"correlation only on short text like the titles of the articles", the information correlated in titles are too little to construct any model

share|improve this answer
Depends on the titles, I guess. Titles of academic papers are usually detailed enough to be useful that way, but other than that the titles will be much too idiosyncratic to be useful. – David Thornley Nov 22 '11 at 14:29
The analysis only on the titles is not a requisite. I just thought it was less resource intensive. – spider Nov 23 '11 at 13:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.