Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am looking to write a Bayes filter that will act as an indicator of topic for a number of topics with a variable number of sources. Given a really big number of RSS feeds and here really big might only be a few hundred only few percent (let's say 2%) are interesting to a given group.

What I want to write (in PHP) is a filter I can train to say this is Topic X and this is not. Then I want selected users to be able to further train the filter. But I want to use only the base training for picking topics to mark as interesting for Topic X and then refine that collection based on the training the user has provided to give the most interesting results for that user by reassessing the probabilities including the user supplied training on the short list per user. Or maybe scoring the short list and picking the "best" items to show first.

So over time I am going to gain a fairly large collection of data (all of the headlines and blurbs from hundreds of feeds) and I am going to have flagged the feed items "yes" and "no" on an unspecified number of topics and have had users have likewise done so.

The headline and blurb will be stored for easy display later but I don't want to hold on to too many items that are "no" (up to 98% I imagine). Given I go with, say, this implementation from GitHub that I found earlier today https://github.com/benwaine/BayesPHP (or can be talked into rolling my own) what is the most efficient way to go about storing the data needed to carry out an assessment? Am I asking for trouble if simply try and store any news headlines etc flagged "yes" but store some sort of summary for the base "no" assessments and then just work it out on the fly each time storing the resultant "yes".

The server I have to work with is already doing a fair amount and is not going to spare me much horse power for this.

So balancing processing efficiency and storage needs what can you recommend?

share|improve this question

2 Answers 2

Wouldn't it be more efficient to store why it is yes rather than the reason for no? Or no because the filter is basically checking to see if it shares any attribute with a no rather than checking against yesses...

So if I understand you correctly, you want to store both the data and a list for each of why it is not a member of a specific topic? Perhaps you should store it in a relational database like MySQL. You could have one primary table containing the summaries, and a table for each containing the reasons why it is not, linked to the primary on an id as primary key. The full text of the data probably would be better stored in a text file or otherwise outside the database, unless you intend to search within it directly.

I hope that helps and I didn't totally misunderstand what you are asking.

share|improve this answer
    
You might have something there. I've been thinking of articles as spam with occasional ham but it would be better to think of them as ham with occasional spam (guess where I worked with this type of filter before) as I am looking for specific positives... good point. –  Matthew Brown May 16 '13 at 12:55

Bayes filters have to be taught. The activity of teaching the filter is a heavy operation, but it only needs to be performed once. After that, testing an article against a Bayes filter is very efficient. You only need to re-teach the filter if the population of accepting/rejected articles change enough to alter the filters outcome.

Teaching a filter that already works, by adding more keywords does little to improve the filter unless the new words introduced are significant in the population.

There is no need to store data for words with low thresholds (i.e. 0.00001%). Since they are most likely a single time occurrence of the word.

To store the filter in the database. I would convert the filter's statistical results to JSON and simply save it in a text field of the database. To test an article against the filter you would just load the data and serialize it back to a PHP array.

Once you have taught the Bayes filter with your sample articles. You can destroy those articles, but if you want to later re-teach you may need them again.

In time, you will have falsely rejected articles, and falsely accepted articles. You'll need to keep track of those, then re-categorize them and teach the Bayes filter again.

Bayes filters are only effective with articles that are significantly different from the rest of the population. Rather than categorize them as in or out of a category. You will have to weigh their membership to a category. An article may be 80% about gun control, 20% about gun registry and 45% about tax fraud. While this might seem like the article is mostly about gun control. It might really be about tax fraud simply because your filter wasn't taught enough about the subject.

share|improve this answer
    
Yes, that makes sense. Weighing membership of categories might allow, for example, the system to recognise when some categories are members of more than one category or might make good supplementary content for a category that is a little "light" on, matching items one day. That's given me quite a bit to think about. Thanks. –  Matthew Brown May 16 '13 at 12:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.