Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are crawling and downloading lots of companies' PDFs and trying to pick out the ones that are Annual Reports. Such reports can be downloaded from most companies' investor-relations pages.

The PDFs are scanned and the database is populated with, among other things, the:

  • Title
  • Contents (full text)
  • Page count
  • Word count
  • Orientation
  • First line

Using this data we are checking for the obvious phrases such as:

  • Annual report
  • Financial statement
  • Quarterly report
  • Interim report

Then recording the frequency of these phrases and others. So far we have around 350,000 PDFs to scan and a training set of 4,000 documents that have been manually classified as either a report or not.

We are experimenting with a number of different approaches including Bayesian classifiers and weighting the different factors available. We are building the classifier in Ruby. My question is: if you were thinking about this problem, where would you start?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

I think you should match phrase in first few (say 500 words) as normally these report contains information whether they are quarterly or annual in first few pages only(like 1Q2012, FY2012 etc). Along with it you can have words which should not be there in annual report.

Much simpler would be to figure out if report is annual or not from the site from where you are downloading this report, so while downloading/crawling only look for this information on the site itself.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.