Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We are crawling and downloading lots of companies' PDFs and trying to pick out the ones that are Annual Reports. Such reports can be downloaded from most companies' investor-relations pages.

The PDFs are scanned and the database is populated with, among other things, the:

  • Title
  • Contents (full text)
  • Page count
  • Word count
  • Orientation
  • First line

Using this data we are checking for the obvious phrases such as:

  • Annual report
  • Financial statement
  • Quarterly report
  • Interim report

Then recording the frequency of these phrases and others. So far we have around 350,000 PDFs to scan and a training set of 4,000 documents that have been manually classified as either a report or not.

We are experimenting with a number of different approaches including Bayesian classifiers and weighting the different factors available. We are building the classifier in Ruby. My question is: if you were thinking about this problem, where would you start?

share|improve this question
up vote 1 down vote accepted

I think you should match phrase in first few (say 500 words) as normally these report contains information whether they are quarterly or annual in first few pages only(like 1Q2012, FY2012 etc). Along with it you can have words which should not be there in annual report.

Much simpler would be to figure out if report is annual or not from the site from where you are downloading this report, so while downloading/crawling only look for this information on the site itself.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.