Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

My task lays in the area of text mining / labelling, more particular extraction of bibliographic citations (references) from the plain text, as shown here:

enter image description here

Basically, for the input text the algorithm should recognize:

  • The beginning and end of citation
  • Journal name, title, author, issue, volume, pages

This task was already approached in e.g. Citation Parsing Using Maximum Entropy and Repairs (Ng Yong Kiat, 2005), which demonstrated that CRF models deal better then MEMM or pattern-based approaches. Currently there are two projects, that do the job:

While the results of mentioned approaches are already known (and are very good), I was given a task to do the same but using the context-free grammars. Indeed, the information looks to be very structured but I am not sure if this approach can outperform CRF on sensitivity, at least I hope it can beat it on accuracy.

My idea is the following:

  • Define the fixed grammar rules for already known types:
    YEAR → (19|20)\d\d | \d\d\d\d
    PAGE → \d{1,3}\-\d{1,3} | e\d{1,3}
    ISSUE → \d{1,3}
  • It is possible to infer from training set the probabilities of rule alternatives e.g.:
    PAGE → \d{1,3}\-\d{1,3} with P=0.8 | e\d{1,3} with P=0.2
  • From the pre-annotated training set generate the high-level rules (all chars between non-terminals are terminals):
    CITATION → AUTHORS, JOURNALTITLE, ISSUE, PAGES, YEAR | AUTHORS, BOOKTITLE, VOLUME, CHAPTER, CHAPTERTITLE, EDITORS, PUBLISHER, PUBPLACE | ...
    CITATIONS → CITATION | CITATIONS; CITATION | CITATIONS; and CITATION

The problematic places are:

  • AUTHORS: There is no fixed rule for them, but one can write a function, that can score that the given text looks like an author. Also the list of possible (English) surnames is not so huge, but if I create the thousands of rules it will slow down the grammar parser. Let's say for authors we have some scoring function, that for given input returns a number [0 .. 1] reflecting how the input really looks like an author. The grammar parser should feed all possible variants for AUTHORS and stop on one with a best score.
  • JOURNALTITLE: The same story. There is obviously no general rule to define it. There is a list of all journals, but then comes also the subproblem of abbreviation: Ann. NY Acad. Sci. = Ann N.Y. Academy Sci = Annals of the New York Academy of Science. Again, we can define a scoring function for that; in worst case the function can measure the distance between input sting and all known journal titles using Damerau–Levenshtein algorithm.
  • The text like "Vol", "Chapter", "edited by" can be very useful as terminators prefixing the corresponding values (and thus reducing the ambiguity of the grammar), but can they be identified and learned automatically?
  • The same for terminators before and after citation, which can be very useful (perhaps will speedup the parsing):
    SUPERCITATION → ( CITATIONS ) | ( see, e.g., CITATION ) | see, for example, CITATION | described in, e.g., CITATION | is disclosed by CITATION
    Does it make sense to "hardcode" them or try to learn them?
  • The "noise" text between citations: how to deal with it?

The goal of the lexical analysis is perhaps not to find the most probable parse tree, but also to maximise AUTHORS and JOURNALTITLE functions.

Can somebody advise me:

  • If SCFG is the right way to approach the problem? If yes, how is it possible to adapt this approach to create a parse tree based on probabilities and also on scoring functions?
  • Are there any other relative works / articles?
  • Are there any libraries that can help in chunking the text (written in a way that can be applied also for e.g. Chinese text)? I have found some monsters like UIMA, Mallet, MinorThird... I wonder about your personal experience when using them for the task like above.

Thank in advance for any reply.

share|improve this question
add comment

1 Answer

One of the starting points you should look at is Proper Noun recognition and Tokenization. One of the well known papers is this one, and also a lot of new submissions into ACM have dealt with this problem. This will help with in-line citations.

@Admin - I wanted to add this as a comment rather than an answer: not sure if I can't do that because of my current "reputation" points. Please move it there if you can.

share|improve this answer
    
You can add either a comment (via add comment under my post) or add the answer, if you think your idea is mature. May I ask you: how do you think the noun detection will help in my case? I do not deal with natural language recognition, but on the other side the part I would like to capture is inside the natural language stream. –  dma_k Aug 18 '11 at 23:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.