Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Let's say I have a big corpus (for example in english or an arbitrary language), and I want to perform some semantic search on it. For example I have the query:

"Be careful: [art] armada of [sg] is coming to [do sg]!"

And the corpus contains the following sentence:

"Be careful: an armada of alien ships is coming to destroy our planet!"

It can be seen that my query string could contain "semantic placeholders", such as:

[art] - some placeholder for articles (for example a / an in English) [sg], [do sg] - some placeholders for NPs and VPs (subjects and predicates) I would like to develop a library which would be capable to handle these queries efficiently. I suspect that some kind of POS-tagging would be necessary for parsing the text, but because I don't want to fully reimplement an already existing full-text search engine to make it work, I'm considering that how could I integrate this behaviour into a search engine like Lucene?

I know there are SpanQueries which could behave similarly in some cases, but as I can see, Lucene doesn't do any semantic stuff with stored texts.

It is possible to implement a behavior like this? Or do I have to write an own search engine?

share|improve this question
add comment

1 Answer

Not a Lucene expert but:

You could create your own tokenizers that indexes e.g. [ART] as a token. That way you might be able to do phrase searches like the example above. This won't work if 'armada' is sometimes a [SG] or some other token. You'd need a specialized wildcard search if that is the case.

Perhaps a conventional wildcard search through Lucene will allow you to gather at least a pretty good result set to start with. With a second pass you could parse out the [ART] qualifiers and such to come to a final result set.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.