0
votes
0answers
136 views

What is the purpose of chunking strings in natural language processing?

I see many examples of libraries that use a 'chunker' and also people asking how to write chunkers, but what is this for and why do we need it? Isn't it enough to split a text by whitespace or other ...
3
votes
2answers
375 views

NLP - Queries using semantic wildcards in full text searching, maybe with Lucene?

Let's say I have a big corpus (for example in english or an arbitrary language), and I want to perform some semantic search on it. For example I have the query: "Be careful: [art] armada of [sg] is ...
6
votes
3answers
1k views

What algorithm(s) can be used to achieve reasonably good next word prediction?

What is a good way of implementing "next-word prediction"? For example, the user types "I am" and the system suggests "a" and "not" (or possibly others) as the next word. I am aware of a method that ...
125
votes
14answers
6k views

Simple method for reliably detecting code in text?

GMail has this feature where it will warn you if you try to send an email that it thinks might have an attachment. Because GMail detected the string see the attached in the email, but no actual ...