8
votes
1answer
827 views

Good technique for search text tokenization

We are looking for a way to tokenize some text in the same or similar way as a search engine would do it. The reason we are doing this is so that we can run some statistical analysis on the tokens. ...