Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have written a program that can rapidly (within 5 sec on a 2GB RAM desktop, 2.33 Ghz CPU) differentiate between structured text (e.g English text) and random alphanumeric strings. It can also provide a probability score for the prediction.

Are there any practical applications/uses of such a program? Note that the program is based on entropy models and does not have any dictionary comparisons in its workflow.

share|improve this question
add comment

4 Answers 4

You could use it to scan through random text blocks and identify patterns and use these patterns for "predicting" the future. Kinda like an automated Nostradamus. :-)

On a more serious note, perhaps you could use it to select word/phrase-like strings from a bunch of randomly generated strings. These are usually desirable passwords because they are easier to remember and usually wouldn't appear in a dictionary.

share|improve this answer
add comment

The typical application for this (categorizing data based on previous training data - assuming that is indeed what this does) would be for an adaptive spam filter. Other things that I can think of:

  • categorize texts by language
  • estimate entropy (the higher the probability for "completely random", the better your entropy)
  • salvage textual content from damaged files or partitions
  • estimate the quality of forum posts (higher probability for "structured text" means more likely to be properly written)
  • find clear-text payload in sniffed network communications
  • categorize texts by some stylistic property, e.g. formality level (slang / informal / formal / ...) or poetic vs. factual (is this bit of text from a novel or from a news article?)

Whether your algorithm is suitable for these problems depends, but if it does what I think it does, you should be able to apply it to any of them.

share|improve this answer
add comment

It could be used as part of a program like the Unix file command to distinguish "English text" from unstructured "data".

share|improve this answer
add comment

You could "reverse" the algorithm, to generate natural-looking phrases. This could be used in games to generate the text on signs and posters, newspapers on the floor, etc. in a procedurally generated world. The text would look intelligible, but still look like natural language.

If I understand your algorithm correctly, you do some kind of measure on entropy, so it won't accurately model which letters should be used by the text. If that is the case, a simple tweak to generate likely alternations of vowels and consonants, like proposed here : http://www.umbrarumregnum.net/articles/creating-names.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.