Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I was told to create two lists concerning most frequently used words from a plain text (10 Mb arbitrary texts) as monograms (for single worded expressions such as human, water, is) and bigrams (for two-worded expressions such as basketball team, united states and etc)

I am stuck here and don't know how I can go about it! And how I can distinguish between these two?

My domain is not English, I only gave those examples to make my intention and meaning more clear.

share|improve this question
2  
You may be over-thinking it... a 'bigram' could simply be any pair of adjacent words, not necessarily a related pair. So in your post, some example bigrams might be "My domain", "domain is", "is not", and "not English". You should probably get clarification. –  Hellion Jul 15 '13 at 16:02
1  
@Hellion there are certain word pairs that have a higher occurrence than random, that might be what is desired... though yes, further clarification is a good idea. –  MichaelT Jul 15 '13 at 16:07
    
Thank you guys, So if we assume those word pairs that have meanings are desired how should i go about it? In first scenario where i can choose any two adjacent words as a pair, i would definitely face problems since i would easily make a meaningful pair into a nonsense ! suppose: I am a basketbal team player that play in UNITED STATES national team I am a basketbal team player that play in UNited States national team! See its really hard to understand where to start to how to go about it.:-/ –  Hossein Jul 15 '13 at 16:13
2  
You might be headed in the direction of the markov chain - specifically, the text generator (or at least creating the structure to run one). –  MichaelT Jul 15 '13 at 16:18
1  
In NLP, "bigrams" are always two adjacent words in an input stream. So counting frequent bigrams is essentially the same as counting frequent words: keep track of how often each pair of adjacent words occurs, and return the ones with the highest count - the only problem is that storing pair counts requires much more memory. –  Kilian Foth Jul 15 '13 at 17:40
show 4 more comments

1 Answer

You can try to read from text word by word and make 2 instances of Dictionary, one for monograms and one for bigrams, having the expression as Key and occurrence as Value. With this you can make some statistics about expressions usage. You can also use database storage for bigger files.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.