How do I separate words in a string?
In the following I have a random sample of words in a string extracted from text file with over a million words.
Here's the string: "intervene Pockets Jerusalemand tissues powerful days gadgets rate invention heated Stewartis find communal working lots internal They caused noise offer goand big jumps lose galaxies All Mall birth child world spin killed produced great center Dont pretend Africa solution times cursor good night professional problem refugees talk"
As you can see, the third word is "Jerusalemand." My goal is to separate "Jerusalem" and "and", and do the same thing for any other words that are stuck together.
Only thing that I could think of as of now is to compare every word in the string to a dictionary (SCOWL perhaps?), and if a segment of the word matches a word in the dictionary, I'll have to split the words to make them independent.
As an example, for "Jerusalemand" I'll loop through every character until a match is found in the dictionary for "Jerusalem," then I can separate that from "and," in turn completing the separation.
Shouldn't there be a better way to do this?