Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 1 down vote favorite

I've got lists of 60,000+ words in a text file. I want to be able to sort the words and do complex computations such as removing duplicates, concatenating every word with a string and other similar tasks.

What language or tool can manipulate large amounts of text from a file?

share|improve this question
There's a so called "Practical Extraction and Report Language", which seems to be ok for this and is available on almost every platform. The abbreviation of this language is P.E.R.L or, which is the official name: Perl. In Perl, I'd guess your requirement: "removing duplicates, concatenating every word with a string" wouldn't exaclty constitute "complex computations" ... Maybe you wouldn't even need a "program" at all. What exactly would you like to do? – rubber boots Mar 20 '12 at 16:50
1  
bash and plain unix commands can do this easily – Jarrod Roberson Mar 20 '12 at 16:56
c# of course:) If seriously, choosing the proper language for some tasks should be done among languages you know. But in your case almost any language will be good. – the_joric Mar 20 '12 at 18:24
2  
Many text editors have that kind of functionality built in. It doesnt make sense to ask what programming language to use, because essentially all of them can be used to do that. – GrandmasterB Mar 20 '12 at 18:36
How about not reinventing the wheel and just use Excel? It can easily solve your issues. – Claudiu Constantin Mar 20 '12 at 18:52
show 2 more comments

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer: please explain why you're recommending it as a solution. Answers that don't explain anything will be deleted. See Good Subjective, Bad Subjective for more information.

5 Answers

up vote 15 down vote

I think you could do this with plain linux bash scripts. 60,000 words isn't really that many, in this age of multi-GHz CPUs. The following Linux command seemed instaneous on an older rack-mounted RHEL server:

4 % wc -l /usr/share/dict/words
479623 /usr/share/dict/words

That's a lot more than 60,000 words. Between sed, awk, xargs, grep, paste, join, sort, uniq you'll have most or all of what you want to do.

My advice is to develop text manipulation commands as pipelines, using more as the final stage:

1 % cat filename1 filename2 | tr -cs "[:alpha:]" "\n" | more   # there's your words
2 % cat filename1 filename2 | tr -cs "[:alpha:]" "\n" | sort | uniq | more # unique words
3 % cat filename1 filename2 | tr -cs "[:alpha:]" "\n" | sort | uniq -c | sort +1 -1 | more

You just use terminal editing features to add another stage to the end of the pipeline until you get what you want. Then, cut-n-paste the final pipeline into a text file, make that into a shell script, and you've got a software tool.

share|improve this answer
2  
+1, And if the processing does end up taking a long time, this can also take advantage of multiple cores, since each step is a separate running process. – Izkata Mar 20 '12 at 18:27
up vote 10 down vote

Practically any language is capable of those operations. However, languages like python, perl, and php are particularly well-suited to those kinds of tasks, which is one of the reasons they are frequently used for web programming. Features like native regular expressions, intuitive string operations like split, easy line-wise file processing, full-featured string formatting, templating, and easy creation of dynamic data structures like hash tables are what make them stand out.

share|improve this answer
4  
+1 Perl was designed to do precisely what @Abdulla is asking for, but you are right that Python is quite good for doing text manipulation as well. – Matthew Flynn Mar 20 '12 at 15:05
2  
I would not really want to use PHP for this, sure you could do it but its really not the best tool for the job. Awk would be my choice – Zachary K Mar 20 '12 at 16:32
There are a couple of Python books that cover this area: Python 2.6 Text Processing Beginners Guide by Jeff McNeil and Text Processing in Python by David Mertz. – tcrosley Mar 20 '12 at 19:04
This is not an answer - despite the votes - please explain why python, perl and php are "particularly well-suited" to these tasks. – ChrisF♦ Mar 20 '12 at 21:39
up vote 5 down vote

I would say check out awk, its old school (really old school) but it will make doing what you want easy. Any unix will have it installed and it should be in cygwin if you use that

You can look at the book "Sed and Awk"

share|improve this answer
A bit more detail would go a long way here. – ChrisF♦ Mar 20 '12 at 21:43
up vote 1 down vote

I'd recommend importing them into an Sqlite base and performing everything you need with a plain SQL. When I wanted some statistics about letters of English alphabet, I will get final sequences and numbers in about 10-15 minutes.

share|improve this answer
up vote 0 down vote

SNOBOL was originally developed for exactly that purpose. See also SNOBOL4.ORG -- Resources.

Decades (probably man-centuries if not man-millenia) of work has been done in LISP for this kind of thing.

share|improve this answer
I used SNOBOL4 extensively on a time-sharing network in the early 1970s -- racking up hundreds of dollars per month in connection fees. The lab never complained because I was so productive. – tcrosley Mar 20 '12 at 19:07
Snobol. Seriously? – Robert Harvey Mar 20 '12 at 21:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.