Tag Info

Hot answers tagged

33

Have you considered localization? It may look simple to write something like: var text = (count > 0) ? "items" : "item" But it's nowhere near that simple when you have to work in multiple languages. Here's an example, just using Google Translate: Language | Singular | Plural ...


20

Every popular language has facilities for text editing and processing; it's mostly a matter of personal preference. Since you are already familiar with PHP, I'd suggest Perl, as the syntax is similar, and PHP's regular expression facilities are heavily based on Perl's. Furthermore, Perl was originally designed as a text processing language, and although it ...


13

This is called Natural Language Processing and it's a huge, complex field. Something like you describe is a monumental achievement, and even the best solutions, like Watson, are nowhere near perfect. Things like this make it challenging: "Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo" a grammatically correct sentence in American ...


11

As you are on Linux, I would suggest looking at Tcl, Perl and Python. Depending on what you actually want to do, these three should be able to handle it. Also educate yourself about sed, AWK and grep, etc... and use Bash, sh, and or tcsh scripts to your advantage also. (There is no reason why you cannot use a sh script to split the work into three bits and ...


10

Perl and Ruby would be good choices. However, what you really want might be AWK, which is rather old, but it will do what you want. There are a few good books on AWK, but all of them are more than 10 years old. However, don't let that phase you; it's still a powerful tool.


6

It's hard to say what is best objectively. My personal favourite for these kind of applications is Clojure, and here are the reasons why I think it is particularly strong on this space: It's a Lisp - so you get all the power of "Beating the Averages". Basically Lisp combines the fact that is is homoiconic with a very sophisticated macro system that ...


6

For natural language processing, a traditional fit is Prolog. The book Prolog Programming in Depth has a single chapter devoted to using Prolog for NLP. According to this book, Prolog is a nice fit for NLP because of it's built-in knowledge representation system and unification mechanism. According to Wikipedia (http://en.wikipedia.org/wiki/Prolog), : ...


5

Solutions to this problem are often ambiguous, and it's sometimes difficult to decide an optimum solution, even for a human. If you need only a single solution where all words exist in a dictionary, then a naive approach will fail as soon as you encounter a word which prefix is also a valid dictionary entry. For example, if your input string was: ...


5

Three candidates here: Python would be my favourite. It has superb string handling and a vast number of libraries to help you parse text. It also has superb support for structures (Dictionaries,linked lists etc.) which ironically you will really need when trying to make sense of unstructured data. perl for much the same reason as Python, the language ...


5

I do not think you will gain much, if anything, in performance, by using memory-mapped files instead of performing normal text-file processing. From the moment that you change the length of a single line even by just one byte, the remainder of the file will need to be read, shifted by one byte, and written back to disk. From the point of view of I/O, this is ...


5

This sounds like an interpreter. It looks like you are worried about the implementation more than the detailed functionality (I am only guessing here). This project if extended is not a trivial task. Make sure you study the scope clearly as this requires engineering approach and not an ad-hoc development approach to get a reliable product rather than a ...


5

Imlementing "1 result(s)" is easier and faster. EDIT: And it makes the code shorter and therefore easier to understand.


4

Though splitting a sentence and determining the grammatical correctness along with solving your first problem is easier than your second problem, many complexities like verb-nouns or gerunds like swimming, programming, etc and other such intricacies, it still is a challenge - See Morons' answer. But your second problem - people have put in huge efforts to ...


4

I'd say do the filtering as part of the processing. Programming in Excel is significantly more painful and limited than any server-side technology you could possibly be using. CSV as an output format is much easier to work with than Excel proper, and virtually every programming language can easily output CSV without requiring any libraries (even writing ...


4

Learn Regular Expressions (regex) Regex is a small technology that you can master in a week or two (1-2 hours/day), but it's so useful that it will pay off that investment of time on the first project you use it on. Most programming languages support them including PHP. There is a wonderful book on regex, Mastering Regular Expressions by Jeff Friedl. It ...


4

A very rough outline how this could be done with Apache Solr. Solr is a full text search engine with many setup options and very flexible ways to handle the indexing and faceting. Using the right combinations of tokenizers (split text in single elements, mostly words) and filters (post process the tokens like removing stop words as "a", "and", "I" etc or ...


4

Yes, there are, and no, they don't work very well. Deducing information about the author from a text is sub-discipline of natural language processing - most NLP applications are about extracting information about the content of a text rather than the author, but the goals, methods and state of the art are actually rather similar (currently this favors ...


3

Actually you can use PHP on the command line, so the assertion that it cannot be used offline is false. If you're already familiar with PHP and its string handling functions you should look into it. I used to work with PHP and there were some behind the scenes stuff done with PHP scripts that could be run on the command line (with cron jobs even). It was a ...


3

It REALLY depends on what you need to do. If you mainly want to recognize things, and your input is lines, each line consisting of fields delimited by spaces, AWK is easy to learn, easy to use, and quite effective for what it does. I've never really studied Perl, so I can't really comment on it. Ditto for Ruby and Python. I don't think there's anything ...


3

In general, I think that it really depends on the aptitude of the non-IT people. Some will have it, others won't. (And my gut feeling is that most of them won't ... given the difficulty that a lot of IT-trained people seem to have with regexes.) Is this a valid goal to make some users regexp-literate through appropriate training ? Define a "valid ...


3

I think you should start reading this Wikipedia article: http://en.wikipedia.org/wiki/Part-of-speech_tagging (it is a research field, don't expect any easy solution for it.)


3

The best language I'm aware of specific to text search and processing is awk. If awk doesn't meet your needs, it's likely nothing will unless you create it yourself. However, if you do need to make your own, you don't need to start completely from scratch for each language. You can use a tool like antlr that can be exported to various languages, or build ...


3

Unless you have an extensive background in natural language processing, I might use an existing library versus rolling your own named-entity recognizer. Even if you're a NLP expert, a heavily-used library will likely have many more testing hours committed to it than your new recognizer. In addition, existing libraries are likely to be more flexible since ...


2

It's better to have a separate LICENSE file instead. However, if you really want to do this, you can use a script: for i in `command_giving_your_PHP_files`; do cp $i $i.bak cat LICENSE > $i cat $i.bak >> $i done Assuming you use Linux or an UNIX environment


2

You should have some kind of templating mechanism in place (at the very least, a header.php and footer.php that you include in every page). If you do, then it should be trivial to include a 'copyright message' in a part of the page that gets included everywhere already (typically, the footer); how exactly you do this depends on the template system. If you ...


2

You can use Lucene Term Vectors. Here is a blog post explaining it in detail. http://sujitpal.blogspot.com/2011/10/computing-document-similarity-using.html Lucene is an indexing library: http://lucene.apache.org/java/docs/index.html


2

Wouldn't you just need to reverse index the words in the various fields back to the page? As a simplistic example, break down each Product Name (say, by spaces) and normalize each keyword (say, lowercase, throw out punctuation), and then map it to the product's URL or record. Searching for "iphone" would then give you close to the right results. You even ...


2

Looks like Stanford has some good references: http://nlp.stanford.edu/software/ With an interesting demo: http://nlp.stanford.edu:8080/parser/ "My dog also likes eating sausage." "My/PRP$ dog/NN also/RB likes/VBZ eating/VBG sausage/NN"


2

Unless you're specifically interested in writing the actual parser for yourself, I'd suggest having a look at one of the parser generator frameworks. For C you have YACC or Bison, but there should be other alternatives for other languages if you prefer. These takes away the complexities of parsing complex grammars and lets you concentrate on the task you ...


2

What you are describing is very close to a stack language. For instance in Factor what you describe would be done like 1 2 + 7 * even? not Or you could define your own words and then use them, like : add ( x y -- sum ) + ; : multiply ( x y -- product ) * ; : odd? ( n -- ? ) even? not ; With these definitions the above example becomes 1 2 add 7 ...



Only top voted, non community-wiki answers of a minimum length are eligible