What does the Jamie Zawinski's quotation about regular expressions mean?

Question

In a message to comp.emacs.xemacs, Jamie Zawinski once said:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

I've always had trouble understanding what he was getting at. What does he mean by this?

Update

The answer that I'm looking for is one which explains what the 2nd problem is. Most answers below are that regexes are hard, which doesn't seem to fit the question.

Answer 1

Regular expressions - particularly non trivial ones - are difficult to code, understand and maintain. You only have to look at the number of questions on Stack Overflow tagged [regex] where the questioner has assumed that the answer is a regex had has got stuck. In a lot of cases the problem can (and perhaps should) be solved a different way.

This means that, if you decide to use a regex you now have two problems:

1. The original problem you wanted to solve.
2. The support of a regex.

Basically, I think he means you should only use a regex if there's no other way of solving your problem. Any other solution is going to be easier to code, maintain and support.

Answer 2

RegularExpressionsarenoworsetoreadormaintainthananyotherunformattedcontent;indeedaregexisprobablyeasiertoreadthanthispieceoftexthere-butunfortunatelytheyhaveabadreputationbecausesomeimplementationsdon'tallowformattingandpeopleingeneraldon'tknowthatyoucandoit.

_{^{(Regular Expressions are no worse to read or maintain than any other unformatted content; indeed a regex is probably easier to read than this piece of text here - but unfortunately they have a bad reputation because some implementations don't allow formatting and people in general don't know that you can do it.)}}

Here's a trivial example:

^(?:[^,]*+,){21}[^,]*+$

Which isn't really that difficult to read or maintain anyway, but is even easier when it looks like this:

(?x)    # enables comments, so this whole block can be used in a regex.
^       # start of string

(?:     # start non-capturing group
  [^,]*+  # as many non-commas as possible, but none required
  ,       # a comma
)       # end non-capturing group
{21}    # 21 of previous entity (i.e. the group)

[^,]*+  # as many non-commas as possible, but none required

$       # end of string

That's a bit of an over-the-top example (commenting $ is akin to commenting i++) but clearly there should be no problem reading, understanding, and maintaining that.

So long as you're clear as to when regular expressions are suited and when they're a bad idea, there's nothing wrong with them, and most times the JWZ quote doesn't really apply.

Answer 3

Regular expressions are very powerful, but they have one small and one big problem; they are hard to write, and near impossible to read.

In a best case the use of the regular expression solves the problem, so then you only have the maintenance problem of the complicated code. If you don't get the regular expression just right, you have both the original problem and the problem with unreadable code that doesn't work.

Sometimes regular expressions are referred to as write-only code. Faced with a regular expression that needs fixing, it's often faster to start from scratch than to try to understand the expression.

Answer 4

In addition to ChrisF's answer - that regular expressions "are difficult to code, understand and maintain", there's worse: they're just powerful enough to trick people into trying to use them to parse things they can't, like HTML. See the numerous questions on SO on "how do I parse HTML?" For instance, the single most epic answer in all of SO!

Answer 5

The meaning has two parts:

• First, you didn't solve the original problem.
  This probably refers to the fact that regular expressions often offer incomplete solutions to common problems.
• Second, you now added additional difficulty associated with the solution you've picked.
  In the case of regular expressions, the additional difficulty probably refers to complexity, maintainability, or the additional difficulty associated with making regular expressions fit a problem it wasn't supposed to solve.

Answer 6

If there is one thing you should learn from computer science, it is Chomsky hierarchy. I would say that all problems with regular expressions come from attempts to parse context-free grammar with it. When you can impose a limit (or think you can impose a limit) to nesting levels in CFG, you get those long and complex regular expressions.

Answer 7

Regular expressions are more suitable for tokenisation than for full-scale parsing.

But, a surprisingly large set of things that programmers need to parse are parseable by a regular language (or, worse, almost parseable by a regular language and if you only write a little more code...).

So if one is habituated to "aha, I need to pick text apart, I'll use a regular expression", it's easy to go down that route, when you need something that's closer to a push-down automaton, a CFG parser or even more powerful grammars. That usually ends in tears.

So, I think the quote isn't so much slamming regexps, they have their use (and well-used, they're very useful indeed), but the over-reliance on regexps (or, specifically, the uncritical choice of them).

Answer 8

jwz is simply off his rocker with that quote. regular expressions are no different than any language feature - easy to screw up, hard to use elegantly, powerful at times, inappropriate at times, often well documented, often useful.

the same could be said for floating point arithmetic, closures, object-orientation, asynchronous I/O, or anything else you can name. if you don't know what you are doing, programming languages can make you sad.

if you think regexes are hard to read, try reading the equivalent parser implementation for consuming the pattern in question. often regexes win because they are more compact than full parsers...and in most languages, they are faster as well.

don't be put off of using regular expressions (or any other language feature) because a self-promoting blogger makes unqualified statements. try things out for yourself and see what works for you.

Answer 9

My favourite, in-depth answer to this is given by the famous Rob Pike in a blog post reproduced from an internal Google code comment: http://commandcenter.blogspot.ch/2011/08/regular-expressions-in-lexing-and.html

The summary is that it's not that they are bad, but they are frequently used for tasks for whcih they are not necessarily suited, especially when it comes to lexing and parsing some input.

Regular expressions are hard to write, hard to write well, and can be expensive relative to other technologies... Lexers, on the other hand, are fairly easy to write correctly (if not as compactly), and very easy to test. Consider finding alphanumeric identifiers. It's not too hard to write the regexp (something like "[a-ZA-Z_][a-ZA-Z_0-9]*"), but really not much harder to write as a simple loop. The performance of the loop, though, will be much higher and will involve much less code under the covers. A regular expression library is a big thing. Using one to parse identifiers is like using a Ferrari to go to the store for milk.

He says a lot more than that, arguing that regular expressions are useful in, e.g. disposable matching of patterns in text editors but should rarely be used in compiled code, and so on. It's worth a read.

Answer 10

This is related to Alan Perlis' epigram #34:

The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

So if your choose the character string as your data structure (and, naturally, regex-based code as the algorithms to manipulate it), you have a problem, even if it works: bad design around an inappropriate representation of data which is hard to extend, and inefficient.

However, often it doesn't work: the original problem isn't solved, and so in that case you have two problems.