Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

There is a popular quote by Jamie Zawinski:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

How is this quote supposed to be understood?

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

34  
The 2nd problem is that they are using regex and still haven't solved the first problem, hence 2 problems. –  Ampt Jan 9 at 17:57
21  
@Euphoric - actually, good code is short - but without being cryptically concise. –  Steve314 Jan 9 at 18:30
20  
@IQAndreas: I think it is intended to be semi-humorous. The comment that's being made is that if you are not careful, using regular expressions can make things worse instead of better. –  FrustratedWithFormsDesigner Jan 9 at 19:24
102  
Some people, when trying to explain something, think "I know, I'll use a Jamie Zawinski quote." Now they have two things to explain. –  detly Jan 9 at 23:22
26  

17 Answers 17

As you ask for it in 2014, it would be interesting to focus on programming languages ideologies of 1997 context comparing to today's context. I will not enter this debate here but opinions about Perl and Perl itself have greatly changed.

However, to stay in a 2013 context (de l'eau a coulé sous les ponts depuis), I would suggest to focus on reenactment in quotes using a famous XKCD comic that is a direct quote of Jamie Zawinski's one :

A comic from XKCD about regexes, Perl and problems

First I had problems to understand this comic because it was a reference to the Zawinski quote, and a quote of a Jay-z song lyrics, and a reference of GNU program --help -z flag2, so, it was too much culture for me to understand it.

I knew it was fun, I was feeling it, but I didn't really know why. People are often doing jokes about Perl and regexes, especially since it's not the hipstiest programming language, don't really know why it is supposed to be fun... Maybe because Perl mongers do funny-weirdo things.

So the initial quote seems to be a sarcastic joke based on real life problems (pain?) caused by programming with tools that hurts. Just like a hammer can hurt a mason, programming with tools that are not the ones that a developer would choose if he could can hurt (the brain, the feelings). Sometimes, great debates about which tool is the best occurs, but it's almost worthless cause it's a problem of your taste or your programming team taste, cultural or economic reasons. Another excellent XKCD comic about this :

A comic from XKCD about programming tools debates

I can understand people feeling pain about regexes, and they do believe that another tool is better suited for what regexes are designed for. As @karl-bielefeldt answers your question with great expressivity comes great responsibility, and regexes are especially concerned by this. If a developer don't care of how s-he deals with regexes, it will eventually be a pain for people who will maintain the code later.

I will finish with this answer about quotes reenactment by a quote showing a typical example from Damian Conw ay's Perl Best Practices (a 2005 book).

He explains that writing a pattern like this:

m{'[^\\']*(?:\\.[^\\']*)*'}

...is no more acceptable than writing a program like this:

sub'x{local$_=pop;sub'_{$_>=$_[0
]?$_[1]:$"}_(1,'*')._(5,'-')._(4
,'*').$/._(6,'|').($_>9?'X':$_>8
?'/':$")._(8,'|').$/._(2,'*')._(
7,'-')._(3,'*').$/}print$/x($=).
x(10)x(++$x/10).x($x%10)while<>;

But it can be rewritten, it's still not pretty, but at least it's now survivable.

# Match a single-quoted string efficiently...
m{ '            # an opening single quote
    [^\\']*     # any non-special chars (i.e., not backslash or single quote)
    (?:         # then all of...`
    \\ .        # any explicitly backslashed char
    [^\\']*     #    followed by any non-special chars
    )*          # ...repeated zero or more times
    '           # a closing single quote
}x

This kind of rectangular shaped code is the second problem not regexes that can be formatted in a clear, maintainable and readable way.

share|improve this answer

Some programming technologies are not generally well-understood by programmers (regular expressions, floating point, Perl, AWK... and others).

These can be amazingly powerful tools for solving the right set of problems. Regular expressions in particular are very useful for matching regular languages. And there is the crux of the problem: few people know how to describe a regular language (it's part of computer science theory / linguistics that uses funny symbols - you can read about it at Chomsky hierarchy).

When dealing with these things, if you use them wrong it is unlikely that you've actually solved your original problem. Using a regular expression to match HTML (a far too common occurrence) will mean that you will miss edge cases. And now, you've still got the original problem that you didn't solve, and another subtle bug floating around that has been introduced by using the wrong solution.

This is not to say that regular expressions shouldn't be used, but rather that one should work to understand what the set of problems they can solve and can't solve and use them judiciously.

The key to maintaining software is writing maintainable code. Using regular expressions can be counter to that goal. When working with regular expressions, you've written a mini computer (specifically a non-deterministic finite state automaton) in a special domain specific language. It's easy to write the 'Hello world' equivalent in this language and gain rudimentary confidence in it, but going further needs to be tempered with the understanding of the regular language to avoid writing additional bugs that can be very hard to identify and fix (because they aren't part of the program that the regular expression is in).

So now you've got a new problem; you chose the tool of the regular expression to solve it (when it is inappropriate), and you've got two bugs now, both of which are harder to find, because they're hidden in another layer of abstraction.

share|improve this answer
4  
I'm not sure perl itself belongs in a list of technologies not well-understood by programmers ;) –  crad Jan 9 at 22:02
7  
@crad its more that it has been said about perl too... Many people have heard it popularized there. I still like the floating point one in the rand talk: "Now you have 2.00000152 problems" –  MichaelT Jan 9 at 22:05
34  
@crad Some people, when confronted with a problem, think "I know, I'll use perl." Now they have $(^@#%()^%)(#) problems. –  Michael Hampton Jan 10 at 0:54
1  
@Jens if anything, the additional power of the PCRE vs traditional regex makes it a more tempting solution and a more difficult to maintain one. The finite automata that the PCRE matches is explored in Extending Finite Automata to Efficiently Match Perl-Compatible Regular Expressions..‌​. and its a non-trivial thing. At least with the traditional regex, one can get their head around it without too much trouble once the necessary concepts are understood. –  MichaelT Jan 10 at 15:23
2  
You make a good point. regular expressions are effectively a second, non-trivial language. Even if the original programmer is competent in the main language and the flavor of regex used, adding in a "second language" means lower odds that maintainers will know both. Not to mention that regex readability is often lower than the "host" language. –  JS. Jan 10 at 18:25

There are a few things going on with this quote.

  1. The quote is a restatement of an earlier joke:

    Whenever faced with a problem, some people say "Lets use AWK." Now, they have two problems. — D. Tilbrook

    It is a joke and a real dig, but it's also a way of highlighting regex as a bad solution by linking it with other bad solutions. It's a great ha ha only serious moment.

  2. To me, mind you this quote is purposely open to interpretation, the meaning is straight forward. Simply announcing the idea of using a regular expression has not solved the problem. In addition, you've increased the cognitive complexity of the code by adding an additional language with rules that stand apart from whatever language you are using.

  3. Although funny as a joke, you need to compare the complexity of a non-regex solution with the complexity of the regex solution + the additional complexity of including regexes. It may be worth while to solve a problem with a regex despite the additional cost of adding regexes.

share|improve this answer

Jeff Atwood brings out a different interpretation in a blog post discussing this very quote: Regular Expressions: Now You Have Two Problems (thanks to Euphoric for the link)

Analyzing the full text of Jamie's posts in the original 1997 thread, we find the following:

Perl's nature encourages the use of regular expressions almost to the exclusion of all other techniques; they are far and away the most "obvious" (at least, to people who don't know any better) way to get from point A to point B.

The first quote is too glib to be taken seriously. But this, I completely agree with. Here's the point Jamie was trying to make: not that regular expressions are evil, per se, but that overuse of regular expressions is evil.

Even if you do fully understand regular expressions, you run into The Golden Hammer problem, trying to solve a problem with regular expressions, when it would have been easier and more clear to do the same thing with regular code (see also CodingHorror: Regex use vs. Regex abuse).

There is another blog post which looks at the context of the quote, and goes into more detail than Atwood: Jeffrey Friedl's Blog: Source of the famous “Now you have two problems” quote

share|improve this answer
1  
This is, to my mind, the best answer because it adds context. jwz's criticism of regexes was as much about Perl as anything. –  Evicatos Jan 9 at 20:15
2  
@Evicatos There was even more research done on the same 1997 thread in another blog post: regex.info/blog/2006-09-15/247 –  IQAndreas Jan 9 at 20:30

It's mostly a tongue-in-cheek joke, albeit with a grain of truth.

There are some tasks for which regular expressions are an excellent fit. I once replaced 500 lines of manually written recursive descent parser code with one regular expression that took around 10 minutes to fully debug. People say regexes are hard to understand and debug, but appropriately-applied ones are not nearly as hard to debug as a huge hand-designed parser. In my example, it took two weeks to debug all the edge cases of the non-regex solution.

However, to paraphrase Uncle Ben:

With great expressivity comes great responsibility.

In other words, regexes add expressivity to your language, but that puts more responsibility on the programmer to choose the most readable mode of expression for a given task.

Some things initially look like a good task for regular expressions, but aren't. For example, anything with nested tokens, like HTML. Sometimes people use a regular expression when a simpler method is more clear. For example, string.endsWith("ing") is easier to understand than the equivalent regex. Sometimes people try to cram a large problem into a single regex, where breaking it into pieces is more appropriate. Sometimes people fail to create appropriate abstractions, repeating a regex over and over instead of creating a well-named function to do the same job (perhaps implemented internally with a regex).

For some reason, regexes have a weird tendency to create a blind spot to normal software engineering principles like single responsibility and DRY. That's why even people who love them find them problematic at times.

share|improve this answer
4  
Didn't Uncle Ben also say "Perfect results, every time"? Maybe that's why people get so trigger happy with regexes... –  Andrzej Doyle Jan 10 at 15:52

The problem is that regex is a complicated beast, and you only solve your problem if you use regex perfectly. If you don't, you end up with 2 problems: your original problem and regex.

You claim that it can do the work of a hundred lines of code, but you could also make the argument that 100 lines of clear, concise code is better than one line of regex.

If you need some proof of this: You can check out this SO Classic or simply comb through the SO Regex Tag

share|improve this answer
7  
Neither of the claims in your first sentence are true. Regex is not particularly complicated, and like no other tool do you need to know it perfectly in order to solve problems with it. That’s just FUD. Your second paragraph is plain ridiculous: of course you can make the argument. But it’s not a good one. –  Konrad Rudolph Jan 10 at 10:21
1  
@KonradRudolph I think the fact that there are numerous regex generation and validation tools goes to show that regex is a complicated mechanism. It's not human readable (by design) and can cause a complete change in flow for someone modifying or writing a piece of code which uses regex. As to the second part, I think it's clear in it's implication from the vast grouping of knowledge on P.SE and by the saying "Debugging code is twice as hard as writing it, so if you write the most clever code you can, you are, by definition, not smart enough to debug it" –  Ampt Jan 10 at 16:30
2  
That’s not a proper argument. Yes, sure regex are complex. But so are other programming languages. Regex is considerably less complex than most other languages, and the tools that exist for regex are dwarfed by development tools for other languages (FWIW I work extensively with regex and I’ve never used such tools …). It’s a simple truth that even complex regex are simpler than equivalent non-regex parsing code. –  Konrad Rudolph Jan 10 at 17:29

Because most people do not fully understand regular expressions, they do not properly solve their problem with them. Thus, your regular expression is a problem and your initial issue is still a problem.

share|improve this answer
15  
Even if you understand them and use them properly, they make the code harder to support for the wider "We don't get regular expressions" masses. –  Satanicpuppy Jan 9 at 18:21
4  
@Steve314: Hell, half the time I can't read my own regex if I did something unusual. It's just so useful though... –  Satanicpuppy Jan 9 at 18:55
7  
“most people” don’t fully understand any programming code. What a bad criterion. –  Konrad Rudolph Jan 10 at 10:21
3  
@KonradRudolph I'd say that most people don't fully understand How to Answer at Programmers - as evidenced by +27/-4 score of this low quality answer –  gnat Jan 10 at 14:01
2  
Why do people keep upvoting this low-quality answer?! It directly contradicts the post notice above: “Don't just give a one-line answer; explain why your answer is right, ideally with citations.” –  Konrad Rudolph Jan 11 at 10:04

Regular expressions are very powerful, but they have one small and one big problem; they are hard to write, and near impossible to read.

In a best case the use of the regular expression solves the problem, so then you only have the maintenance problem of the complicated code. If you don't get the regular expression just right, you have both the original problem and the problem with unreadable code that doesn't work.

Sometimes regular expressions are referred to as write-only code. Faced with a regular expression that needs fixing, it's often faster to start from scratch than to try to understand the expression.

share|improve this answer
1  
The real problem is that regexps cannot implement e.g. a parser since they cannot count how deeply nested they currently are. –  user1249 Aug 7 '11 at 9:21
4  
@Thorbjørn Ravn Andersen: That's more of a limitation than a problem. It's only a problem if you try to use regular expressions for that, and then it's not a problem with the regular expressions, it's a problem with your choise of method. –  Guffa Aug 7 '11 at 11:28
1  
You can use REs just fine for the lexer (well, for most languages) but assembling the token stream into a parse tree (i.e., parsing) is formally beyond them. –  Donal Fellows Dec 6 '11 at 15:56

This is related to Alan Perlis' epigram #34:

The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

So if your choose the character string as your data structure (and, naturally, regex-based code as the algorithms to manipulate it), you have a problem, even if it works: bad design around an inappropriate representation of data which is hard to extend, and inefficient.

However, often it doesn't work: the original problem isn't solved, and so in that case you have two problems.

share|improve this answer

My favourite, in-depth answer to this is given by the famous Rob Pike in a blog post reproduced from an internal Google code comment: http://commandcenter.blogspot.ch/2011/08/regular-expressions-in-lexing-and.html

The summary is that it's not that they are bad, but they are frequently used for tasks for whcih they are not necessarily suited, especially when it comes to lexing and parsing some input.

Regular expressions are hard to write, hard to write well, and can be expensive relative to other technologies... Lexers, on the other hand, are fairly easy to write correctly (if not as compactly), and very easy to test. Consider finding alphanumeric identifiers. It's not too hard to write the regexp (something like "[a-ZA-Z_][a-ZA-Z_0-9]*"), but really not much harder to write as a simple loop. The performance of the loop, though, will be much higher and will involve much less code under the covers. A regular expression library is a big thing. Using one to parse identifiers is like using a Ferrari to go to the store for milk.

He says a lot more than that, arguing that regular expressions are useful in, e.g. disposable matching of patterns in text editors but should rarely be used in compiled code, and so on. It's worth a read.

share|improve this answer

The meaning has two parts:

  • First, you didn't solve the original problem.
    This probably refers to the fact that regular expressions often offer incomplete solutions to common problems.
  • Second, you now added additional difficulty associated with the solution you've picked.
    In the case of regular expressions, the additional difficulty probably refers to complexity, maintainability, or the additional difficulty associated with making regular expressions fit a problem it wasn't supposed to solve.
share|improve this answer

If there is one thing you should learn from computer science, it is Chomsky hierarchy. I would say that all problems with regular expressions come from attempts to parse context-free grammar with it. When you can impose a limit (or think you can impose a limit) to nesting levels in CFG, you get those long and complex regular expressions.

share|improve this answer
1  
Yes! People who learning regular expressions without that part of CS background don't always understand that there are just some things that a regex mathematically cannot do. –  benzado Jun 22 '11 at 16:49

Regular expressions are more suitable for tokenisation than for full-scale parsing.

But, a surprisingly large set of things that programmers need to parse are parseable by a regular language (or, worse, almost parseable by a regular language and if you only write a little more code...).

So if one is habituated to "aha, I need to pick text apart, I'll use a regular expression", it's easy to go down that route, when you need something that's closer to a push-down automaton, a CFG parser or even more powerful grammars. That usually ends in tears.

So, I think the quote isn't so much slamming regexps, they have their use (and well-used, they're very useful indeed), but the over-reliance on regexps (or, specifically, the uncritical choice of them).

share|improve this answer

jwz is simply off his rocker with that quote. regular expressions are no different than any language feature - easy to screw up, hard to use elegantly, powerful at times, inappropriate at times, often well documented, often useful.

the same could be said for floating point arithmetic, closures, object-orientation, asynchronous I/O, or anything else you can name. if you don't know what you are doing, programming languages can make you sad.

if you think regexes are hard to read, try reading the equivalent parser implementation for consuming the pattern in question. often regexes win because they are more compact than full parsers...and in most languages, they are faster as well.

don't be put off of using regular expressions (or any other language feature) because a self-promoting blogger makes unqualified statements. try things out for yourself and see what works for you.

share|improve this answer
1  
FWIW, floating point arithmetic is waaay more tricky than REs, but appears simpler. Beware! (At least tricky REs tend to look dangerous.) –  Donal Fellows Dec 6 '11 at 15:59

In addition to ChrisF's answer - that regular expressions "are difficult to code, understand and maintain", there's worse: they're just powerful enough to trick people into trying to use them to parse things they can't, like HTML. See the numerous questions on SO on "how do I parse HTML?" For instance, the single most epic answer in all of SO!

share|improve this answer

Regular expressions - particularly non trivial ones - are difficult to code, understand and maintain. You only have to look at the number of questions on Stack Overflow tagged [regex] where the questioner has assumed that the answer is a regex had has got stuck. In a lot of cases the problem can (and perhaps should) be solved a different way.

This means that, if you decide to use a regex you now have two problems:

  1. The original problem you wanted to solve.
  2. The support of a regex.

Basically, I think he means you should only use a regex if there's no other way of solving your problem. Any other solution is going to be easier to code, maintain and support.

share|improve this answer
18  
And worse: they're just powerful enough to trick people into trying to use them to parse things they can't, like HTML. See the numerous questions on SO on "how do I parse HTML?" –  Frank Shearar Oct 11 '10 at 14:53
4  
For certain situations regex is awesome. In many other cases not so much. At the other end it is a horrifying pit of despair. The problem often arises when someone learns about them for the first time and starts to see applications everywhere. Another famous saying: "When the only tool you have is a hammer, everything looks like a nail." –  Todd Williamson Oct 11 '10 at 15:10
1  
Does this mean that by the number of questions in the SO [c#] tag, it is the hardest programming language to understand? –  Roger Pate Oct 29 '10 at 9:45
2  
I would much rather see a complex regular expression than a long series of calls to string methods. OTOH, I really hate seeing regular expressions misused to parse complex languages. –  kevin cline May 20 '11 at 3:57
3  
"Basically, I think he means you should only use a regex if there's no other way of solving your problem. Any other solution is going to be easier to code, maintain and support." - seriously disagree.. Regexes are excellent tools, you just have to know their limits. A lot of tasks can be coded more elegantly with regexes. (but, just to make an example, you shouldn't use them to parse HTML) –  Karoly Horvath Dec 5 '11 at 23:30

RegularExpressionsarenoworsetoreadormaintainthananyotherunformattedcontent;indeedaregexisprobablyeasiertoreadthanthispieceoftexthere-butunfortunatelytheyhaveabadreputationbecausesomeimplementationsdon'tallowformattingandpeopleingeneraldon'tknowthatyoucandoit.

(Regular Expressions are no worse to read or maintain than any other unformatted content; indeed a regex is probably easier to read than this piece of text here - but unfortunately they have a bad reputation because some implementations don't allow formatting and people in general don't know that you can do it.)


Here's a trivial example:

^(?:[^,]*+,){21}[^,]*+$


Which isn't really that difficult to read or maintain anyway, but is even easier when it looks like this:

(?x)    # enables comments, so this whole block can be used in a regex.
^       # start of string

(?:     # start non-capturing group
  [^,]*+  # as many non-commas as possible, but none required
  ,       # a comma
)       # end non-capturing group
{21}    # 21 of previous entity (i.e. the group)

[^,]*+  # as many non-commas as possible, but none required

$       # end of string

That's a bit of an over-the-top example (commenting $ is akin to commenting i++) but clearly there should be no problem reading, understanding, and maintaining that.


So long as you're clear as to when regular expressions are suited and when they're a bad idea, there's nothing wrong with them, and most times the JWZ quote doesn't really apply.

share|improve this answer
1  
Sure, but I'm not looking for discussions of the merits of regexs, and I wouldn't like to see this discussion go that way. I'm just trying to understand what he was getting at. –  Paul Biggar Oct 11 '10 at 14:05
1  
Then the link in livibetter's comment tells you what you need to know. This response is just pointing out that regexes do not need to be obscure, and thus the quote is nonsense. –  Peter Boughton Oct 11 '10 at 14:57
6  
What’s the point of using *+? How is that any different (functionally) from just *? –  Timwi Jan 12 '11 at 18:38
1  
While what you say may be true, it doesn't answer this specific question. Your answer boils down to "in my opinion that quote usually isn't true". The question isn't about whether it's true or not, but what the quote means. –  Bryan Oakley Dec 5 '11 at 23:40
1  
There's literally no point in doing *+ in this case; everything is anchored and can be matched in a single pass by an automaton that can count up to 22. The correct modifier on those non-comma sets is just plain old *. (What's more, there should also be no differences between greedy and non-greedy matching algorithms here. It's an extremely simple case.) –  Donal Fellows Apr 8 '13 at 8:08

protected by World Engineer Jan 10 at 3:44

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.