Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Many programmers know the joy of whipping up a quick regular expression, these days often with help of some web service, or more traditionally at interactive prompt, or perhaps writing a small script which has the regular expression under development, and a collection of test cases. In either case the process is iterative and fairly quick: keep hacking at the cryptic-looking string until it matches and captures what you want and will reject what you don't want.

For a simple case result might be something like this, as a Java regexp:

Pattern re = Pattern.compile(
  "^\\s*(?:(?:([\\d]+)\\s*:\\s*)?(?:([\\d]+)\\s*:\\s*))?([\\d]+)(?:\\s*[.,]\\s*([0-9]+))?\\s*$"
);

Many programmers also know the pain of needing to edit a regular expression, or just code around a regular expression in a legacy code base. With a bit editing to split it up, above regexp is still very easy to comprehend for anyone reasonably familiar with regexps, and a regexp veteran should see right away what it does (answer at the end of the post, in case someone wants the exercise of figuring it out themselves).

However, things don't need to get much more complex for a regexp to become truly write-only thing, and even with diligent documentation (which everybody of course does for all complex regexps they write...), modifying the regexps becomes a daunting task. It can be a very dangerous task too, if regexp is not carefully unit tested (but everybody of course has comprehensive unit tests for all their complex regexps, both positive and negative...).

So, long story short, is there a write-read solution/alternative for regular expressions without losing their power? How would the above regexp look like with an alternative approach? Any language is fine, though a multi-language solution would be best, to the degree regexps are multi-language.


And then, what the earlier regexp does is this: parse a string of numbers in format 1:2:3.4, capturing each number, where spaces are allowed and only 3 is required.

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

2  
related thing on SO: stackoverflow.com/a/143636/674039 –  wim Apr 15 '13 at 16:32
15  
Reading / editing regexes is actually trivial if you know what they're supposed to capture. You might have heard of this rarely used feature of most languages called "comments". If you don't put one above a complex regex explaining what it does you'll pay the price later. Also, code review. –  TC1 Apr 15 '13 at 19:08
1  
Two options to clean this up without actually breaking it into smaller pieces. Their presence or absence varies from language to language. (1) extended line regexes, where whitespace in the regex is ignored (unless escaped) and a single-line comment form is added, so you can break it out into logical chunks with indentation, line spacing, and comments. (2) named capture groups, where you can give a name to each parenthetical, which both adds some self-documentation, and automatically populates a hash of matches -- way better than either a numerically indexed array of matches or $N variables. –  Ben Lee Apr 15 '13 at 20:46
2  
Part of the problem is the regex language itself, and the bad historic choices in its design which are dragged along like baggage. In a sane language, grouping parentheses are purely a syntactic device to shape the parse tree. But in regex implemenations going back to Unix they have semantics: binding registers to subexpression matches. So then you need some more complicated, ugly brackets just to achieve pure grouping! –  Kaz Apr 16 '13 at 3:51
1  
@SK-logic Well, one being supported natively or by standard library, while other is not, is kind of a reason. But why don't you add an answer showing an example of PEG in action in a mainstream language, it would certainly fit within the question scope. –  hyde Apr 16 '13 at 13:59

10 Answers 10

up vote 53 down vote accepted

A number of people have mentioned composing from smaller parts, but no one's provided an example yet, so here's mine:

string number = "(\\d+)";
string unit = "(?:" + number + "\\s*:\\s*)";
string optionalDecimal = "(?:\\s*[.,]\\s*" + number + ")?";

Pattern re = Pattern.compile(
  "^\\s*(?:" + unit + "?" + unit + ")?" + number + optionalDecimal + "\\s*$"
);

Not the most readable, but I feel like it's clearer than the original.

Also, C# has the @ operator which can be prepended to a string in order to indicate that it is to be taken literally (no escape characters), so number would be @"([\d]+)";

share|improve this answer
1  
@Heinzi - Good point. I'd actually missed the fact that it supported multiple-digit numbers, since I was just treating that bit of the regex as a single group, without really considering what it did. Updated the names. –  Bobson Apr 15 '13 at 20:50
    
Just now noticed how both [\\d]+ and [0-9]+ should be just \\d+ (well, some may find [0-9]+ more readable). I'm not going to edit the question, but you may want to fix this answer. –  hyde Apr 17 '13 at 9:59
    
@hyde - Good catch. Technically they're not quite the same thing - \d will match anything that's considered a number, even in other numbering systems (Chinese, Arabic, etc.), while [0-9] will just match the standard digits. I did standardize on \\d, though, and factored it into the optionalDecimal pattern. –  Bobson Apr 17 '13 at 13:30
    
I've been doing this for a while, and it makes life much easier. +1 –  gsingh2011 Apr 19 '13 at 19:17

The key to documenting the regular expression is documenting it. Far too often people toss in what appears to be line noise and leave it at that.

Within perl the /x operator at the end of the regular expression suppresses whitespace allowing one to document the regular expression.

The above regular expression would then become:

$re = qr/
  ^\s*
  (?:
    (?:       
      ([\d]+)\s*:\s*
    )?
    (?:
      ([\d]+)\s*:\s*
    )
  )?
  ([\d]+)
  (?:
    \s*[.,]\s*([\d]+)
  )?
  \s*$
/x;

Yes, its a bit consuming of vertical whitespace, though one could shorten it up without sacrificing too much readability.

And then, what the earlier regexp does is this: parse a string of numbers in format 1:2:3.4, capturing each number, where spaces are allowed and only 3 is required.

Looking at this regular expression one can see how it works (and doesn't work). In this case, this regex will match the string 1.

Similar approaches can be taken in other language. The python re.VERBOSE option works there.

Perl6 (the above example was for perl5) takes this further with the concept of rules which leads to even more powerful structures than the PCRE (it provides access to other grammars (context free and context sensitive) than just regular and extended regular ones).

In Java (where this example draws from), one can use string concatenation to form the regex.

Pattern re = Pattern.compile(
  "^\\s*"+
  "(?:"+
    "(?:"+
      "([\\d]+)\\s*:\\s*"+  // Capture group #1
    ")?"+
    "(?:"+
      "([\\d]+)\\s*:\\s*"+  // Capture group #2
    ")"+
  ")?"+ // First groups match 0 or 1 times
  "([\\d]+)"+ // Capture group #3
  "(?:\\s*[.,]\\s*([0-9]+))?"+ // Capture group #4 (0 or 1 times)
  "\\s*$"
);

Admittedly, this creates many more " in the string possibly leading to some confusion there, can be more easily read (especially with syntax highlighting on most IDEs) and documented.

The key is recognizing the power and "write once" nature that regular expressions often fall into. Writing the code to defensively avoid this so that the regular expression remains clear and understandable is key. We format Java code for clarity - regular expressions are no different when the language gives you the option to do so.

share|improve this answer
6  
There's a big difference between "documenting" and "adding line breaks". –  Jon of All Trades Apr 15 '13 at 19:47
4  
@JonofAllTrades Making the code able to be read is the first step to anything. Adding line breaks also allows one to add comments for that subset of the RE on the same line (something that is more difficult to do on a single long line of regular expression text). –  MichaelT Apr 15 '13 at 19:53
2  
@JonofAllTrades, I disagree pretty strongly. "Documenting" and "adding line breaks" are not that different in that they both serve the same purpose -- making the code easier to understand. And for poorly-formatted code, "adding line breaks" serves that purpose much better than adding documentation would. –  Ben Lee Apr 15 '13 at 20:50
1  
Adding line breaks is a start, but it's about 10% of the job. Other answers give more specifics, which is helpful. –  Jon of All Trades Apr 15 '13 at 21:03

The "verbose" mode offered by some languages and libraries is one of the answers to these concerns. In this mode, whitespace in regexp string is stripped out (so you need to use \s) and comments are possible. Here's a short example in Python which supports this by default:

email_regex = re.compile(r"""
    ([\w\.\+]+) # username (captured)
    @
    \w+         # minimal viable domain part
    (?:\.w+)    # rest of the domain, after first dot
""", re.VERBOSE)

In any language that doesn't, implementing a translator from verbose to "normal" mode should be a simple task. If you're concerned about your regexps' readability, you would probably justify this time investment pretty easily.

share|improve this answer
    
+1 for the simple translator/comment stripper idea –  hyde Apr 15 '13 at 18:15

Every language that uses regexes allows you to compose them from simpler blocks to make reading easier, and with anything more complicated than (or as complicated as) your example, you should definitely take advantage of that option. The particular trouble with Java and many other languages is that they don't treat regular expressions as "first-class" citizens, instead requiring them to sneak into the language via string literals. This means many quotation marks and backslashes that aren't actually part of regex syntax and make things hard to read, and it also means that you can't get much more readable than that without effectively defining your own mini-language and interpreter.

The prototypical better way of integrating regular expressions was of course Perl, with its whitespace option and regex-quoting operators. Perl 6 extends the concept of building up regexes from parts to actual recursive grammars, which is so much better to use it's really no comparison at all. The language may have missed the boat of timeliness, but its regex support was The Good Stuff(tm).

share|improve this answer
    
By "simpler blocks" mentioned at the start of the answer, do you mean just string concatenation, or something more advanced? –  hyde Apr 15 '13 at 14:08
6  
I meant defining sub-expressions as shorter string literals, assigning them to local variables with meaningful names, and then concatenating. I find the names are more important to readability than just the layout improvement. –  Kilian Foth Apr 15 '13 at 14:27

I like to use Expresso: http://www.ultrapico.com/Expresso.htm

This free application has the following features that I find useful over time:

  • You can simply copy and paste your regex and the application will parse it for you
  • Once your regex is written, you can test it directly from the application (the application will give you the list of captures, replacements...)
  • Once you have tested it, it will generate the C# code to implement it (note that the code will contain the explanations about your regex).

For example, with the regex you just submitted, it would would look like: Sample screen with the initially given regex

Of course, giving it a try is worth a thousand word describing it. Please also note that I'm note related in any way with the editor of this application.

share|improve this answer
3  
would you mind explaining about this in more detail - how and why does it answer the question asked? "Link-only answers" are not quite welcome at Stack Exchange –  gnat Apr 15 '13 at 15:37
3  
@gnat Sorry about that. You're absolutely right. I hope that my edited answer does provide more insights. –  Jaepetto Apr 15 '13 at 19:49

For some things, it might help to just use a grammar like BNF. These can be much easier to read than regular expressions. A tool such as GoldParser Builder can then convert the grammar into a parser that does the heavy lifting for you.

The BNF, EBNF, etc. grammars can be much easier to read and make than a complicated regular expression. GOLD is one tool for such things.

The c2 wiki link below has a list of possible alternatives which can be googled, with some discussion on them included. It is basically a "see also" link to top off my grammar engine recommendation:

Alternatives To Regular Expressions

Taking "alternative" to mean "semantically equivalent facility with different syntax", there are at least these alternatives to/with RegularExpressions:

  • Basic regular expressions
  • "Extended" regular expressions
  • Perl-compatible regular expressions
  • ... and many other variants...
  • SNOBOL-style RE syntax (SnobolLanguage, IconLanguage)
  • SRE syntax (RE's as EssExpressions)
  • different FSM syntaces
  • Finite-state intersection grammars (quite expressive)
  • ParsingExpressionGrammars, as in OMetaLanguage and LuaLanguage (http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html)
  • The parse mode of RebolLanguage
  • ProbabilityBasedParsing...
share|improve this answer
    
would you mind explaining more on what this link does and what it's good for? "Link-only answers" are not quite welcome at Stack Exchange –  gnat Apr 15 '13 at 18:36
1  
Welcome to Programmers, Nick P. Please ignore the downvote/r, but do read the page on meta that @gnat linked to. –  Christoffer Lette Apr 15 '13 at 22:56
1  
+1: EBNF grammar is much easier to maintain than very long/complex regular expressions. –  Paulo Scardine Apr 16 '13 at 6:40
    
@ Christoffer Lette Appreciate your reply. Will try to keep this in mind in future posts. @ gnat Paulo Scardine's comment reflects my posts' intent. The BNF, EBNF, etc. grammars can be much easier to read and make than a complicated regular expression. GOLD is one tool for such things. The c2 link has a list of possible alternatives which can be googled, with some discussion on them included. It was basically a "see also" link to top off my grammar engine recommendation. –  Nick P Apr 16 '13 at 8:07

The simplest way would be to still use regex but build your expression from composing simpler expresssions with descriptive names e.g. http://www.martinfowler.com/bliki/ComposedRegex.html (and yes this is from string concat)

however as an alternative you could also use a parser combinator library e.g. http://jparsec.codehaus.org/ which will give you a full recursive decent parser. again the real power here comes from composition (this time functional composition).

share|improve this answer

I wrote Pi Regex, a small Java library that provides a kind of fluent API to compose regexes. It might help you with complex expressions.

The following example creates a pattern that matches an IP address in dotted decimal format:

final Regex octet1Digit = CharClass.DIGIT;
final Regex octet2Digits = CharClass.oneIn('1', '9').then(CharClass.DIGIT);
final Regex octet100 = Regex.literal("1").then(CharClass.DIGIT.repeat(2));
final Regex octet200 = Regex.literal("2").then(CharClass.oneIn('0', '4')).then(CharClass.DIGIT);
final Regex octet200High = Regex.literal("25").then(CharClass.oneIn('0', '5'));
final Regex octet = Regex.or(octet1Digit, octet2Digits, octet100, octet200, octet200High);
final Regex ipAddress = octet.then(Regex.literal(".")).repeat(3).then(octet);
final Pattern pattern = ipAddress.toPattern();

The generated pattern is:

(?:(?:\d|(?:[1-9]\d)|(?:(?:1)\d{2})|(?:(?:(?:2)[0-4])\d)|(?:(?:25)[0-5]))(?:\.)){3}(?:\d|(?:[1-9]\d)|(?:(?:1)\d{2})|(?:(?:(?:2)[0-4])\d)|(?:(?:25)[0-5]))

To learn and use Pi Regex, please see the following links:

share|improve this answer
3  
That Regex is an example on why you shouldn't use a regex for every parsing/matching task. It still misses the capture of the octets into groups, though that would not be difficult to add. –  Paŭlo Ebermann Apr 16 '13 at 17:28
    
@PaŭloEbermann Granted, there are more efficient ways to validate an IP address, but the point here is to give an example of a not-so-trivial regex. As for capturing the octets, it would indeed be very easy, especially with such a library. –  Laurent Pireyn Apr 18 '13 at 8:46

I thought it'd be worth mentioning logstash's grok expressions. Grok builds upon the idea of composing long parsing expressions from shorter ones. It allows convenient testing of these building blocks and comes prepackaged with over 100 commonly used patterns. Other than these patterns, it allows use of all regular expressions syntax.

The above pattern expressed in grok is (I tested in the debugger app but could have blundered):

"(( *%{NUMBER:a} *:)? *%{NUMBER:b} *:)? *%{NUMBER:c} *(. *%{NUMBER:d} *)?"

The optional parts and spaces make it seem a bit uglier than usual, but both here and in other cases, using grok can make one's life much nicer.

share|improve this answer

First, understand that code that merely works is bad code. Good code also needs to accurately report any errors encountered.

For example, if you're writing a function to transfer cash from one user's account to another user's account; you wouldn't just return a "worked or failed" boolean because that doesn't give the caller any idea of what went wrong and doesn't allow the caller to inform the user properly. Instead, you might have a set of error codes (or a set of exceptions): couldn't find destination account, insufficient funds in source account, permission denied, can't connect to database, too much load (retry later), etc.

Now think about your "parse a string of numbers in format 1:2:3.4" example. All the regex does is report a "pass/fail" that doesn't allow adequate feedback to be presented to the user (whether this feedback is an error message in a log, or an interactive GUI where the errors are shown in red as the user types, or whatever else). What types of errors does it fail to describe properly? Bad character in first number, first number too large, missing colon after first number, etc.

To convert "bad code that merely works" into "good code that provides adequately descriptive errors" you have to break the regex up into many smaller regexes (typically, regexes that are so small that it's easier to do it without regexes in the first place).

Making the code readable/maintainable is just an accidental consequence of making the code good.

share|improve this answer
    
Can I assume that the downvoters are just people that consistently write bad code? –  Brendan Apr 19 '13 at 6:08
1  
Probably not a good assumption. Mine is because A) This doesn't address the question (How to make it readable?), B) Regular expression matching is pass/fail, and if you break it down to the point where you can say exactly why it failed, you lose a lot of power and speed, and increase the complexity, C) There's no indication from the question that there's even the possibility of the match failing - it's simply a question about making the Regex readable. When you have control of the data going in and/or validate it before hand, you can assume it's valid. –  Bobson Apr 19 '13 at 14:01
    
A) Breaking it into smaller pieces makes it more readable (as a consequence of making it good). C) Where unknown/unvalidated strings enter a piece of software a sane developer would parse (with error reporting) at that point and convert the data into a form that doesn't need reparsing - regex is not needed after that. B) is nonsense that only applies to bad code (refer to points A and C). –  Brendan Apr 20 '13 at 5:20
    
Going from your C: What if this is his validation logic? The OP's code could be exactly what you're suggesting - validating the input, reporting if it's not valid, and converting it to a usable form (via the captures). All we have is the expression itself. How would you suggest parsing it other than with a regex? If you add some sample code that will accomplish the same result, I'll remove my downvote. –  Bobson Apr 21 '13 at 15:40
    
If this is "C: Validating (with error reporting)" then it's bad code because the error reporting is bad. If it fails; was it because the string was NULL, or because the first number had too many digits, or because the first separator wasn't :? Imagine a compiler that only had one error message ("ERROR") that was too stupid to tell the user what the problem is. Now imagine thousands of web sites that are just as stupid and display (e.g.) "Bad email address" and nothing more. –  Brendan Apr 22 '13 at 8:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.