Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am planning to give a presentation/demonstration on regular expressions at work to encourage young developers to add this powerful and important tool in their toolbox.

Just teaching syntax doesn't cut it. I often see people say nice. After the presentation, they get on with their programming lives without ever thinking of using it mostly.

I am raking my grey matter to come up with some solid examples, not just problems that matches 'cat' and 'cut'. I missed to note down the occasions of my regex enlightenments to use here. :^(

Do you have some inspiring problems to share that could be solved with regex?

Update: Wow! I didn't expect so many wonderful answers. Thank you. For now, I will mark @tdammers' answer as accepted. Bring those regex memories keep coming :-)

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

closed as too broad by gnat, MichaelT, GlenH7, Dynamic, Yusubov Aug 12 '13 at 14:45

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

15  
If they need it they will remember it and use it. The last thing we need is a bunch of programmers with a Regex hammer looking for an HTML parsing nail. –  JohnFx Nov 18 '11 at 4:43
    
Very true. The problem is more fundamental. I have noticed that they go with doing things by manual repetition, than considering automation or regex. This is where I thought they want to make them excited about the possibilities of regex. –  ragu.pattabi Nov 18 '11 at 4:58
7  
If you can't think of a time you use it at work then it's not necessarily a bad thing if they don't know them. Half the time I see regular expressions they're doing something they shouldn't be (checking that a date looks like mm/dd/yyyy). If you want an inspiring example xkcd.com/208. –  WuHoUnited Nov 18 '11 at 5:13
3  
You may be inspired by email checking using regex according to the RFC822 : ex-parrot.com/pdw/Mail-RFC822-Address.html –  deadalnix Nov 18 '11 at 11:04
    
@WvHoUnited I love that cartoon :-). I don't think there would be a programmer (or even computer user) who wouldn't see the necessity of regexes if they know it. –  ragu.pattabi Nov 18 '11 at 14:22
show 1 more comment

10 Answers 10

up vote 13 down vote accepted

A few examples:

  • vi(m) uses regular expressions for all search-and-replace commands, and I wouldn't want to miss it anymore
  • semi-automated code editing, such as 'remove all python line comments' (s/#.*$//'), 'remove trailing whitespace at end of line' (s/\s+$//), 'remove redundant newlines' (s/\n\n+/\n/), etc. etc.
  • massaging output from one command to feed to another (pretty common in the *nix world)
  • Real life example from last week: converting an SQL dump between two incompatible SQL dialects. The dump was way too large to edit by hand (5 GB), but sed worked well.
  • Simple input validation - things like 'make sure this value contains only numeric digits'.
share|improve this answer
3  
search and replace... now there is one awesome use for regex. Some of us sometimes have to deal with big data-set that are not always very clean. Knowing and using reg-ex in this case is the equivalent of safe sex for data import. in a CSV file Which lines contain phone numbers that are not formatted as expected ? Names come in a single field with last, first middle but I want them in separate fields, write a reg-ex for that. Small, quick, efficient. –  Newtopian Nov 18 '11 at 7:17
add comment

The killer application for regular expressions for programmers today (as I see it) is in mining log files. Typical examples:

  • All lines containing "Frobozz finished"
  • All lines containing "Failure" but not "Frobozz finished"
  • All lines containing "Took number seconds" (where number is one or more digits)

This is a typical grep job. For multiple logfiles the advantage of prepending with the file name is really nice.

For more advanced stuff, you need a programming language supporting regular expressions which can be easily started from the command line. This is where Unix tools truly shine. The default standard Unix program fitting this description is awk but these days perl is a more powerful tool which is installed by default on most systems.

This allows you to do stuff like:

  • All lines containing "Frobozz finished" with a timestamp after 22:00.
  • All lines containing "Failure" but not "Frobozz finished" where the previous line contained "Frobozz started".
  • All lines containing "Took number seconds" (where number is one or more digits) where number is more than 10.

Being able to ask such questions allow you to ask questions which quickly dig out those log entries which indicate trouble. Besides being helpful in your daily work, it can be a life-saver in a situation you have a production server with problems that needs to be fixed as soon as possible.

share|improve this answer
    
+1 I was about to add this example. Additionally, regex + sed makes it very easy to strip unnecessary stuff from logs, e.g. timestamps when you want to compare two logs created by the same app at different times. –  Péter Török Nov 18 '11 at 8:13
    
For more advanced stuff you can also (in addition to knowing regexs) use more specific tools such as logstash. –  Alb Nov 18 '11 at 17:55
add comment

A real life example: in order to achieve PCI compliance, we need to ensure that no credit card numbers are published by our system in any way, including log files. So we implemented a log filter to identify and mask out anything in our log messages which looks like a credit card number.

For more details see How to mask credit card numbers in log files with Log4J?

share|improve this answer
add comment

I really love the example made in the book "Mastering regular expressions" Google Books chapter 1 Amazon

Let people think about the problem themselves and how they would solve it using their language of choice and than present the RegEx solution.

Find doubled words in website pages like "this this", and the solution has to

  • accept any number of files , report each line of each file that has double wordes, highlight each doubled word
  • work across lines
  • be case insensitive, and allow different amounts of whitespace (spaces, tabs, newlines) between the doubled words
  • don't care about html tags in between the words. "very very"

The book gives a perl and java example of the solution

Perl:

$/ = ".\n";
while (<>) {
  next if !s/\b([a-z]+)((?:\s|<[^>]+>(\1\b)/\e[7m$2\e[m$2\e[7m$3\e[m/ig;
  s/^(?:[^\e]*\n)+//mg; # remove any unmarked lines.
  s/^/$ARGV: /mg;       # Ensure lines begins with filename.
  print;
}

What I love about this example:

  • the question seems pretty simple, so that every body can understand it.
  • Most people can think up how they would write the code without regex and would find it challenging
  • The solution shows the power of regex, so much can be done in a few lines of code
  • The solution also shows it weakness, given only the solution try to reason on just what the program is doing.
share|improve this answer
1  
Your example show very clearly why regular expressions need comments. It is very hard to show in code why a given expression is there. –  user1249 Nov 18 '11 at 11:06
add comment

Once we had to migrate all our strcpy statements to strncpy_s. Most of them were in the format strcpy(dest, src) and we wanted to them to basically look like strncpy_s(dest, sizeof(dest), src, _TRUNCATE). There were hundreds of these calls, and after setting off bravely to change them manually I began to explore the options in Search and Replace in Visual Studio. The RegEx codes in VS seem to be quite different from other RegEx engines, but I had a lot of fun working out step by step how to move things around and make them look like I wanted. I even gave a short presentation on Regular Expressions to my co-workers, and after that became the go-to guy for regex.

share|improve this answer
add comment

I don't know how inspiring it is, but it is a practical one: formatting my code. I often will copy/paste stuff from the web and then run a series of regular expressions doing replace to format it to my standards. This way I can strip out unnecessary white-space after a }, or numerous other things like enforcing that there be exactly one space between a function name and the opening {.

share|improve this answer
    
How much code does one need to copy exactly as is from the web? –  Daniel Little Nov 18 '11 at 4:39
    
When I am an active answerer on StackOverflow? A lot. I usually hate their syntax, so you can darn well bet that I am going to clean it up. –  Levi Morrison Nov 18 '11 at 4:43
    
hmmm, while I was reading the answer the following thoughts emerged "When a bit of metal is sticking out I invented this awesome usefull thing. Basically a metal flat head on top of a short metal rod. Grab the rod, give a good whack and boom, all even now". Code formatting is indeed a problem but it's been a long time since I had to do more that 5 minutes googling to find a working formatter to do the job. Most times it comes embedded right in my IDE... spend time and energy create a reg-ex or... ctrf-shift-f, which is faster ? –  Newtopian Nov 18 '11 at 7:13
2  
@Levi, why don't you use an editor that can format code to your taste with a simple key command? –  user1249 Nov 18 '11 at 7:19
    
@THorbjornRavnAndersen Well, mainly because I use VIM but haven't bothered to look into it. Aren't regular expressions so much cooler, anyway? –  Levi Morrison Nov 18 '11 at 18:00
show 1 more comment

About a year or two ago, I wrote a piece of code that made heavy use of regular expressions to parse free-form text fields that represented street addresses into structured data (separate fields for street number, suite number, street direction, street type (avenue, road, street, etc...), street names (with multiple words, some of which could have also been part of the direction or street type)). It was a bit of a nightmare but mostly worked on > 80% of the input fields.

Databases that support regular expressions are extremely useful when you're searching for data that is just a bit too complex for LIKE 'something%somethingelse'.

These days I don't do a lot of coding with regular expressions, but I do use text editors that allow searching with regular expressions, which makes searching through large error logs much easier. You can search for all lines containing FoobarFramework(Fatal)?(Exception|Error)(123|456) to search for a few different classes of errors, which otherwise would take 3 or 4 searches through the file.

And then there's input validation... SOO much can be done there with regular expressions. Postal codes and phone numbers seem to be the canonical examples, but there are others, but it will probably depend on your input.

If I weren't so tired, I'd probably have even better examples... maybe tomorrow I'll come up with some more.

share|improve this answer
    
Postal codes and phone numbers are also canonical examples of things which most people validate really badly because they don't take into account things like the world containing more than just the country they live in. –  Peter Taylor Nov 18 '11 at 10:01
    
@PeterTaylor: True, though if your application is intended for use in a specific region where there are well-accepted standards, regular expressions can make the validation much easier. If the application lets users choose a country, there is usually one postal code regex for the postal code patterns of different postal systems. –  FrustratedWithFormsDesigner Nov 18 '11 at 14:24
add comment

Following tdammers answer: I program in Visual Studio but when I need to do some complex editing I open vi(m) to use regular expressions. Besides removing / adding leading comments (Python, bash, C++, etc), removing trailing spaces, I can transform and reuse pieces of codes in complex ways.

Example snippet (adapted from some real code I had to edit):

// A class with several member variables T1 x1, Tk xk, ..., Tn xn (n > k)
// where Ti is the type of the variable, and xi is the name of the variable.
class A;

void foo(const A &a)
{
    // Section 1.
    // Save x1, xk to local variables.
    T1 temp1 = a.x1;
    T2 temp2 = a.x2;
    ...
    Tk tempk = a.xk;

    // Section 2.
    // Do some complex computation with a, which might change member
    // variables x1, ..., xk.

    // Section 3.
    // Restore x1, xk from local variables.
    a.x1 = temp1;
    a.x2 = temp2;
    ...
    a.xk = tempk;
}

With vi(m), I can copy / paste section 1 and produce section 3 by using a regular expression like

s/T[0-9][0-9]* \(temp[0-9][0-9]*\) = \(a.x[0-9][0-9]*\);/\2 = \1;/

I type one line instead of going through all lines and adapting each of them. Even if this example is a bit artificial (normally, there are no types numbered from 1 to n in real code), I think you should get the idea that regular expressions are good at matching sub-strings and moving them around. I use this very often.

share|improve this answer
add comment

A CSS Selector engine in JavaScript.

Granted, making a selector engine is lot more than just regular expressions, but the application of regexes is indispensable for such a system. There's a parser (or lexer) that converts the CSS selector expression into something intermediate that can be easier to select DOM elements with. And you just can't make a non-trivial parser without regexes.

Also, since it's just plain JavaScript, your young programmers can wrestle with regexes without having to build/compile anything. (I guess it helps when you're trying some new language or API, not to be bothered by anything that can be avoided.)

It is also very exciting, because they can just extend the selector syntax in their own arbitrary (but meaningful) ways, and then implement them. (jQuery has a lot of custom selector facilities).

share|improve this answer
add comment

Instead of coming up with "fake" but real-world example. If I were you, I'd go back to your own source repository. If you show people how 200 lines of their own code is reduced down to 5, even if you don't convince everyone, it is very likely you'll convince the author of that code. Then you have 2 people on the team advocating good practices.

Even if you find good examples, I don't think anything will inspire more than seeing your own product's code being simplified, made cleaner and reduced in size by 90%. Better yet, bring a laptop to the meeting and do it in front of the team to illustrate how something that probably took a full day to write could've been reduced to 15 minutes.

And as WuHoUnited noted, if you can't find such examples, you might be doing premature optimization :)

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.