Inspiring problems to show off the importance of regular expressions?

Question

I am planning to give a presentation/demonstration on regular expressions at work to encourage young developers to add this powerful and important tool in their toolbox.

Just teaching syntax doesn't cut it. I often see people say nice. After the presentation, they get on with their programming lives without ever thinking of using it mostly.

I am raking my grey matter to come up with some solid examples, not just problems that matches 'cat' and 'cut'. I missed to note down the occasions of my regex enlightenments to use here. :^(

Do you have some inspiring problems to share that could be solved with regex?

Update: Wow! I didn't expect so many wonderful answers. Thank you. For now, I will mark @tdammers' answer as accepted. Bring those regex memories keep coming :-)

Answer 1

A few examples:

• vi(m) uses regular expressions for all search-and-replace commands, and I wouldn't want to miss it anymore
• semi-automated code editing, such as 'remove all python line comments' (s/#.*$//'), 'remove trailing whitespace at end of line' (s/\s+$//), 'remove redundant newlines' (s/\n\n+/\n/), etc. etc.
• massaging output from one command to feed to another (pretty common in the *nix world)
• Real life example from last week: converting an SQL dump between two incompatible SQL dialects. The dump was way too large to edit by hand (5 GB), but sed worked well.
• Simple input validation - things like 'make sure this value contains only numeric digits'.

Answer 2

The killer application for regular expressions for programmers today (as I see it) is in mining log files. Typical examples:

• All lines containing "Frobozz finished"
• All lines containing "Failure" but not "Frobozz finished"
• All lines containing "Took number seconds" (where number is one or more digits)

This is a typical grep job. For multiple logfiles the advantage of prepending with the file name is really nice.

For more advanced stuff, you need a programming language supporting regular expressions which can be easily started from the command line. This is where Unix tools truly shine. The default standard Unix program fitting this description is awk but these days perl is a more powerful tool which is installed by default on most systems.

This allows you to do stuff like:

• All lines containing "Frobozz finished" with a timestamp after 22:00.
• All lines containing "Failure" but not "Frobozz finished" where the previous line contained "Frobozz started".
• All lines containing "Took number seconds" (where number is one or more digits) where number is more than 10.

Being able to ask such questions allow you to ask questions which quickly dig out those log entries which indicate trouble. Besides being helpful in your daily work, it can be a life-saver in a situation you have a production server with problems that needs to be fixed as soon as possible.

Answer 3

A real life example: in order to achieve PCI compliance, we need to ensure that no credit card numbers are published by our system in any way, including log files. So we implemented a log filter to identify and mask out anything in our log messages which looks like a credit card number.

For more details see How to mask credit card numbers in log files with Log4J?

Answer 4

I really love the example made in the book "Mastering regular expressions" Google Books chapter 1 Amazon

Let people think about the problem themselves and how they would solve it using their language of choice and than present the RegEx solution.

Find doubled words in website pages like "this this", and the solution has to

• accept any number of files , report each line of each file that has double wordes, highlight each doubled word
• work across lines
• be case insensitive, and allow different amounts of whitespace (spaces, tabs, newlines) between the doubled words
• don't care about html tags in between the words. "very very"

The book gives a perl and java example of the solution

Perl:

$/ = ".\n";
while (<>) {
  next if !s/\b([a-z]+)((?:\s|<[^>]+>(\1\b)/\e[7m$2\e[m$2\e[7m$3\e[m/ig;
  s/^(?:[^\e]*\n)+//mg; # remove any unmarked lines.
  s/^/$ARGV: /mg;       # Ensure lines begins with filename.
  print;
}

What I love about this example:

• the question seems pretty simple, so that every body can understand it.
• Most people can think up how they would write the code without regex and would find it challenging
• The solution shows the power of regex, so much can be done in a few lines of code
• The solution also shows it weakness, given only the solution try to reason on just what the program is doing.

Answer 5

Once we had to migrate all our strcpy statements to strncpy_s. Most of them were in the format strcpy(dest, src) and we wanted to them to basically look like strncpy_s(dest, sizeof(dest), src, _TRUNCATE). There were hundreds of these calls, and after setting off bravely to change them manually I began to explore the options in Search and Replace in Visual Studio. The RegEx codes in VS seem to be quite different from other RegEx engines, but I had a lot of fun working out step by step how to move things around and make them look like I wanted. I even gave a short presentation on Regular Expressions to my co-workers, and after that became the go-to guy for regex.

Answer 6

I don't know how inspiring it is, but it is a practical one: formatting my code. I often will copy/paste stuff from the web and then run a series of regular expressions doing replace to format it to my standards. This way I can strip out unnecessary white-space after a }, or numerous other things like enforcing that there be exactly one space between a function name and the opening {.

Answer 7

About a year or two ago, I wrote a piece of code that made heavy use of regular expressions to parse free-form text fields that represented street addresses into structured data (separate fields for street number, suite number, street direction, street type (avenue, road, street, etc...), street names (with multiple words, some of which could have also been part of the direction or street type)). It was a bit of a nightmare but mostly worked on > 80% of the input fields.

Databases that support regular expressions are extremely useful when you're searching for data that is just a bit too complex for LIKE 'something%somethingelse'.

These days I don't do a lot of coding with regular expressions, but I do use text editors that allow searching with regular expressions, which makes searching through large error logs much easier. You can search for all lines containing FoobarFramework(Fatal)?(Exception|Error)(123|456) to search for a few different classes of errors, which otherwise would take 3 or 4 searches through the file.

And then there's input validation... SOO much can be done there with regular expressions. Postal codes and phone numbers seem to be the canonical examples, but there are others, but it will probably depend on your input.

If I weren't so tired, I'd probably have even better examples... maybe tomorrow I'll come up with some more.

Answer 8

Following tdammers answer: I program in Visual Studio but when I need to do some complex editing I open vi(m) to use regular expressions. Besides removing / adding leading comments (Python, bash, C++, etc), removing trailing spaces, I can transform and reuse pieces of codes in complex ways.

Example snippet (adapted from some real code I had to edit):

// A class with several member variables T1 x1, Tk xk, ..., Tn xn (n > k)
// where Ti is the type of the variable, and xi is the name of the variable.
class A;

void foo(const A &a)
{
    // Section 1.
    // Save x1, xk to local variables.
    T1 temp1 = a.x1;
    T2 temp2 = a.x2;
    ...
    Tk tempk = a.xk;

    // Section 2.
    // Do some complex computation with a, which might change member
    // variables x1, ..., xk.

    // Section 3.
    // Restore x1, xk from local variables.
    a.x1 = temp1;
    a.x2 = temp2;
    ...
    a.xk = tempk;
}

With vi(m), I can copy / paste section 1 and produce section 3 by using a regular expression like

s/T[0-9][0-9]* \(temp[0-9][0-9]*\) = \(a.x[0-9][0-9]*\);/\2 = \1;/

I type one line instead of going through all lines and adapting each of them. Even if this example is a bit artificial (normally, there are no types numbered from 1 to n in real code), I think you should get the idea that regular expressions are good at matching sub-strings and moving them around. I use this very often.

Answer 9

A CSS Selector engine in JavaScript.

Granted, making a selector engine is lot more than just regular expressions, but the application of regexes is indispensable for such a system. There's a parser (or lexer) that converts the CSS selector expression into something intermediate that can be easier to select DOM elements with. And you just can't make a non-trivial parser without regexes.

Also, since it's just plain JavaScript, your young programmers can wrestle with regexes without having to build/compile anything. (I guess it helps when you're trying some new language or API, not to be bothered by anything that can be avoided.)

It is also very exciting, because they can just extend the selector syntax in their own arbitrary (but meaningful) ways, and then implement them. (jQuery has a lot of custom selector facilities).

Answer 10

Instead of coming up with "fake" but real-world example. If I were you, I'd go back to your own source repository. If you show people how 200 lines of their own code is reduced down to 5, even if you don't convince everyone, it is very likely you'll convince the author of that code. Then you have 2 people on the team advocating good practices.

Even if you find good examples, I don't think anything will inspire more than seeing your own product's code being simplified, made cleaner and reduced in size by 90%. Better yet, bring a laptop to the meeting and do it in front of the team to illustrate how something that probably took a full day to write could've been reduced to 15 minutes.

And as WuHoUnited noted, if you can't find such examples, you might be doing premature optimization :)