Regular Expressions

Most text editing applications include Find and Replace features, although these can only replace a specific text string in the current document or selection by alternative text. For example, you could find every instance of the word cat and replace it by dog. Most software has a Match Entire Word option, preventing, for example the changing of catwalk to dogwalk.

Some applications also accommodate a wildcard character, such as an * (asterisk). If you then search for c*t you’ll be able, for example, to replace both cot and cat by dog, should that be required. Unfortunately, things can get confused if the wildcard character itself is contained in the text.

Other programs support the Global Regular Expression Parser (GREP), which accommodates a much greater manipulation of text. For example, if you have a database of names as tab-delimited text you can change their form from Mr. Fred James Bloggs to BLOGGS, FJ. A regular expression contains a search pattern of characters that’s compared with the target text, usually an entire document or the currently selected text. An appropriate replacement pattern is then applied.

Expression Elements

A regular expression is defined using combinations of the following:-

The following sections consider the various types of metacharacters and shortcuts, taking the form of anchors, character classes, quantifiers, alternation and groups. In the examples shown below, single metacharacters are shown in red while special strings and the matching text is shown in blue. The author is indebted to the Regex documentation, which forms the basis of these notes.

Anchors

An anchor ensures that a match only occurs at a specific position in the target text.

\A  Start of Text

This matches the notional ‘empty string’ that exists at the beginning of the target text, thereby anchoring a search pattern to the start of the material. Here are two examples, where P is the search pattern, T is the target text and R is the result, the latter showing the actual matching text in blue:-

\Z  End of Text

This matches the notional ‘empty string’ that exists at the end of the target text, thereby anchoring a pattern to the end of the material.

^ (Caret)  Start of Line

Often known as a wildcard metacharacter, this matches the notional ‘empty string’ at the start of the text or after a new line character, thereby anchoring a pattern to the beginning of a line. The \r shown here denotes the new line character, corresponding to a CR (carriage return) in the Classic Mac OS.

$ (Dollar)  End of Line

Sometimes known as a wildcard metacharacter, this matches the notional ‘empty string’ at the end of the text or before a new line character, thereby anchoring a pattern to the end of a line.

This metacharacter can be used with the caret to identify a matching line, as in:-

\<  Word Start Boundary

This matches the ‘empty string’ at the start of a word, anchoring a pattern to this point, although this isn’t recognised by all GREP applications. The vertical bar shown here indicates the anchor point, although this character doesn’t actually appear in the result.

\>  Word End Boundary

This matches the ‘empty string’ at the end of a word, anchoring a pattern to this point, although this isn’t recognised by all GREP applications.

\b  Word Boundary

This matches the ‘empty string’ at the beginning or end of a word, anchoring a pattern to this point.

Such boundaries exist between a \w and \W character (see below) in either direction.

\B  Not Word Boundary

This matches the ‘empty string’ in a word, anchoring a pattern to a position that’s not word boundary.

Character Classes

These elements let you create patterns for matching particular classes of characters.

. (Dot)  Match-All

Often known as a wildcard metacharacter, this matches any class of character, apart from the NUL control code character, the latter also identified as \0 in regular expressions.

[...]  Match in List

This character class, known as a list or bracket expression, consists of a list of one or more metacharacters or literal characters contained within square brackets. Note that the effect of metacharacters used in such a list is different to those outside of character classes. For example, a . (dot) used inside a list represents a literal dot, not the match-all character described above.

• To include a ] (square bracket) in a list you should either enter it as the first item, as in []b] or precede it with a \ (backslash), as in [b\]].
• It’s important to remember that each list represents a single character.

List Characters and Metacharacters

The following items can be found in a list:-

[[:...:]]  Match POSIX Class

POSIX stands for Portable Operating System Interface, which allows standard terminology to be used across different operating systems. These classes, which must be enclosed in double brackets, are identified by unique strings of characters, as shown below.

The enhanced version of POSIX used with the Perl language also allows negation of classes, as in [[:^digit:]], which detects a non-numerical character.

\...  Match Shortcut

These use normal characters preceded by a \ (backslash), as shown here:-

\...  Match Other Shortcut

The following character shortcuts are used by Frontier and other applications:-

Quantifiers

A quantifier or repeat sub-pattern describes how many times a preceding match should occur.

* (Asterisk)  Match None or More

This metacharacter accepts any number of preceding matches or none at all, as shown here:-

The next example shows how valid HTML tags that contain spaces can be detected:-

Remember, this operator always matches something, even the nominal ‘empty string’ that occurs at the beginning of the target text, as shown here:-

although you should note that the result is at the boundary of the text but isn’t the text itself.

+ (Plus)  Match One or More

This accepts one or more preceding matches, as shown below:-

When trying to match more than one occurrence, this more effective than the asterisk operator.

? (Question Mark)  Match One or None

Similar to the asterisk metacharacter but only accepting one preceding match or none at all.

{...}  Match Specified Number of Repeats

Lets you set how many times a preceding match is allowed to occur. The following forms are used:-

where COUNT, MIN and MAX must be replaced by integer numbers. Here’s an example:-

The Longest Match Issue

Unfortunately, all the quantifiers described above are greedy. In other words, they find the longest match. Consider the following HTML example:-

which, instead of matching the individual HTML tags, finds all the text in the outer < and > brackets. This is caused by the quantifier, which takes the search to the end of the line, then works backwards until it hits the last > bracket. You can get round this problem by using the pattern shown here:-

or by using a non-greedy quantifier, as provided in applications such as BBEdit 6.5:-

As you you can see, the quantifier is changed into this form by adding a question mark.

Alternation

Alternation allows the use of different expressions in a list, any of which could be a match. The list of expressions is delimited by a vertical bar, as shown in the following example:-

Although regular expressions are greedy, causing them to obtain the longest possible first match, the alternation metacharacter has the lowest priority of any operator. This is demonstrated by the following example, which doesn’t give the expected result:-

To fix this problem you can use grouping brackets (see below), as shown here:-

Groups

Enclosing a pattern within curved brackets creates a group, also known as sub-pattern or sub-expression. As mentioned above, this limits the scope of the alternation metacharacter. It also lets you identify elements in a search pattern or apply a quantifier to an entire group, as in this case:-

where the ‘match one or none’ operator is used to ensure that both numbers are detected successfully, even though the first value doesn’t contain a decimal fraction.

Back-References

A back-reference refers to earlier groups or sub-patterns. It consists of a \ (backslash) followed by an integer number, the latter obtained by counting the previous sub-patterns from left to right. The following example detects doubled words:-

In this instance, the sub-pattern that captures the letters of a word is referred to as \1. The doubled word can then be replaced in a replacement string (see below) by simply entering \1.

• Groups can also be contained within other groups, in which case the outside brackets are numbered first. In some applications, you can prevent a group from being included in the counting by entering ?: immediately after the opening brackets of the group.
• Some applications can only use back-reference numbers from 1 to 9, while others permit values up to 99. Some software uses a ^ (caret) prefix instead of \ for back-references.

Applications often use Perl-style pattern extensions, some of which are described below, and all of which begin with a curved open bracket followed by a question mark.

(?=...)  Exclude Group from Back-Reference Count

As mentioned above, back-references are counted from left to right. However, any groups in this form are excluded from the count, allowing you use convenient groupings with logical numbering. Technically speaking, this is known as cluster-only parenthesis or a none capturing group.

(?#...)  Comment

This kind of group, which can’t be enclosed in other groups, contains comment text, as shown here:-

(?i)  Case Insensitive

Here’s an example where the matching of text is forced to be insensitive:-

The search can be made sensitive to case by inserting a ‘negating’ hyphen:-

Perl also accommodates other search modifier letters than can be used in combination with i. Note that a colon can also be added after such letters if you don’t want the group included in back-references.

(?=...)  Positive Look-Ahead Assertion

In common with all the other positional assertions that follow, this extension doesn’t include the look-ahead or look-behind material in the result. In this example, the look-ahead function detects the semicolon after a word, giving a result of cat but without including the actual semicolon:-

Such a result, in this case lacking the semicolon, can also be used in a back-reference.

(?!...)  Negative Look-Ahead Assertion

This extension looks for something that doesn’t exist after a pattern, as in:-

Once again, the look-ahead element isn’t included in the result.

(?<=...)  Positive Look-Behind Assertion

This example looks for one immediately before two and returns the latter:-

Strings matched by a look-behind assertion must have a fixed length, which means that patterns creating results of varying lengths can’t be used within the brackets.

(?<!...)  Negative Look-Behind Assertion

This example looks for one that’s not before two and returns the latter:-

Once again, the matched strings must have a fixed length.

(? Conditionals

These take the following forms:-

Here’s an example of how the IF-THEN-ELSE conditional can be used:-

(?>...)  Once-only Match

Under normal circumstances, searching for a pattern such as /d+xx in a string such as 123yy involves multiple searches. Having matched the three numbers, the search engine finds that the letters don’t correspond so tries again, looking for the letters after the first two numbers. It repeats this until the match finally fails. Normally, this isn’t a problem. However, where large amount of text are being processed, this extra processing can be time-consuming. To prevent such repeated searching you can use the once-only form, which for the above example is (?>\d+)xx.

References

Regex Documentation, Script Meridian, 1998

BBEdit 6.5 Help file, Bare Bones Software Inc, 2002

©Ray White 2004.