1.3.11 · D1Python Intermediate

Foundations — Regular expressions — re module, patterns, groups, findall, sub

3,821 words17 min readBack to topic

Before you can read the parent note, you must be able to read every squiggle it throws at you. This page introduces each symbol one at a time, in an order where nothing appears before it is built. We start with the single most important picture: the row of boxes the engine walks across. To keep things honest, the very first thing we define below is where the engine actually stands — the position and the cursor — before we lean on those words anywhere else.


0. The cursor picture — what "matching" means

Figure — Regular expressions — re module, patterns, groups, findall, sub

Figure 1 (alt text): the string the dog drawn as seven character boxes, with the between-character positions numbered 0–7 underneath. An orange arrow marks the cursor sitting at position 0 with the caption "match starting here?"; a blue arrow along the top shows the cursor sliding rightward when a match fails.

At position 0 the engine asks "starting here, does my pattern fit?" If not, it moves to position 1, then 2, … until the string ends. This single mental model explains the difference between the two most confused functions in the whole topic (we meet them below): one function only asks at position 0; the other slides all the way along.


1. A literal character — the atom that means itself

Picture: the pattern dog is three literal boxes that must line up, in order, with three text boxes. This is the "shape" in its simplest form — an exact copy.


2. . — "any one character"

Picture: . is a box that any character can slot into. The pattern d.g matches dog, dig, d9g — three boxes, middle one free.


3. Escape characters and the backslash \

The backslash does two jobs, and you must keep them apart.

Job A — turn a plain letter into a special code. d is the literal d; \d (backslash-d) is "any digit". Here the backslash says "read the next letter as a special code, not as itself."

Job B — turn a special symbol back into a literal. Symbols like . * + ? ( ) [ ] { } ^ $ | \ are metacharacters — they mean something to the engine. To match one of them literally you put a backslash in front. So \. matches a real dot, \* a real star, \( a real open-paren, \\ a real backslash.


4. Character classes [ ] — pick your own set

Picture: [ ] is a box with a little menu taped inside — the character in that text box must be on the menu. \d is just a built-in shortcut for the menu [0-9], and \D is its negated shortcut [^0-9].


5. Quantifiers * + ? {m,n} — "how many times"

First, the word we've been using all along:

A quantifier attaches to the atom immediately before it and says how many copies to allow.

Figure — Regular expressions — re module, patterns, groups, findall, sub

Figure 3 (alt text): a table with three rows — patterns a*, a+, a? — and four test columns — empty string, a, aa, aaa. Green ticks and red crosses show what each pattern accepts: a* accepts all four; a+ accepts everything except the empty string; a? accepts only the empty string and a single a.

Read the figure across: for the atom a, the * lane accepts the empty string and any run of as; the + lane rejects empty but accepts one-or-more; the ? lane accepts only "" or a single a.


6. Greedy vs lazy — how far a quantifier reaches

Picture: on "<a><b>" the greedy <.*> shoots its marker all the way to the last >, matching <a><b> in one gulp; the lazy <.*?> stops at the first >, giving <a> then <b> separately.


7. Groups ( ) and (?: ) — drawing boxes on the rule card

Picture: parentheses are little labelled boxes drawn on the rule card. When the engine matches, it writes down what landed inside each box. group(0) is by convention the whole match; group(1), group(2), … are the boxes.


8. Anchors ^ $ \A \Z, boundaries \b \B, and alternation |

These symbols all match a position, not a character — they are zero-width assertions. They check where the cursor stands and consume no box.

Figure — Regular expressions — re module, patterns, groups, findall, sub

Figure 2 (alt text): the string a dog! drawn as character boxes. Each box is labelled \w (green) for word chars or "not w" (gray) otherwise. Vertical red seams mark every \b word boundary: at the string start before a, on both sides of the space, and at the seam between g and !. The caption notes these seams are zero-width — they mark positions, not characters.

Look at the seams described in Figure 2: \b matches the gap at the edge of a word, exactly where a \w box touches a non-\w box. \B is the opposite — it matches the gaps that are not edges (between two letters of the same word). Both consume no box; they only assert where you stand.


9. Lookaround (?= ) (?! ) (?<= ) (?<! ) — check without consuming

Anchors check for the string edge. Lookaround lets you check for any pattern to the right or left of the cursor, again without eating any characters. There are four, split by direction (ahead/behind) and polarity (must match / must not match).

Picture: lookaround is like the cursor peeking through a window without stepping forward. Example: \d+(?= dollars) matches the digits in "50 dollars" but the match is just 50 — the word dollars was only checked, not captured. And (?<=\$)\d+ matches the digits in "$50" only because a $ sits just behind.


10. Flags (?i), (?m), (?s), (?x), and scoped (?i:...)


11. The five verbs, now that you can read patterns

With every symbol defined, the parent note's five functions read cleanly:


Prerequisite map

Positions and the cursor

Literal characters

Dot any one char

Backslash escapes and metacharacters

Raw strings r-prefix

Character classes

Quantifiers star plus question

Greedy vs lazy

Groups and backreferences

Anchors and boundaries

Lookaround assertions

Flags

Regex re module and the five verbs

This feeds straight into the parent topic. If you also want the string-slicing side of text handling, see String methods — split, join, format; and for where you apply all this, Python Intermediate — File I/O and parsing logs. Under the hood the engine that walks these positions is a finite state machine.


Equipment checklist

Test yourself — you should be able to answer each before reading the parent note.

What is a "position" in a regex scan?
A gap between characters, numbered 0 to len(s); the cursor lives there, not on a character.
What does a literal character match?
Exactly itself.
What does . match, and how many characters?
Any single character except newline — exactly one.
What is a metacharacter, and how do you match one literally?
A symbol the engine treats as an instruction (. * + ? ( ) [ ] { } ^ $ | \); prefix it with \ (e.g. \. matches a real dot).
Why must patterns use raw strings r"..."?
So Python doesn't interpret the backslashes; the regex engine receives \d, \b intact.
What does \d mean, and what is \D?
\d = any digit; \D = any non-digit.
In Python 3, does \w equal [A-Za-z0-9_]?
Not by default — \w matches all Unicode word characters; use the re.ASCII flag (or (?a)) to restrict it to [A-Za-z0-9_].
What does [^abc] match?
One character that is NOT a, b, or c.
The ^ inside [ ] versus outside — two meanings?
Inside brackets it means "not"; outside it anchors the start of the string.
What is an atom?
The smallest matchable unit a quantifier can attach to: a literal, ., a shorthand, a character class, or a group.
What atom does * attach to, and what does it mean?
The atom immediately to its left; repeat that atom 0 or more times.
Difference between +, *, and ? as quantifiers?
+ = 1 or more, * = 0 or more, ? = 0 or 1.
What makes a quantifier lazy, and does it work on {m,n} too?
A ? placed after it (*?, +?, {m,n}?); it matches as few characters as possible.
What does ( ) capture, and how are groups numbered?
The text matched inside it; numbered by opening parenthesis, left to right, starting at 1.
What is group(0)?
The entire matched substring.
What does \1 mean inside a pattern versus inside a replacement?
In a pattern it matches the same text group 1 captured; in a replacement it inserts that captured text.
How do you write and reference a named group?
Define with (?P<name>...); reference in-pattern with (?P=name), retrieve with m.group("name").
What is a zero-width assertion?
A pattern piece that succeeds based on the cursor's position and consumes no characters (anchors, \b, lookaround).
What do \A and \Z anchor, and how do they differ from ^/$?
Absolute start/end of the string; unlike ^/$ they never shift to line boundaries in multiline mode.
What is \b versus \B?
\b = a word boundary (seam between a word char and a non-word char); \B = a position that is not such a seam.
What do the four lookarounds do?
(?=X) positive lookahead, (?!X) negative lookahead, (?<=X) positive lookbehind, (?<!X) negative lookbehind — each checks context without consuming characters.
Does a lookahead show up in group(0) or findall?
No — it matches zero characters, so it only gates the match, it does not capture.
What does the inline flag (?i) do, and what is (?i:...)?
(?i) makes the whole pattern case-insensitive; (?i:...) applies it to only the parenthesised slice.
What does (?x) (verbose) enable?
Whitespace in the pattern is ignored and # starts a comment, so long patterns can be spread over lines with notes.
Why is re.match("dog","the dog") None?
match only tries at position 0, where the character is t; the cursor never slides.