Regular expressions — re module, patterns, groups, findall, sub: Visual walkthrough

We use exactly one running example so nothing ever surprises you:

$pattern = \ d+, text = "a12b"$

Step 1 — The cursor: text is a strip of numbered cells

WHAT. Before any matching happens, picture the text not as a word but as a row of boxes, one character per box, each with a position number between the boxes (0, 1, 2, 3, 4). The engine holds a single pointer — call it the cursor — that sits at one of those numbered gaps.

WHY a numbered strip and not just "a string"? Because every regex decision is really the question "at which position am I, and what character sits just to my right?" Positions (not characters) are what ^, $, and word boundaries \b talk about. If you cannot see the positions, anchors feel like magic; once you see them, they are obvious.

PICTURE. The teal boxes are characters; the plum numbers are positions. The cursor (orange) starts at position 0.

Figure — Regular expressions — re module, patterns, groups, findall, sub

Step 2 — What one atom means: `\d` is a yes/no test on one box

WHAT. The pattern \d+ is made of two pieces: the atom \d and the quantifier +. Ignore + for now. The atom \d is a single test: "look at the one character immediately to the right of the cursor — is it a digit 0–9? yes/no."

WHY split atom from quantifier? Because the engine only ever knows how to do two things: (1) test one atom against one character, and (2) repeat that test as the quantifier tells it. Every huge regex is just these two moves stacked. Keeping them separate is the whole secret.

PICTURE. At position 0 the char to the right is a. Run the test \d: is a a digit? No (red cross). The atom fails here.

Step 3 — Failing at the start: the engine slides right and retries

WHAT. re.search did not find a match at position 0 (because a is not a digit). So it does the one thing it always does on failure at a start position: advance the cursor by one and try the entire pattern again from position 1.

WHY slide instead of give up? This is exactly the difference the parent note flagged between search and match. re.match tries only position 0 and stops. re.search retries at 1, 2, 3, … until the pattern fits somewhere or the text runs out. Sliding is how "find it anywhere" is built out of "test it here."

PICTURE. The cursor jumps from 0 to 1. Now the char to the right is 1 — a digit. The \d test passes (green check). This is our first success.

Step 4 — `+` is greedy: it eats digits as long as it can

WHAT. The + means "one or more of the previous atom." Having matched one digit (1) at position 1, the engine advances to position 2, tests \d again on 2 → passes, advances to position 3, tests \d on b → fails. It cannot eat b, so the + stops. It has consumed 1 and 2.

WHY does it grab as much as possible? Because + is greedy — it always tries the largest repetition count first (the parent's formula $A^{+} = A^{{1, \infty}}$ , largest-first). It keeps swallowing matching characters until the atom finally fails, then hands control back.

PICTURE. Watch the orange bracket stretch: it grabs 1, grabs 2, then bounces off b. The matched span is positions 1→3, the substring "12".

Step 5 — Backtracking: what happens when greed goes too far

WHAT. Our pattern is only \d+, which ended cleanly. But greed can overshoot. Consider the stretch case \d+b on "a12b". The \d+ greedily eats 12, then the pattern still needs to match a literal b. The cursor is at position 3 — and b is there, so it works. But imagine the pattern were \d+2: greed eats 1 and 2, leaving the cursor at b, and now the required 2 cannot be found. The engine must backtrack: give one digit back (release the 2), re-check if 2 matches at that returned position — and it does.

WHY show backtracking? Because it is the reason greedy vs lazy matters (parent Example 5, <.*> vs <.*?>). Greedy grabs everything, then rewinds one step at a time until the rest of the pattern can also succeed. Without seeing the rewind, greedy/lazy is memorized; with it, it is understood.

PICTURE. Top row: \d+ greedily takes 12, then the required 2 has nothing left → dead end (red). Bottom row: the engine gives back one char, the cursor steps left, and now 2 matches (green).

Step 6 — Degenerate case: zero matches and the empty text

WHAT. Two edge cases you must never be surprised by.

No digits at all: re.search(r"\d+", "abc"). The cursor slides 0→1→2→3, the \d test fails at every position, the text ends → the engine returns ==None==.
Empty string: re.search(r"\d+", ""). There is no character to the right of position 0, so \d fails immediately → None. But note: a pattern that allows zero repetitions, like \d*, matches the empty span at position 0 and returns a match of ''.

WHY cover these? Because the number-one crash for beginners is calling .group() on a None. Knowing exactly when the engine returns None (nothing fit anywhere) versus an empty match (a zero-count quantifier succeeded on nothing) prevents that crash.

PICTURE. Left: cursor tries every gap of "abc", all red, result None. Right: on "", \d+ gives None but \d* gives an empty green match at position 0.

Step 7 — Groups: the same cursor, but with remembered spans

WHAT. Add parentheses: (\d+) on "a12b". The match is identical ("12" at positions 1→3), but the engine now also records the start and end position of whatever the ( ) matched. group(0) = the whole match's span; group(1) = the recorded span of the first pair of parentheses.

WHY does findall return tuples with groups? Because with groups the engine has more than one remembered span per match. With 0 groups it hands you the whole match; with 1 group it hands you that group's text; with 2+ groups it can't collapse them, so it hands you a tuple of spans — exactly the parent's Example 3 rule. The tuple is just "here are all the boxes I remembered."

PICTURE. The full match span (orange) and the captured group-1 span (plum) drawn as two brackets over the same "12"; a side table shows group(0)='12', group(1)='12', and for a two-group pattern (\d)(\d) the recorded pair ('1','2').

The one-picture summary

Everything above is one cursor doing four moves: slide (advance on failure at a start), test (one atom vs one char), repeat greedily (the +), and backtrack (give characters back until the rest fits) — while optionally remembering spans (groups). This single diagram compresses the whole crawl of \d+ over "a12b".

Recall Feynman retelling — the whole walkthrough in plain words

Picture your text as a row of boxes with numbered gaps between them, and one finger (the cursor) pointing at a gap. The regex \d+ is a tiny rule: "the box to my right must be a digit, and grab as many digits in a row as you can." I put my finger at gap 0 — the box is a, not a digit, fail. So I slide my finger one gap right and try again. Now the box is 1, a digit — success! Greedily I keep going: next box 2 is a digit too, grab it; next box b is not, so I stop. I've grabbed "12". If my rule had extra parts after the digits and I grabbed too much, I'd hand the last digit back and try again — that's backtracking. If the text had no digits anywhere, I'd slide to the end and report nothing found (None), which is why you always check before asking for .group(). And if I wrap the digits in parentheses, I additionally remember where they started and ended, so I can pull that piece out later or paste it into a replacement. That's the entire engine: slide, test, grab, backtrack, remember.

Connections

Parent topic
Finite State Machines — the compiled form of this slide-and-test cursor
Greedy vs Backtracking algorithms — the grab-then-undo pattern seen in Steps 4–5
Escape characters and raw strings — why the atom is written r"\d"
String methods — split, join, format
Python Intermediate — File I/O and parsing logs