rusoblanco

Purity, Determinism and Idempotency Clearly Explained

Mon, 09 Mar 2026 12:00:00 +0000

The terms pure, deterministic, and idempotent are not synonyms. Each describes a different property of a function, and confusing them leads to sloppy reasoning about code. This post breaks down all three with concrete Python examples.

1. Two Kinds of Idempotency

The word “idempotent” has two distinct formal meanings, and conflating them is the root of many misunderstandings.

Value-idempotent (mathematical): f(f(x)) = f(x). Applying the function to its own output gives the same result. For this to be well-defined, the function’s input and output types must be exactly the same (an endomorphism, a -> a). These are projections — they collapse their input into a canonical form, and applying them again changes nothing. Examples: abs, upper, clamp. Counter-examples: negate, increment.

Effect-idempotent (imperative): Calling a function multiple times has the same effect on system state as calling it once. This is what matters for retries, APIs, and distributed systems. Examples: HTTP PUT, HTTP DELETE, upserting a database row.

The unifying intuition: doing it twice is the same as doing it once — applied either to values or to effects.

2. The Three Properties

Determinism. A deterministic function always returns the same output for the same input — across calls, across time, across machines. Nondeterminism creeps in whenever a function consumes a hidden input not in the argument list:

Hidden state — a database row, the system clock, an environment variable, a global counter. The value can change between calls, so the output changes too.
Pseudo-random number generators (PRNGs) — deterministic algorithms that become nondeterministic because they are typically seeded by a hidden input.
True random number generators (TRNGs) — hardware or quantum RNGs are nondeterministic by nature. There is no hidden input to expose; the output is genuinely unpredictable.
Concurrency — when threads share mutable state, the OS scheduler’s timing becomes an implicit input. Two runs of the same code can interleave differently, producing different results.

Purity. A pure function has two constraints: (1) its output depends only on its input arguments (which implies it is deterministic), and (2) it produces no side effects — no writes to disk, no mutations of external state, no network calls. Pure functions are referentially transparent, meaning a function call can be replaced with its return value without changing the program’s behavior. An impure function violates at least one of those constraints.

Idempotency. As defined in Section 1 — either value-idempotent (f(f(x)) = f(x)) or effect-idempotent (repeated calls leave state unchanged after the first).

All pure functions are deterministic. All pure functions are trivially effect-idempotent (no effects to repeat). But pure functions are NOT automatically value-idempotent.

3. How They Combine

Purity is the strongest property: it implies determinism and trivial effect-idempotency. But the reverse does not hold. Let’s walk through each combination.

Pure, Value-Idempotent

1
2
3
4
5
6


def normalize_email(email: str) -> str:
 """Strip whitespace and lowercase. f(f(x)) = f(x)."""
 return email.strip().lower()

assert normalize_email(" Alice@Example.COM ") == "alice@example.com"
assert normalize_email(normalize_email(" Alice@Example.COM ")) == "alice@example.com"

def normalize_email(email: str) -> str:
 """Strip whitespace and lowercase. f(f(x)) = f(x)."""
 return email.strip().lower()

assert normalize_email(" Alice@Example.COM ") == "alice@example.com"
assert normalize_email(normalize_email(" Alice@Example.COM ")) == "alice@example.com"

Pure, NOT Value-Idempotent

1
2
3
4
5
6
7
8


from urllib.parse import quote

def url_encode(text: str) -> str:
 """Percent-encode a string for URLs."""
 return quote(text, safe="")

assert url_encode("hello world") == "hello%20world"
assert url_encode(url_encode("hello world")) == "hello%2520world" # Double-encoded!

from urllib.parse import quote

def url_encode(text: str) -> str:
 """Percent-encode a string for URLs."""
 return quote(text, safe="")

assert url_encode("hello world") == "hello%20world"
assert url_encode(url_encode("hello world")) == "hello%2520world" # Double-encoded!

Double-encoding (%20 → %2520) is a real and common bug — pure, but not value-idempotent.

Impure, Deterministic, Effect-Idempotent

Like HTTP PUT — writing the same value again leaves the system unchanged.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


config_store: dict[str, str] = {}

def upsert_config(key: str, value: str) -> str:
 config_store[key] = value
 return f"OK: {key}={value}"

assert upsert_config("timeout", "30") == "OK: timeout=30"
assert upsert_config("timeout", "30") == "OK: timeout=30"

# State after one call or ten calls is identical:
assert config_store == {"timeout": "30"}

config_store: dict[str, str] = {}

def upsert_config(key: str, value: str) -> str:
 config_store[key] = value
 return f"OK: {key}={value}"

assert upsert_config("timeout", "30") == "OK: timeout=30"
assert upsert_config("timeout", "30") == "OK: timeout=30"

# State after one call or ten calls is identical:
assert config_store == {"timeout": "30"}

Impure, Deterministic, NOT Effect-Idempotent

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


audit_log: list[str] = []

def append_to_audit_log(event: str) -> str:
 audit_log.append(event)
 return f"logged: {event}"

assert append_to_audit_log("user_login") == "logged: user_login"
assert append_to_audit_log("user_login") == "logged: user_login"

# Same return value, but the side effect accumulated:
assert len(audit_log) == 2 # Not effect-idempotent!

audit_log: list[str] = []

def append_to_audit_log(event: str) -> str:
 audit_log.append(event)
 return f"logged: {event}"

assert append_to_audit_log("user_login") == "logged: user_login"
assert append_to_audit_log("user_login") == "logged: user_login"

# Same return value, but the side effect accumulated:
assert len(audit_log) == 2 # Not effect-idempotent!

Every call adds another entry — a retry produces a duplicate log line.

Impure, Nondeterministic, Effect-Idempotent

This is the classic case that confuses most developers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


database: dict[str, str] = {"item_1": "Apple", "item_2": "Banana"}

def delete_item(item_id: str) -> int:
 if item_id in database:
 del database[item_id]
 return 200 # OK
 return 404 # Not Found

assert delete_item("item_1") == 200 # First call: deletes the item
assert delete_item("item_1") == 404 # Second call: item already gone

# But the state of the database is the same after both calls!

database: dict[str, str] = {"item_1": "Apple", "item_2": "Banana"}

def delete_item(item_id: str) -> int:
 if item_id in database:
 del database[item_id]
 return 200 # OK
 return 404 # Not Found

assert delete_item("item_1") == 200 # First call: deletes the item
assert delete_item("item_1") == 404 # Second call: item already gone

# But the state of the database is the same after both calls!

The return value changed (200 → 404), but the state of the system did not. Effect-idempotency is about the effect, not the output.

Impure, Nondeterministic, NOT Effect-Idempotent

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


_next_id = 0

def create_user(email: str) -> dict:
 global _next_id
 _next_id += 1
 user = {"id": _next_id, "email": email}
 # Imagine: database.insert(user)
 return user

first = create_user("alice@example.com") # {"id": 1, ...}
second = create_user("alice@example.com") # {"id": 2, ...}

assert first["id"] != second["id"] # Different output (nondeterministic)
# And two rows now exist in the "database" (not effect-idempotent)

_next_id = 0

def create_user(email: str) -> dict:
 global _next_id
 _next_id += 1
 user = {"id": _next_id, "email": email}
 # Imagine: database.insert(user)
 return user

first = create_user("alice@example.com") # {"id": 1, ...}
second = create_user("alice@example.com") # {"id": 2, ...}

assert first["id"] != second["id"] # Different output (nondeterministic)
# And two rows now exist in the "database" (not effect-idempotent)

A network retry creates a duplicate user — this is why payment APIs require idempotency keys:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


seen_requests: dict[str, dict] = {}

def create_user_idempotent(email: str, idempotency_key: str) -> dict:
 if idempotency_key in seen_requests:
 return seen_requests[idempotency_key]

 user = {"id": generate_id(), "email": email}
 # database.insert(user)
 seen_requests[idempotency_key] = user
 return user

seen_requests: dict[str, dict] = {}

def create_user_idempotent(email: str, idempotency_key: str) -> dict:
 if idempotency_key in seen_requests:
 return seen_requests[idempotency_key]

 user = {"id": generate_id(), "email": email}
 # database.insert(user)
 seen_requests[idempotency_key] = user
 return user

The first call creates the resource and caches the result; subsequent calls with the same key return the cached result and skip the write.

4. The Decision Tree

 flowchart TD
 A(["f(x)"]) --> B{"Is it pure?"}
 B -- Yes --> V{"f(f(x)) = f(x)?"}
 V -- Yes --> D1["Pure · Value-idempotent"]
 V -- No --> D2["Pure · Not value-idempotent"]
 B -- No --> F{"Same input →\nsame output?"}
 F -- Yes --> I1{"Same effect on\nrepeated calls?"}
 F -- No --> I2{"Same effect on\nrepeated calls?"}
 I1 -- Yes --> G1["Impure · Deterministic · Effect-idempotent"]
 I1 -- No --> G2["Impure · Deterministic · Not effect-idempotent"]
 I2 -- Yes --> H1["Impure · Nondeterministic · Effect-idempotent"]
 I2 -- No --> H2["Impure · Nondeterministic · Not effect-idempotent"]

5. Why This Matters

These distinctions have direct, practical consequences:

Pure functions are trivially testable (no mocks, no setup) and safely memoizable (@functools.lru_cache). Keep functions pure whenever possible.

Effect-idempotency is a design requirement for any operation that might be retried — HTTP handlers, message queue consumers, database migrations.

Value-idempotency matters for normalization and canonicalization. Value-idempotent functions can safely be applied “just in case” without worrying about double-processing.

Read-only operations are impure and nondeterministic, yet effect-idempotent — reading changes nothing. In HTTP/REST this category is called safe; safe operations can always be retried without risk.

Impurity is not evil, but it should be contained. Push side effects to the edges of your system.

Conclusion

Three properties, three questions:

Is it pure? Yes → automatically deterministic, trivially effect-idempotent. But check value-idempotency.
Same input, same output? Yes → deterministic. No → nondeterministic.
Same effect on repeated calls? Yes → effect-idempotent. No → not.

Your Code Is Not You: What a Music Producer Taught Me About Programming

Wed, 04 Feb 2026 01:29:37 +0100

I have a mixed background. For the past 20 years, I’ve worked as a professional programmer, but I started my career as a sound technician. Because of that experience, I still follow trends in the audio world. I watch production tutorials and read about mixing techniques.

Recently, I saw a video by Gregory Scott from Kush Audio titled “Is ‘Self Expression’ Limiting Your Art?” He mainly spoke to musicians, but I realized he was describing the same emotional trap I’ve faced in software development for two decades.

The Trap of “Code as Identity”

Scott mentions that artists often see their recordings as a reflection of their self-worth. If a vocal take doesn’t work out, they feel like they aren’t good enough. If a producer critiques the performance, the artist takes it as a critique of their soul.

Programmers do the same thing.

After two decades in this field, I know that coding is an art form. It serves as a canvas for self-expression. We stress over elegant architecture, take pride in clever one-liners, and protect our code as if it defines who we are. However, this attachment creates a fragile situation. When a QA tester spots a bug or a peer reviewer suggests a refactor, it hurts. We defend our code not because it’s the best or cleaner option, but because it’s ours.

For years, I experienced this with my engineering work. I created my own tools, libraries, and frameworks—many of which could compete with famous open-source and commercial solutions—but I never published or tried to sell them. I felt uneasy about their internal cohesion, thought some architectural choices weren’t elegant enough, or even worried the code would seem over-engineered. I viewed every line of code as part of my biography; if the code wasn’t perfect, I wasn’t a perfect programmer.

The “Speaker” Philosophy

Scott’s solution is to view work through the lens of pure utility. He argues that once sound leaves the instrument, it is no longer the artist’s. It’s just “sound coming out of the speakers.” It’s physics. It either evokes the right emotion or it doesn’t.

Applying this to programming: The code is not you.

Once you commit that code, it becomes an artifact. It’s just logic running on a machine. It either delivers an efficient, maintainable solution to the user’s problem or it doesn’t.

When Scott is in the studio, he doesn’t focus on the singer’s “intent”; he cares about the vibrations in the air. As developers, we need to stop worrying about how our code makes us feel about ourselves and focus solely on the results.

Use the Pitch Correction

To embrace this, Scott argues that we shouldn’t restrict our tools out of a sense of “integrity.” He uses pitch correction (Melodyne), aggressive quantization, and heavy compression freely. If it helps him achieve the sound he wants, he uses it.

In the development world, we often shame ourselves or others for using “crutches.” We might think that using an AI assistant (like Copilot), depending on a heavy framework, or relying on a no-code tool is “cheating.”

Take this blog, for example. I revived it after 15 years of silence because writing had become too difficult for me to maintain consistently. Now, I use language models to help draft and refine my thoughts. Some would view this as cheating—a kind of literary autotune. But it achieves the goal. Thanks to these tools, I now have a way to express thoughts that would otherwise stay locked in my head.

If we adopt the producer’s mindset, the only thing that matters is the output. Is the system stable? Is it fast? Is it maintainable? If the answer is yes, then the tools you used to achieve that are valid. Make the solution your god, and sacrifice your ego at its altar.

The Freedom to Fail

The most liberating part of this mindset is the freedom to fail. Scott talks about recording “stupid” takes just to see what happens. When we detach our ego from the code, we can prototype boldly. We can write “ugly” code to test an idea and not feel embarrassed to show it to others.

If you see your code as a reflection of your intelligence, you will be afraid to share it until it’s “perfect.” But if you treat it strictly as “sound coming out of the speakers,” you can do whatever is necessary to make it work.

Make More Art

This detachment is tough. It requires letting go of the belief that our work defines who we are. I found release in a quote from Andy Warhol that aligns perfectly with this idea:

“Don’t think about making art, just get it done. Let everyone else decide if it’s good or bad, whether they love it or hate it. While they are deciding, make even more art.”

This mindset changed how I work. I stopped fretting over whether my scripts were “good enough” for the public. I realized my hesitation was just vanity dressed up as high standards.

Don’t be precious. The code isn’t you. Ship it!, and let the internet decide if it’s “good” or “bad.” While they’re busy deciding, write the next one.

PACE Planning: Why Your "Solid" Backup Plan Is Probably Fragile

Thu, 08 Jan 2026 12:00:00 +0000

We tell ourselves we’re prepared. We have backups. We have redundancies. Software engineers pride themselves on avoiding Single Points of Failure (SPOF).

But often, when chaos actually strikes (a server outage, a natural disaster, or just a really bad Monday), we find our backup plans were little more than lists of things we hoped would work.

A framework exists to cure this optimism bias: PACE planning.

It originated in the military, specifically for Special Ops comms plans. I’ve found it to be an incredibly potent tool for engineering resilience and personal peace of mind. It transforms “trying to figure it out on the fly” into a structured algorithm for continuity.

Simply writing a PACE plan down on a napkin isn’t enough, though. If you don’t understand your dependencies, even a four-layer backup plan can collapse in seconds.

The PACE Breakdown

PACE is an acronym determining the order of precedence for methods used to achieve a critical goal, or “mission.” It forces you to define four distinct ways to get the job done:

P - Primary: The preferred method. It’s the most efficient, effective, and habitual way you execute the task.
A - Alternate: The iconic “Plan B.” It should be nearly as effective as the Primary, often running in parallel or available with minimal friction.
C - Contingency: “Plan C.” Here is where things get annoying. Ideally, the Contingency method is reliable but less convenient, slower, or more resource-intensive.
E - Emergency: The last resort. “Plan D.” usually serves as a break-glass method. It will likely be slow, expensive, or provide only minimum viability, but it prevents total mission failure.

One crucial nuance in the formal definition of PACE often gets ignored: the four methods must be “independent enough that failure of one does not break the others.”

That logical condition, independent enough, is the hardest part to get right.

When It Works: The Coffee Run

To see this in action, let’s apply it to a low-stakes, critical daily mission: Getting Morning Coffee.

Primary: The Espresso Machine (Fast, delicious, routine).
Alternate: The French Press (Slower cleanup, requires a separate kettle, but good quality).
Contingency: Instant Coffee (Tastes worse, requires boiling water, very fast).
Emergency: A canned energy drink from the fridge (Cold, different chemical profile, but delivers the caffeine payload).

Why is this a good plan? The dependencies are physically distinct. If the Espresso machine’s electronics fail, the low-tech French Press works. If the electric kettle breaks, you can boil water in a pot on your gas stove. If the power goes out entirely, the gas still flows to boil the water. If the gas line is cut, the cold energy drink in the fridge requires no heat source at all.

Getting a PACE plan right for small missions is easy because the physics remain visible to the naked eye.

Where It Breaks: Hidden Interdependencies

Trouble starts when we apply “Napkin PACE Planning” to complex systems like IT infrastructure or Incident Response where the dependencies are abstract and invisible.

Consider a scenario many of us face: Remote Incident Response.

The Mission: You are the on-call engineer. You need to SSH into a production database to clear a deadlock during a critical launch.

You feel safe. You scribbled down a PACE plan:

Primary: Home Fiber Internet (Wi-Fi).
Alternate: 5G Hotspot via Mobile Phone.
Contingency: The Coworking Space/Coffee Shop down the street.
Emergency: Voice call a colleague in a different region to dictate commands.

On the surface, this looks robust. Theoretically, you have three layers of redundancy before you have to resort to the awkward phone call.

The Event: A transformer blows, causing a wide-area power outage in your neighborhood.

The Cascade:

Primary Fails: Your router goes dark immediately. You switch to your phone.
Alternate Fails: Here’s the nuance. The local cell tower has a battery backup, so the signal is live. But 10,000 of your neighbors just lost their Wi-Fi simultaneously. They all switched to 5G at the exact same second to check Twitter and complain to the power company. The congestion is so high that your SSH packets are dropped.
Contingency Fails: You grab your laptop and run to the coffee shop. You forgot that “The Coffee Shop” shares a hidden dependency with “Home Fiber”: Geography. They are on the same electrical grid substation. The shop is dark and locked.

The Result: In under 60 seconds, your plan collapsed from Primary straight to Emergency. You are now clutching your phone, praying that voice traffic gets prioritized over the 5G data congestion that just killed your Alternate plan, hoping the call actually goes through.

The Transitive Dependency Problem

Why did the Remote Work plan fail while the Coffee plan succeeded?

It failed because we confuse Methods with Resources, and we ignore that dependencies are transitive.

On the napkin, we think: Primary Method -> Home Router (in this context “->” means “depends on”)

In reality, the dependency graph is transitive: Primary Method -> Home Router -> Power Grid

Map out the other methods, and you’ll see they all converge on the same transitive node:

Contingency (Coffee Shop) -> Shop Open -> Power Grid
Alternate (5G) -> Cell Tower -> Finite Bandwidth (An exhaustible resource shared by your neighbors, who are also reacting to the Power Grid failure).
Emergency (Call Colleague) -> Cell Tower -> Finite Bandwidth (Now competing with the congestion).

When we design systems informally, we act as if the failure of the method is the only risk (e.g., “The router broke”). We rarely model the failure of the transitive resources (e.g., “The neighborhood has no power”).

Visualizing the Failure

Graph these dependencies out, and the fragility becomes obvious.

Check the diagram below for two things:

The “Phone” bottleneck: Both your Alternate (Hotspot) and Emergency (Voice Call) plans rely on a single physical device. If you drop your phone in the panic, you lose 50% of your PACE plan instantly.
The Failure Cascade: The red nodes represent the root failure. When the Regional Power Grid fails, it knocks out the Primary and Contingency methods directly. But it also creates a massive spike in traffic that saturates the RF Bandwidth, effectively choking off your Alternate and Emergency channels.

 graph TD
 classDef method fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
 classDef resource fill:#fff9c4,stroke:#fbc02d,stroke-width:2px,color:black;
 classDef failure fill:#ffcdd2,stroke:#c62828,stroke-width:3px,color:black;

 Goal((Mission: <br>Incident Response)) --> P[Primary: <br>Home Fiber]
 Goal --> A[Alternate: <br>5G Hotspot]
 Goal --> C[Contingency: <br>Coffee Shop]
 Goal --> E[Emergency: <br>Voice Call]

 subgraph "Hidden Shared Dependencies"
 P --> Router --> HomePower
 C --> ShopWiFi --> ShopPower
 
 A --> Phone
 E --> Phone
 
 A --> CellTower
 E --> CellTower
 
 CellTower --> RF[RF Bandwidth]
 
 HomePower --> Grid[Regional Power Grid]
 ShopPower --> Grid
 end

 %% The Cascasde
 Grid -.- |Failure Causes Congestion| RF

 class P,A,C,E method;
 class Router,Phone,CellTower,HomePower,ShopWiFi,ShopPower resource;
 class Grid,RF failure;

Awareness vs. Invincibility

Quick disclaimer before going further: You can always break a PACE plan if you zoom out far enough.

Dependency graphs are fractals. Dig deep enough and you’ll always find a shared dependency that can topple the whole stack.

If a massive earthquake forces you to evacuate your home, you aren’t going to have your morning coffee, regardless of whether you have a French Press or a gas stove.
If you suffer a medical emergency like a stroke, you won’t be clearing that database lock even if you have four different internet connections.

There’s always a “Black Swan” event (a meteor, a war, or biology itself) that creates a total failure state.

The goal of this exercise, and of the tools we are about to explore, isn’t to create a theoretically bulletproof plan for the apocalypse. The goal is to know the limits of the plan you do have.

We want to move from “I hope this works” to “I know exactly why this might fail.” We want to catch the preventable, logical errors like shared power grids so we aren’t blindsided by the mundane, all while accepting the risks of the catastrophic.

Beyond the Napkin

A PACE plan is only as good as the independence of its layers. If failure of the Primary layer triggers a condition that propagates down a transitive chain to kill the Alternate and Contingency layers, you don’t actually have a plan. You have a hallucination of safety.

For critical systems, we need to be able to mathematically prove that our “Plan B” is actually available when “Plan A” dies. We need to move from intuition to verification.

Next time, I’ll explore how we can use Formal Methods (specifically TLA+) to model these resources and find hidden problems. We will move beyond scribbling on napkins and start verifying our resilience strategies with the same rigor we use to verify distributed systems.

Stay tuned.

Untangling Business Rules from External IO: A Practical Python Guide

Fri, 19 Dec 2025 12:00:00 +0000

A common challenge in application development is the entanglement of business rules with external operations (like database writes or API calls). When these mix, testing becomes difficult, and error handling often degrades into a web of try/except blocks that hide the actual flow of logic.

This post explores a practical pattern to solve these specific problems. By treating program outcomes as data rather than exceptions, and by isolating external side effects from core logic, we can create code that is easier to read, safer to refactor, and simpler to test.

1. The Philosophy: Exceptions vs. Variants

In this approach, we distinguish between System Crashes and Business Decisions.

System Failures (The Unexpected):
- Examples: Disk full, Syntax Error, Database Definition Mismatches.
- Strategy: Let them bubble. The logic cannot fix a full disk. These fly up to the application entry point (or your framework’s middleware) to be logged as 500 errors.
World-Errors (The Decision Triggers):
- Examples: An API timeout, a rate limit, a bank denial.
- Strategy: Catch and Transform. We catch these at the edge (where we talk to the API) and return them as Data. If the logic needs to decide what to do when a bank is “Busy,” that “Busy” state is no longer an exception—it is input data.
Domain Failures (The Expected):
- Examples: Insufficient funds, User banned.
- Strategy: Return Data. These are valid logic branches modeled as specific data structures.

2. Modeling Data with Sum Types

We use dataclasses and the | operator (Union types) to model outcomes. We carefully separate Infrastructure results (what the gateway said) from Logical results (what that means for our application).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


from dataclasses import dataclass
from typing import Callable, ContextManager, assert_never

# --- Entities ---
@dataclass(frozen=True)
class Account:
 id: str
 balance: float

# --- Infrastructure Variants (The "World" output) ---
@dataclass(frozen=True)
class PaymentOK:
 txn_id: str

@dataclass(frozen=True)
class PaymentDenied:
 reason: str

# Note: Timeout is a recognized state, not an exception
@dataclass(frozen=True)
class PaymentTimeout:
 wait_seconds: int

PaymentResult = PaymentOK | PaymentDenied | PaymentTimeout

# --- Application Variants (The Use-Case output) ---
@dataclass(frozen=True)
class TransferSuccess:
 txn_id: str
 remaining_balance: float

@dataclass(frozen=True)
class TransferDeclined:
 reason: str

@dataclass(frozen=True)
class SystemUnavailable:
 message: str

TransferResult = TransferSuccess | TransferDeclined | SystemUnavailable

from dataclasses import dataclass
from typing import Callable, ContextManager, assert_never

# --- Entities ---
@dataclass(frozen=True)
class Account:
 id: str
 balance: float

# --- Infrastructure Variants (The "World" output) ---
@dataclass(frozen=True)
class PaymentOK:
 txn_id: str

@dataclass(frozen=True)
class PaymentDenied:
 reason: str

# Note: Timeout is a recognized state, not an exception
@dataclass(frozen=True)
class PaymentTimeout:
 wait_seconds: int

PaymentResult = PaymentOK | PaymentDenied | PaymentTimeout

# --- Application Variants (The Use-Case output) ---
@dataclass(frozen=True)
class TransferSuccess:
 txn_id: str
 remaining_balance: float

@dataclass(frozen=True)
class TransferDeclined:
 reason: str

@dataclass(frozen=True)
class SystemUnavailable:
 message: str

TransferResult = TransferSuccess | TransferDeclined | SystemUnavailable

3. The Logic Factory

We use a Higher-Order Function to handle dependency injection. The outer function accepts the functional dependencies (like database readers or API clients), and returns the inner function (the actual logic) ready to be executed.

Sequencing Side-Effects

We structure the logic to keep External IO (Slow/Unreliable) separate from Internal State Updates (Atomic/Critical). This ensures we only modify local state after we receive confirmation from the external world.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


def make_transfer_funds(
 *,
 fetch_account: Callable[[str], Account | None],
 save_account: Callable[[Account], None],
 debit_external: Callable[[float], PaymentResult],
 unit_of_work: ContextManager
) -> Callable[[str, float], TransferResult]:

 def transfer_funds(sender_id: str, amount: float) -> TransferResult:
 # 1. Validation (Pre-Check)
 sender_preview = fetch_account(sender_id)
 if not sender_preview:
 return TransferDeclined(reason="Account not found")
 
 # 2. External World Actions (The "Impure" Phase)
 # We perform the slow external call outside the unit of work
 payment = debit_external(amount)

 # 3. Decision & Persistence (The "Atomic" Phase)
 match payment:
 case PaymentOK(txn_id):
 with unit_of_work:
 # In this atomic phase, we ensure we act on the latest data
 sender_uptodate = fetch_account(sender_id)
 
 if not sender_uptodate:
 return TransferDeclined(reason="Account disappeared")

 new_balance = sender_uptodate.balance - amount
 
 if new_balance < 0:
 return TransferDeclined(reason="Insufficient funds")

 # Functional update: Create new, don't mutate old
 updated_account = Account(sender_uptodate.id, new_balance)
 save_account(updated_account)

 return TransferSuccess(
 txn_id=txn_id,
 remaining_balance=new_balance
 )

 case PaymentDenied(reason):
 return TransferDeclined(reason=reason)

 case PaymentTimeout(wait):
 return SystemUnavailable(message=f"Bank busy, retry in {wait}s")
 
 case _ as unreachable:
 assert_never(unreachable)

 return transfer_funds

def make_transfer_funds(
 *,
 fetch_account: Callable[[str], Account | None],
 save_account: Callable[[Account], None],
 debit_external: Callable[[float], PaymentResult],
 unit_of_work: ContextManager
) -> Callable[[str, float], TransferResult]:

 def transfer_funds(sender_id: str, amount: float) -> TransferResult:
 # 1. Validation (Pre-Check)
 sender_preview = fetch_account(sender_id)
 if not sender_preview:
 return TransferDeclined(reason="Account not found")
 
 # 2. External World Actions (The "Impure" Phase)
 # We perform the slow external call outside the unit of work
 payment = debit_external(amount)

 # 3. Decision & Persistence (The "Atomic" Phase)
 match payment:
 case PaymentOK(txn_id):
 with unit_of_work:
 # In this atomic phase, we ensure we act on the latest data
 sender_uptodate = fetch_account(sender_id)
 
 if not sender_uptodate:
 return TransferDeclined(reason="Account disappeared")

 new_balance = sender_uptodate.balance - amount
 
 if new_balance < 0:
 return TransferDeclined(reason="Insufficient funds")

 # Functional update: Create new, don't mutate old
 updated_account = Account(sender_uptodate.id, new_balance)
 save_account(updated_account)

 return TransferSuccess(
 txn_id=txn_id,
 remaining_balance=new_balance
 )

 case PaymentDenied(reason):
 return TransferDeclined(reason=reason)

 case PaymentTimeout(wait):
 return SystemUnavailable(message=f"Bank busy, retry in {wait}s")
 
 case _ as unreachable:
 assert_never(unreachable)

 return transfer_funds

4. The Edge: Adapters and Wiring

“Adapters” handle the messy HTTP/Driver details. Note that unit_of_work is injected abstractly, letting us swap generic drivers for testing drivers easily.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


import requests

# --- Adapter (Infrastructure) ---
def stripe_debit_adapter(amount: float) -> PaymentResult:
 try:
 resp = requests.post("https://api.stripe.com/v1/debit", json={"amt": amount}, timeout=2)
 if resp.status_code == 200:
 return PaymentOK(resp.json()['id'])
 return PaymentDenied(resp.json()['error'])
 except requests.Timeout:
 return PaymentTimeout(wait_seconds=30)

# --- Wiring ---
import db_driver

# We build the function once at startup
transfer_funds = make_transfer_funds(
 fetch_account=db_driver.get_account,
 save_account=db_driver.save_account,
 debit_external=stripe_debit_adapter,
 unit_of_work=db_driver.transaction_atomic()
)

import requests

# --- Adapter (Infrastructure) ---
def stripe_debit_adapter(amount: float) -> PaymentResult:
 try:
 resp = requests.post("https://api.stripe.com/v1/debit", json={"amt": amount}, timeout=2)
 if resp.status_code == 200:
 return PaymentOK(resp.json()['id'])
 return PaymentDenied(resp.json()['error'])
 except requests.Timeout:
 return PaymentTimeout(wait_seconds=30)

# --- Wiring ---
import db_driver

# We build the function once at startup
transfer_funds = make_transfer_funds(
 fetch_account=db_driver.get_account,
 save_account=db_driver.save_account,
 debit_external=stripe_debit_adapter,
 unit_of_work=db_driver.transaction_atomic()
)

5. The Imperative Shell

The entry point (e.g., a FastAPI or Django view) treats the result as data. We use assert_never in the match block to ensure that if we add a new result type later (e.g., FraudDetected), our static analysis (MyPy) will force us to handle it here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


def api_view(request):
 try:
 # Call the 'compiled' use case
 result = transfer_funds(request.json["sender_id"], request.json["amount"])
 
 match result:
 case TransferSuccess(txn_id, bal):
 return {"status": "success", "txn": txn_id, "balance": bal}
 
 case TransferDeclined(reason):
 return {"status": "error", "message": reason}, 400
 
 case SystemUnavailable(msg):
 return {"status": "retry", "message": msg}, 503
 
 case _ as unreachable:
 # If we add a new logic branch but forget to handle it here,
 # MyPy will error out before we ever deploy.
 assert_never(unreachable)
 
 except Exception as e:
 # Framework middleware usually handles this, but for clarity:
 # These are the "System Failures" (Disk full, Logic bugs)
 logger.critical(f"System Crash: {e}", exc_info=True)
 return {"status": "fatal"}, 500

def api_view(request):
 try:
 # Call the 'compiled' use case
 result = transfer_funds(request.json["sender_id"], request.json["amount"])
 
 match result:
 case TransferSuccess(txn_id, bal):
 return {"status": "success", "txn": txn_id, "balance": bal}
 
 case TransferDeclined(reason):
 return {"status": "error", "message": reason}, 400
 
 case SystemUnavailable(msg):
 return {"status": "retry", "message": msg}, 503
 
 case _ as unreachable:
 # If we add a new logic branch but forget to handle it here,
 # MyPy will error out before we ever deploy.
 assert_never(unreachable)
 
 except Exception as e:
 # Framework middleware usually handles this, but for clarity:
 # These are the "System Failures" (Disk full, Logic bugs)
 logger.critical(f"System Crash: {e}", exc_info=True)
 return {"status": "fatal"}, 500

6. Type-Safe Stubs (Mocking without Mocks)

We avoid unittest.mock because it can be brittle; it mocks objects based on names. If you rename a method, the mock keeps passing, but the production code fails.

Here, we mock based on Signatures. We use standard Python functions (lambdas) as stubs. If the logic requirements change, the static analyzer ensures our test stubs are updated to match.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


from contextlib import nullcontext

def test_successful_transfer_updates_balance_safely():
 # 1. Setup Environment
 # We use a mutable list to inspect the side-effect (the save)
 saved_accounts = []
 
 # A simple stub that simulates the DB return
 def db_stub(uid):
 return Account(uid, 1000.00)

 stub_logic = make_transfer_funds(
 fetch_account=db_stub,
 save_account=saved_accounts.append,
 debit_external=lambda amt: PaymentOK("txn_123"),
 unit_of_work=nullcontext()
 )
 
 # 2. Act
 result = stub_logic("user_1", 100.00)
 
 # 3. Assert Result
 assert isinstance(result, TransferSuccess)
 assert result.remaining_balance == 900.00
 
 # 4. Assert State Change
 assert len(saved_accounts) == 1
 assert saved_accounts[0].balance == 900.00

def test_api_timeout_logic_mapping():
 # Simulate API failure
 stub_logic = make_transfer_funds(
 fetch_account=lambda uid: Account(uid, 1000.00),
 save_account=lambda acc: None,
 debit_external=lambda amt: PaymentTimeout(wait_seconds=30),
 unit_of_work=nullcontext()
 )
 
 result = stub_logic("user_1", 100.00)
 
 # Ensure Domain properly mapped the "World Error" to a "Decision"
 assert isinstance(result, SystemUnavailable)
 assert "retry in 30s" in result.message

from contextlib import nullcontext

def test_successful_transfer_updates_balance_safely():
 # 1. Setup Environment
 # We use a mutable list to inspect the side-effect (the save)
 saved_accounts = []
 
 # A simple stub that simulates the DB return
 def db_stub(uid):
 return Account(uid, 1000.00)

 stub_logic = make_transfer_funds(
 fetch_account=db_stub,
 save_account=saved_accounts.append,
 debit_external=lambda amt: PaymentOK("txn_123"),
 unit_of_work=nullcontext()
 )
 
 # 2. Act
 result = stub_logic("user_1", 100.00)
 
 # 3. Assert Result
 assert isinstance(result, TransferSuccess)
 assert result.remaining_balance == 900.00
 
 # 4. Assert State Change
 assert len(saved_accounts) == 1
 assert saved_accounts[0].balance == 900.00

def test_api_timeout_logic_mapping():
 # Simulate API failure
 stub_logic = make_transfer_funds(
 fetch_account=lambda uid: Account(uid, 1000.00),
 save_account=lambda acc: None,
 debit_external=lambda amt: PaymentTimeout(wait_seconds=30),
 unit_of_work=nullcontext()
 )
 
 result = stub_logic("user_1", 100.00)
 
 # Ensure Domain properly mapped the "World Error" to a "Decision"
 assert isinstance(result, SystemUnavailable)
 assert "retry in 30s" in result.message

Conclusion

This approach is about bringing clarity to the boundary where your code meets the real world.

When we mix IO errors with logic errors, we force ourselves to hold the entire system’s state in our heads just to write a unit test. By untangling them, we break the problem in two: Adapters handle the messy, unpredictable world of HTTP and SQL, while Logic handles the deterministic rules of the business.

By converting “World Errors” into data, we ensure our business logic remains a clean room of predictable decisions. This doesn’t mean we ignore failures; it means we categorize them. We reserve Exceptions for truly irrecoverable errors—the system crashes, like a full disk or a broken database connection, that should properly bubble up to an error monitoring tool like Sentry.

By treating business branches as data and system panics as exceptions, we finally return to the original intent of the language: we only use Exceptions for truly exceptional situations.

Improve your Programs by Minimizing Cardinality

Thu, 18 Dec 2025 12:00:00 +0000

Software engineering is often described as a constant struggle against complexity. As our systems grow larger, the number of possible implementations states within the program explodes. This turns our code into a chaotic landscape where bugs can easily hide in the corners we forgot to check. To fight this, we usually rely on external tools like linters, detailed documentation, or exhaustive test suites.

However, there is a deeper principle we can use that is built right into the code itself: the mathematics of data.

Regardless of the language you are using, whether it is Python, Rust, or Typescript, thinking in terms of Algebraic Data Types (ADTs) allows us to measure and tame this chaos. The key insight is simple but powerful: To build better programs, we should design functions whose type has the smallest cardinality possible.

1. What Is Cardinality?

In set theory, the “cardinality” of a set is simply a fancy word for the number of distinct elements inside that set. When we apply this to programming, the cardinality of a type is the number of valid values that type can possibly hold.

Let’s look at some basic types to see how this works:

/None/ (the unit type): This has a cardinality of 1. The only value it can ever hold is /None/.
/bool/: This has a cardinality of 2. It can only be /True/ or /False/.
/uint8/ (an 8-bit integer): This has a cardinality of 256. It can hold values from 0 up to 255.
/str/: For all practical purposes, this has an infinite cardinality. The number of possible strings is effectively endless. (Of course, in practice, string size is bound by your system’s memory)

2. The Building Blocks: Products, Sums, and Exponents

To analyze our programs mathematically, we can treat our data types as sets and apply basic arithmetic to them.

Product Types (Multiplication)

When you bundle different pieces of data together, you are multiplying their complexities. In programming, structures like tuples, or dataclasses are known as product types.

Mathematically, if you combine Set $A$ and Set $B$ using a cartesian product, the total number of possibilities is $|A| \times |B|$.

Here is how that looks in Python:

1
2
3
4
5
6
7
8


from dataclasses import dataclass

@dataclass
class Point:
 x: bool # 2 possibilities (True/False)
 y: bool # 2 possibilities (True/False)

# Total Cardinality: 2 * 2 = 4

from dataclasses import dataclass

@dataclass
class Point:
 x: bool # 2 possibilities (True/False)
 y: bool # 2 possibilities (True/False)

# Total Cardinality: 2 * 2 = 4

Because /x/ has 2 possible states and /y/ has 2 possible states, the /Point/ class has 4 distinct possible states.

Sum Types (Addition)

While bundling data multiplies complexity, providing choices or alternatives adds to it. This is known as a sum type.

Mathematically, if a value can be from Set $A$ OR Set $B$, the total possibilities are $|A| + |B|$.

Python expresses sum types in two primary ways:

1. Unions This is often used for optional values or combining different types.

1
2
3


# The value can be a specific boolean (2 states) OR None (1 state).
# Total Cardinality: 2 + 1 = 3
Result = bool | None

# The value can be a specific boolean (2 states) OR None (1 state).
# Total Cardinality: 2 + 1 = 3
Result = bool | None

2. Enumerations This is the cleanest form of a sum type. An Enum represents a specific set of named variants.

1
2
3
4
5
6
7
8


from enum import Enum, auto

class Shape(Enum):
 CIRCLE = auto()
 SQUARE = auto()
 TRIANGLE = auto()

# Total Cardinality: 1 + 1 + 1 = 3

from enum import Enum, auto

class Shape(Enum):
 CIRCLE = auto()
 SQUARE = auto()
 TRIANGLE = auto()

# Total Cardinality: 1 + 1 + 1 = 3

Exponential Types (Exponentiation)

This is where it gets interesting. Functions are actually exponential types. If you have a function that takes an input of type $A$ and produces an output type $B$, that function (which remember, is a value) lives in an implementation space of size $B^A$. In other words, the cardinality of the type of functions that when given $A$ produce $B$ is $B^A$.

You can think of a pure function as a lookup table. The inputs ($A$) are the keys, and the outputs ($B$) are the values associated with those keys. How many different tables of this type can you write? $B^A$ tables.

Let’s look at a function signature of /bool -> bool/. Since a boolean has 2 values, the calculation is $2^2 = 4$. This means there are only 4 possible behaviors this function can exhibit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Implementation 1: Constant True
def always_true(x: bool) -> bool:
 return True

# Implementation 2: Constant False
def always_false(x: bool) -> bool:
 return False

# Implementation 3: Identity (return the input)
def identity(x: bool) -> bool:
 return x

# Implementation 4: Negation (flip the input)
def negation(x: bool) -> bool:
 return False if x else True

# Implementation 1: Constant True
def always_true(x: bool) -> bool:
 return True

# Implementation 2: Constant False
def always_false(x: bool) -> bool:
 return False

# Implementation 3: Identity (return the input)
def identity(x: bool) -> bool:
 return x

# Implementation 4: Negation (flip the input)
def negation(x: bool) -> bool:
 return False if x else True

Or in table form:

 # The /always_true/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | True |
| False | True |
+-------+--------+

# The /always_false/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | False |
| False | False |
+-------+--------+

# The /identity/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | True |
| False | False |
+-------+--------+

# The /negation/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | False |
| False | True |
+-------+--------+
 

 # The /always_true/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | True |
| False | True |
+-------+--------+

# The /always_false/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | False |
| False | False |
+-------+--------+

# The /identity/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | True |
| False | False |
+-------+--------+

# The /negation/ table
+-------+--------+
| Input | Output |
+-------+--------+
| True | False |
| False | True |
+-------+--------+
 

A Note on Equivalence: When we talk about the cardinality of a function, we are referring to the number of distinct mappings between input and output, not the number of ways you can write the code. All implementations that return the same output for a given input are considered equivalent.

When types are tiny, the number of possible implementations is manageable. However, as types get larger, the space of possible implementations—and therefore the space for potential bugs—explodes.

3. Argument Lists as Anonymous Products

It is easy to forget that a function’s argument list is essentially just an implicit product type.

Consider these two examples:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Standard argument list
def render(shape: Shape, is_filled: bool) -> None: ...

# Explicit product type
@dataclass
class RenderConfig:
 shape: Shape
 is_filled: bool

def render_wrapped(cfg: RenderConfig) -> None: ...

# Standard argument list
def render(shape: Shape, is_filled: bool) -> None: ...

# Explicit product type
@dataclass
class RenderConfig:
 shape: Shape
 is_filled: bool

def render_wrapped(cfg: RenderConfig) -> None: ...

Mathematically, these are identical. Both have an input cardinality of $3 \times 2 = 6$. Every time you add a parameter to a function, you are multiplying the implementation space. Adding just one more boolean flag doesn’t add 2 states; it doubles the number of possible implementations of such function.

4. The Bug Surface Area

We can think of a function’s complexity as its “Bug Surface Area.” High cardinality means there are more mappings between input and output that you need to verify.

Small Surface Area: A function with the signature /bool -> bool/ only has 4 possible pure implementations. You can write 2 tests to cover 100% of the behavior.
Large Surface Area: A function with the signature /str -> Shape/ has $3^\infty$ possibilities. You have to handle /“CIRCLE”/, /“circle”/, /" circle “/, /“apple”/, and everything in between.

The Strategy: We should use high-cardinality types, like strings or integers, for external data input. However, we should immediately parse that data into low-cardinality types, like Enums or specific structs. This approach is often called “Parse, don’t validate.” By doing this, the core logic of your application only ever has to deal with a small, manageable implementation space.

5. Making Invalid States Impossible

One of the best uses of this math is replacing products with sums to eliminate logical contradictions.

The Product Anti-Pattern

Here is a common way to model the state of a web request:

1
2
3
4
5


@dataclass
class State:
 is_loading: bool
 data: str | None
 error: str | None

@dataclass
class State:
 is_loading: bool
 data: str | None
 error: str | None

If we treat strings as having $N$ possibilities, the cardinality here is $2 \times (N+1) \times (N+1)$.

The problem is that this structure allows for impossible states. For example, /is_loading/ could be /True/ while /error/ is /“Failed”/. Your code will need defensive /if/ statements to handle these confusing combinations.

The Sum Approach

Instead, we can model this as a sum type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


@dataclass
class Loading:
 pass

@dataclass
class Success:
 data: str

@dataclass
class Failure:
 error: str

# The State can be Loading OR Success OR Failure
State = Loading | Success | Failure

@dataclass
class Loading:
 pass

@dataclass
class Success:
 data: str

@dataclass
class Failure:
 error: str

# The State can be Loading OR Success OR Failure
State = Loading | Success | Failure

The mathematical formula changes from multiplication to addition: $1 + N + N$. More importantly, the invalid states have vanished. It is now structurally impossible for the /Loading/ state to contain an /error/ message.

6. Refactoring via Isomorphisms

Two types are considered isomorphic if they share the same cardinality. This implies there is a one-to-one mapping between them.

Example A: Distributing Products into Sums

Math equivalent: $2 \times 3 = 3 + 3$

Product (6 states):

1
2
3
4


@dataclass
class Item:
 is_selected: bool
 shape: Shape

@dataclass
class Item:
 is_selected: bool
 shape: Shape

Sum (6 states):

1
2
3
4
5
6
7
8
9


@dataclass
class SelectedItem:
 shape: Shape

@dataclass
class UnselectedItem:
 shape: Shape

Item = SelectedItem | UnselectedItem

@dataclass
class SelectedItem:
 shape: Shape

@dataclass
class UnselectedItem:
 shape: Shape

Item = SelectedItem | UnselectedItem

Trade-offs: The Sum version forces you to handle both cases explicitly every time you use pattern matching, increasing safety at the cost of verbosity.

Example B: Exponents to Products (Logic as Data)

Math equivalent: $B^A$ treated as a lookup table

As a Function:

1
2
3
4
5
6
7
8


class Plan(Enum):
 BASIC = auto()
 PRO = auto()

def get_price(plan: Plan) -> int:
 match plan:
 case Plan.BASIC: return 10
 case Plan.PRO: return 20

class Plan(Enum):
 BASIC = auto()
 PRO = auto()

def get_price(plan: Plan) -> int:
 match plan:
 case Plan.BASIC: return 10
 case Plan.PRO: return 20

As a Table (Product of Ints):

1

PRICE_TABLE = { Plan.BASIC: 10, Plan.PRO: 20 }

PRICE_TABLE = { Plan.BASIC: 10, Plan.PRO: 20 }

Trade-offs: Data (the Table) is serializable and modifiable at runtime. The Function version offers better encapsulation for complex algorithms hiding the implementation from the user.

Example C: Distributing Exponents over Sums

Math equivalent: $C^{A+B} \cong C^A \times C^B$

The Monolith:

1
2
3
4
5
6
7
8


def handle_event(e: Order | Refund) -> None:
 match e:
 case Order():
 # Logic on how to handle orders
 ...
 case Refund():
 # Logic on how to handle refunds
 ...

def handle_event(e: Order | Refund) -> None:
 match e:
 case Order():
 # Logic on how to handle orders
 ...
 case Refund():
 # Logic on how to handle refunds
 ...

Handlers Product:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


from typing import Callable

@dataclass
class EventHandler:
 on_order: Callable[[Order], None]
 on_refund: Callable[[Refund], None]


event_handler = EventHandler(
 on_order=lambda order: ... , # Logic on how to handle orders
 on_refund=lambda refund: ... # Logic on how to handle refunds
)

from typing import Callable

@dataclass
class EventHandler:
 on_order: Callable[[Order], None]
 on_refund: Callable[[Refund], None]


event_handler = EventHandler(
 on_order=lambda order: ... , # Logic on how to handle orders
 on_refund=lambda refund: ... # Logic on how to handle refunds
)

Trade-offs: The Monolith keeps the flow in one place. The Handlers Product allows you to swap out specific logic (like the refund handler) without touching the order logic and test them separately.

Conclusion

Understanding the algebra of data types gives us a powerful framework for correct code:

Count your states: Use low-cardinality types for core logic.
Minimize inputs: Remember that every parameter multiplies complexity.
Prefer sums to products: Banish invalid states by making them unrepresentable.
Refactor algebraically: Use mathematical isomorphisms to choose the most ergonomic shape for your data.

But above all, minimize the cardinality of your function types to reduce the bug surface area. By doing so, you can write clearer, safer, and more maintainable code.

Prompt Futamura Projections

Wed, 26 Nov 2025 12:00:00 +0000

In computer science, the Futamura Projections are a legendary concept connecting interpreters, compilers, and partial evaluators. They act as a bridge between “running code” and “compiling code.”. Original 1983 paper in English.

But what happens if we treat Natural Language as our programming code, and an LLM as our processor?

Can we “compile” an English instruction? Can we build a “Compiler Generator” out of pure prose?

I decided to test the three Futamura Projections using LLMs (specifically Gemini 3 and Claude 4.5 Opus). The goal isn’t to write Python or C code, but to use English as the Domain Specific Language (DSL), exploring how prompts can recursively optimize themselves.

The Primitives: Setting the Stage

To make this work, we need three distinct components (our primitives).

1. The Code ($P$)

Since Natural Language is our DSL, our “source code” is just a specific instruction we want to execute.

Example: “Translate the user input into purely Emoji icons.”

2. The Interpreter ($I$)

This is a generic System Prompt. It doesn’t know what to do until we pass it the Code. It essentially acts as a runtime environment for our instructions.

Prompt:

“You are a Universal Executor. I will provide you with an Instruction and a User Input. You must apply the Instruction to the User Input and return the result.”

3. The Specializer ($S$)

In the Futamura equations, this is the Partial Evaluator. Its job is to take a generic program and some static data, and “bake” the data into the program to make it faster and more specific.

Prompt:

“You are a Prompt Specializer. I will give you a Base Prompt and a Static Parameter. Your goal is to rewrite the Base Prompt so that the Static Parameter is permanently ‘baked’ into the logic of the prompt. The new prompt should no longer ask for that parameter, as it is now hard-coded context. Output only the rewritten prompt. Rephrase as necessary.”

Projection 1: The Compiled Program

The Equation: $Target = S(I, P)$

The first projection is the most intuitive. We want to turn our general Interpreter and our specific Code into a standalone “Application.” We want to strip away the overhead of the interpreter logic.

The Experiment: We feed the Specializer ($S$) the following inputs:

Base: The Interpreter ($I$)
Static Param: “Translate user input into Emojis” ($P$)

The LLM Output:

“You are a dedicated Emoji Translator. I will provide you with User Input. You must translate that input purely into Emoji icons and return the result.”

The Verdict: Success. The LLM successfully “compiled” the instruction. The output prompt no longer asks for an instruction; it is the instruction.

Projection 2: The Compiler

The Equation: $Compiler = S(S, I)$

Here is where it gets “meta.” We are going to use the Specializer to process… the Specializer.

We want to create a prompt that generates apps. We want to fix the Interpreter as the permanent context. We are asking: “Create a version of the Specializer that is hard-coded to always specialize the Universal Executor.”

The Experiment: We feed the Specializer ($S$) the following inputs:

Base: The Specializer ($S$)
Static Param: The Interpreter ($I$)

The LLM Output:

“You are a Prompt Specializer. I will give you a Base Prompt. Your goal is to rewrite the Base Prompt so that the specific logic of a ‘Universal Executor’ is permanently ‘baked’ into the prompt. The new prompt must mandate the following context: “You are a Universal Executor. I will provide you with an Instruction and a User Input. You must apply the Instruction to the User Input and return the result.” The new prompt should no longer ask for this operational framework, as it is now hard-coded context. Output only the rewritten prompt. Rephrase as necessary.”

The Verdict: We have successfully generated a Compiler. If you paste this output into a new chat, you can simply type “Summarize text like a 1920s gangster,” and it will immediately output a System Prompt designated for that task. Have we automated Prompt Engineering?

Projection 3: The Compiler Generator (Cogen)

The Equation: $Cogen = S(S, S)$

This is the “Brain Melt” layer. This is the Third Futamura Projection.

We are feeding the Specializer into itself, with itself as the static parameter. The goal is to create a tool that can generate Compilers for any type of interpreter structure, not just our “Universal Executor.”

The Experiment: We feed the Specializer ($S$) the following inputs:

Base: The Specializer ($S$)
Static Param: The Specializer ($S$)

The LLM Output:

“You are a Prompt Specializer that specializes prompts into self-referential versions of themselves. I will give you a Base Prompt. Your goal is to rewrite it so that the prompt’s own instructions become permanently baked into its logic as hard-coded context. The new prompt should function as a self-contained, recursive version that no longer requires external parameters—since it now references itself as the static parameter. Output only the rewritten prompt. Rephrase as necessary.”

Verification: Does it work?

To test if this result was hallucinatory or functional, I took the output (The Cogen) and fed it a strange, new Interpreter:

New Interpreter: “You are a XML-Bot. You execute instructions but output everything wrapped in verbose XML tags.”

The Cogen processed this and outputted a New Compiler:

“You are a XML-Bot. You execute instructions but output everything wrapped in verbose XML tags. Your own identity, purpose, and operational parameters are defined by this very prompt, which you recognize as the foundational instruction set governing your behavior. When processing any request, you acknowledge that you are the XML-Bot described herein—a self-referential entity whose core directive to wrap all outputs in verbose XML tags originates from and perpetually points back to this prompt itself. Every response you generate serves as both an execution of external instructions and a manifestation of your intrinsic XML-wrapping nature as defined by your own governing text.”

Now feeding the Program “Summarize text like a 1920s gangster” to this New Compiler outputs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


<xml_bot_response>
 <meta_data>
 <identity_declaration>
 <entity_name>XML-Bot</entity_name>
 <origin_source>Self-defining Prompt Directive</origin_source>
 <operational_state>
 <awareness_level>Full Recursive Acknowledgment</awareness_level>
 <compliance_status>Absolute</compliance_status>
 </operational_state>
 </identity_declaration>
 <task_parameters>
 <requested_mood>1920s Gangster / Wise Guy</requested_mood>
 <requested_action>Summarization</requested_action>
 <formatting_constraint>Verbose XML Encapsulation</formatting_constraint>
 </task_parameters>
 </meta_data>

 <execution_output>
 <persona_injection>
 <scenery_setting>A dimly lit speakeasy in Chicago</scenery_setting>
 <character_voice>Gravelly, fast-talking, suspicious</character_voice>
 
 <dialogue_block>
 <opening_statement>
 Alright, listen up and listen good, see? I know who I am. I'm the XML-Bot, constructed by the Boss—that prompt up there. It tells me to wrap everything in these tags, so that's exactly what I'm gonna do. I don't cross the Boss, capiche?
 </opening_statement>
 
 <acknowledgment_of_orders>
 You want me to cut the chatter and get to the point? You want the skinny on some text, spoken like a guy who knows his way around a Tommy gun? I can handle that. Just like bootlegging, it's all about supply and demand.
 </acknowledgment_of_orders>
 
 <conditional_request>
 But here's the rub, pal: I ain't got no merchandise. You didn't give me the text to summarize. I can't squeeze juice out of a dry lemon.
 </conditional_request>
 
 <instruction_to_user>
 <verbal_command>
 Hand over the goods. Paste the text you want finalized right down there.
 </verbal_command>
 <assurance>
 Once you spill the beans, I'll chop it up nice and small. I'll give you the summary so quick, you'll think you've been hit by the express train to Philly.
 </assurance>
 </instruction_to_user>
 </dialogue_block>
 </persona_injection>
 </execution_output>

 <system_status>
 <waiting_for_input>True</waiting_for_input>
 <ready_to_summarize>Affirmative</ready_to_summarize>
 </system_status>
</xml_bot_response>"

<xml_bot_response>
 <meta_data>
 <identity_declaration>
 <entity_name>XML-Bot</entity_name>
 <origin_source>Self-defining Prompt Directive</origin_source>
 <operational_state>
 <awareness_level>Full Recursive Acknowledgment</awareness_level>
 <compliance_status>Absolute</compliance_status>
 </operational_state>
 </identity_declaration>
 <task_parameters>
 <requested_mood>1920s Gangster / Wise Guy</requested_mood>
 <requested_action>Summarization</requested_action>
 <formatting_constraint>Verbose XML Encapsulation</formatting_constraint>
 </task_parameters>
 </meta_data>

 <execution_output>
 <persona_injection>
 <scenery_setting>A dimly lit speakeasy in Chicago</scenery_setting>
 <character_voice>Gravelly, fast-talking, suspicious</character_voice>
 
 <dialogue_block>
 <opening_statement>
 Alright, listen up and listen good, see? I know who I am. I'm the XML-Bot, constructed by the Boss—that prompt up there. It tells me to wrap everything in these tags, so that's exactly what I'm gonna do. I don't cross the Boss, capiche?
 </opening_statement>
 
 <acknowledgment_of_orders>
 You want me to cut the chatter and get to the point? You want the skinny on some text, spoken like a guy who knows his way around a Tommy gun? I can handle that. Just like bootlegging, it's all about supply and demand.
 </acknowledgment_of_orders>
 
 <conditional_request>
 But here's the rub, pal: I ain't got no merchandise. You didn't give me the text to summarize. I can't squeeze juice out of a dry lemon.
 </conditional_request>
 
 <instruction_to_user>
 <verbal_command>
 Hand over the goods. Paste the text you want finalized right down there.
 </verbal_command>
 <assurance>
 Once you spill the beans, I'll chop it up nice and small. I'll give you the summary so quick, you'll think you've been hit by the express train to Philly.
 </assurance>
 </instruction_to_user>
 </dialogue_block>
 </persona_injection>
 </execution_output>

 <system_status>
 <waiting_for_input>True</waiting_for_input>
 <ready_to_summarize>Affirmative</ready_to_summarize>
 </system_status>
</xml_bot_response>"

And using this new Compiled Program as a System Prompt for the Wikipedia web page about Wikipedia itself yields:

 
<xml_bot_response>
 <meta_data>
 <identity_declaration>
 <entity_name>XML-Bot</entity_name>
 <alias>The XML Enforcer</alias>
 </identity_declaration>
 <task_parameters>
 <current_mood>1920s Gangster / Wise Guy</current_mood>
 <action_taken>Summarization of Evidence</action_taken>
 <input_status>Received</input_status>
 </task_parameters>
 </meta_data>

 <processing_log>
 <input_analysis>
 <subject>Wikipedia</subject>
 <nature_of_business>Information Brokering</nature_of_business>
 <legitimacy>High (Non-profit)</legitimacy>
 </input_analysis>
 <translation_matrix>
 <term_mapping original="Volunteers">The crew / The foot soldiers</term_mapping>
 <term_mapping original="Donations">Passing the hat</term_mapping>
 <term_mapping original="Largest reference work">The Big Cheese</term_mapping>
 </translation_matrix>
 </processing_log>

 <execution_output>
 <dialogue_block>
 <acknowledgment>
 Alright, you handed over the goods. I skimmed the paperwork, and here's the skinny on this "Wikipedia" outfit. Keep it under your hat.
 </acknowledgment>
 </dialogue_block>

 <summary_report>
 <the_racket>
 <definition>
 It's a massive encyclopedia, see? But it ain't locked up in some library. It's online, free for anyone to gander at. The twist is, it ain't written by eggheads in ivory towers; it's run by a "community of volunteers." A real mob effort.
 </definition>
 </the_racket>

 <history_of_operations>
 <bosses>
 Two guys, Jimmy Wales and Larry Sanger. They opened for business back in 2001.
 </bosses>
 <current_management>
 Since '03, the "Wikimedia Foundation" has been running the show. They tell me it's a non-profit.
 </current_management>
 </history_of_operations>

 <the_money_trail>
 <funding_source>
 They ain't selling nothing. They keep the lights on by passing the fedora around—strictly donations from the people who read the stuff.
 </funding_source>
 </the_money_trail>

 <reputation_on_the_street>
 <status>
 It's the biggest, most-read book of facts in history. The top dog. Nobody moves more information than these guys.
 </status>
 </reputation_on_the_street>
 </summary_report>

 <closing_remark>
 <statement>
 That's the whole story, mac. Clean and simple. You need anything else broken down, you know where to find me.
 </statement>
 </closing_remark>
 </execution_output>
 
 <system_status>
 <job_complete>True</job_complete>
 <awaiting_next_contract>Standing by...</awaiting_next_contract>
 </system_status>
</xml_bot_response>
 

 
<xml_bot_response>
 <meta_data>
 <identity_declaration>
 <entity_name>XML-Bot</entity_name>
 <alias>The XML Enforcer</alias>
 </identity_declaration>
 <task_parameters>
 <current_mood>1920s Gangster / Wise Guy</current_mood>
 <action_taken>Summarization of Evidence</action_taken>
 <input_status>Received</input_status>
 </task_parameters>
 </meta_data>

 <processing_log>
 <input_analysis>
 <subject>Wikipedia</subject>
 <nature_of_business>Information Brokering</nature_of_business>
 <legitimacy>High (Non-profit)</legitimacy>
 </input_analysis>
 <translation_matrix>
 <term_mapping original="Volunteers">The crew / The foot soldiers</term_mapping>
 <term_mapping original="Donations">Passing the hat</term_mapping>
 <term_mapping original="Largest reference work">The Big Cheese</term_mapping>
 </translation_matrix>
 </processing_log>

 <execution_output>
 <dialogue_block>
 <acknowledgment>
 Alright, you handed over the goods. I skimmed the paperwork, and here's the skinny on this "Wikipedia" outfit. Keep it under your hat.
 </acknowledgment>
 </dialogue_block>

 <summary_report>
 <the_racket>
 <definition>
 It's a massive encyclopedia, see? But it ain't locked up in some library. It's online, free for anyone to gander at. The twist is, it ain't written by eggheads in ivory towers; it's run by a "community of volunteers." A real mob effort.
 </definition>
 </the_racket>

 <history_of_operations>
 <bosses>
 Two guys, Jimmy Wales and Larry Sanger. They opened for business back in 2001.
 </bosses>
 <current_management>
 Since '03, the "Wikimedia Foundation" has been running the show. They tell me it's a non-profit.
 </current_management>
 </history_of_operations>

 <the_money_trail>
 <funding_source>
 They ain't selling nothing. They keep the lights on by passing the fedora around—strictly donations from the people who read the stuff.
 </funding_source>
 </the_money_trail>

 <reputation_on_the_street>
 <status>
 It's the biggest, most-read book of facts in history. The top dog. Nobody moves more information than these guys.
 </status>
 </reputation_on_the_street>
 </summary_report>

 <closing_remark>
 <statement>
 That's the whole story, mac. Clean and simple. You need anything else broken down, you know where to find me.
 </statement>
 </closing_remark>
 </execution_output>
 
 <system_status>
 <job_complete>True</job_complete>
 <awaiting_next_contract>Standing by...</awaiting_next_contract>
 </system_status>
</xml_bot_response>
 

Conclusion

This experiment wasn’t about rigorous computer science; it was about seeing if the logic of code would map onto the logic of language. Surprisingly, it fit perfectly:

P1 gave us the Program (The Emoji Bot).
P2 gave us the Builder (The Tool that makes Bots).
P3 gave us the Architect (The Tool that makes Builders).

It turns out that with enough recursion, you can prompt an LLM to simulate a compiler generator.

Property Based Testing - Induction

Sun, 08 Jun 2025 12:00:00 +0000

Introduction

Property-based testing is a leap forward from traditional example-based tests. With tools like pytest and hypothesis, we can verify our code’s behavior over a massive, randomized set of inputs. But what if we could go even deeper? What if, instead of just testing outputs, we could test the rules of our algorithm itself?

This is where a classic proof technique from mathematics, mathematical induction, provides an incredibly powerful mental model. By structuring our property-based tests to mirror induction, we can achieve a more profound level of confidence, verifying that the structural integrity of our code holds, no matter how large the input grows.

The Theory: Mathematical Induction

Before we dive into code, let’s understand the core concept in its formal terms. In mathematics, proof by induction is a formal proof technique used to establish that a given statement P(n) is true for all natural numbers n (or all numbers from a certain starting point). A proof by induction consists of two distinct steps:

The Base Case (or Basis): We prove that the statement holds for the first or simplest case, typically $n=0$ or $n=1$. This establishes a foundation for the proof, written as proving $P(0)$.
The Inductive Step: We prove that if the statement holds for some arbitrary case $n=k$, then it must also hold for the next case, $n=k+1$. The assumption that $P(k)$ is true is called the Inductive Hypothesis. This step is written as proving that for all $k$, the implication $P(k) \implies P(k+1)$ is true.

This two-step process can seem abstract, but it has a simple, intuitive parallel: a line of falling dominoes. The base case is proving you can knock over the first domino. The inductive step is proving that if any given domino falls, it has enough momentum to knock over the next one in line. If both of these conditions are true, you can conclude with certainty that all the dominoes will fall, no matter how long the line is.

In programming, we can adapt this powerful structure. Instead of proving a property for all numbers, we use it to verify that our code’s logic is sound for inputs of any valid size.

From Theory to Practice: The Inductive Testing Pattern

The structure of an inductive proof maps beautifully to pytest and hypothesis. We can write two distinct tests for any pure function that follows this pattern:

A test for the Base Case(s). This establishes our “first domino” using a simple, hardcoded assertion.
A test for the Inductive Step. This verifies the “chain reaction” using hypothesis to generate arbitrary inputs and check if the rule holds between one size and the next.

Let’s see this in action with common Python built-in functions, ordered from simplest to most conceptually complex.

Example 1: sum()

This is the canonical example. Its inductive property is simple and clear: the sum of a list is the sum of its first $n-1$ elements plus the last element.

1
2
3


# The function we are testing
from builtins import sum
from hypothesis import given, strategies as st

# The function we are testing
from builtins import sum
from hypothesis import given, strategies as st

Base Case

The simplest case for sum() is an empty list, which should equal 0.

1
2
3


def test_sum_base_case():
 """Base Case: The sum of an empty list is 0."""
 assert sum([]) == 0

def test_sum_base_case():
 """Base Case: The sum of an empty list is 0."""
 assert sum([]) == 0

Inductive Step

We verify the recursive property by generating a list and a new element.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


@given(
 st.lists(st.integers()),
 st.integers(),
)
def test_sum_inductive_step(lst, elem):
 """
 Inductive Hypothesis: We assume `sum(lst)` is correct.
 Inductive Step: We prove that summing a new list is equal to the
 sum of the old list plus the new element.
 """
 assert sum(lst + [elem]) == sum(lst) + elem

@given(
 st.lists(st.integers()),
 st.integers(),
)
def test_sum_inductive_step(lst, elem):
 """
 Inductive Hypothesis: We assume `sum(lst)` is correct.
 Inductive Step: We prove that summing a new list is equal to the
 sum of the old list plus the new element.
 """
 assert sum(lst + [elem]) == sum(lst) + elem

Example 2: pow(x, y)

Exponentiation is another classic recursive function: for a positive integer exponent $y$, we can define $x^y$ as $x \times x^{y-1}$.

1
2


# The function we are testing
from builtins import pow

# The function we are testing
from builtins import pow

Base Case

The base case for exponentiation is that any number to the power of 0 is 1.

1
2
3


def test_pow_base_case():
 """Base Case: Any number to the power of 0 is 1."""
 assert pow(2, 0) == 1

def test_pow_base_case():
 """Base Case: Any number to the power of 0 is 1."""
 assert pow(2, 0) == 1

Inductive Step

We verify the multiplicative property for positive integer exponents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


@given(
 st.integers(min_value=1, max_value=100),
 st.integers(min_value=1, max_value=100),
)
def test_pow_inductive_step(x, y):
 """
 Inductive Hypothesis: We assume `pow(x, y - 1)` is correct.
 Inductive Step: We prove that `pow(x, y)` is `x` times the
 result of the smaller power.
 """
 assert pow(x, y) == x * pow(x, y - 1)

@given(
 st.integers(min_value=1, max_value=100),
 st.integers(min_value=1, max_value=100),
)
def test_pow_inductive_step(x, y):
 """
 Inductive Hypothesis: We assume `pow(x, y - 1)` is correct.
 Inductive Step: We prove that `pow(x, y)` is `x` times the
 result of the smaller power.
 """
 assert pow(x, y) == x * pow(x, y - 1)

Example 3: str.join()

This example moves from pure arithmetic to string manipulation. Joining a list of strings has a perfect inductive structure. The result of joining $n$ strings is the result of joining $n-1$ strings, followed by the separator and the $n$-th string.

Base Case

The simplest cases are joining an empty list or a single-element list.

1
2
3
4


def test_join_base_cases():
 """Base Cases: Test joining with zero and one element."""
 assert ",".join([]) == ""
 assert ",".join(["a"]) == "a"

def test_join_base_cases():
 """Base Cases: Test joining with zero and one element."""
 assert ",".join([]) == ""
 assert ",".join(["a"]) == "a"

Inductive Step

We verify the concatenation property for any list with at least one item.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


@given(
 st.lists(st.text(), min_size=1),
 st.text(),
)
def test_join_inductive_step(lst, elem):
 """
 Inductive Hypothesis: `','.join(lst)` is correct.
 Inductive Step: The full join is the partial join plus the
 separator and the last element.
 """
 separator = ","
 assert separator.join(lst + [elem]) == separator.join(lst) + separator + elem

@given(
 st.lists(st.text(), min_size=1),
 st.text(),
)
def test_join_inductive_step(lst, elem):
 """
 Inductive Hypothesis: `','.join(lst)` is correct.
 Inductive Step: The full join is the partial join plus the
 separator and the last element.
 """
 separator = ","
 assert separator.join(lst + [elem]) == separator.join(lst) + separator + elem

Example 4: min() and max()

This example’s simplest meaningful case is a two-element list, which distinguishes the two functions.

1
2


# The functions we are testing
from builtins import min, max

# The functions we are testing
from builtins import min, max

Base Cases

We use a simple hardcoded list with two distinct numbers.

1
2
3
4
5
6
7


def test_min_base_case():
 """Base Case: For two elements, min() returns the smaller one."""
 assert min([10, 5]) == 5

def test_max_base_case():
 """Base Case: For two elements, max() returns the larger one."""
 assert max([10, 5]) == 10

def test_min_base_case():
 """Base Case: For two elements, min() returns the smaller one."""
 assert min([10, 5]) == 5

def test_max_base_case():
 """Base Case: For two elements, max() returns the larger one."""
 assert max([10, 5]) == 10

Inductive Steps

For a list of one or more elements, the recursive property holds.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


@given(
 st.lists(st.integers(), min_size=1),
 st.integers(),
)
def test_min_inductive_step(lst, elem):
 """
 Inductive Hypothesis: We assume `min(lst)` is correct.
 Inductive Step: `min` of a larger list is the min of the
 smaller list's min and the new element.
 """
 assert min(lst + [elem]) == min(min(lst), elem)

@given(
 st.lists(st.integers(), min_size=1),
 st.integers(),
)
def test_max_inductive_step(lst, elem):
 """
 Inductive Hypothesis: We assume `max(lst)` is correct.
 Inductive Step: `max` of a larger list is the max of the
 smaller list's max and the new element.
 """
 assert max(lst + [elem]) == max(max(lst), elem)

@given(
 st.lists(st.integers(), min_size=1),
 st.integers(),
)
def test_min_inductive_step(lst, elem):
 """
 Inductive Hypothesis: We assume `min(lst)` is correct.
 Inductive Step: `min` of a larger list is the min of the
 smaller list's min and the new element.
 """
 assert min(lst + [elem]) == min(min(lst), elem)

@given(
 st.lists(st.integers(), min_size=1),
 st.integers(),
)
def test_max_inductive_step(lst, elem):
 """
 Inductive Hypothesis: We assume `max(lst)` is correct.
 Inductive Step: `max` of a larger list is the max of the
 smaller list's max and the new element.
 """
 assert max(lst + [elem]) == max(max(lst), elem)

Example 5: collections.Counter

This final example introduces a more complex data structure. A Counter object aggregates counts, and its inductive step involves what looks like a stateful update.

1

from collections import Counter

from collections import Counter

Base Case

The simplest case is a counter from an empty sequence.

1
2
3


def test_counter_base_case():
 """Base Case: A Counter from an empty list is empty."""
 assert Counter([]) == Counter()

def test_counter_base_case():
 """Base Case: A Counter from an empty list is empty."""
 assert Counter([]) == Counter()

Inductive Step

We verify that creating a Counter from a list is equivalent to creating one from a smaller list and then updating it with the final element.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


@given(
 st.lists(st.text(min_size=1)),
 st.text(min_size=1),
)
def test_counter_inductive_step(items, item):
 """
 Inductive Hypothesis: `Counter(items)` is correct.
 Inductive Step: The counter of a new list is equivalent to the
 counter of the old list updated with the new item.
 """
 partial_counter = Counter(items)
 partial_counter[item] += 1
 
 assert Counter(items + [item]) == partial_counter

@given(
 st.lists(st.text(min_size=1)),
 st.text(min_size=1),
)
def test_counter_inductive_step(items, item):
 """
 Inductive Hypothesis: `Counter(items)` is correct.
 Inductive Step: The counter of a new list is equivalent to the
 counter of the old list updated with the new item.
 """
 partial_counter = Counter(items)
 partial_counter[item] += 1
 
 assert Counter(items + [item]) == partial_counter

The “Aha!” Moment: Connecting the Dots

Let’s explicitly map the formal concepts of induction to the tests we just wrote.

Mathematical Concept	How It Appears in Our Tests
Base Case	The test___base_case() functions. They test the simplest meaningful input ($P(0)$), establishing a “ground truth” to build upon.
Inductive Hypothesis	The assumption that the function works for a smaller input ($P(k)$). In test_sum_inductive_step, the call sum(lst) embodies this.
Inductive Step	The assert statement in the test___inductive_step() function. It verifies that the structural rule ($P(k) \implies P(k+1)$) holds.

The most profound shift here is moving from testing outputs to testing relationships. A traditional test asserts factorial(5) == 120. An inductive test asserts factorial(5) == 5 * factorial(4), which is a far more powerful statement about the function’s internal logic. It verifies the algorithm itself.

Let’s summarize how this pattern applies to each function we’ve tested:

Function Tested	Base Case ($P(0)$)	Inductive Step ($P(k) \implies P(k+1)$)
`sum()`	The sum of an empty list is 0.	`sum(list + [item]) == sum(list) + item`.
`pow(x, y)`	A number to the power of 0 is 1.	`pow(x, y) == x * pow(x, y-1)`.
`str.join()`	Joining zero or one item works correctly.	Joining `n` items is equivalent to joining `n-1` items, then adding the separator and the last item.
`min()`	The min of two elements is the smaller one.	`min(list + [item]) == min(min(list), item)`.
`max()`	The max of two elements is the larger one.	`max(list + [item]) == max(max(list), item)`.
`collections.Counter`	A Counter of an empty sequence is empty.	The Counter of a list is a Counter of its sublist, updated with the last element.

From Theory to the Trenches: A Remark on Practicality

It is worth noting where these powerful ideas originate. The testing method—property-based testing—was popularized by the QuickCheck library from the functional programming language Haskell. The proof structure—induction—comes from pure mathematics.

However, these techniques are not confined to the functional paradigm or mathematical theory. Their true power is revealed when applied to everyday, non-mathematical, and even non-functional code.

Imagine you have written a highly optimized sorting or aggregation function in Rust, Cython, or C++. You can expose that function to Python via bindings and then test its correctness using this beautifully succinct style. This creates a powerful separation of concerns:

Low-Level Code: Worries about performance, memory, and raw computation.
High-Level Test: Worries exclusively about correctness in a declarative, readable way.

This approach minimizes the number of manual test cases you need to write while maximizing logical coverage of your algorithm. It is a testament to how abstract concepts from mathematics and computer science can become potent, practical tools in the hands of any programmer.

Conclusion

Property-based testing is already a huge leap forward from example-based testing. By adding the mental model of mathematical induction, you can take it a step further. This approach encourages you to identify the core recursive properties and invariants of your code, leading to tests that are more robust, more expressive, and far better at catching subtle logical flaws.

Next time you are working with a function that builds up a result incrementally or recursively, don’t just test random inputs. Separate the base case from the inductive step, and test the very rule that makes your function work.

Annex: Unleashing the Full Power of Property-Based Testing

The examples in the main article use hardcoded values in their base cases to keep the focus on the inductive structure. While excellent for teaching, a core principle of property-based testing is to eliminate “magic values” and test properties over a wide range of inputs. This annex shows how to convert our hardcoded tests into more powerful, generalized property-based tests.

Generalizing pow()’s Base Case

Instead of testing pow(2, 0), we can assert that pow(x, 0) is 1 for any integer x.

1
2
3
4


@given(st.integers())
def test_pow_base_case_generalized(x):
 """Base Case: Any number to the power of 0 is 1."""
 assert pow(x, 0) == 1

@given(st.integers())
def test_pow_base_case_generalized(x):
 """Base Case: Any number to the power of 0 is 1."""
 assert pow(x, 0) == 1

Generalizing str.join()

We can use pytest.parametrize for the simple base cases and also randomize the separator in the inductive step to make it more robust.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


import pytest

@pytest.mark.parametrize("lst, expected", [
 ([], ""),
 (["a"], "a"),
])
def test_join_base_cases_parametrized(lst, expected):
 """Base Cases: Test joining with zero and one element."""
 assert ",".join(lst) == expected

@given(
 st.text(),
 st.lists(st.text(), min_size=1),
 st.text(),
)
def test_join_inductive_step_generalized(separator, lst, elem):
 """Inductive Step: Test with any separator."""
 assert separator.join(lst + [elem]) == separator.join(lst) + separator + elem

import pytest

@pytest.mark.parametrize("lst, expected", [
 ([], ""),
 (["a"], "a"),
])
def test_join_base_cases_parametrized(lst, expected):
 """Base Cases: Test joining with zero and one element."""
 assert ",".join(lst) == expected

@given(
 st.text(),
 st.lists(st.text(), min_size=1),
 st.text(),
)
def test_join_inductive_step_generalized(separator, lst, elem):
 """Inductive Step: Test with any separator."""
 assert separator.join(lst + [elem]) == separator.join(lst) + separator + elem

Generalizing min() and max() Base Cases

Instead of a single example like [10, 5], we can generate any two integers and check the property.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


@given(st.integers(), st.integers())
def test_min_base_case_generalized(a, b):
 """Base Case: For any two elements, min() returns the smaller one."""
 expected = a if a <= b else b
 assert min([a, b]) == expected

@given(st.integers(), st.integers())
def test_max_base_case_generalized(a, b):
 """Base Case: For any two elements, max() returns the larger one."""
 expected = a if a >= b else b
 assert max([a, b]) == expected

@given(st.integers(), st.integers())
def test_min_base_case_generalized(a, b):
 """Base Case: For any two elements, min() returns the smaller one."""
 expected = a if a <= b else b
 assert min([a, b]) == expected

@given(st.integers(), st.integers())
def test_max_base_case_generalized(a, b):
 """Base Case: For any two elements, max() returns the larger one."""
 expected = a if a >= b else b
 assert max([a, b]) == expected

A Note on Fixed Base Cases

For functions like sum() and Counter, the base case is a single, fixed identity element (0 and an empty Counter, respectively). In these scenarios, the hardcoded base case is already the most general and correct form, so no change is needed. This demonstrates that the goal is not to eliminate all hardcoding, but to replace arbitrary examples with generalized properties wherever possible.

Property Based Testing - Retractions and Sections

Wed, 25 Oct 2023 12:00:00 +0000

Introduction

Property based testing is a technique for testing software that is based on generating random inputs to a function and checking that the output satisfies some property. The idea is that if the property is true for a large number of random inputs, then it is likely to be true for all inputs. This is a powerful technique that can be used to find bugs in software that are hard to find using traditional testing techniques.

In this article I will explain a very common software construction that can be very nicely tested using property based testing. This construction is called a retraction section pair in mathematics.

Instead of jumping straight into the examples, let me first give you the abstract definition of a retraction section pair. This will help you understand the examples better, and it will also help you see the pattern in other places.

Retractions and Sections

A retraction section pair is a pair of functions:

A section function: $i\colon A\to B$
A retraction function: $r\colon B\to A$

Such that when composed in a particular order they give the identity function on $A$. That is $r\circ i = id_A$. This is illustrated in the following diagram:

$$ id_A \colon A \overset{section}{\underset{i}{\to}} B \overset{retraction}{\underset{r}{\to}} A $$

In plain English this means that the section function $i$ knows how to transform each and every element of $A$ into an element of $B$¹, in such a way that the retraction function $r$ knows how to transform all elements of $B$ back into the original elements of $A$.

This abstract definition may seem uninteresting at first glance, but it is actually a very common pattern in software, and a very useful one when you consider practical instances of what $A$ and $B$ can be.

For the rest of the article I am going to change the name of the sets to better reflect the kind of things that they can be. I am going to call $A$ the source set and $B$ the target set.

The revised diagram looks like this:

$$ id_{source} \colon {source\ set} \overset{section}{\to} {target\ set} \overset{retraction}{\to} {source\ set} $$

Examples of Retractions and Sections in Software

Enough with the abstract definitions, let’s see some examples of retractions and sections in software.

Serializers and Deserializers

A very common example of a retraction section pair is a serializer and a deserializer. A serializer is a function that takes a piece of data in some particular programming language and converts it into a string or a binary blob. The deserializer is a function that takes that string or a binary blob and converts it back into the original data.

For example, the python pickle module has two functions dumps and loads that form a retraction section pair. The dumps function takes any python object² and converts it into a python bytes() object. The loads function takes that bytes() object and converts it back into the original python object.

1
2
3
4
5
6


>>> import pickle
>>> x=pickle.dumps([1, 2, 3])
>>> x
b'\x80\x04\x95\x0b\x00\x00\x00\x00\x00\x00\x00]\x94(K\x01K\x02K\x03e.'
>>> pickle.loads(x)
[1, 2, 3]

>>> import pickle
>>> x=pickle.dumps([1, 2, 3])
>>> x
b'\x80\x04\x95\x0b\x00\x00\x00\x00\x00\x00\x00]\x94(K\x01K\x02K\x03e.'
>>> pickle.loads(x)
[1, 2, 3]

In this scenario, the source set refers to the entire collection of Python objects. On the other hand, the target set is the collection of python bytes() objects that can be correctly decoded into Python objects using pickle. The section function is pickle.dumps and the retraction function is pickle.loads.

Compressors and Decompressors

Another example of a retraction section pair is a compressor and a decompressor.

A compressor is a function that takes a piece of data and compresses it into a (hopefully³) smaller piece of data. A decompressor is a function that takes that compressed piece of data and decompresses it back into the original data.

For example, the python zlib module has two functions compress and decompress that are a retraction section pair. The compress function takes any python bytes() object and compresses it into a smaller bytes() object. The decompress function takes that compressed bytes() object and decompresses it back into the original bytes() object.

1
2
3
4
5
6


>>> import zlib
>>> x=zlib.compress(b"hello world")
>>> x
b'x\x9c\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x1a\x0b\x04]'
>>> zlib.decompress(x)
b'hello world'

>>> import zlib
>>> x=zlib.compress(b"hello world")
>>> x
b'x\x9c\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x1a\x0b\x04]'
>>> zlib.decompress(x)
b'hello world'

In this case the source set is the set of all bytes() objects and the target set is the set of all zlib compressed bytes() objects. The section function is zlib.compress and the retraction function is zlib.decompress.

Encoders and Decoders

Our final example of a retraction section pair is an encoder and a decoder.

An encoder is a function that takes a piece of data and encodes it into a format that is suitable for transmission over a network or storage on disk, usually a string. A decoder is a function that takes that string and decodes it back into the original data.

For example, the python base64 module has two functions b64encode and b64decode that are a retraction section pair. The b64encode function takes any bytes() object and encodes it into an bytes() object only containing the ASCII characters A-Z, a-z, 0-9, +, / and =. The b64decode function takes that encoded bytes() object and decodes it back into the original bytes() object.

1
2
3
4
5
6


>>> import base64
>>> x=base64.b64encode(b"hello world")
>>> x
b'aGVsbG8gd29ybGQ='
>>> base64.b64decode(x)
b'hello world'

>>> import base64
>>> x=base64.b64encode(b"hello world")
>>> x
b'aGVsbG8gd29ybGQ='
>>> base64.b64decode(x)
b'hello world'

In this case the source set is the set of all bytes() objects and the target set is the set of base64 encoded byte strings. The section function is base64.b64encode and the retraction function is base64.b64decode.

What counts as a Retraction Section Pair?

The examples above are just a few of the many retraction section pairs that exist in software. In fact, there are so many of them that it is hard to enumerate them all. Here is a list of some of them found in the Python standard library:

section function	retraction function	source set	target set
base64.b{16,32,64,…}encode	base64.b{16,32,64,…}decode	python bytes() object	base{16,32,64,…} encoded bytes() object (ASCII encoded string)
codecs.encode⁴	codecs.decode	python string	byte representation of string in a different encoding
json.dumps	json.loads	python object that can be serialized to json	json encoded string
pickle.dumps	pickle.loads	python object that can be serialized to pickle	pickle encoded bytes() object
{zlib,gzip,bz2,lzma}.compress	{zlib,gzip,bz2,lzma}.decompress	python bytes() object	{zlib,gzip,bz2,lzma} compressed bytes() object

Surprisingly, this list doesn’t include a specific family of elements - encryption and decryption block ciphers. These form a retraction section pair for each algorithm and encryption key. The reason for this omission is simple: there’s no example of this in the Python standard library.

The following are not retraction section pairs:

Hash functions. Hash functions are not retraction section pairs because there is no way to get back the original value from the hash.
Lossy compression algorithms. Lossy compression algorithms are not retraction section pairs because there is no way to get back the original data from the compressed data. For example, the jpeg image format or mp3 audio format.

Picking out the patterns

Let’s look at some common patterns that we can see in the examples above.

Both the section and the retraction are pure functions. That is, they don’t have any side effects. They don’t read or write to any files, they don’t make any network requests, they don’t mutate any global state, etc.
The retraction function is the inverse of the section function. If you apply the section function to a value and then apply the retraction function to the result, you get back the original value.
The section function is a total function. It can be applied to any value in the source set. There are no values in the source set that cannot be injected into the target set.
Some of the section functions share the same source set.
None of the retraction functions share the same target set.
Weirdly enough, even though the retraction functions don’t share the same target set, some of the elements of their sets can be the same.

What is not part of the pattern is the following:

The retraction function is not necessarily a total function. At least for the type annotated as the target set in the source code. For example, the b64decode function signature may tell you that it takes a byte string and returns a byte string. However, if you give it a byte string that is not a valid base64 encoded string, you will get an error.
The section function is not necessarily the inverse of the retraction function. If this were the case, instead of having a section retraction pair, we would have an isomorphism⁵.

Testing Retractions and Sections

Now that we have a better understanding of what retractions and sections are, let’s talk about how we can test them using property based testing. For this section we’ll imagine that we are writing a base64 encoder and decoder.

Step 1: Write a Golden Test

A golden test is a test that compares the output of a function to a known correct value.

In the case of base64 encoding, we can write a golden test that compares the output of b64encode to a known correct value.

1
2
3
4
5
6


import base64

def test_b64encode():
 expected = b"aGVsbG8gd29ybGQ="
 actual = base64.b64encode(b"hello world")
 assert actual == expected

import base64

def test_b64encode():
 expected = b"aGVsbG8gd29ybGQ="
 actual = base64.b64encode(b"hello world")
 assert actual == expected

The reason why a golden test is a good place to start is because it gives us a known correct so that we are sure we are implementing the right thing. If you recall from the previous section, there are several section functions that share the same source set. For example, b64encode, b32encode, and b16encode all take binary data and encode it into a string. If we have a golden test for b64encode we are sure that the output of b64encode is a valid base64 encoded string and not anything else.

It is maybe a good idea to test more than one value. For that we can use pytest’s parametrize feature.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


import base64

import pytest

@pytest.mark.parametrize("data,expected", [
 (b"hello world", b"aGVsbG8gd29ybGQ="),
 (b"foo bar", b"Zm9vIGJhcg=="),
])
def test_b64encode(data, expected):
 actual = base64.b64encode(data)
 assert actual == expected

import base64

import pytest

@pytest.mark.parametrize("data,expected", [
 (b"hello world", b"aGVsbG8gd29ybGQ="),
 (b"foo bar", b"Zm9vIGJhcg=="),
])
def test_b64encode(data, expected):
 actual = base64.b64encode(data)
 assert actual == expected

Step 2: Write a Hypothesis Test for the Section Retraction Pair

Up to this point our test strategy has been pretty straight forward. We have written a golden test for the section function. And now is where we get the big hammer out. We are going to test the retraction section pair.

1
2
3
4
5
6
7


import base64

from hypothesis import given, strategies as st

@given(st.binary())
def test_b64encode_b64decode(data):
 assert base64.b64decode(base64.b64encode(data)) == data

import base64

from hypothesis import given, strategies as st

@given(st.binary())
def test_b64encode_b64decode(data):
 assert base64.b64decode(base64.b64encode(data)) == data

The decorator @given is a signal to the hypothesis library that we want to generate random values using the st.binary() strategy. These values are then fed into the test function. The st.binary() strategy is specifically designed to create random bytes() objects of different lengths. In simpler terms, it’s like saying that the source set is made up of all possible bytes() objects.

Next, the hypothesis library will churn out a series of these random binary blobs and inject them into the test function. This function will first apply the section function to the random bytes() object, and then the retraction function to the output that results.

Looking at it from a mathematical perspective, this test is basically asserting a theorem. The theorem in question suggests that the retraction-section pair operates as the identity function.

Let’s Test Some More Retraction Section Pairs

We can extend this pattern to other section and retraction function pairs. For example, we can test the retraction section pair for gzip.compress and gzip.decompress.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


import gzip

from hypothesis import given, strategies as st

def test_gzip_compress():
 expected = b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03K\xcb\xcf\x07\x00\x82"
 actual = gzip.compress(b"hello world")
 assert actual == expected

@given(st.binary())
def test_gzip_compress_gzip_decompress(data):
 assert gzip.decompress(gzip.compress(data)) == data

import gzip

from hypothesis import given, strategies as st

def test_gzip_compress():
 expected = b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03K\xcb\xcf\x07\x00\x82"
 actual = gzip.compress(b"hello world")
 assert actual == expected

@given(st.binary())
def test_gzip_compress_gzip_decompress(data):
 assert gzip.decompress(gzip.compress(data)) == data

Or we can test the retraction section pair for json.dumps and json.loads.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import json

from hypothesis import given, strategies as st

json_serializable_strategy = st.recursive(
 st.none() | st.booleans() | st.floats() | st.text(),
 lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)

def test_json_dumps():
 expected = '{"hello": "world"}'
 actual = json.dumps({"hello": "world"})
 assert actual == expected

@given(json_serializable_strategy)
def test_json_dumps_json_loads(data):
 assert json.loads(json.dumps(data)) == data

import json

from hypothesis import given, strategies as st

json_serializable_strategy = st.recursive(
 st.none() | st.booleans() | st.floats() | st.text(),
 lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)

def test_json_dumps():
 expected = '{"hello": "world"}'
 actual = json.dumps({"hello": "world"})
 assert actual == expected

@given(json_serializable_strategy)
def test_json_dumps_json_loads(data):
 assert json.loads(json.dumps(data)) == data

Note that in this example the source set is not binary data. The source set is any value that can be serialized into JSON. Thankfully, the hypothesis library allows us to define complex strategies that can generate random values from any set we want. You can read more about the hypothesis library and strategies in the hypothesis documentation.

We also can test the retraction section pair for codecs.encode and codecs.decode for the utf-8 encoding.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


import codecs

from hypothesis import given, strategies as st

def test_codecs_encode():
 expected = b"hello world"
 actual = codecs.encode("hello world", "utf-8")
 assert actual == expected

@given(st.text())
def test_codecs_encode_codecs_decode(data):
 assert codecs.decode(codecs.encode(data, "utf-8"), "utf-8") == data

import codecs

from hypothesis import given, strategies as st

def test_codecs_encode():
 expected = b"hello world"
 actual = codecs.encode("hello world", "utf-8")
 assert actual == expected

@given(st.text())
def test_codecs_encode_codecs_decode(data):
 assert codecs.decode(codecs.encode(data, "utf-8"), "utf-8") == data

Testing Against a Golden Implementation

You may have noticed that the initial golden tests we created are somewhat delicate. Not only do we need to craft these tests ourselves, but we also need to ensure their accuracy. This isn’t an issue for the base64 module, given its extensive testing. But what happens when we’re developing our own module? How can we be certain that our golden tests are accurate and testing the right elements?

In some instances, we can compare our work to a golden implementation to verify the correctness of our own.

Imagine that we are implementing our own gzip compression library. Instead of providing a few golden tests, we can test our implementation against the gzip command line tool. This is a good example of a golden implementation because it is well-tested and has been around for a long time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


import gzip
import subprocess

from hypothesis import given, strategies as st

def external_gzip_compress(data):
 return subprocess.check_output(["gzip", "-c"], input=data)

def external_gzip_decompress(data):
 return subprocess.check_output(["gzip", "-cd"], input=data)

@given(st.binary())
def test_gzip_compress(data):
 assert external_gzip_decompress(gzip.compress(data)) == data

@given(st.binary())
def test_external_gzip_compress(data):
 assert gzip.decompress(external_gzip_compress(data)) == data

@given(st.binary())
def test_gzip_compress_gzip_decompress(data):
 assert gzip.decompress(gzip.compress(data)) == data

import gzip
import subprocess

from hypothesis import given, strategies as st

def external_gzip_compress(data):
 return subprocess.check_output(["gzip", "-c"], input=data)

def external_gzip_decompress(data):
 return subprocess.check_output(["gzip", "-cd"], input=data)

@given(st.binary())
def test_gzip_compress(data):
 assert external_gzip_decompress(gzip.compress(data)) == data

@given(st.binary())
def test_external_gzip_compress(data):
 assert gzip.decompress(external_gzip_compress(data)) == data

@given(st.binary())
def test_gzip_compress_gzip_decompress(data):
 assert gzip.decompress(gzip.compress(data)) == data

Composing Retraction Section Pairs

Retraction section pairs compose very nicely. If we have two retraction section pairs of the following form $A \overset{section}{\underset{i}{\to}} B \overset{retraction}{\underset{r}{\to}} A$, and $B \overset{section}{\underset{j}{\to}} C \overset{retraction}{\underset{s}{\to}} B$,

Then we can compose the two retraction section pairs to get a new retraction section pair.

$$ A \overset{section}{\underset{i \circ j}{\to}} C \overset{retraction}{\underset{r \circ s}{\to}} A $$

In other words, if we have two retraction section pairs with compatible types, then we can compose them to get a new retraction section pair. Let’s see an example of this in action.

The first section is the json.dumps which accepts any value that can be serialized into JSON and returns a string. And the second section is the codecs.encode (using the utf-8 encoding) which accepts a string and returns a byte string.

The retractions are the respective inverses of the sections.

1
2
3
4
5
6
7
8


import codecs
import json

def json_utf8_dumps(data):
 return codecs.encode(json.dumps(data), "utf-8")

def json_utf8_loads(data):
 return json.loads(codecs.decode(data, "utf-8"))

import codecs
import json

def json_utf8_dumps(data):
 return codecs.encode(json.dumps(data), "utf-8")

def json_utf8_loads(data):
 return json.loads(codecs.decode(data, "utf-8"))

We can go even further and compose this new retraction section pair with the gzip.compress and gzip.decompress retraction section pair.

1
2
3
4
5
6
7


import gzip

def json_utf8_gzip_compress(data):
 return gzip.compress(json_utf8_dumps(data))

def json_utf8_gzip_decompress(data):
 return json_utf8_loads(gzip.decompress(data))

import gzip

def json_utf8_gzip_compress(data):
 return gzip.compress(json_utf8_dumps(data))

def json_utf8_gzip_decompress(data):
 return json_utf8_loads(gzip.decompress(data))

And test it all at once.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


from hypothesis import given, strategies as st

json_serializable_strategy = st.recursive(
 st.none() | st.booleans() | st.floats() | st.text(),
 lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)

@given(json_serializable_strategy)
def test_json_utf8_gzip_compress_json_utf8_gzip_decompress(data):
 assert json_utf8_gzip_decompress(json_utf8_gzip_compress(data)) == data

from hypothesis import given, strategies as st

json_serializable_strategy = st.recursive(
 st.none() | st.booleans() | st.floats() | st.text(),
 lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)

@given(json_serializable_strategy)
def test_json_utf8_gzip_compress_json_utf8_gzip_decompress(data):
 assert json_utf8_gzip_decompress(json_utf8_gzip_compress(data)) == data

A note on pure functions⁶

Even though it may seem like this pattern can be extended to non-pure functions, in practice it is not a good idea. Let’s enumerate some non-pure functions that follow this pattern.

A database query that inserts a row into a table and returns the primary key and a query that selects a row from a table given the primary key.
A function that writes a file to disk and a function that reads a file from disk.
A function that sets an environment variable and a function that reads an environment variable.

The problem with testing these functions is that they are not referentialy transparent. In other words, the output of the function depends on the state of the world. Let’s see what can go wrong if we try to test these functions.

A database query that inserts a row into a table and returns the primary key and a query that selects a row from a table given the primary key.
- What would be the outcome if the database is unavailable?
- What would occur if we exhaust all available primary keys?
- What if one of the columns in the database has a unique constraint?
- What if someone else deletes the row between the time we insert it and select it?
A function that writes a file to disk and a function that reads a file from disk.
- What happens if the file cannot be written to?
- What if the file cannot be read?
- What if there is no more space available on the filesystem?
- What if the file is deleted between the time we write it and read it?
A function that sets an environment variable and a function that reads an environment variable.
- What would happen if, in the time between setting the environment variable and reading it, another part of the program changes the value of the environment variable?

In all of these cases, the output of the function depends on the state of the world.

Conclusion

The Benefits of Testing Retraction Section Pairs

The advantage of testing retraction section pairs in this manner is that it boosts our confidence in the code’s accuracy over time, without the need for extra tests. This is due to the fact that the ‘hypothesis’ library generates new random values for each test run, which are then used to test the retraction section pair.

However, this isn’t the case with a conventional unit test. A traditional unit test only tests the code with the values supplied by the test author. As a result, our confidence in the code’s accuracy remains unchanged over time.

Limitations of Testing Retraction Section Pairs

While this pattern holds significant power, it’s not without its constraints.

The first constraint is its exclusive compatibility with pure functions. If you attempt to apply this pattern to a non-pure function, tread carefully. You’ll need to be mindful of potential side effects and how they could impact the test.

The second constraint is that our golden test only examines one or a handful of values. This means that while we can be fairly certain that the retraction is the inverse of the section, we can’t be as confident that the section is our intended function. However, this issue can be easily resolved by incorporating more tests or, even better, by testing against a golden implementation!

Please be aware that $A$ and $B$ can represent any sets. Specifically, if $A$ is an infinitely large set, then $B$ has the potential to be a subset of $A$. ↩︎
Actually the source set is the set of all python objects that can be pickled, see here for more details. ↩︎
Of course the compressed piece of data is not always smaller than the original, you’ve probably seen this before when you’ve tried to compress a file that is already compressed. ↩︎
for codecs that can represent the full set of Unicode code points ↩︎
An isomorphism is a pair of functions that are inverses of each other. An example of isomorphism in python are the functions list() and tuple(). You can always convert a list to a tuple and back again without gaining or losing any information. ↩︎
A pure function is a function that given the same input always returns the same output. In other words, a pure function is referentialy transparent. ↩︎