Instead, we should have helpful error messages explaining why a property does not hold; which parts of the specification failed and which concrete values from the trace were involved. Not false, unsat, or even assertion error: x != y. We should get the full story. I started exploring this space a few years ago when I worked actively on Quickstrom, but for some reason it went on the shelf half-finished. Time to tie up the loose ends!

The starting point was Picostrom, a minimal Haskell version of the checker in Quickstrom, and Error Reporting Logic (ERL), a paper introducing a way of rendering natural-language messages to explain propositional logic counterexamples. I ported it to Rust mostly to see what it turned into, and extended it with error reporting supporting temporal operators. The code is available at codeberg.org/owi/picostrom-rs under the MIT license.

Between the start of my work and picking it back up now, A Language for Explaining Counterexamples was published, which looks closely related, although it’s focused on model checking with CTL. If you’re interested in other related work, check out A Systematic Literature Review on Counterexample Explanation in Model Checking.

All right, let’s dive in!

QuickLTL and Picostrom

A quick recap on QuickLTL is in order before we go into the Picostrom code. QuickLTL operates on finite traces, making it suitable for testing. It’s a four-valued logic, meaning that a formula evaluates to one of these values:

• definitely true
• definitely false
• probably true
• probably false

It extends propositional logic with temporal operators, much like LTL:

next_d(P)

P must hold in the next state, demanding a next state is available. This forces the evaluator to draw a next state.

next_f(P)

P must hold in the next state, defaulting to definitely false if no next state is available.

next_t(P)

P must hold in the next state, defaulting to probably true if no next state is available.

eventually_N(P)

P must hold in the current or a future state. It demands at least N states, evaluating on all available states, finally defaulting to probably false.

always_N(P)

P must hold in the current and all future states. It demands at least N states, evaluating on all available states, finally defaulting to probably true.

You can think of eventually_N(P) as unfolding into a sequence of N nested next_d, wrapping an infinite sequence of next_f, all connected by ∨. Let’s define that inductively with a coinductive base case:

$$ \begin{align} \text{eventually}_0(P) & = P \lor \text{next}_F(\text{eventually}_0(P)) \\ \text{eventually}_(N + 1)(P) & = P \lor \text{next}_D(\text{eventually}_N(P)) \\ \end{align} $$

And similarly, always_N(P) can be defined as:

$$ \begin{align} \text{always}_0(P) & = P \land \text{next}_T(\text{always}_0(P)) \\ \text{always}_(N + 1)(P) & = P \land \text{next}_D(\text{always}_N(P)) \\ \end{align} $$

This is essentially how the evaluator expands these temporal operators, but for error reporting reasons, not exactly.

Finally, there are atoms, which are domain-specific expressions embedded in the AST, evaluating to ⊤ or ⊥. The AST is parameterized on the atom type, so you can plug in an atom language of choice. An atom type must implement the Atom trait, which in simplified form looks like this:

trait Atom {  type State;   fn eval(&self, state: &Self::State) -> bool;   fn render(  &self,   mode: TextMode,   negated: bool,  ) -> String;   fn render_actual(  &self,   negated: bool,   state: &Self::State,  ) -> String; }

For testing the checker, and for this blog post, I’m using the following atom type:

enum TestAtom {  Literal(u64),  Select(Identifier),  Equals(Box<TestAtom>, Box<TestAtom>),  LessThan(Box<TestAtom>, Box<TestAtom>),  GreaterThan(Box<TestAtom>, Box<TestAtom>), }  enum Identifier {  A,  B,  C, }

Evaluation

The first step, like in ERL, is transforming the formula into negation normal form (NNF), which means pushing down all negations into the atoms:

enum Formula<Atom> {  Atomic {  negated: bool,  atom: Atom,  },  // There's no `Not` variant here!  ... }

This makes it much easier to construct readable sentences, in addition to another important upside which I’ll get to in a second. The NNF representation is the one used by the evaluator internally.

Next, the eval function takes an Atom::State and a Formula, and produces a Value:

enum Value<'a, A: Atom> {  True,  False { problem: Problem<'a, A> },  Residual(Residual<'a, A>), }

A value is either immediately true or false, meaning that we don’t need to evaluate on additional states, or a residual, which describes how to continue evaluating a formula when given a next state. Also note how the False variant holds a Problem, which is what we’d report as definitely false. The True variant doesn’t need to hold any such information, because due to NNF, it can’t be negated and “turned into a problem.”

I won’t cover every variant of the Residual type, but let’s take one example:

 enum Residual<'a, A: Atom> {  // ...  AndAlways {  start: Numbered<&'a A::State>,  left: Box<Residual<'a, A>>,  right: Box<Residual<'a, A>>,  },  // ... }

When such a value is returned, the evaluator checks if it’s possible to stop at this point, i.e. if there are no demanding operators in the residual. If not possible, it draws a new state and calls step on the residual. The step function is analogous to eval, also returning a Value, but it operates on a Residual rather than a Formula.

The AndAlways variant describes an ongoing evaluation of the always operator, where the left and right residuals are the operands of ∧ in the inductive definition I described earlier. The start field holds the starting state, which is used when rendering error messages. Similarly, the Residual enum has variants for ∨, ∧, ⟹, next, eventually, and a few others.

When the stop function deems it possible to stop evaluating, we get back a value of this type:

enum Stop<'a, A: Atom> {  True,  False(Problem<'a, A>), }

Those variants correspond to probably true and probably false. In the false case, we get a Problem which we can render. Recall how the Value type returned by eval and step also had True and False variants? Those are the definite cases.

Rendering Problems

The Problem type is a tree structure, mirroring the structure of the evaluated formula, but only containing the parts of it that contributed to its falsity.

enum Problem<'a, A: Atom> {  And {  left: Box<Problem<'a, A>>,  right: Box<Problem<'a, A>>,  },  Or {  left: Box<Problem<'a, A>>,  right: Box<Problem<'a, A>>,  },  Always {  state: Numbered<&'a A::State>,  problem: Box<Problem<'a, A>>,  },  Eventually {  state: Numbered<&'a A::State>,  formula: Box<Formula<A>>,  },  // A bunch of others... }

I’ve written a simple renderer that walks the Problem tree, constructing English error messages. When hitting the atoms, it uses the render and render_actual methods from the Atom trait I showed you before.

The mode is very much like in the ERL paper, i.e. whether it should be rendered in deontic (e.g. “x should equal 4”) or indicative (e.g. “x equals 4”) form:

enum TextMode {  Deontic,  Indicative, }

The render method should render the atom according to the mode, and render_actual should render relevant parts of the atom in a given state, like its variable assignments.

With all these pieces in place, we can finally render some error messages! Let’s say we have this formula:

eventually₁₀(B = 3 ∧ C = 4)

If we run a test and never see such a state, the rendered error would be:

Probably false: eventually B must equal 3 and C must equal 4, but it was not observed starting at state 0

Neat! This is the kind of error reporting I want for my stateful tests.

Implication

You can trace why some subformula is relevant by using implication. A common pattern in state machine specs and other safety properties is:

precondition ⟹ before ∧ next_t(after)

So, let’s say we have this formula:

always_N((A > 0) ⟹ (B > 5 ∧ next_t(C < 10)))

If B or C are false, the error includes the antecedent:

Definitely false: B must be greater than 5 and in the next state, C must be less than 10 since A is greater than 0, […]

Small Errors, Short Tests

Let’s consider a conjunction of two invariants. We could of course combine the two atomic propositions with conjunction inside a single always(...), but in this case we have the formula:

always(A < 3) ∧ always(B < C)

An error message, where both invariants fail, might look the following:

Definitely false: it must always be the case that A is less than 3 and it must always be the case that B is greater than C, but A=3 in state 3 and B=0 in state 3

If only the second invariant (B < C) fails, we get a smaller error:

Definitely false: it must always be the case that B is greater than C, but B=0 and C=0 in state 0

And, crucially, if one of the invariants fail before the other we also get a smaller error, ignoring the other invariant. While single-state conjunctions evaluate both sides, possibly creating composite errors, conjunctions over time short-circuit in order to stop tests as soon as possible.

Diagrams

Let’s say we have a failing safety property like the following:

next_d(always₈(B < C))

The textual error might be:

Definitely false: in the next state, it must always be the case that B is greater than C, but B=13 and C=15 in state 6

But with some tweaks we could also draw a diagram, using the Problem tree and the collected states:

Or for a liveness property like next_d(eventually₈(B = C)), where there is no counterexample at a particular state, we could draw a diagram showing how we give up after some time:

These are only sketches, but I think they show how the Problem data structure can be used in many interesting ways. What other visualizations would be possible? An interactive state space explorer could show how problems evolve as you navigate across time. You could generate spreadsheets or HTML documents, or maybe even annotate the relevant source code of some system-under-test? I think it depends a lot on the domain this is applied to.

No Loose Ends

It’s been great to finally finish this work! I’ve had a lot of fun working through the various head-scratchers in the evaluator, getting strange combinations of temporal operators to render readable error messages. I also enjoyed drawing the diagrams, and almost nerd-sniped myself into automating that. Maybe another day. I hope this is interesting or even useful to someone out there. LTL is really cool and should be used more!

The code, including many rendering tests cases, is available at codeberg.org/owi/picostrom-rs.

A special thanks goes to Divyanshu Ranjan for reviewing a draft of this post.

This is not a sponsored post.

Why do I even bother, you might ask? Sunlight makes me energetic and alert, which I need when I work. Living in the Nordics, 50% of the year is primarily dark, so any direct daylight I can get becomes really important. I usually run light mode on my Framework laptop during the day, but working in actual daylight with these displays, or plain old paper, is even better.

The Setup

Here are the main components of this coding environment:

• Daylight DC-1: an Android-based tablet with a “Live Paper” display (Reflective LCD, not E-Ink)
• 8BitDo Retro Mechanical Keyboard: a mechanical Bluetooth-enabled keyboard, with Kailh key switches and USB-C charging and optional connection
• Termux: a terminal emulator for Android, with a package collection based on apt
• SSH, tmux, and Neovim: nothing surprising here

I use a slimmed-down version of my regular dotfiles, because this setup doesn’t use Nix. I’ve manually installed Neovim, tmux, and a few other essentials, using the package manager that comes with Termux. I’ve configured Termux to not show its virtual keyboard when a physical keyboard is connected (the Bluetooth keyboard). The Termux theme is “E-Ink” and the font is JetBrains Mono, all built into Termux. Neovim uses the built-in quiet colorscheme for maximum contrast.

Certain work requires a more capable environment, and in those cases I connect to my workstation using SSH and run tmux in there. For writing or simpler programming projects (I’ve even done Rust work with Cargo, for instance), the local Termux environment is fine.

Sometimes I want to go really minimalist, so I hide the tmux status bar and run Goyo in Neovim. Deep breaths. Feel the fresh air in your lungs. This is especially nice for writing blog posts like this one.

My blog editing works locally in Termux, with a live reloading Chrome in a split window, here during an evening writing session with the warm backlight enabled:

There’s the occasional Bluetooth connection problem with the 8BitDo keyboard. I also don’t love the layout, and I’m considering getting the Kinesis Freestyle2 Blue instead. I already have the wired version for my workstation, and the ergonomics are great.

Daylight DC-1 vs Boox Tab Ultra

What about the Boox? I’ve had this device for longer and I really like it too, but not for the same tasks. The E-Ink display is, quite frankly, a lot nicer to read on; EPUB books, research PDFs, web articles, etc. The 227 PPI instead of the Daylight’s 190 PPI makes a difference, and I like the look of E-Ink better overall.

However, the refresh rate and ghosting make it a bit frustrating for typing. Same goes for drawing, which I’ve used the Daylight for a lot. Most of my home renovation blueprints are sketched on the Daylight. The refresh rate makes it possible.

When reading at night with a more direct bedside lamp, often in combination with a subtle backlight, the Boox is much better. The Daylight screen can glare quite a bit, so the only option is backlight only. And at that point, a lot of the paperlike quality goes away.

You can also get some glare when there’s direct sunlight at a particular angle:

Even if I don’t write or program directly on the Boox, I’ve experimented with using it as a secondary display, like for the live reload blog preview:

To sum up, these devices are good for different things, in my experience. I’ve probably spent more time on the Boox, because I’ve had it for longer and I’ve read a lot on it, but the Daylight has been much better for typing and drawing.

Another thing I’d like to try is a larger E-Ink monitor for my workstation, like the one Zack is hacking on. I’m hoping this technology continues to improve on refresh rate, because I love E-Ink. Until then, the Daylight is a good compromise.

The goal is to get one of Amp’s most central pieces, the ThreadWorker, under heavy scrutiny. We’ve had a few perplexing bug reports, where users experienced corrupted threads, LLM API errors from invalid tool calls, and more vague issues like “it seems like it’s spinning forever.” Reproducing such problems manually is usually somewhere between impractical and impossible. I want to reproduce them deterministically, and in a way where we can debug and fix them. And beyond the known ones, I’d like to find the currently unknown ones before our users hit them.

Generative testing to the rescue!

Approach: Lightweight DST in TypeScript

Amp is written in TypeScript, which is an ecosystem currently not drowning in fuzzing tools. My starting point was using jsfuzz, which I hadn’t used before but it looked promising. However, I had a bunch of problems getting it to run together with our Bun stack. One could use fast-check, but as far as I can tell, the model-based testing they support doesn’t fit with our needs. We don’t have a model of the system, and we need to generate values in multiple places as the test runs. So, I decided to build something from scratch for our purposes.

I borrowed an idea I got from matklad last year: instead of passing a seeded PRNG to generate test input, we generate an entropy Buffer with random contents, and track our position in that array with a cursor. Drawing a random byte consumes the byte at the current position and increments the cursor. We don’t know up-front how many bytes we need for a given fuzzer, so the entropy buffer grows dynamically when needed, appending more random bytes. This, together with a bunch of methods for drawing different types of values, is packaged up in an Entropy class:

class Entropy {  random(count): UInt8Array { ... }  randomRange(minIncl: number, maxExcl: number): number { ... }  // ... lots of other stuff }

A fuzzer is an ES module written in TypeScript, exporting a single function:

export async function fuzz(entropy: Entropy) {  // test logic here }

Any exception thrown by fuzz is considered a test failure. We use the node:assert module for our test assertions, but it could be anything.

Another program, the fuzz runner, imports a built fuzzer module and runs as many tests it can before a given timeout. If it finds a failure, it prints out the command to reproduce that failure:

Fuzzing example.fuzzer.js iteration 1000... Fuzzing example.fuzzer.js iteration 2000... Fuzzer failed: AssertionError [ERR_ASSERTION]: 3 != 4 at [...] Reproduce with: bun --console-depth=10 scripts/fuzz.ts \ dist/example.fuzzer.js \ --verbose \ --reproduce=1493a513f88d0fd9325534c33f774831

Why use this Entropy rather than a seed? More about that at the end of the post!

The ThreadWorker Fuzzer

In the fuzzer for our ThreadWorker, we stub out all IO and other nondeterministic components, and we install fake timers to control when and how asynchronous code is run. In effect, we have determinism and simulation to run tests in, so I guess it qualifies as DST.

The test simulates a sequence of user actions (send message, cancel, resume, and wait). Similarly, it simulates responses from tool calls (like the agent reading a file) and from inference backends (like the Anthropic API). We inject faults and delays in both tool calls and inference requests to test our error handling and possible race conditions.

After all user actions have been executed, we make sure to approve any pending tool calls that require confirmation. Next, we tell the fake timer to run all outstanding timers until the queue is empty; like fast-forwarding until there’s nothing left to do. Finally, we check that the thread is idle, i.e. that there’s no ongoing inference and that all tool calls have terminated. This is a liveness property.

After the liveness property, we check a bunch of safety properties:

• all messages posted by the user are present in the thread
• all message pairs involving tools calls are valid according to Anthropic’s API specification
• all tool calls have settled in expected terminal states

Some of these are targeted at specific known bugs, while some are more general but have found bugs we did not expect.

Here’s a highly simplified version of the fuzzer:

export async function fuzz(entropy: Entropy) {  const clock = sinon.useFakeTimers({  loopLimit: 1_000_000,  })  const worker = setup() // including stubbing IO, etc   try {  const resumed = worker.resume()  await clock.runAllAsync()  await resumed   async function run() {  for (let round = 0; round < entropy.randomRange(1, 50); round++) {  const action = await generateNextAction(entropy, worker)  switch (action.type) {  case 'user-message':  await worker.handle({  ...action,  type: 'user:message',  })  break  case 'cancel':  await worker.cancel()  break  case 'resume':  await worker.resume()  break  case 'sleep':  await sleep(action.milliseconds)  break  case 'approve': {  await approveTool(action.threadID, action.toolUseID)  break  }  }  }   // Approve any remaining tool uses to ensure termination into an   // idle thread state  const blockedTools = await blockedToolUses()  await Promise.all(blockedTools.map(approve))  }   const done = run()  await clock.runAllAsync()  await done   // check liveness and safety properties  // ...  } finally {  sinon.restore()  } }

Now, let’s dig into the findings!

Results

Given I’ve been working on this for about a week in total, I’m very happy with the outcome. Here are some issues the fuzzer found:

Corrupted thread due to eagerly starting tool calls during streaming

While streaming tool use blocks from the Anthropic API, we invoked tools eagerly, while not all of them were finished streaming. This, in combination with how state was managed, led to tool results being incorrectly split across messages. Anthropic’s API would reject any further requests, and the thread would essentially be corrupted. This was reported by a user and was the first issue we found and fixed using the fuzzer.

Another variation, which the fuzzer also found, this was a race condition where user messages interfered at a particular timing with ongoing tool calls, splitting them up incorrectly.

Subagent tool calls not terminating when subthread tool calls were rejected

Due to a recent change in behavior, where we don’t run inference automatically after tool call rejection, subagents could end up never signalling their termination, which led to the main thread never reaching an idle state.

I confirmed this in both VSCode and the CLI: infinite spinners, indeed.

Tool calls blocked on user not getting cancelled after user message

Due to how some tool calls require confirmation, like reading files outside the workspace or running some shell commands, in combination how we represent and track termination of tools, there’s a possibility for such tools to be resumed and then, after an immediate user cancellation, not be properly cancelled. This leads to incorrect mutations of the thread data.

I’ve not yet found the cause of this issue, but it’s perfectly reproducible, so that’s a start.

Furthermore, we were able to verify an older bug fix, where Anthropic’s API would send an invalid message with an empty tool use block array. That used to get the agent into an infinite loop. With the fuzzer, we verified and improved the old fix which had missed another case.

How about number of test runs and timeouts? Most of these bugs were found almost immediately, i.e. within a second. The last one in the list above takes longer, around a minute normally. We run a short version of each fuzzer in every CI build, and longer runs on a nightly basis. This is up for a lot of tuning and experimentation.

Why the Entropy Buffer?

So why the entropy buffer instead of a seeded PRNG? The idea is to use that buffer to mutate the test input, instead of just bombarding with random data every time. If we can track which parts of the entropy was used where, we can make those slices “smaller” or “bigger.” We can use something like gradient descent or simulated annealing to optimize inputs, maximizing some objective function set by the fuzzer. Finally, we might be able to minimize inputs by manipulating the entropy.

In case the JavaScript community gets some powerful fuzzing framework like AFL+, that could also just be plugged in. Who knows, but I find this an interesting approach that’s worth exploring. I believe the entropy buffer approach is also similar to how Hypothesis works under the hood. Someone please correct me if that’s not the case.

Anyhow, that’s today’s report from the generative testing mines. Cheers!

I know there are probably tons of blog posts by the newly converted. I’m not trying to offer any grand insights, I’m just documenting my process and current ideas.

Gradually, Then Suddenly

I’ve been a typical sceptic of Copilot and similar tools. Sure, it’s nice to generate boilerplate and throw-away scripts, but that’s a minor part of what we do all day, right? I even took a break from using them for many months, and I’ve had serious qualms with their use in some areas outside coding.

After messing around with copilot.lua in Neovim, I tried Cursor. Their vision and what they’ve already built opened my eyes, especially the shadow workspaces, Tab, and the rule files. At the same time, a critical mass of friends and peers were building new products on top of these models; things I highly respect and can see massive value in.

Since then I’ve actively been looking for how I can use LLMs — beyond the auto-complete and chat interaction modes, beyond making me a slightly more productive developer. Don’t get me wrong, I love being alone coding for hours in a cozy room. It’s great. But I’m also curious to see how far I can push this myself, and of course how far and where the industry goes.

Taking on New Projects

One project that I’ve been working on goes under the working name site2doc. It’s a tool that converts entire web sites into EPUB and PDF books. Mostly because I want it for myself. I’d like to read online material, typeset minimalistically and beautifully, offline on my ebook reader. It turned out others want that too. There are great tools for converting single pages, but not for entire sites.

My main problem is that the web is highly unstructured and diverse. To be frank, a lot of sites have really bad markup. No titles at all, identical titles across pages, <h1> elements used inconsistently. The list goes on. This makes it very difficult for site2doc to generate a useful table of contents.

A friend suggested using LLMs to extract the information, and I experimented with using screenshots as input to classify pages. Both Claude 3.5 Sonnet and Gemini 2.0 Flash performed well, but I haven’t been able to generalize the approach across many sites. There’s just too much variability in how websites are structured, and I’m not sure how to handle it. I’m open to suggestions!

The other project, temporarily named converge, is a bit closer to what everyone else is doing: using LLMs for programming. It’s an agent that, given some file or module, covers it with a generated test suite, and then goes on to optimize it. The key idea is that the original code is the source of truth. The particular optimization goal could be performance, resource usage, readability, safety, or robustness. So far I’ve focused only on performance, partly because evaluation is straightforward.

Going beyond example-based test suites, I’ve been thinking about how property-based testing (PBT) might fit in. The obvious approach is to have the LLM generate property tests rather than examples. I don’t know how well this would work, if the LLM can generate meaningful properties.

A more interesting way is to generate an oracle property that compares the behavior of new code generated by the LLM to the original code: ∀i.old(i) = new(i), where i is some generated input. This provides a rigorous way to verify that optimizations preserve the original behavior. I’m curious to see how PBT’s shrinking could guide the LLM to iteratively fix the generated code.

Another random idea: have the LLM explain existing code in natural language, then generate new code based only on the description. Run the old and new code side-by-side, and see how they differ, functionally and non-functionally.

I’ve only run converge on toy examples and snippets so far. I’m sure there are major challenges in applying it to larger code bases. Here’s what it currently does to Achilles numbers in Rosetta Code:

» converge -input AchillesNumbers.java time=2025-02-11T09:35:44.877+01:00 level=INFO msg="test suite created" time=2025-02-11T09:35:45.271+01:00 level=WARN msg="tests failed" attempt=1 time=2025-02-11T09:35:53.308+01:00 level=INFO msg="test suite modified" time=2025-02-11T09:35:53.709+01:00 level=WARN msg="tests failed" attempt=2 time=2025-02-11T09:36:02.457+01:00 level=INFO msg="test suite modified" time=2025-02-11T09:36:02.885+01:00 level=INFO msg="tests passed" time=2025-02-11T09:36:09.508+01:00 level=INFO msg="increasing benchmark iterations" duration=0.038339998573064804 attempt=0 time=2025-02-11T09:36:15.056+01:00 level=INFO msg="increasing benchmark iterations" duration=0.2295980006456375 attempt=1 time=2025-02-11T09:36:21.225+01:00 level=INFO msg="increasing benchmark iterations" duration=0.5998150110244751 attempt=2 time=2025-02-11T09:36:28.010+01:00 level=INFO msg="benchmark run" time=2025-02-11T09:36:35.737+01:00 level=INFO msg="code optimized" attempt=0 time=2025-02-11T09:38:19.094+01:00 level=INFO msg="tests failed" ... time=2025-02-11T09:38:33.798+01:00 level=INFO msg="code optimized" attempt=9 optimization succeeded: 1.870953s -> 1.337585s (-28.51%)

That took about three minutes. It does all sorts of tricks to make it faster, but one that caught my eye was the conversion from HashMap<Integer, Boolean> to HashSet<Integer>, and finally to BitSet. Here’s part of the diff:

...  public class AchillesNumbers { - - private Map<Integer, Boolean> pps = new HashMap<>(); + private final BitSet pps = new BitSet(); + private final BitSet achillesCache = new BitSet(); + private static final byte[] SMALL_PRIMES = {2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47}; + private static final int[] POWERS_OF_TEN = {1, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000}; + private static final short[] SQUARES = new short[317]; + private static final int[] CUBES = new int[47]; + private static final short[] TOTIENTS = new short[1000];   ...  }

Unlearning

I’m surprised to find myself as excited as I am now. I did not see it coming! Just during the last few days, I’ve realized how much I need to unlearn in order to make better use of what these models have learned.

I was implementing control flow, using Claude to generate various bits of code for the converge tool. Then I realized that, hey, maybe it should be the other way around? Claude plans the control flow, and my tool just provides the ways of interacting with the environment (modifying source files, running tests, etc). It’s not a revelation, but an example of how one might need to think differently.

Even more down to earth, things like saying “do X for me” rather than asking “how do I do X?”. Instead of asking for some chunk of code, I tell it to solve a problem for me. Of course, I still review the changes. Cursor and Cody have both been great ways of changing my thinking.

What other habits and thought patterns might need to change? I don’t know how programming will look in the future, but I’m actively working on keeping an open mind and hopefully playing a small role in shaping it.

Comment on Hacker News or X.

This morphed into a technical challenge, while still having that creative aspect that I started with. Subsequent screenshots with the fixed grid and responsive tables sparked a lot of interest. About a week later I published the source, and since then there’s been a lot of forks. People use it for their personal web sites, mostly, but also for apps.

I’d like to explain how it works. Not everything, just focusing on the most interesting parts.

The Fixed Grid

This design aligns everything, horizontally and vertically, to a fixed grid. Like a table with equal-size cells. Every text character should be exactly contained in a cell in that grid. Borders and other visual elements may span cells more freely.

The big idea here is to use the ch unit in CSS. I actually did not know about it before this experiment. The ch unit is described in CSS Values and Units Module Level 4:

Represents the typical advance measure of European alphanumeric characters, and measured as the used advance measure of the “0” (ZERO, U+0030) glyph in the font used to render it.

Further, it notes:

This measurement is an approximation (and in monospace fonts, an exact measure) of a single narrow glyph’s advance measure, thus allowing measurements based on an expected glyph count.

Fantastic! A cell is thus 1ch wide. And the cell height is equal to the line-height. In order to refer to the line height in calculations, it’s extracted as a CSS variable:

:root {  --line-height: 1.2rem; }

So far there’s no actual grid on the page. This is just a way of measuring and sizing elements based on the width and height of monospace characters. Every element must take up space evenly divisible by the dimensions of a cell; if it’s top left corner starts at a cell, then its bottom right must do so as well. That means that all elements, given that their individual sizes respect the grid, line up properly.

The Font

I’ve chosen JetBrains Mono for the font. It looks great, sure, but there’s a more important reason for this particular choice: it has good support for box-drawing characters at greater line heights. Most monospace fonts I tried broke at line heights above 110% or so. Lines and blocks were chopped up vertically. With JetBrains Mono, I can set it to 120% before it starts to become choppy.

I suspect Pragmata Pro or Berkeley Mono might work in this regard, but I haven’t tried them yet.

Also, if you want to use this design and with a greater line height, you can probably do so if you don’t need box-drawing characters. Then you may also consider pretty much any monospace font. Why not use web-safe ones, trimming down the page weight by about 600kB!

To avoid alternate glyphs for numbers, keeping everything aligned to the grid, I set:

:root {  font-variant-numeric: tabular-nums lining-nums; }

And, for a unified thickness of borders, text, and underlines:

:root {  --border-thickness: 2px;  font-weight: 500;  text-decoration-thickness: var(--border-thickness); }

This gives the design that thick and sturdy feel.

The Body

The body element is the main container in the page. It is at most 80 characters wide. (Huh, I wonder where that number came from?)

Now for one of the key tricks! I wanted this design to be reasonably responsive. For a viewport width smaller than 84ch (80ch and 4ch of padding), the body width needs to be some smaller width that is still evenly divisible by the cell dimensions. This can be accomplished with CSS rounding:

body {  padding: var(--line-height) 2ch;  max-width: calc(min(80ch, round(down, 100%, 1ch))); }

This way, the body shrinks in steps according to the grid.

The Horizontal Rule

Surprisingly, the custom horizontal rule styling was a fiddly enterprise. I wanted it to feel heavy, with double lines. The lines are vertically centered around the break between two cells:

To respect the grid, the top and bottom spacing needs to be calculated. But padding won’t work, and margin interacts with adjacent elements’ margins, so this required two elements:

hr {  position: relative;  display: block;  height: var(--line-height);  margin: calc(var(--line-height) * 1.5) 0;  border: none;  color: var(--text-color); }  hr:after {  display: block;  content: "";  position: absolute;  top: calc(var(--line-height) / 2 - var(--border-thickness));  left: 0;  width: 100%;  border-top: calc(var(--border-thickness) * 3) double var(--text-color);  height: 0; }

The hr itself is just a container that takes up space; 4 lines in total. The hr:after pseudo-element draws the two lines, using border-top-style, at the vertical center of the hr.

The Table

Table styling was probably the trickiest. Recalling the principles from the beginning, every character must be perfectly aligned with the grid. But I wanted vertical padding of table cells to be half the line height. A full line height worth of padding is way too airy.

This requires the table being horizontally offset by half the width of a character, and vertically offset by half the line height.

table {  position: relative;  top: calc(var(--line-height) / 2);  width: calc(round(down, 100%, 1ch));  border-collapse: collapse; }

Cell padding is calculated based on cell size and borders, to keep grid alignment:

th, td {  border: var(--border-thickness) solid var(--text-color);  padding:   calc((var(--line-height) / 2))  calc(1ch - var(--border-thickness) / 2)  calc((var(--line-height) / 2) - (var(--border-thickness)))  ;  line-height: var(--line-height); }

Finally, the first row must have slightly less vertical padding to compensate for the top border. This is hacky, and would be nicer to solve with some kind of negative margin on the table. But then I’d be back in margin interaction land, and I don’t like it there.

table tbody tr:first-child > * {  padding-top: calc(  (var(--line-height) / 2) - var(--border-thickness)  ); }

Another quirk is that columns need to have set sizes. All but one should use the width-min class, and the remaining should use width-auto. Otherwise, cells divide the available width in a way that doesn’t align with the grid.

The Layout Grid

I also included a grid class for showcasing how a grid layout helper could work. Much like the 12-column systems found in CSS frameworks, but funkier. To use it, you simply slap on a grid class on a container. It uses a glorious hack to count the children in pure CSS:

.grid > * {  flex: 0 0 calc(  round(  down,   (100% - (1ch build-feed.sh build-index.sh flake.lock flake.nix generate-redirects.sh LICENSE Makefile README.md src target watch.sh (var(--grid-cells) - 1))) / var(--grid-cells),  1ch  )  ); } .grid:has(> :last-child:nth-child(1)) { --grid-cells: 1; } .grid:has(> :last-child:nth-child(2)) { --grid-cells: 2; } .grid:has(> :last-child:nth-child(3)) { --grid-cells: 3; } .grid:has(> :last-child:nth-child(4)) { --grid-cells: 4; } .grid:has(> :last-child:nth-child(5)) { --grid-cells: 5; } .grid:has(> :last-child:nth-child(6)) { --grid-cells: 6; } .grid:has(> :last-child:nth-child(7)) { --grid-cells: 7; } .grid:has(> :last-child:nth-child(8)) { --grid-cells: 8; } .grid:has(> :last-child:nth-child(9)) { --grid-cells: 9; }

Look at it go!

Unlike with tables, the layout grid rows don’t have to fill the width. Depending on your viewport width, you’ll see a ragged right margin. However, by setting flex-grow: 1; on one of the children, that one grows to fill up the remaining width.

The Media Elements

Images and video grow to fill the width. But they have their own proportions, making vertical alignment a problem. How many multiples of the line height should the height of the media be? I couldn’t figure out a way to calculate this with CSS alone.

One option was a preprocessor step that would calculate and set the ratio of every such element as a CSS variable, and then have CSS calculate a padding-bottom based on the ratio and the width:

<img style="--ratio: 0.821377" ... >

However, I eventually settled for small chunk of JavaScript to calculate the difference, and set an appropriate padding-bottom. Ending up in JavaScript was inevitable, I suppose.

Summary

There are many small things I haven’t shown in detail, including:

• the debug grid overlay, which you see in the screenshots
• ordered list numbering
• the tree-rendered list
• various form controls and buttons
• the custom details element

But I think I’ve covered the most significant bits. For a full tour, have a look at the source code.

There are still bugs, like alignment not working the same in all browsers, and not working at all in some cases. I’m not sure why yet, and I might try to fix it in at least Firefox and Chromium. Those are the ones I can test with easily.

This has been a fun project, and I’ve learned a bunch of new CSS tricks. Also, the amount of positive feedback has been overwhelming. Of course, there’s been some negative feedback as well, not to be dismissed. I do agree with the concern about legibility. Monospace fonts can be beautiful and very useful, but they’re probably not the best for prose or otherwise long bodies of text.

Some have asked for reusable themes or some form of published artifact. I won’t spend time maintaining such packages, but I encourage those who would. Fork it, tweak it, build new cool things with it!

Finally, I’ll note that I’m happy with how the overall feel of this design turned out, even setting aside the monospace aspect. Maybe it would work with a proportional, or perhaps semi-proportional, font as well?

In the eternal search of a better text editor, I’ve recently gone back to Neovim. I’ve taken the time to configure it myself, with as few plugins and other cruft as possible. My goal is a minimalist editing experience, tailored for exactly those tasks that I do regularly, and nothing more. In this post, I’ll give a brief tour of my setup and its motivations.

Over the years, I’ve been through a bunch of editors. Here are most of them, in roughly chronological order:

• Adobe Dreamweaver
• Sublime Text
• Atom
• Vim/Neovim
• IntelliJ IDEA
• VS Code
• Emacs

The majority were used specifically for the work I was doing at the time. VS Code for web development, IntelliJ IDEA for JVM languages, Emacs for Lisps. Vim and Emacs have been the most generally applicable, and the ones that I’ve enjoyed the most.

I’ve also evaluated Zed recently, but it hasn’t quite stuck. The start-up time and responsiveness is impressive. However, it seems to insist on being a global application. I want my editor to be local to each project, with the correct PATH from direnv, and I want multiple isolated instances to run simultaneously. Maybe I’ll revisit it later.

Returning to Neovim

Yeah, so I’m back in Neovim. I actually started with the LazyVim distribution based on recommendation. On the positive side, it got me motivated to use Neovim again. But I had some frustrations with the distribution.

The start-up time wasn’t great. I guess it did some lazy loading of plugins to speed things up, but the experience was still that of an IDE taking its time to get ready. Not the Neovim experience I was hoping for.

More importantly, it was full of distractions; popups, status messages, news, and plugins I didn’t need. I guess it takes a batteries-included approach. That might make sense for beginners and those just getting into Neovim, but I realized quickly that I wanted something different.

Supposedly I could strip things out, but instead I decided to start from scratch and build the editor I wanted. One that I understand. Joran Dirk Greef talks about two different types of artists, sculptors and painters, and this is an exercise in painting.

More concretely, my main goals are:

Plugins

I want to keep plugins to an absolute minimum. My editor is meant for coding and writing, and for what I work on right now. I might add or remove plugins and configuration over time, and that’s fine.

User interface

It should be as minimalist as I can make it. Visual distractions kept at a minimum. This includes busy colorschemes, which I find add little value. A basic set of typographical conventions in monochrome works well for me.

Start-up time

With the way I use Neovim, it needs to start fast. I quit and start it all the time, jumping between directories, working on different parts of the system or a project. Often I put it in the background with C-z, but not always. Making it faster seems to be mainly an exercise in minimizing plugins.

I have to mention Nix, you know?

I manage dotfiles and other personal configuration using Nix and home-manager. The Neovim configuration is no exception, so I’ll include some of the Nix bits as well.

The vim.nix module declares that I want Neovim installed and exposed as vim:

programs.neovim = {  enable = true;  vimAlias = true;  ... };

In that attribute set, there are two other important parts; the plugins list and the extraConfig.

Plugins

Let’s start with the plugins:

plugins = with pkgs.vimPlugins; [  nvim-lspconfig  (nvim-treesitter.withPlugins(p: [  p.bash  p.json  p.lua  p.markdown  p.nix  p.python  p.rust  p.zig  p.vimdoc  ]))  conform-nvim  neogit  fzf-vim ];

Basically it’s five plugins, not counting the various treesitter parsers:

nvim-lspconfig

LSP is included in Neovim nowadays, but it doesn’t know about specific language servers. The lspconfig plugins helps with configuring Neovim for use with various servers.

nvim-treesitter

This provides better highlighting for Neovim. I’ve only included the languages I use right now. No nice-to-haves. I could probably remove this plugin, but I haven’t tried yet.

conform-nvim

Auto-formatting is useful and I don’t want to think about it. Of course, I could run :%!whatever-formatter, but I’d rather have the editor do it for me.

neogit

Magit was the reason I clung to Emacs for so long. Neogit is, for my purposes, a worthy replacement. It enabled me to finally make the switch.

fzf-vim

A wrapper around the awesome fzf fuzzy finder. I use :Files and :GFiles, as quick jump-to-file commands (think C-p in VS Code or Zed). They are bound to <Leader>ff and <Leader>gf, respectively. This might be another plugin I could do without, writing a small helper around fzf, or just making do with :find and **/ wildcards. On the other hand, I’m trying out :Buffers instead of stumbling around with :bnext and :bprev.

One great thing with the Nix setup is I don’t need a package manager in Neovim itself.

Batteries Are Included

Many things I don’t need plugins for. For instance, there are a ton of plugins for auto-completion, but Neovim has most of that built in, and I prefer triggering it manually:

• File name completions with C-x C-f
• Omnicomplete (rebound to C-Space in my case)
• Buffer completions with C-x C-n
• Spelling suggestions with C-x C-s or z=

I’ve tried various snippet engines many times, but not found them very useful. Most of my time is spent reading or modifying existing code, not churning out new boilerplate. Instead they tend to clutter the auto-completion list. Snippets might make more sense for things like HTML, but I don’t write HTML often, and in that case I’d prefer some emmet/zen-coding plugin.

You can get great mileage from learning how to use the Quickfix list. I’m no expert, but I prefer investing in composable primitives that I can reuse in different ways. Project-wide search-and-replace is such an example:

:grep whatever :cfdo s/whatever/something else/g | update :cfdo :bd

Here we search (:grep, which I’ve configured to use rg), substitute and save each file, and delete those buffers afterwards.

I also use :make and :compiler a lot. Neovim is cool.

Life in Monochrome

Maybe I’m just growing old, but I prefer a monochrome colorscheme. Right now I’m using the built-in quiet scheme with a few extra tweaks:

set termguicolors set bg=dark colorscheme quiet highlight Keyword gui=bold highlight Comment gui=italic highlight Constant guifg=#999999 highlight NormalFloat guibg=#333333

It’s black-and-white, but keywords are bold, comments are darker and italic, and literals are light gray. Here’s how it looks with some Zig code:

Maybe I’m coming off as nostalgic or conservative, but I do find it more readable this way.

Another thing I’m going to try soon is writing on the Daylight Computer, hopefully in Neovim. Being comfortable with a monochrome colorscheme should come in handy.

The Full Configuration

My config uses a Vimscript entrypoint (extraConfig in the Nix code). This part is based on my near-immortal config from the good old Vim days. Early on, it calls vim.loader.enable() to improve startup time. I use Lua scripts for configuring the plugins and related keymap bindings. Maybe everything could be Lua, but I haven’t gotten that far yet. However, it’s nice to have the base config somewhat portable; I can just copy-paste it onto a server or some other temporary environment and have a decent /usr/bin/vim experience.

You’ll find the full configuration in nix.vim and the Lua bits inside the vim/ directory.

That’s about it! I’m really happy with how fast and minimalistic it is. It starts in just above 100ms. And I can understand all of my configuration (even if I don’t understand all of Neovim.) Perhaps I’ve spent more time on it than I’ve saved, but at least I’m happy so far.

I’m writing and publishing this on my birthday. What a treat to find time to blog on such an occasion!

What I’m describing in this post is a trade-off that I find comfortable to use in Python, especially with the new features that I’ll describe. Much of this works nicely in Kotlin and Typescript, too, with minor adaptions.

The principles I try to use are:

Declarative rather than imperative operations on data

Transform or fold data using map, reduce, or for-comprehensions instead of for-loops

Functions with destructuring pattern-matching to dispatch based on data

Use when statements rather than if instanceof(...) or inheritance and dynamic dispatch

Programs structured mainly around data and functions

Programs are trees of function invocations on data, rather than class hierarchies, dependency injection, and overuse of exceptions

Effects pushed to the outer layers of the program (hexagonal architecture)

Within reasonable bounds, functions in the guts of programs are pure and return data, whereas the outer layers interpret that data and manage effects (IO, non-determinism, etc)

This list is not exhaustive, but I’m trying to keep this focused. Also, I’m intentionally not taking this in the direction of Haskell, with typeclass hierarchies, higher-kinded types, and so on. I don’t believe cramming such constructs in would benefit Python programs in practice. Furthermore, I won’t be talking about effects and hexagonal architecture in this post.

The examples are all type-annotated and checked with Pyright. You could do all of this without static type-checking, as far as I know.

Finally, note that I consider myself a Python rookie, as I mostly use it for small tools and scripts. The largest program I’ve written in Python is Quickstrom. This post is meant to inspire and trigger new ideas, not to tell anyone how to write Python code.

All right, let’s get started and see what’s possible!

Preliminaries

First, let’s get some boilerplate stuff out of the way. I’m running Python 3.12. Some of the things I’ll show can be done with earlier versions, but not all.

We’ll not be using any external packages, only modules from the standard library:

from typing import * from dataclasses import dataclass

You might not want to use wildcard imports in more serious programs, but it’s acceptable for these examples.

Pattern Matching

Let’s start with a classic example from the functional programming world: an evaluator for a simple expression-based language. It only supports a few operations in order to keep it simple. First, we the different types of expressions there are using dataclasses and a union type:

type Expr = int | bool | str | BinOp | Let | If   @dataclass class BinOp:  op: Literal["<"] | Literal[">"]  lhs: Expr  rhs: Expr   @dataclass class Let:  name: str  value: Expr  body: Expr   @dataclass class If:  cond: Expr  t: Expr  f: Expr

The syntax for type aliases and the union type operator (|) are both new additions. You could create type aliases before using regular top-level bindings, but mutually recursive types required some types to be enclosed in strings. Otherwise, Python would complain that the second type (e.g. BinOp in the code above) wasn’t defined. It’s a bit cleaner now.

Note that we existing primitive types from Python (int, bool, and str), combined with dataclasses for complex expressions. The str is interpreted as a reference to a name bound in the lexical scope, not as a string literal, as we’ll see in the following snippet.

The evaluator tracks bindings in the Env. Evaluating an expression results in a value that is either an integer or a boolean.

type Env = dict[str, Value]  type Value = int | bool

Now, let’s look at the eval function. Here we pattern-match on the expression, which is a union. For literals, we just return the value:

def eval(env: Env, expr: Expr) -> Value:  match expr:  case int() | bool():  return expr  ...

References are looked up in the environment:

case str():  return env[expr]

Let-bindings create a new environment with the new binding:

case Let(name, value, body):  new_env = env | {name: eval(env, value)}  return eval(new_env, body)

Finally, the BinOp and If branches pattern-match on the evaluated nested expressions to make sure they’re of the correct types:

case BinOp(op, lhs, rhs):  l = eval(env, lhs)  r = eval(env, rhs)  match op, l, r:  case "<", int(), int():  return l < r  case ">", int(), int():  return l > r  case _:  raise ValueError(  f"Invalid binary operation {op} on {lhs} and {rhs}"  )  case If(cond, t, f):  match eval(env, cond):  case True:  return eval(env, t)  case False:  return eval(env, f)  case c:  raise ValueError(f"Expected bool condition, got: {c}")

All right, let’s try it out:

>>> example = Let("x", 1, If(BinOp("<", "x", 2), 42, 0)) >>> eval({}, example) 42

Nice! But this is far from a robust evaluator. If we run it with a deep enough expression, we’d get a RecursionError saying that the maximum recursion depth was exceeded. This is a commonly occurring problem when writing recursive functions in Python.¹ The eval function could be rewritten with an explicit stack for operations and operands, but it’s a bit fiddly.

In some cases, you can restructure a recursive function as tail-recursive, and then manually convert it to a loop. Perhaps you could automatically optimize tail-calls, or use a trampoline. Some solutions avoid stack overflows, at the expense of increased heap memory usage. In simpler cases, combinators like map and reduce might suffice, instead of explicit recursion.

Either way, recursive functions and stack overflow is something to watch out for.

Generics

Since Python 3.12, it’s also much nicer to work with generic types. Previously, you had to define type variables before using them in type signatures. This felt very awkward to me.

Let’s look at an example that models a rose tree data type. To spice it up a little, I’m including a map function for both types of the tree nodes. This is equivalent to fmap in Haskell, but without the typeclass and higher-order types.

type RoseTree[T] = Branch[T] | Leaf[T]   @dataclass class Branch[A]:  branches: list[RoseTree[A]]   def map[B](self, f: Callable[[A], B]) -> Branch[B]:  return Branch([b.map(f) for b in self.branches])   @dataclass class Leaf[A]:  value: A   def map[B](self, f: Callable[[A], B]) -> Leaf[B]:  return Leaf(f(self.value))

Let’s print these trees using pattern matching. Here’s a function that’s written as a loop, maintaining a list of remaining sub-trees to print:

def print_tree[T](tree: RoseTree[T]):  trees = [(tree, 0)]  while trees:  match trees.pop(0):  case Branch(branches), level:  print(" " * level * 2 + "*")  trees = [(branch, level + 1) for branch in branches] + trees  case Leaf(value), level:  print(" " * level * 2 + "- " + repr(value))

It could be even simpler using plain recursion, but then we could run into stack depth issues again. Anyway, let’s try it out:

example = Branch(  [  Leaf(1),  Leaf(2),  Branch([Leaf(3), Leaf(4)]),  Branch([Leaf(5), Leaf(6)]),  Leaf(7),  ] ) >>> print_tree(example.map(str)) *  - '1'  - '2'  *  - '3'  - '4'  *  - '5'  - '6'  - '7'

As you can see from the repr being printed, all the values are mapped to strings.

Protocols

As a last example, I’d like to show how you can do basic structural subtyping using protocols. This is useful in cases where you don’t want to define all variants of a union in a single place. For instance, you might have many types of events that can be emitted in an application. Centrally defining each type of event adds unwanted coupling. Breaking apart a base class for events and the code that later on consumes the events decreases cohesion. In such cases, a protocol might be a better option.

We’ll need a new import:

from datetime import datetime

Now, consider the following events in module A:

@dataclass class Increment[Time]:  id: str  time: Time   def description(self: Self) -> str:  return "Incremented counter"   @dataclass class Reset[Time]:  id: str  time: Time   def description(self: Self) -> str:  return "Reset counter"

We made them generic just to showcase the combination of protocols and generics. The Time type parameter isn’t instantiated in any other way than datetime in this example.

In another module B — that doesn’t depend on A, and isn’t depended upon by A — the protocol is defined, along with the log_event function:

class Event[Time](Protocol):  id: str  time: Time   def description(self: Self) -> str: ...  def log_event(event: Event[datetime]):  print(f"Got {event.id} at {event.time}: {event.description()}")

Increment and Decrement both implement the Event protocol by virtue of being structurally compatible. They can both be passed to log_event:

log_event(  Increment("foo", datetime.now()), )

If we annotate the Event protocol with @runtime_checkable, we can check it with isinstance and use it in match cases:

@runtime_checkable class Event[Time](Protocol):  ...  def log(x: Any):  match x:  case Event() if isinstance(datetime, x.time):  log_event(x)  case _:  print(x)

Pretty neat!

That’s all I have for now. Maybe more Python hacking and blog posts will pop up if there’s interest. I’m positive to the evolution of Python and functional programming, as it’s something I use quite regularly.

Join the discussion on Twitter, Hacker News, or Lobsters.

Thank you @tusharisanerd for reviewing a draft of this post.

This post focuses on how to use LTL to specify systems in terms of state machines. It’s a brief overview that avoids going into too much detail. For more information on how to test web applications using such specifications, see the Quickstrom documentation.

To avoid possible confusion, I want to start by pointing out that a state machine specification in this context is not the same as a model in TLA+ (or similar modeling languages.) We’re not building a model to prove or check properties against. Rather, we’re defining properties in terms of state machine transitions, and the end goal is to test actual system behavior (e.g. web applications, desktop applications, APIs) by checking that recorded traces match our specifications.

Linear Temporal Logic

In this post, we’ll be using an LTL language. It’s a sketch of a future specification language for Quickstrom.

A formula (plural formulae) is a logical expression that evaluates to true or false. We have the constants:

• true
• false

We combine formulae using the logical connectives, e.g:

• &&
• ||
• not
• ==>

The ==> operator is implication. So far we have propositional logic, but we need a few more things.

Temporal Operators

At the core of our language we have the notion of state. Systems change state over time, and we’d like to express that in our specifications. But the formulae we’ve seen so far do not deal with time. For that, we use temporal operators.

To illustrate how the temporal operators work, I’ll use diagrams to visualize traces (sequences of states). A filled circle (●) denotes a state in which the formula is true, and an empty circle (○) denotes a state where the formula is false.

For example, let’s say we have two formulae, P and Q, where:

• P is true in the first and second state
• Q is true in the second state

Both formulae are false in all other states. The formulae and trace would be visualized as follows:

P ●───●───○ Q ○───●───○

Note that in these diagrams, we assume that the last state repeats forever. This might seem a bit weird, but drawing an infinite number of states is problematic.

All of the examples explaining operators have links to the Linear Temporal Logic Visualizer, in which you can interactively experiment with the formulae. The syntax is not the same as in the article, but hopefully that’s not a problem.

Next

The next operator takes a formula as an argument and evaluates it in the next state.

next P ●───○───○ P ○───●───○

The next operator is relative to the current state, not the first state in the trace. This means that we can nest nexts to reach further into the future.

next (next P) ●───●───○───○───○ next P ○───●───●───○───○ P ○───○───●───●───○

Next for State Transitions

All right, time for a more concrete example, something we’ll evolve throughout this post. Let’s say we have a formula gdprConsentIsVisible which is true when the GDPR consent screen is visible. We specify that the screen should be visible in the current and next state like so:

gdprConsentIsVisible && next gdprConsentIsVisible

A pair of consecutive states is called a step. When specifying state machines, we use the next operator to describe state transitions. A state transition formula is a logical predicate on a step.

In the GDPR example above, we said that the consent screen should stay visible in both states of the step. If we want to describe a state change in the consent screen’s visibility, we can say:

gdprConsentIsVisible && next (not gdprConsentIsVisible)

The formula describes a state transition from a visible to a hidden consent screen.

Always

But interesting state machines usually have more than one possible transition, and interesting behaviors likely contain multiple steps.

While we could nest formulae containing the next operator, we’d be stuck with specifications only describing a finite number of transitions.

Consider the following, where we like to state that the GDPR consent screen should always be visible:

gdprConsentIsVisible && next (gdprConsentIsVisible && next ...)

This doesn’t work for state machines with cycles, i.e. with possibly infinite traces, because we can only nest a finite number of next operators. We want state machine specifications that describe any number of transitions.

This is where we pick up the always operator. It takes a formula as an argument, and it’s true if the given formula is true in the current state and in all future states.

always P ●───●───●───●───● P ●───●───●───●───● always Q ○───○───●───●───● Q ●───○───●───●───●

Note how always Q is true in the third state and onwards, because that’s when Q becomes true in the current and all future states.

Let’s revisit the always-visible consent screen specification. Instead of trying to nest an infinite amount of next formulae, we instead say:

always gdprConsentIsVisible

Neat! This is called an invariant property. Invariants are assertions on individual states, and an invariant property says that it must hold for every state in the trace.

Always for State Machines

Now, let’s up our game. To specify the system as a state machine, we can combine transitions with disjunction (||) and the always operator. First, we define the individual transition formulae open and close:

let open = not gdprConsentIsVisible && next gdprConsentIsVisible; let close = gdprConsentIsVisible && next (not gdprConsentIsVisible);

Our state machine formula says that it always transitions as described by open or close:

always (open || close)

We have a state machine specification! Note that this specification only allows for transitions where the visibility of the consent screen changes back and forth.

So far we’ve only seen examples of safety properties. Those are properties that specify that “nothing bad happens.” But we also want to specify that systems somehow make progress. The following two temporal operators let us specify liveness properties, i.e. “good things eventually happen.”

Quickstrom does not support liveness properties yet.¹

Eventually

We’ve used next to specify transitions, and always to specify invariants and state machines. But we might also want to use liveness properties in our specifications. In this case, we are not talking about specific steps, but rather goals.

The temporal operator eventually takes a formula as an argument, and it’s true if the given formula is true in the current or any future state.

eventually P ○───○───○───○───○ P ○───○───○───○───○ eventually Q ●───●───●───●───○ Q ○───○───○───●───○

For instance, we could say that the consent screen should initially be visible and eventually be hidden:

gdprConsentIsVisible && eventually (not gdprConsentIsVisible)

This doesn’t say that it stays hidden. It may become visible again, and our specification would allow that. To specify that it should stay hidden, we use a combination of eventually and always:

gdprConsentIsVisible && eventually (always (not gdprConsentIsVisible))

Let’s look at a diagram to understand this combination of temporal operators better:

eventually (always P) ○───○───○───○───○ P ○───○───●───●───○ eventually (always Q) ●───●───●───●───● Q ○───○───●───●───●

The formula eventually (always P) is not true in any state, because P never starts being true forever. The other formula, eventually (always Q), is true in all states because Q becomes true forever in the third state.

Until

The last temporal operator I want to discuss is until. For P until Q to be true, P must be true until Q becomes true.

P until Q ●───●───●───●───○ P ●───●───○───○───○ Q ○───○───●───●───○

Just as with the eventually operator, the stop condition (Q) doesn’t have to stay true forever, but it has to be true at least once.

The until operator is more expressive than always and eventually, and they can both be defined using until.²

Anyway, let’s get back to our running example. Suppose we have another formula supportChatVisible that is true when the support chat button is shown. We want to make sure it doesn’t show up until after the GDPR consent screen is closed:

not supportChatVisible until not gdprConsentIsVisible

The negations make it a bit harder to read, but it’s equivalent to the informal statement: “the support chat button is hidden at least until the GDPR consent screen is hidden.” It doesn’t demand that the support chat button is ever visible, though. For that, we instead say:

gdprConsentIsVisible until (supportChatVisible && not gdprConsentIsVisible)

In this formula, supportChatVisible has to become true eventually, and at that point the consent screen must be hidden.

Until for State Machines

We can use the until operator to define a state machine formula where the final transition is more explicit.

Let’s say we want to specify the GDPR consent screen more rigorously. Suppose we already have the possible state transition formulae defined:

• allowCollectedData
• disallowCollectedData
• submit

We can then put together the state machine formula:

let gdprConsentStateMachine = gdprConsentIsVisible && (allowCollectedData || disallowCollectedData) until (submit && next (not gdprConsentIsVisible));

In this formula we allow any number of allowCollectedData or disallowCollectedData transitions, until the final submit resulting in a closed consent screen.

What’s next?

We’ve looked at some temporal operators in LTL, and how to use them to specify state machines. I’m hoping this post has given you some ideas and inspiration!

Another blog post worth checking out is TLA+ Action Properties by Hillel Wayne. It’s written specifically for TLA+, but most of the concepts are applicable to LTL and Quickstrom-style specifications.

I intend to write follow-ups, covering atomic propositions, queries, actions, and events. If you want to comment, there are threads on GitHub, Twitter, and on Lobsters. You may also want to sponsor my work.

Thank you Vitor Enes, Andrey Mokhov, Pascal Poizat, and Liam O’Connor for reviewing drafts of this post.

Edits

• 2021-05-28: Added links to the Linear Temporal Logic Visualizer matching the relevant examples. Note that the syntax is different in the visualizer.

Footnotes

What is Quickstrom?

Quickstrom is a new autonomous testing tool for the web. It can find problems in any type of web application that renders to the DOM. Quickstrom automatically explores your application and presents minimal failing examples. Focus your effort on understanding and specifying your system, and Quickstrom can test it for you.

Past and future

I started writing Quickstrom on April 2, 2020, about a week after our first child was born. Somehow that code compiled, and evolved into a capable testing tool. I’m now happy and excited to share it with everyone!

In the future, when Quickstrom is more robust and has a greater mileage, I might build a commercial product on top of it. This one of the reasons I’ve chosen an AGPL-2.0 license for the code, and why contributors must sign a CLA before pull requests can be merged. The idea is to keep the CLI test runner AGPL forever, but I might need a license exception if I build a closed-source SaaS product later on.

Learning more

Interested in Quickstrom? Start by checking out any of these resources:

• Main website
• Project documentation, including installation instructions and usage guides
• Source code

And keep an eye out for updates by signing up for the newsletter, or by following me on Twitter. Documentation should be significantly improved soon.

Comments

If you have any comments or questions, please reply to any of the following threads:

• Twitter
• Lobste.rs

WebCheck

During the last three months I’ve used my spare time to build WebCheck. It’s a browser testing framework combining ideas from:

• property-based testing (PBT)
• TLA+ and linear temporal logic
• functional programming

In WebCheck, you write a specification for your web application, instead of manually writing test cases. The specification describes the intended behavior as a finite-state machine and invariants, using logic formulae written in a language inspired by the temporal logic of actions (PDF) from TLA+. WebCheck generates hundreds or thousands of tests and runs them to verify if your application is accepted by the specification.

The tests generated by WebCheck are sequences of possible actions, determined from the current state of the DOM at the time of each action. You can think of WebCheck as exploring the state space automatically, based on DOM introspection. It can find user behaviors and problems that we, the biased and lazy humans, are unlikely to think of and to write tests for. Our job is instead to think about the requirements and to improve the specification over time.

Specifications vs Models

In property-based testing, when testing state machines using models, the model should capture the essential complexity of the system under test (SUT). It needs to be functionally complete to be a useful oracle. For a system that is conceptually simple, e.g. a key-value database engine, this is not a problem. But for a system that is inherently complex, e.g. a business application with a big pile of rules, a useful model tends to grow as complex as the system itself.

In WebCheck, the specification is not like such a model. You don’t have to implement a complete functional model of your system. You can leave out details and specify only the most important aspects of your application. As an example, I wrote a specification that states that “there should at no time be more than one spinner on a page”, and nothing else. Again, this is possible to specify in PBT in general, but not with model-based PBT, from what I’ve seen.

TodoMVC as a Benchmark

Since the start of this project, I’ve used the TodoMVC implementations as a benchmark of WebCheck, and developed a general specification for TodoMVC implementations. The TodoMVC contribution documentation has a high-level feature specification, and the project has a Cypress test suite, but I was curious if I could find anything new using WebCheck.

Early on, checking the mainstream framework implementations, I found that both the Angular and Mithril implementations were rejected by my specification, and I submitted an issue in the TodoMVC issue tracker. Invigorated by the initial success, I decided to check the remaining implementations and gradually improve my specification.

I’ve generalized the specification to work on nearly all the implementations listed on the TodoMVC website. Some of them use the old markup, which uses IDs instead of classes for most elements, so I had to support both variants.

The Specification

Before looking at the tests results, you might want to have a look at the WebCheck specification that I’ve published as a gist:

TodoMVC.spec.purs

The gist includes a brief introduction to the WebCheck specification language and how to write specifications. I’ll write proper documentation for the specification language eventually, but this can give you a taste of how it works, at least. I’ve excluded support for the old TodoMVC markup to keep the specification as simple as possible.

The specification doesn’t cover all features of TodoMVC yet. Most notably, it leaves out the editing mode entirely. Further, it doesn’t cover the usage of local storage in TodoMVC, and local storage is disabled in WebCheck for now.

I might refine the specification later, but I think I’ve found enough to motivate using WebCheck on TodoMVC applications. Further, this is likely how WebCheck would be used in other projects. You specify some things and you leave out others.

The astute reader might have noticed that the specification language looks like PureScript. And it pretty much is PureScript, with some WebCheck-specific additions for temporal modalities and DOM queries. I decided not to write a custom DSL, and instead write a PureScript interpreter. That way, specification authors can use the tools and packages from the PureScript ecosystem. It works great so far!

Test Results

Below are the test results from running WebCheck and my TodoMVC specification on the examples listed on the TodoMVC website. I’ll use short descriptions of the problems (some of which are recurring), and explain in more detail further down.

✓ Passed, ❌ Failed, – Not testable

Filters not implemented

There’s no way of switching between “All”, “Active”, and “Completed” items. This is specified in the TodoMVC documentation under Routing.

Race condition during initialization

The event listeners are attached some time after the .new-todo form is rendered. Although unlikely, if you’re quick enough you can focus the input, press Return, and post the form. This will navigate the user agent to the same page but with a query parameter, e.g. index.html?text=. In TodoMVC it’s not the end of the world, but there are systems where you do not want this to happen.

Inconsistent first render

The application briefly shows an inconsistent view, then renders the valid initial state. KnockoutJS + RequireJS shows an empty list items and “0 left” in the bottom, even though the footer should be hidden when there are no items.

Needs a custom readyWhen condition

The specification awaits an element matching .todoapp (or #todoapp for the old markup) in the DOM before taking any action. In this case, the implementation needs a modified specification that instead awaits a framework-specific class, e.g. .ng-scope. This is an inconvenience in testing the implementation using WebCheck, rather than an error.

No input field

There’s no input field to enter TODO items in. I’d argue this defeats the purpose of a TODO list application, and it’s indeed specified in the official documentation.

Adds pending item on other interaction

When there’s a pending item in the input field, and another action is taken (toggle all, change filter, etc), the pending item is submitted automatically without a Return key press.

.todo-count strong element is missing

An element matching the selector .todo-count strong must be present in the DOM when there are items, showing the number of active items, as described in the TodoMVC documentation.

State cannot be cleared

This is not an implementation error, but an issue where the test implementation makes it hard to perform repeated isolated testing. State cannot (to my knowledge) be cleared between tests, and so isolation is broken. This points to a key requirement currently placed by WebCheck: the SUT must be stateless, with respect to a new private browser window. In future versions of WebCheck, I’ll add hooks to let the tester clear the system state before each test is run.

Missing/broken link

The listed implementation seems to be moved or decommissioned.

Note that I can’t decide which of these problems are bugs. That’s up to the TodoMVC project maintainers. I see them as problems, or disagreements between the implementations and my specification. A good chunk of humility is in order when testing systems designed and built by others.

The Future is Bright

I’m happy with how effective WebCheck has been so far, after only a few months of spare-time prototyping. Hopefully, I’ll have something more polished that I can make available soon. An open source tool that you can run yourself, a SaaS version with a subscription model, or maybe both. When that time comes, maybe WebCheck can be part of the TodoMVC project’s testing. And perhaps in your project?

If you’re interested in WebCheck, please sign up for the newsletter. I’ll post regular project updates, and definitely no spam. You can also follow me on Twitter.

Comments

If you have any comments or questions, please reply to any of the following threads:

• Twitter
• Lobste.rs

Thanks to Hillel Wayne, Felix Holmgren, Martin Janiczek, Tom Harding, and Martin Clausen for reviewing drafts of this post.

This tutorial was originally written as a book chapter, and later extracted as a standalone piece. Since I’m not expecting to finish the PBT book any time soon, I decided to publish the chapter here.

System Under Test: User Signup Validation

The business logic we’ll test is the validation of a website’s user signup form. The website requires users to sign up before using the service. When signing up, a user must pick a valid username. Users must be between 18 and 150 years old.

Stated formally, the validation rules are:

$$ \begin{aligned} 0 \leq \text{length}(\text{name}) \leq 50 \\ 18 \leq \text{age} \leq 150 \end{aligned} \qquad(1)$$

The signup and its validation is already implemented by previous programmers. There have been user reports of strange behaviour, and we’re going to locate and fix the bugs using property tests.

Poking around the codebase, we find the data type representing the form:

data SignupForm = SignupForm  { formName :: Text  , formAge :: Int  } deriving (Eq, Show)

And the existing validation logic, defined as validateSignup. We won’t dig into to the implementation yet, only its type signature:

validateSignup  :: SignupForm -> Validation (NonEmpty SignupError) Signup

It’s a pure function, taking SignupForm data as an argument, and returning a Validation value. In case the form data is valid, it returns a Signup data structure. This data type resembles SignupForm in its structure, but refines the age as a Natural when valid:

data Signup = Signup  { name :: Text  , age :: Natural  } deriving (Eq, Show)

In case the form data is invalid, validateSignup returns a non-empty list of SignupError values. SignupError is a union type of the possible validation errors:

data SignupError  = NameTooShort Text  | NameTooLong Text  | InvalidAge Int  deriving (Eq, Show)

The Validation Type

The Validation type comes from the validation package. It’s parameterized by two types:

1. the type of validation failures
2. the type of a successfully validated value

The Validation type is similar to the Either type. The major difference is that it accumulates failures, rather than short-circuiting on the first failure. Failures are accumulated when combining multiple Validation values using Applicative.

Using a non-empty list for failures in the Validation type is common practice. It means that if the validation fails, there’s at least one error value.

Validation Property Tests

Let’s add some property tests for the form validation, and explore the existing implementation. We begin in a new test module, and we’ll need a few imports:

import Data.List.NonEmpty (NonEmpty (..)) import Data.Text (Text) import Data.Validation import Hedgehog import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range

Also, we’ll need to import the implementation module:

import Validation

We’re now ready to define some property tests.

A Positive Property Test

The first property test we’ll add is a positive test. That is, a test using only valid input data. This way, we know the form validation should always be successful. We define prop_valid_signup_form_succeeds:

prop_valid_signup_form_succeeds = property $ do  let genForm = SignupForm <$> validName <*> validAge ➊  form <- forAll genForm ➋   case validateSignup form of ➌  Success{} -> pure ()  Failure failure' -> do  annotateShow failure'  failure

First, we define genForm (➊), a generator producing form data with valid names and ages. Next, we generate form values from our defined generator (➋). Finally, we apply the validateSignup function and pattern match on the result (➌):

• In case it’s successful, we have the test pass with pure ()
• In case it fails, we print the failure' and fail the test

The validName and validAge generators are defined as follows:

validName :: Gen Text validName = Gen.text (Range.linear 1 50) Gen.alphaNum  validAge :: Gen Int validAge = Gen.integral (Range.linear 18 150)

Recall the validation rules (eq. 1). The ranges in these generators yielding valid form data are defined precisely in terms of the validation rules.

The character generator used for names is alphaNum, meaning we’ll only generate names with alphabetic letters and numbers. If you’re comfortable with regular expressions, you can think of genValidName as producing values matching [a-zA-Z0-9]+.

Let’s run some tests:

λ> check prop_valid_signup_form_succeeds ✓ <interactive> passed 100 tests.

Hooray, it works.

Negative Property Tests

In addition to the positive test, we’ll add negative tests for the name and age, respectively. Opposite to positive tests, our negative tests will only use invalid input data. We can then expect the form validation to always fail.

First, let’s test invalid names.

prop_invalid_name_fails = property $ do  let genForm = SignupForm <$> invalidName <*> validAge ➊  form <- forAll genForm   case validateSignup form of ➋  Failure (NameTooLong{} :| []) -> pure ()  Failure (NameTooShort{} :| []) -> pure ()  other -> do ➌  annotateShow other  failure

Similar to our the positive property test, we define a generator genForm (➊). Note that we use invalidName instead of validName.

Again, we pattern match on the result of applying validateSignup (➋). In this case we expect failure. Both NameTooLong and NameTooShort are expected failures. If we get anything else, the test fails (➌).

The test for invalid age is similar, expect we use the invalidAge generator, and expect only InvalidAge validation failures:

prop_invalid_age_fails = property $ do  let genForm = SignupForm <$> validName <*> invalidAge  form <- forAll genForm  case validateSignup form of  Failure (InvalidAge{} :| []) -> pure ()  other -> do  annotateShow other  failure

The invalidName and invalidAge generators are also defined in terms of the validation rules (eq. 1), but with ranges ensuring no overlap with valid data:

invalidName :: Gen Text invalidName =  Gen.choice [mempty, Gen.text (Range.linear 51 100) Gen.alphaNum]  invalidAge :: Gen Int invalidAge = Gen.integral (Range.linear minBound 17)

Let’s run our new property tests:

λ> check prop_invalid_name_fails ✓ <interactive> passed 100 tests. λ> check prop_invalid_age_fails ✓ <interactive> passed 100 tests.

All good? Maybe not. The astute reader might have noticed a problem with one of our generators. We’ll get back to that later.

Accumulating All Failures

When validating the form data, we want all failures returned to the user posting the form, rather than returning only one at a time. The Validation type accumulates failures when combined with Applicative, which is exactly what we want. Yet, while the hard work is handled by Validation, we still need to test that we’re correctly combining validations in validateSignup.

We define a property test generating form data, where all fields are invalid (➊). It expects the form validation to fail, returning two failures (➋).

prop_two_failures_are_returned = property $ do  let genForm = SignupForm <$> invalidName <*> invalidAge ➊  form <- forAll genForm  case validateSignup form of  Failure failures | length failures == 2 -> pure () ➋  other -> do  annotateShow other  failure

This property is weak. It states nothing about which failures should be returned. We could assert that the validation failures are equal to some expected list. But how do we know if the name is too long or too short? I’m sure you’d be less thrilled if we replicated all of the validation logic in this test.

Let’s define a slightly stronger property. We pattern match, extract the two failures (➊), and check that they’re not equal (➋).

prop_two_different_failures_are_returned = property $ do  let genForm = SignupForm <$> invalidName <*> invalidAge  form <- forAll genForm  case validateSignup form of  Failure (failure1 :| [failure2]) -> ➊  failure1 /== failure2 ➋  other -> do  annotateShow other  failure

We’re still not being specific about which failures should be returned. But unlike prop_two_failures_are_returned, this property at least makes sure there are no duplicate failures.

The Value of a Property

Is there a faulty behaviour that would slip past prop_two_different_failures_are_returned? Sure. The implementation could have a typo or copy-paste error, and always return NameTooLong failures, even if the name is too short. Does this mean our property is bad? Broken? Useless?

In itself, this property doesn’t give us strong confidence in the correctness of validateSignup. In conjuction with our other properties, however, it provides value. Together they make up a stronger test suite.

Let’s look at it in another way. What are the benefits of weaker properties over stronger ones? In general, weak properties are beneficial in that they are:

1. easier to define
2. likely to catch simple mistakes early
3. less coupled to the SUT

A small investment in a set of weak property tests might catch a lot of mistakes. While they won’t precisely specify your system and catch the trickiest of edge cases, their power-to-weight ratio is compelling. Moreover, a set of weak properties is better than no properties at all. If you can’t formulate the strong property you’d like, instead start simple. Lure out some bugs, and improve the strength and specificity of your properties over time.

Coming up with good properties is a skill. Practice, and you’ll get better at it.

Testing Generators

Remember how in Negative Property Tests we noted that there’s a problem? The issue is, we’re not covering all validation rules in our tests. But the problem is not in our property definitions. It’s in one of our generators, namely genInvalidAge. We’re now in a perculiar situation: we need to test our tests.

One way to test a generator is to define a property specifically testing the values it generates. For example, if we have a generator positive that is meant to generate only positive integers, we can define a property that asserts that all generated integers are positive:

positive :: Gen Int positive = Gen.integral (Range.linear 1 maxBound)  prop_integers_are_positive = property $ do  n <- forAll positive  assert (n >= 1)

We could use this technique to check that all values generated by validAge are valid. How about invalidAge? Can we check that it generates values such that all boundaries of our validation function are hit? No, not using this technique. Testing the correctness of a generator using a property can only find problems with individual generated values. It can’t perform assertions over all generated values. In that sense, it’s a local assertion.

Instead, we’ll find the generator problem by capturing statistics on the generated values and performing global assertions. Hedgehog, and a few other PBT frameworks, can measure the occurences of user-defined labels. A label in Hedgehog is a Text value, declared with an associated condition. When Hedgehog runs the tests, it records the percentage of tests in which the condition evaluates to True. After the test run is complete, we’re presented with a listing of percentages per label.

We can even have Hedgehog fail the test unless a certain percentage is met. This way, we can declare mininum coverage requirements for the generators used in our property tests.

Adding Coverage Checks

Let’s check that we generate values covering enough cases, based on the validation rules in eq. 1 . In prop_invalid_age_fails, we use cover to ensure we generate values outside the boundaries of valid ages. 5% is enough for each, but realistically they could both get close to 50%.

prop_invalid_age_fails = property $ do  let genForm = SignupForm <$> validName <*> invalidAge  form <- forAll genForm  cover 5 "too young" (formAge form <= 17)  cover 5 "too old" (formAge form >= 151)  case validateSignup form of  Failure (InvalidAge{} :| []) -> pure ()  other -> do  annotateShow other  failure

Let’s run some tests again.

λ> check prop_invalid_age_fails ✗ <interactive> failed after 100 tests. too young 100% ████████████████████ ✓ 5% ⚠ too old 0% ···················· ✗ 5% ┏━━ test/Validation/V1Test.hs ━━━ 63 ┃ prop_invalid_age_fails = property $ do 64 ┃ let genForm = SignupForm <$> validName <*> invalidAge 65 ┃ form <- forAll genForm 66 ┃ cover 5 "too young" (formAge form <= 17) 67 ┃ cover 5 "too old" (formAge form >= 151) ┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ┃ │ Failed (0% coverage) 68 ┃ case validateSignup form of 69 ┃ Failure (InvalidAge{} :| []) -> pure () 70 ┃ other -> do 71 ┃ annotateShow other 72 ┃ failure Insufficient coverage.

100% too young and 0% too old. The invalidAge generator is clearly not good enough. Let’s have a look at its definition again:

invalidAge :: Gen Int invalidAge = Gen.integral (Range.linear minBound 17)

We’re only generating invalid ages between the minimum bound of Int and 17. Let’s fix that, by using Gen.choice and another generator for ages greater than 150:

invalidAge :: Gen Int invalidAge = Gen.choice  [ Gen.integral (Range.linear minBound 17)  , Gen.integral (Range.linear 151 maxBound)  ]

Running tests again, the coverage check stops complaining. But there’s another problem:

λ> check prop_invalid_age_fails ✗ <interactive> failed at test/Validation/V1Test.hs:75:7 after 3 tests and 2 shrinks. too young 67% █████████████▎······ ✓ 5% ⚠ too old 0% ···················· ✗ 5% ┏━━ test/Validation/V1Test.hs ━━━ 66 ┃ prop_invalid_age_fails = property $ do 67 ┃ let genForm = SignupForm <$> validName <*> invalidAge 68 ┃ form <- forAll genForm ┃ │ SignupForm { formName = "a" , formAge = 151 } 69 ┃ cover 5 "too young" (formAge form <= 17) 70 ┃ cover 5 "too old" (formAge form >= 151) 71 ┃ case validateSignup form of 72 ┃ Failure (InvalidAge{} :| []) -> pure () 73 ┃ other -> do 74 ┃ annotateShow other ┃ │ Success Signup { name = "a" , age = 151 } 75 ┃ failure ┃ ^^^^^^^

OK, we have an actual bug. When the age is 151 or greater, the form is deemed valid. It should cause a validation failure. Looking closer at the implementation, we see that a pattern guard is missing the upper bound check:

 validateAge age' | age' >= 18 = Success (fromIntegral age')  | otherwise = Failure (pure (InvalidAge age'))

If we change it to age' >= 18 && age' <= 150, and rerun the tests, they pass.

λ> check prop_invalid_age_fails ✓ <interactive> passed 100 tests. too young 53% ██████████▌········· ✓ 5% too old 47% █████████▍·········· ✓ 5%

We’ve fixed the bug.

Measuring and declaring requirements on coverage is a powerful tool in Hedgehog. It gives us visibility into the generative tests we run, making it practical to debug generators. It ensures our tests meet our coverage requirements, even as implementation and tests evolve over time.

From Ages to Birth Dates

So far, our efforts have been successful. We’ve fixed real issues in both implementation and tests. Management is pleased. They’re now asking us to modify the signup system, and use our testing skills to ensure quality remains high.

Instead of entering their age, users will enter their birth date. Let’s suppose this information is needed for something important, like sending out birthday gifts. The form validation function must be modified to check, based on the supplied birth date date, if the user signing up is old enough.

First, we import the Calendar module from the time package:

import Data.Time.Calendar

Next, we modify the SignupForm data type to carry a formBirthDate of type Date, rather than an Int.

data SignupForm = SignupForm  { formName :: Text  , formBirthDate :: Day  } deriving (Eq, Show)

And we make the corresponding change to the Signup data type:

data Signup = Signup  { name :: Text  , birthDate :: Day  } deriving (Eq, Show)

We’ve also been requested to improve the validation errors. Instead of just InvalidAge, we define three constructors for various invalid birthdates:

data SignupError  = NameTooShort Text  | NameTooLong Text  | TooYoung Day  | TooOld Day  | NotYetBorn Day  deriving (Eq, Show)

Finally, we need to modify the validateSignup function. Here, we’re faced with an important question. How should the validation function obtain today’s date?

Keeping Things Deterministic

We could make validateSignup a non-deterministic action, which in Haskell would have the following type signature:

validateSignup  :: SignupForm -> IO (Validation (NonEmpty SignupError) Signup)

Note the use of IO. It means we could retrieve the current time from the system clock, and extract the Day value representing today’s date. But this approach has severe drawbacks.

If validateSignup uses IO to retrieve the current date, we can’t test it with other dates. What it there’s a bug that causes validation to behave incorrectly only on a particular date? We’d have to run the tests on that specific date to trigger it. If we introduce a bug, we want to know about it immediately. Not weeks, months, or even years after the bug was introduced. Furthermore, if we find such a bug with our tests, we can’t easily reproduce it on another date. We’d have to rewrite the implementation code to trigger the bug again.

Instead of using IO, we’ll use a simply technique for keeping our function pure: take all the information the function needs as arguments. In the case of validateSignup, we’ll pass today’s date as the first argument:

validateSignup  :: Day -> SignupForm -> Validation (NonEmpty SignupError) Signup

Again, let’s not worry about the implementation just yet. We’ll focus on the tests.

Generating Dates

In order to test the new validateSignup implementation, we need to generate Day values. We’re going to use a few functions from a separate module called Data.Time.Gen, previously written by some brilliant developer in our team. Let’s look at their type signatures. The implementations are not very interesting.

The generator, day, generates a day within the given range:

day :: Range Day -> Gen Day

A day range is constructed with linearDay:

linearDay :: Day -> Day -> Range Day

Alternatively, we might use exponentialDay:

exponentialDay :: Day -> Day -> Range Day

The linearDay and exponentialDay range functions are analoguous to Hedgehog’s linear and exponential ranges for integral numbers.

To use the generator functions from Data.Time.Gen, we first add an import, qualified as Time:

import qualified Data.Time.Gen as Time

Next, we define a generator anyDay:

anyDay :: Gen Day anyDay =  let low = fromGregorian 1900 1 1  high = fromGregorian 2100 12 31  in Time.day (Time.linearDay low high)

The date range [1900-01-01, 2100-12-31] is arbitrary. We could pick any centuries we like, provided the time package supports the range. But why not make it somewhat realistic?

Rewriting Existing Properties

Now, it’s time to rewrite our existing property tests. Let’s begin with the one testing that validating a form with all valid data succeeds:

prop_valid_signup_form_succeeds = property $ do  today <- forAll anyDay ➊  let genForm = SignupForm <$> validName <*> validBirthDate today  form <- forAll genForm ➋   case validateSignup today form of  Success{} -> pure ()  Failure failure' -> do  annotateShow failure'  failure

A few new things are going on here. We’re generating a date representing today (➊), and generating a form with a birth date based on today’s date (➋). Generating today’s date, we’re effectively time travelling and running the form validation on that date. This means our validBirthDate generator must know which date is today, in order to pick a valid birth date. We pass today’s date as a parameter, and generate a date within the range of 18 to 150 years earlier:

validBirthDate :: Day -> Gen Day validBirthDate today = do  n <- Gen.integral (Range.linear 18 150)  pure (n `yearsBefore` today)

We define the helper function yearsBefore in the test suite. It offsets a date backwards in time by a given number of years:

yearsBefore :: Integer -> Day -> Day yearsBefore years = addGregorianYearsClip (negate years)

The Data.Time.Calendar module exports the addGregorianYearsClip function. It adds a number of years, clipping February 29th (leap days) to February 28th where necessary.

Let’s run tests:

λ> check prop_valid_signup_form_succeeds ✓ <interactive> passed 100 tests.

Let’s move on to the next property, checking that invalid birth dates do not pass validation. Here, we use the same pattern as before, generating today’s date, but use invalidBirthDate instead:

prop_invalid_age_fails = property $ do  today <- forAll anyDay  form <- forAll (SignupForm <$> validName <*> invalidBirthDate today)   cover 5 "not yet born" (formBirthDate form > today)  cover 5 "too young" (formBirthDate form > 18 `yearsBefore` today)  cover 5 "too old" (formBirthDate form < 150 `yearsBefore` today)   case validateSignup today form of  Failure (TooYoung{} :| []) -> pure ()  Failure (NotYetBorn{} :| []) -> pure ()  Failure (TooOld{} :| []) -> pure ()  other -> do  annotateShow other  failure

Notice that we’ve also adjusted the coverage checks. There’s a new label, “not born yet,” for birth dates in the future. Running tests, we see the label in action:

λ> check prop_invalid_age_fails ✓ <interactive> passed 100 tests. not yet born 18% ███▌················ ✓ 5% too young 54% ██████████▊········· ✓ 5% too old 46% █████████▏·········· ✓ 5%

Good coverage, all tests passing. We’re not quite done, though. There’s a particular set of dates that we should be sure to cover: “today” dates and birth dates that are close to, or exactly, 18 years apart.

Within our current property test for invalid ages, we’re only sure that generated birth dates include at least 5% too old, and at least 5% too young. We don’t know how far away from the “18 years” validation boundary they are.

We could tweak our existing generators to produce values close to that boundary. Given a date T, exactly 18 years before today’s date, then:

• invalidBirthDate would need to produce birth dates just after but not equal to T
• validBirthDate would need to produce birth dates just before or equal to T

There’s another option, though. Instead of defining separate properties for valid and invalid ages, we’ll use a single property for all cases. This way, we only need a single generator.

A Single Validation Property

In Building on developers’ intuitions to create effective property-based tests, John Hughes talks about “one property to rule them all.” Similarly, we’ll define a single property prop_validates_age for birth date validation. We’ll base our new property on prop_invalid_age_fails, but generalize to cover both positive and negative tests:

prop_validates_age = property $ do  today <- forAll anyDay  form <- forAll (SignupForm <$> validName <*> anyBirthDate today) ➊   let tooYoung = formBirthDate form > 18 `yearsBefore` today ➋  notYetBorn = formBirthDate form > today  tooOld = formBirthDate form < 150 `yearsBefore` today  oldEnough = formBirthDate form <= 18 `yearsBefore` today  exactly age = formBirthDate form == age `yearsBefore` today  closeTo age =  let diff' =  diffDays (formBirthDate form) (age `yearsBefore` today)  in abs diff' `elem` [0 .. 2]   cover 10 "too young" tooYoung  cover 1 "not yet born" notYetBorn  cover 1 "too old" tooOld   cover 20 "old enough" oldEnough ➌  cover 1 "exactly 18" (exactly 18)  cover 5 "close to 18" (closeTo 18)   case validateSignup today form of ➍  Failure (NotYetBorn{} :| []) | notYetBorn -> pure ()  Failure (TooYoung{} :| []) | tooYoung -> pure ()  Failure (TooOld{} :| []) | tooOld -> pure ()  Success{} | oldEnough -> pure ()  other -> annotateShow other >> failure

There are a few new things going on here:

1. Instead of generating exclusively invalid or valid birth dates, we’re now generating any birth date based on today’s date
2. The boolean expressions are used both in coverage checks and in asserting, so we separate them in a let binding
3. We add three new labels for the valid cases
4. Finally, we assert on both valid and invalid cases, based on the same expressions used in coverage checks

Note that our assertions are more specific than in prop_invalid_age_fails. The failure cases only pass if the corresponding label expressions are true. The oldEnough case covers all valid birth dates. Any result other than the four expected cases is considered incorrect.

The anyBirthDate generator is based on today’s date:

anyBirthDate :: Day -> Gen Day anyBirthDate today =  let ➊  inPast range = do  years <- Gen.integral range  pure (years `yearsBefore` today)  inFuture = do  years <- Gen.integral (Range.linear 1 5)  pure (addGregorianYearsRollOver years today)  daysAroundEighteenthYearsAgo = do  days <- Gen.integral (Range.linearFrom 0 (-2) 2)  pure (addDays days (18 `yearsBefore` today))  in ➋  Gen.frequency  [ (5, inPast (Range.exponential 1 150))  , (1, inPast (Range.exponential 151 200))  , (2, inFuture)  , (2, daysAroundEighteenthYearsAgo)  ]

We defines helper functions (➊) for generating dates in the past, in the future, and close to 18 years ago. Using those helper functions, we combine four generators, with different date ranges, using a Gen.frequency distribution (➋). The weights we use are selected to give us a good coverage.

Let’s run some tests:

λ> check prop_validates_age ✓ <interactive> passed 100 tests. too young 62% ████████████▍······· ✓ 10% not yet born 20% ████················ ✓ 1% too old 4% ▊··················· ✓ 1% old enough 38% ███████▌············ ✓ 20% exactly 18 16% ███▏················ ✓ 1% close to 18 21% ████▏··············· ✓ 5%

Looks good! We’ve gone from testing positive and negative cases separately, to instead have a single property covering all cases, based on a single generator. It’s now easier to generate values close to the valid/invalid boundary of our SUT, i.e. around 18 years from today’s date.

February 29th

For the fun of it, let’s run some more tests. We’ll crank it up to 20000.

λ> check (withTests 20000 prop_validates_age) ✗ <interactive> failed at test/Validation/V3Test.hs:141:64 after 17000 tests and 25 shrinks. too young 60% ████████████········ ✓ 10% not yet born 20% ███▉················ ✓ 1% too old 9% █▉·················· ✓ 1% old enough 40% ███████▉············ ✓ 20% exactly 18 14% ██▊················· ✓ 1% close to 18 21% ████▎··············· ✓ 5% ┏━━ test/Validation/V3Test.hs ━━━ 114 ┃ prop_validates_age = property $ do 115 ┃ today <- forAll anyDay ┃ │ 1956 - 02 - 29 116 ┃ form <- forAll (SignupForm <$> validName <*> anyBirthDate today) ➊ ┃ │ SignupForm { formName = "aa" , formBirthDate = 1938 - 03 - 01 } 117 ┃ 118 ┃ let tooYoung = formBirthDate form > 18 `yearsBefore` today ➋ 119 ┃ notYetBorn = formBirthDate form > today 120 ┃ tooOld = formBirthDate form < 150 `yearsBefore` today 121 ┃ oldEnough = formBirthDate form <= 18 `yearsBefore` today 122 ┃ exactlyEighteen = formBirthDate form == 18 `yearsBefore` today 123 ┃ closeToEighteen = 124 ┃ let diff' = 125 ┃ diffDays (formBirthDate form) (18 `yearsBefore` today) 126 ┃ in abs diff' `elem` [0 .. 2] 127 ┃ 128 ┃ cover 10 "too young" tooYoung 129 ┃ cover 1 "not yet born" notYetBorn 130 ┃ cover 1 "too old" tooOld 131 ┃ 132 ┃ cover 20 "old enough" oldEnough ➌ 133 ┃ cover 1 "exactly 18" exactlyEighteen 134 ┃ cover 5 "close to 18" closeToEighteen 135 ┃ 136 ┃ case validateSignup today form of ➍ 137 ┃ Failure (NotYetBorn{} :| []) | notYetBorn -> pure () 138 ┃ Failure (TooYoung{} :| []) | tooYoung -> pure () 139 ┃ Failure (TooOld{} :| []) | tooOld -> pure () 140 ┃ Success{} | oldEnough -> pure () 141 ┃ other -> annotateShow other >> failure ┃ │ Success Signup { name = "aa" , birthDate = 1938 - 03 - 01 } ┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Failure! Chaos! What’s going on here? Let’s examine the test case:

• Today’s date is 1956-02-29
• The birth date is 1938-03-01
• The validation function considers this valid (it returns a Success value)
• The test does considers this invalid (oldEnough is False)

This means that when the validation runs on a leap day, February 29th, and the person would turn 18 years old the day after (on March 1st), the validation function incorrectly considers the person old enough. We’ve found a bug.

Test Count and Coverage

Two things led us to find this bug:

1. Most importantly, that we generate today’s date and pass it as a parameter. Had we used the actual date, retrieved with an IO action, we’d only be able to find this bug every 1461 days. Pure functions are easier to test.
2. That we ran more tests than the default of 100. We might not have found this bug until much later, when the generated dates happened to trigger this particular bug. In fact, running 20000 tests does not always trigger the bug.

Our systems are often too complex to be tested exhaustively. Let’s use our form validation as an example. Between 1900-01-01 and 2100-12-31 there are 73,413 days. Selecting today’s date and the birth date from that range, we have more than five billion combinations. Running that many Hedgehog tests in GHCi on my laptop (based on some quick benchmarks) would take about a month. And this is a simple pure validation function!

To increase coverage, even if it’s not going to be exhaustive, we can increase the number of tests we run. But how many should we run? On a continuous integration server we might be able to run more than we do locally, but we still want to keep a tight feedback loop. And what if our generators never produce inputs that reveal existing bugs, regardless of the number of tests we run?

If we can’t test exhaustively, we need to ensure our generators cover interesting combinations of inputs. We need to carefully design and measure our tests and generators, based on the edge cases we already know of, as well as the ones that we discover over time. PBT without measuring coverage easily turns into a false sense of security.

In the case of our leap day bug, we can catch it with fewer tests, and on every test run. We need to make sure we cover leap days, used both as today’s date and as the birth date, even with a low number of tests.

Covering Leap Days

To generate inputs that cover certain edge cases, we combine specific generators using Gen.frequency:

(today, birthDate') <- forAll  (Gen.frequency  [ (5, anyDayAndBirthDate) ➊   , (2, anyDayAndBirthDateAroundYearsAgo 18) ➋  , (2, anyDayAndBirthDateAroundYearsAgo 150)   , (1, leapDayAndBirthDateAroundYearsAgo 18) ➌  , (1, leapDayAndBirthDateAroundYearsAgo 150)   , (1, commonDayAndLeaplingBirthDateAroundYearsAgo 18) ➍  , (1, commonDayAndLeaplingBirthDateAroundYearsAgo 150)  ]  )

Arbitrary values for today’s date and the birth date are drawn most frequently (➊), with a weight of 5. Next, with weights of 2, are generators for cases close to the boundaries of the validation function (➋). Finally, with weights of 1, are generators for special cases involving leap days as today’s date (➌) and leap days as birth date (➍).

Note that these generators return pairs of dates. For most of these generators, there’s a strong relation between today’s date and the birth date. For example, we can’t first generate any today’s date, pass that into a generator function, and expect it to always generate a leap day that occured 18 years ago. Such a generator would have to first generate the leap day and then today’s date.

Let’s define the generators. The first one, anyDayAndBirthDate, picks any today’s date within a wide date range. It also picks a birth date from an even wider date range, resulting in some future birth dates and some ages above 150.

anyDayAndBirthDate :: Gen (Day, Day) anyDayAndBirthDate = do  today <- Time.day  (Time.linearDay (fromGregorian 1900 1 1)  (fromGregorian 2020 12 31)  )  birthDate' <- Time.day  (Time.linearDay (fromGregorian 1850 1 1)  (fromGregorian 2050 12 31)  )  pure (today, birthDate')

Writing automated tests with a hard-coded year 2020 might scare you. Won’t these tests fail when run in the future? No, not these tests. Remember, the validation function is deterministic. We control today’s date. The actual date on which we run these tests doesn’t matter.

Similar to the previous generator is anyDayAndBirthDateAroundYearsAgo. First, it generates any date as today’s date (➊). Next, it generates an arbitrary date approximately some number of years ago (➋), where the number of years is an argument of the generator.

anyDayAndBirthDateAroundYearsAgo :: Integer -> Gen (Day, Day) anyDayAndBirthDateAroundYearsAgo years = do  today <- Time.day ➊  (Time.linearDay (fromGregorian 1900 1 1)  (fromGregorian 2020 12 31)  )  birthDate' <- addingApproxYears (negate years) today ➋  pure (today, birthDate')

The addingApproxYearsAgo generator adds a number of years to a date, and offsets it between two days back and two days forward in time.

addingApproxYears :: Integer -> Day -> Gen Day addingApproxYears years today = do  days <- Gen.integral (Range.linearFrom 0 (-2) 2)  pure (addDays days (addGregorianYearsRollOver years today))

The last two generators used in our frequency distribution cover leap day edge cases. First, let’s define the leapDayAndBirthDateAroundYearsAgo generator. It generates a leap day used as today’s date, and a birth date close to the given number of years ago.

leapDayAndBirthDateAroundYearsAgo :: Integer -> Gen (Day, Day) leapDayAndBirthDateAroundYearsAgo years = do  today <- leapDay (Range.linear 1904 2020)  birthDate' <- addingApproxYears (negate years) today  pure (today, birthDate')

The leapDay generator uses mod to only generate years divisible by 4 and constructs dates on February 29th. That alone isn’t enough to only generate valid leap days, though. Years divisible by 100 but not by 400 are not leap years. To keep the generator simple, we discard those years using the already existing isLeapDay predicate as a filter.

leapDay :: Range Integer -> Gen Day leapDay yearRange = Gen.filter isLeapDay $ do  year <- Gen.integral yearRange  pure (fromGregorian (year - year `mod` 4) 2 29)

In general, we should be careful about discarding generated values using filter. If we discard too much, Hedgehog gives up and complains loudly. In this particular case, discarding a few generated dates is fine. Depending on the year range we pass it, we might not discard any date.

Finally, we define the commonDayAndLeaplingBirthDateAroundYearsAgo generator. It first generates a leap day used as the birth date, and then a today’s date approximately the given number of years after the birth date.

commonDayAndLeaplingBirthDateAroundYearsAgo :: Integer -> Gen (Day, Day) commonDayAndLeaplingBirthDateAroundYearsAgo years = do  birthDate' <- leapDay (Range.linear 1904 2020)  today <- addingApproxYears years birthDate'  pure (today, birthDate')

That’s it for the generators. Now, how do we know that we’re covering the edge cases well enough? With coverage checks!

 cover 5 ➊  "close to 18, validated on common day"  (closeTo 18 && not (isLeapDay today)) cover 1  "close to 18, validated on leap day"  (closeTo 18 && isLeapDay today)  cover 5 ➋  "close to 150, validated on common day"  (closeTo 150 && not (isLeapDay today)) cover 1  "close to 150, validated on leap day"  (closeTo 150 && isLeapDay today)  cover 5 ➌  "exactly 18 today, born on common day"  (exactly 18 && not (isLeapDay birthDate')) cover ➍  1  "legally 18 today, born on leap day"  ( isLeapDay birthDate'  && (addGregorianYearsRollOver 18 birthDate' == today)  )

We add new checks to the property test, checking that we hit both leap day and regular day cases around the 18th birthday (➊) and the 150th birthday (➋). Notice that we had similar checks before, but we were not discriminating between leap days and common days.

Finally, we check the coverage of two leap day scenarios that can occur when a person legally turns 18: a person born on a common day turning 18 on a leap day (➌), and a leapling turning 18 on a common day (➍).

Running the modified property test, we get the leap day counter-example every time, even with as few as a hundred tests. For example, we might see today’s date being 1904-02-29 and the birth date being 1886-03-01. The validation function deems the person old enough. Again, this is incorrect.

Now that we can quickly and reliably reproduce the failing example we are in a great position to find the error. While we could use a fixed seed to reproduce the particular failing case from the 20000 tests run, we are now more confident that the property test would catch future leap day-related bugs, if we were to introduce new ones. Digging into the implementation, we’ll find a boolean expression in a pattern guard being the culprit:

birthDate' <= addGregorianYearsRollOver (-18) today

The use of addGregorianYearsRollOver together with adding a negative number of years is the problem, rolling over to March 1st instead of clipping to February 28th. Instead, we should use addGregorianYearsClip:

birthDate' <= addGregorianYearsClip (-18) today

Running 100 tests again, we see that they all pass, and that our coverage requirements are met.

λ> Hedgehog.check prop_validates_age ✓ <interactive> passed 100 tests. too young 17% ███▍················ ✓ 10% not yet born 7% █▍·················· ✓ 1% too old 19% ███▊················ ✓ 1% old enough 83% ████████████████▌··· ✓ 20% close to 18, validated on common day 30% ██████·············· ✓ 5% close to 18, validated on leap day 2% ▍··················· ✓ 1% close to 150, validated on common day 31% ██████▏············· ✓ 5% close to 150, validated on leap day 6% █▏·················· ✓ 1% exactly 18 today, born on common day 17% ███▍················ ✓ 5% legally 18 today, born on leap day 5% █··················· ✓ 1%

Summary

In this tutorial, we started with a simple form validation function, checking the name and age of a person signing up for an online service. We defined property tests for positive and negative tests, learned how to test generators with coverage checks, and found bugs in both the test suite and the implementation.

When requirements changed, we had to start working with dates. In order to keep the validation function deterministic, we had to pass in today’s date. This enabled us to simulate the validation running on any date, in combination with any reported birth date, and trigger bugs that could otherwise take years to find, if ever. Had we not made it deterministic, we would likely not have found the leap day bug later on.

To generate inputs that sufficiently test the validation function’s boundaries, we rewrote our separate positive and negative properties into a single property, and used coverage checks to ensure the quality of our generators. The trade-off between multiple disjoint properties and a single more complicated property is hard.

With multiple properties, for example split between positive and negative tests, both generators and assertions can be simpler and more targeted. On the other hand, you run a risk of missing certain inputs. The set of properties might not cover the entire space of inputs. Furthermore, performing coverage checks across multiple properties, using multiple targeted generators, can be problematic.

Ensuring coverage of generators in a single property is easier. You might even get away with a naive generator, depending on the system you’re testing. If not, you’ll need to combine more targeted generators, for example with weighted probabilities. The drawback of using a single property is that the assertion not only becomes more complicated, it’s also likely to mirror the implementation of the SUT. As we saw with our single property testing the validation function, the assertion duplicated the validation rules. You might be able to reuse the coverage expressions in assertions, but still, there’s a strong coupling.

The choice between single or multiple properties comes down to how you want to cover the boundaries of the SUT. Ultimately, both approaches can achieve the same coverage, in different ways. They both suffer from the classic problem of a test suite mirroring the system it’s testing.

Finally, running a larger number of tests, we found a bug related to leap days. Again, without having made the validation function deterministic, this could’ve only been found on a leap day. We further refined our generators to cover leap day cases, and found the bug reliably with as few as 100 tests. The bug was easy to find and fix when we had the inputs pointing directly towards it.

That’s it for this tutorial. Thanks for reading, and happy property testing and time travelling!

1. Introduction
2. Timeline Flattening
3. Video Scene Classification

This is the final case study in the “Property-Based Testing in a Screencast Editor” series. It covers property-based integration testing and its value during aggressive refactoring work within Komposition.

A History of Two Stacks

In Komposition, a project’s state is represented using an in-memory data structure. It contains the hierarchical timeline, the focus, import and render settings, project storage file paths, and more. To let users navigate backwards and forwards in their history of project edits, for example when they have made a mistake, Komposition supplies undo and redo commands.

The undo/redo history was previously implemented as a data structure recording project states, compromised of:

• a current state variable
• a stack of previous states
• a stack of possible future states

The undo/redo history data structure held entire project state values. Each undoable and redoable user action created a new state value. Let’s look a bit closer at how this worked.

Performing Actions

When a user performed an undoable/redoable action, the undo/redo history would:

• push the previous state onto the undo stack
• perform the action and replace the current state
• discard all states in the redo stack

This can be visualized as in the following diagram, where the state d is being replaced with a new state h, and d being pushed onto the undo stack. The undo/redo history to the left of the dividing line is the original, and the one to the right is the resulting history.

Again, note that performing new actions discarded all states in the redo stack.

Undoing Actions

When the user chose to undo an action, the undo/redo history would:

• pop the undo stack and use that state as the current state
• push the previous state onto the redo stack

The following diagram shows how undoing the last performed action’s resulting state, d, pushes d onto the redo stack, and pops c from the undo stack to use that as the current state.

Redoing Actions

When the user chose to redo an action, the undo/redo history would:

• pop the redo stack and use that state as the current state
• push the previous state onto the undo stack

The last diagram shows how redoing, recovering a previously undone state, pops g from the redo stack to use that as the current state, and pushes the previous state d onto the undo stack.

Note that not all user actions in Komposition are undoable/redoable. Actions like navigating the focus or zooming are not recorded in the history.

Dealing With Performance Problems

While the “two stacks of states” algorithm was easy to understand and implement, it failed to meet my non-functional requirements. A screencast project compromised of hundreds or thousands of small edits would consume gigabytes of disk space when stored, take tens of seconds to load from disk, and consume many gigabytes of RAM when in memory.

Now, you might think that my implementation was incredibly naive, and that the performance problems could be fixed with careful profiling and optimization. And you’d probably be right! I did consider going down that route, optimizing the code, time-windowing edits to compact history on the fly, and capping the history at some fixed size. Those would all be interesting pursuits, but in the end I decided to try something else.

Refactoring with Property-Based Integration Tests

Instead of optimizing the current stack-based implementation, I decided to implement the undo/redo history in terms of inverse actions. In this model, actions not only modify the project state, they also return another action, its inverse, that reverses the effects of the original action. Instead of recording a new project state data structure for each edit, the history only records descriptions of the actions themselves.

I realized early that introducing the new undo/redo history implementation in Komposition was not going to be a small task. It would touch the majority of command implementation code, large parts of the main application logic, and the project binary serialization format. What it wouldn’t affect, though, was the module describing user commands in abstract.

To provide a safety net for the refactoring, I decided to cover the undo/redo functionality with tests. As the user commands would stay the same throughout my modifications, I chose to test at that level, which can be characterized as integration-level testing. The tests run Komposition, including its top-level application control flow, but with the user interface and some other effects stubbed out. Making your application testable at this level is hard work, but the payoff can be huge.

With Komposition featuring close to twenty types of user commands, combined with a complex hierarchical timeline and navigation model, the combinatory explosion of possible states was daunting. Relying on example-based tests to safeguard my work was not satisfactory. While PBT couldn’t cover the entire state space either, I was confident it would improve my chances of finding actual bugs.

Undo/Redo Tests

Before I began refactoring, I added tests for the inverse property of undoable/redoable actions. The first test focuses on undoing actions, and is structured as follows:

1. Generate an initial project and application state
2. Generate a sequence of undoable/redoable commands (wrapped in events)
3. Run the application with the initial state and the generated events
4. Run an undo command for each original command
5. Assert that final timeline is equal to the initial timeline

Let’s look at the Haskell Hedgehog property test:

hprop_undo_actions_are_undoable = property $ do   -- 1. Generate initial timeline and focus  timelineAndFocus <- forAllWith showTimelineAndFocus $  Gen.timelineWithFocus (Range.linear 0 10) Gen.parallel   -- ... and initial application state  initialState <- forAll (initializeState timelineAndFocus)   -- 2. Generate a sequence of undoable/redoable commands  events <- forAll $  Gen.list (Range.exponential 1 100) genUndoableTimelineEvent   -- 3. Run 'events' on the original state  beforeUndos <- runTimelineStubbedWithExit events initialState   -- 4. Run as many undo commands as undoable commands  afterUndos <- runTimelineStubbedWithExit (undoEvent <$ events) beforeUndos   -- 5. That should result in a timeline equal to the one we started  -- with  timelineToTree (initialState ^. currentTimeline)  === timelineToTree (afterUndos ^. currentTimeline)

The second test, focusing on redoing actions, is structured very similarly to the previous test:

1. Generate an initial project and application state
2. Generate a sequence of undoable commands (wrapped in events)
3. Run the application with the initial state and the generated events
4. Run an undo commands for each original command
5. Run an redo commands for each original command
6. Assert that final timeline is equal to the timeline before undoing actions

The test code is also very similar:

hprop_undo_actions_are_redoable = property $ do   -- 1. Generate the initial timeline and focus  timelineAndFocus <- forAllWith showTimelineAndFocus $  Gen.timelineWithFocus (Range.linear 0 10) Gen.parallel   -- ... and the initial application state  initialState <- forAll (initializeState timelineAndFocus)   -- 2. Generate a sequence of undoable/redoable commands  events <- forAll $  Gen.list (Range.exponential 1 100) genUndoableTimelineEvent   -- 3. Run 'events' on the original state  beforeUndos <- runTimelineStubbedWithExit events initialState   -- 4. Run undo commands corresponding to all original commands  afterRedos <-  runTimelineStubbedWithExit (undoEvent <$ events) beforeUndos  -- 5. Run redo commands corresponding to all original commands  >>= runTimelineStubbedWithExit (redoEvent <$ events)   -- 6. That should result in a timeline equal to the one we had  -- before undoing actions  timelineToTree (beforeUndos ^. currentTimeline)  === timelineToTree (afterRedos ^. currentTimeline)

Note that these tests only assert on the equality of timelines, not entire project states, as undoable commands only operate on the timeline.

All Tests Passing, Everything Works

The undo/redo tests were written and run on the original stack-based implementation, kept around during a refactoring that took me two weeks of hacking during late nights and weekends, and finally run and passing with the new implementation based on inverse actions. Except for a few minimal adjustments to data types, these tests stayed untouched during the entire process.

The confidence I had when refactoring felt like a super power. Two simple property tests made the undertaking possible. They found numerous bugs, including:

• Off-by-one index errors in actions modifying the timeline
• Inconsistent timeline focus:
  • focus was incorrectly restored on undoing an action
  • focus was outside of the timeline bounds
• Non-inverse actions:
  • actions returning incorrectly constructed inverses
  • the inverse of splitting a sequence is joining sequences, and joining them back up didn’t always work

After all tests passed, I ran the application with its GUI, edited a screencast project, and it all worked flawlessly. It’s almost too good to be true, right?

Property testing is not a silver bullet, and there might still be bugs lurking in my undo/redo history implementation. The tests I run are never going to be exhaustive and my generators might be flawed. That being said, they gave me a confidence in refactoring that I’ve never had before. Or maybe I just haven’t hit that disastrous edge case yet?

Why Test With Properties?

This was the last case study in the “Property-Based Testing in a Screencast Editor” series. I’ve had a great time writing these articles and giving talks on the subject. Before I wrap up, I’ll summarize my thoughts on PBT in general and my experience with it in Komposition.

Property-based testing is not only for pure functions; you can use it to test effectful actions. It is not only for unit testing; you can write integration tests using properties. It’s not only for functional programming languages; there are good frameworks for most popular programming languages.

Properties describe the general behavior of the system under test, and verify its correctness using a variety of inputs. Not only is this an effective way of finding errors, it’s also a concise way of documenting the system.

The iterative process in property-based testing, in my experience, comes down to the following steps:

1. Think about the specification of your system under test
2. Think about how generators and tests should work
3. Write or modify generators, tests, and implementation code, based on steps 1 and 2
4. Get minimal examples of failing tests
5. Repeat

Using PBT within Komposition has made it possible to confidently refactor large parts of the application. It has found errors in my thinking, my generators, my tests, and in my implementation code. Testing video scene classification went from a time consuming, repetitive, and manual verification process to a fast, effective, and automated task.

In short, it’s been a joy, and I look forward to continue using PBT in my work and in my own projects. I hope I’ve convinced you of its value, and inspired you to try it out, no matter what kind of project you’re working on and what programming language you are using. Involve your colleagues, practice writing property tests together, and enjoy finding complicated bugs before your users do!

Buy the Book

This series is now available as an ebook on Leanpub. While the content is mostly the same, there are few changes bringing it up-to-date. Also, if you’ve already enjoyed the articles, you might want support my work by purchasing this book. Finally, you might enjoy a nicely typeset PDF, or an EPUB book, over a web page.

If you haven’t read the introduction or the first case study yet, I recommend checking them out!

Classifying Scenes in Imported Video

Komposition can automatically classify scenes when importing video files. This is a central productivity feature in the application, effectively cutting recorded screencast material automatically, letting the user focus on arranging the scenes of their screencast. Scenes are segments that are considered moving, as opposed to still segments:

• A still segment is a sequence of at least S seconds of near-equal frames
• A moving segment is a sequence of non-equal frames, or a sequence of near-equal frames with a duration less than S

S is a preconfigured minimum still segment duration in Komposition. In the future it might be configurable from the user interface, but for now it’s hard-coded.

Equality of two frames f₁ and f₂ is defined as a function E(f₁, f₂), described informally as:

• comparing corresponding pixel color values of f₁ and f₂, with a small epsilon for tolerance of color variation, and
• deciding two frames equal when at least 99% of corresponding pixel pairs are considered equal.

In addition to the rules stated above, there are two edge cases:

1. The first segment is always a considered a moving segment (even if it’s just a single frame)
2. The last segment may be a still segment with a duration less than S

The second edge case is not what I would call a desirable feature, but rather a shortcoming due to the classifier not doing any type of backtracking. This could be changed in the future.

Manually Testing the Classifier

The first version of the video classifier had no property tests. Instead, I wrote what I thought was a decent classifier algorithm, mostly messing around with various pixel buffer representations and parallel processing to achieve acceptable performance.

The only type of testing I had available, except for general use of the application, was a color-tinting utility. This was a separate program using the same classifier algorithm. It took as input a video file, and produced as output a video file where each frame was tinted green or red, for moving and still frames, respectively.

In the recording above you see the color-tinted output video based on a recent version of the classifier. It classifies moving and still segments rather accurately. Before I wrote property tests and fixed the bugs that I found, it did not look so pretty, flipping back and forth at seemingly random places.

At first, debugging the classifier with the color-tinting tool way seemed like a creative and powerful technique. But the feedback loop was horrible, having to record video, process it using the slow color-tinting program, and inspecting it by eye. In hindsight, I can conclude that PBT is far more effective for testing the classifier.

Video Classification Properties

Figuring out how to write property tests for video classification wasn’t obvious to me. It’s not uncommon in example-based testing that tests end up mirroring the structure, and even the full implementation complexity, of the system under test. The same can happen in property-based testing.

With some complex systems it’s very hard to describe the correctness as a relation between any valid input and the system’s observed output. The video classifier is one such case. How do I decide if an output classification is correct for a specific input, without reimplementing the classification itself in my tests?

The other way around is easy, though! If I have a classification, I can convert that into video frames. Thus, the solution to the testing problem is to not generate the input, but instead generate the expected output. Hillel Wayne calls this technique “oracle generators” in his recent article.¹

The classifier property tests generate high-level representations of the expected classification output, which are lists of values describing the type and duration of segments.

Next, the list of output segments is converted into a sequence of actual frames. Frames are two-dimensional arrays of RGB pixel values. The conversion is simple:

• Moving segments are converted to a sequence of alternating frames, flipping between all gray and all white pixels
• Still frames are converted to a sequence of frames containing all black pixels

The example sequence in the diagram above, when converted to pixel frames with a frame rate of 10 FPS, can be visualized like in the following diagram, where each thin rectangle represents a frame:

By generating high-level output and converting it to pixel frames, I have input to feed the classifier with, and I know what output it should produce. Writing effective property tests then comes down to writing generators that produce valid output, according to the specification of the classifier. In this post I’ll show two such property tests.

Testing Still Segment Minimum Length

As stated in the beginning of this post, classified still segments must have a duration greater than or equal to S, where S is the minimum still segment duration used as a parameter for the classifier. The first property test we’ll look at asserts that this invariant holds for all classification output.

hprop_classifies_still_segments_of_min_length = property $ do   -- 1. Generate a minimum still segment length/duration  minStillSegmentFrames <- forAll $ Gen.int (Range.linear 2 (2 * frameRate))  let minStillSegmentTime = frameCountDuration minStillSegmentFrames   -- 2. Generate output segments  segments <- forAll $  genSegments (Range.linear 1 10)  (Range.linear 1  (minStillSegmentFrames * 2))  (Range.linear minStillSegmentFrames  (minStillSegmentFrames * 2))  resolution   -- 3. Convert test segments to actual pixel frames  let pixelFrames = testSegmentsToPixelFrames segments   -- 4. Run the classifier on the pixel frames  let counted = classifyMovement minStillSegmentTime (Pipes.each pixelFrames)  & Pipes.toList  & countSegments   -- 5. Sanity check  countTestSegmentFrames segments === totalClassifiedFrames counted   -- 6. Ignore last segment and verify all other segments  case initMay counted of  Just rest ->  traverse_ (assertStillLengthAtLeast minStillSegmentTime) rest  Nothing -> success  where  resolution = 10 :. 10

This chunk of test code is pretty busy, and it’s using a few helper functions that I’m not going to bore you with. At a high level, this test:

1. Generates a minimum still segment duration, based on a minimum frame count (let’s call it n) in the range [2, 20]. The classifier currently requires that n ≥ 2, hence the lower bound. The upper bound of 20 frames is an arbitrary number that I’ve chosen.
2. Generates valid output segments using the custom generator genSegments, where
   • moving segments have a frame count in [1, 2n], and
   • still segments have a frame count in [n, 2n].
3. Converts the generated output segments to actual pixel frames. This is done using a helper function that returns a list of alternating gray and white frames, or all black frames, as described earlier.
4. Count the number of consecutive frames within each segment, producing a list like [Moving 18, Still 5, Moving 12, Still 30].
5. Performs a sanity check that the number of frames in the generated expected output is equal to the number of frames in the classified output. The classifier must not lose or duplicate frames.
6. Drops the last classified segment, which according to the specification can have a frame count less than n, and asserts that all other still segments have a frame count greater than or equal to n.

Let’s run some tests.

> :{ | hprop_classifies_still_segments_of_min_length | & Hedgehog.withTests 10000 | & Hedgehog.check | :} ✓ <interactive> passed 10000 tests.

Cool, it looks like it’s working.

Sidetrack: Why generate the output?

Now, you might wonder why I generate output segments first, and then convert to pixel frames. Why not generate random pixel frames to begin with? The property test above only checks that the still segments are long enough!

The benefit of generating valid output becomes clearer in the next property test, where I use it as the expected output of the classifier. Converting the output to a sequence of pixel frames is easy, and I don’t have to state any complex relation between the input and output in my property. When using oracle generators, the assertions can often be plain equality checks on generated and actual output.

But there’s benefit in using the same oracle generator for the “minimum still segment length” property, even if it’s more subtle. By generating valid output and converting to pixel frames, I can generate inputs that cover the edge cases of the system under test. Using property test statistics and coverage checks, I could inspect coverage, and even fail test runs where the generators don’t hit enough of the cases I’m interested in.²

Had I generated random sequences of pixel frames, then perhaps the majority of the generated examples would only produce moving segments. I could tweak the generator to get closer to either moving or still frames, within some distribution, but wouldn’t that just be a variation of generating valid scenes? It would be worse, in fact. I wouldn’t then be reusing existing generators, and I wouldn’t have a high-level representation that I could easily convert from and compare with in assertions.

Testing Moving Segment Time Spans

The second property states that the classified moving segments must start and end at the same timestamps as the moving segments in the generated output. Compared to the previous property, the relation between generated output and actual classified output is stronger.

hprop_classifies_same_scenes_as_input = property $ do  -- 1. Generate a minimum still still segment duration  minStillSegmentFrames <- forAll $ Gen.int (Range.linear 2 (2 * frameRate))  let minStillSegmentTime = frameCountDuration minStillSegmentFrames   -- 2. Generate test segments  segments <- forAll $ genSegments (Range.linear 1 10)  (Range.linear 1  (minStillSegmentFrames * 2))  (Range.linear minStillSegmentFrames  (minStillSegmentFrames * 2))  resolution   -- 3. Convert test segments to actual pixel frames  let pixelFrames = testSegmentsToPixelFrames segments   -- 4. Convert expected output segments to a list of expected time spans  -- and the full duration  let durations = map segmentWithDuration segments  expectedSegments = movingSceneTimeSpans durations  fullDuration = foldMap unwrapSegment durations   -- 5. Classify movement of frames  let classifiedFrames =  Pipes.each pixelFrames  & classifyMovement minStillSegmentTime  & Pipes.toList   -- 6. Classify moving scene time spans  let classified =  (Pipes.each classifiedFrames  & classifyMovingScenes fullDuration)  >-> Pipes.drain  & Pipes.runEffect  & runIdentity   -- 7. Check classified time span equivalence  expectedSegments === classified   where  resolution = 10 :. 10

Steps 1–3 are the same as in the previous property test. From there, this test:

4. Converts the generated output segments into a list of time spans. Each time span marks the start and end of an expected moving segment. Furthermore, it needs the full duration of the input in step 6, so that’s computed here.
5. Classify the movement of each frame, i.e. if it’s part of a moving or still segment.
6. Run the second classifier function called classifyMovingScenes, based on the full duration and the frames with classified movement data, resulting in a list of time spans.
7. Compare the expected and actual classified list of time spans.

While this test looks somewhat complicated with its setup and various conversions, the core idea is simple. But is it effective?

Bugs! Bugs everywhere!

Preparing for a talk on property-based testing, I added the “moving segment time spans” property a week or so before the event. At this time, I had used Komposition to edit multiple screencasts. Surely, all significant bugs were caught already. Adding property tests should only confirm the level of quality the application already had. Right?

Nope. First, I discovered that my existing tests were fundamentally incorrect to begin with. They were not reflecting the specification I had in mind, the one I described in the beginning of this post.

Furthermore, I found that the generators had errors. At first, I used Hedgehog to generate the pixels used for the classifier input. Moving frames were based on a majority of randomly colored pixels and a small percentage of equally colored pixels. Still frames were based on a random single color.

The problem I had not anticipated was that the colors used in moving frames were not guaranteed to be distinct from the color used in still frames. In small-sized examples I got black frames at the beginning and end of moving segments, and black frames for still segments, resulting in different classified output than expected. Hedgehog shrinking the failing examples’ colors towards 0, which is black, highlighted this problem even more.

I made my generators much simpler, using the alternating white/gray frames approach described earlier, and went on to running my new shiny tests. Here’s what I got:

What? Where does 0s–0.6s come from? The classified time span should’ve been 0s–1s, as the generated output has a single moving scene of 10 frames (1 second at 10 FPS). I started digging, using the annotate function in Hedgehog to inspect the generated and intermediate values in failing examples.

I couldn’t find anything incorrect in the generated data, so I shifted focus to the implementation code. The end timestamp 0.6s was consistently showing up in failing examples. Looking at the code, I found a curious hard-coded value 0.5 being bound and used locally in classifyMovement.

The function is essentially a fold over a stream of frames, where the accumulator holds vectors of previously seen and not-yet-classified frames. Stripping down and simplifying the old code to highlight one of the bugs, it looked something like this:

classifyMovement minStillSegmentTime =  case ... of  InStillState{..} ->  if someDiff > minEqualTimeForStill  then ...  else ...  InMovingState{..} ->  if someOtherDiff >= minStillSegmentTime  then ...  else ...  where  minEqualTimeForStill = 0.5

Let’s look at what’s going on here. In the InStillState branch it uses the value minEqualTimeForStill, instead of always using the minStillSegmentTime argument. This is likely a residue from some refactoring where I meant to make the value a parameter instead of having it hard-coded in the definition.

Sparing you the gory implementation details, I’ll outline two more problems that I found. In addition to using the hard-coded value, it incorrectly classified frames based on that value. Frames that should’ve been classified as “moving” ended up “still”. That’s why I didn’t get 0s–1s in the output.

Why didn’t I see 0s–0.5s, given the hard-coded value 0.5? Well, there was also an off-by-one bug, in which one frame was classified incorrectly together with the accumulated moving frames.

The classifyMovement function is 30 lines of Haskell code juggling some state, and I managed to mess it up in three separate ways at the same time. With these tests in place I quickly found the bugs and fixed them. I ran thousands of tests, all passing.

Finally, I ran the application, imported a previously recorded video, and edited a short screencast. The classified moving segments where notably better than before.

Summary

A simple streaming fold can hide bugs that are hard to detect with manual testing. The consistent result of 0.6, together with the hard-coded value 0.5 and a frame rate of 10 FPS, pointed clearly towards an off-by-one bug. I consider this is a great showcase of how powerful shrinking in PBT is, consistently presenting minimal examples that point towards specific problems. It’s not just a party trick on ideal mathematical functions.

Could these errors have been caught without PBT? I think so, but what effort would it require? Manual testing and introspection did not work for me. Code review might have revealed the incorrect definition of minEqualTimeForStill, but perhaps not the off-by-one and incorrect state handling bugs. There are of course many other QA techniques, I won’t evaluate all. But given the low effort that PBT requires in this setting, the amount of problems it finds, and the accuracy it provides when troubleshooting, I think it’s a clear win.

I also want to highlight the iterative process that I find naturally emerges when applying PBT:

1. Think about how your system is supposed to work. Write down your specification.
2. Think about how to generate input data and how to test your system, based on your specification. Tune your generators to provide better test data. Try out alternative styles of properties. Perhaps model-based or metamorphic testing fits your system better.
3. Run tests and analyze the minimal failing examples. Fix your implementation until all tests pass.

This can be done when modifying existing code, or when writing new code. You can apply this without having any implementation code yet, perhaps just a minimal stub, and the workflow is essentially the same as TDD.

Coming Up

The final post in this series will cover testing at a higher level of the system, with effects and multiple subsystems being integrated to form a full application. We will look at property tests that found many bugs and that made a substantial refactoring possible.

1. Introduction
2. Timeline Flattening
3. Video Scene Classification
4. Integration Testing

Until then, thanks for reading!

Credits

Thank you Ulrik Sandberg, Pontus Nagy, and Fredrik Björeman for reviewing drafts of this post.

Buy the Book

This series is now available as an ebook on Leanpub. While the content is mostly the same, there are few changes bringing it up-to-date. Also, if you’ve already enjoyed the articles, you might want support my work by purchasing this book. Finally, you might enjoy a nicely typeset PDF, or an EPUB book, over a web page.

Footnotes

If you haven’t read the introductory post, I suggest doing so before continuing with this one. You’ll need an understanding of what PBT is for this case study to make sense.

This post is the first case study in the series, covering the timeline flattening process in Komposition and how it’s tested using PBT. The property tests aren’t integration-level tests, but rather unit tests. This case study serves as a warm-up to the coming, more advanced, ones.

Before we look at the tests, we need to learn more about Komposition’s hierarchical timeline and how the flattening process works.

The Hierarchical Timeline

Komposition’s timeline is hierarchical. While many non-linear editing systems have support for some form of nesting¹ they are primarily focused on flat timeline workflows. The timeline structure and the keyboard-driven editing in Komposition is optimized for the screencast editing workflow I use.

It’s worth emphasizing that Komposition is not a general video editor. In addition to its specific editing workflow, you may need to adjust your recording workflow to use it effectively².

Video and Audio in Parallels

At the lowest level of the timeline are clips and gaps. Those are put within the video and audio tracks of parallels. The following diagram shows a parallel consisting of two video clips and one audio clip.

The tracks of a parallel are played simultaneously (in parallel), as indicated by the arrows in the above diagram. The tracks start playing at the same time. This makes parallels useful to synchronize the playback of specific parts of a screencast, and to group closely related clips.

Gaps

When editing screencasts made up of separate video and audio recordings you often end up with differing clip duration. The voice-over audio clip might be longer than the corresponding video clip, or vice versa. A useful default behaviour is to extend the short clips. For audio, this is easy. Just pad with silence. For video, it’s not so clear what to do. In Komposition, shorter video tracks are padded with repeated still frame sections called gaps.

The following diagram shows a parallel with a short video clip and a longer audio clip. The dashed area represents the implicit gap.

You can also add gaps manually, specifying a duration of the gap and inserting it into a video or audio track. The following diagram shows a parallel with manually added gaps in both video and audio tracks.

Manually added gaps (called explicit gaps) are padded with still frames or silence, just as implicit gaps that are added automatically to match track duration.

Sequences

Parallels are put in sequences. The parallels within a sequence are played sequentially; the first one is played in its entirety, then the next one, and so on. This behaviour is different from how parallels play their tracks. Parallels and sequences, with their different playback behaviors, make up the fundamental building blocks of the compositional editing in Komposition.

The following diagram shows a sequence of two parallels, playing sequentially:

The Timeline

Finally, at the top level, we have the timeline. Effectively, the timeline is a sequence of sequences; it plays every child sequence in sequence. The reason for this level to exist is for the ability to group larger chunks of a screencast within separate sequences.

I use separate sequences within the timeline to delimit distinct parts of a screencast, such as the introduction, the different chapters, and the summary.

Timeline Flattening

Komposition currently uses FFmpeg to render the final media. This is done by constructing an ffmpeg command invocation with a filter graph describing how to fit together all clips, still frames, and silent audio parts.

FFmpeg doesn’t know about hierarchical timelines; it only cares about video and audio streams. To convert the hierarchical timeline into a suitable representation to build the FFmpeg filter graph from, Komposition performs timeline flattening.

The flat representation of a timeline contains only two tracks; audio and video. All gaps are explicitly represented in those tracks. The following graph shows how a hierarchical timeline is flattened into two tracks.

Notice in the graphic above how the implicit gaps at the ends of video and audio tracks get represented with explicit gaps in the flat timeline. This is because FFmpeg does not know how to render implicit gaps. All gaps are represented explicitly, and are converted to clips of still frames or silent audio when rendered with FFmpeg.

Property Tests

To test the timeline flattening, there’s a number of properties that are checked. I’ll go through each one and their property test code.

These properties were primarily written after I already had an implementation. They capture some general properties of flattening that I’ve come up with. In other cases, I’ve written properties before beginning on an implementation, or to uncover an existing bug that I’ve observed.

Thinking about your system’s general behaviour and expressing that as executable property tests is hard. I believe, like with any other skill, that it requires a lot of practice. Finding general patterns for properties, like the ones Scott Wlaschin describe in Choosing properties for property-based testing, is a great place to start. When you struggle with finding properties of your system under test, try applying these patterns and see which work for you.

Property: Duration Equality

Given a timeline t, where all parallels have at least one video clip, the total duration of the flattened t must be equal to the total duration of t. Or, in a more dense notation,

∀t ∈ T → duration(flatten(t)) = duration(t)

where T is the set of timelines with at least one video clip in each parallel.

The reason that all parallels must have at least one video clip is because currently the flattening algorithm can only locate still frames for video gaps from within the same parallel. If it encounters a parallel with no video clips, the timeline flattening fails. This limitation is discussed in greater detail at the end of this article.

The test for the duration equality property is written using Hedgehog, and looks like this:

hprop_flat_timeline_has_same_duration_as_hierarchical =  property $ do  -- 1. Generate a timeline with video clips in each parallel  timeline' <- forAll $  Gen.timeline (Range.exponential 0 5) Gen.parallelWithClips   -- 2. Flatten the timeline and extract the result  let Just flat = Render.flattenTimeline timeline'   -- 3. Check that hierarchical and flat timeline duration are equal  durationOf AdjustedDuration timeline'  === durationOf AdjustedDuration flat

It generates a timeline using forAll and custom generators (1). Instead of generating timelines of any shape and filtering out only the ones with video clips in each parallel, which would be very inefficient, this test uses a custom generator to only obtain inputs that satisfy the invariants of the system under test.

The range passed as the first argument to Gen.timeline is used as the bounds of the generator, such that each level in the generated hierarchical timeline will have at most 5 children.

Gen.timeline takes as its second argument another generator, the one used to generate parallels, which in this case is Gen.parallelWithClips. With Hedgehog generators being regular values, it’s practical to compose them like this. A “higher-order generator” can be a regular function taking other generators as arguments.

As you might have noticed in the assertion (3), durationOf takes as its first argument a value AdjustedDuration. What’s that about? Komposition supports adjusting the playback speed of video media for individual clips. To calculate the final duration of a clip, the playback speed needs to taken into account. By passing AdjustedDuration we take playback speed into account for all video clips.

Sidetrack: Finding a Bug

Let’s say I had introduced a bug in timeline flattening, in which all video gaps weren’t added correctly to the flat video tracks. The flattening is implemented as a fold, and it would not be unthinkable that the accumulator was incorrectly constructed in a case. The test would catch this quickly and present us with a minimal counter-example:

Hedgehog prints the source code for the failing property. Below the forAll line the generated value is printed. The difference between the expected and actual value is printed below the failing assertion. In this case it’s a simple expression of type Duration. In case you’re comparing large tree-like structures, this diff will highlight only the differing expressions. Finally, it prints the following:

This failure can be reproduced by running: > recheck (Size 23) (Seed 16495576598183007788 5619008431246301857) <property>

When working on finding and fixing the fold bug, we can use the printed size and seed values to deterministically rerun the test with the exact same inputs.

Property: Clip Occurence

Slightly more complicated than the duration equality property, the clip occurrence property checks that all clips from the hierarchical timeline, and no other clips, occur within the flat timeline. As discussed in the introduction on timeline flattening, implicit gaps get converted to explicit gaps and thereby add more gaps, but no video or audio clips should be added or removed.

hprop_flat_timeline_has_same_clips_as_hierarchical =  property $ do  -- 1. Generate a timeline with video clips in each parallel  timeline' <- forAll $  Gen.timeline (Range.exponential 0 5) Gen.parallelWithClips   -- 2. Flatten the timeline  let flat = Render.flattenTimeline timeline'   -- 3. Check that all video clips occur in the flat timeline  flat ^.. _Just . Render.videoParts . each . Render._VideoClipPart  === timelineVideoClips timeline'   -- 4. Check that all audio clips occur in the flat timeline  flat ^.. _Just . Render.audioParts . each . Render._AudioClipPart  === timelineAudioClips timeline'

The hierarchical timeline is generated and flattened like before (1, 2). The two assertions check that the respective video clips (3) and audio clips (4) are equal. It’s using lenses to extract clips from the flat timeline, and the helper functions timelineVideoClips and timelineAudioClips to extract clips from the original hierarchical timeline.

Still Frames Used

In the process of flattening, the still frame source for each gap is selected. It doesn’t assign the actual pixel data to the gap, but a value describing which asset the still frame should be extracted from, and whether to pick the first or the last frame (known as still frame mode.) This representation lets the flattening algorithm remain a pure function, and thus easier to test. Another processing step runs the effectful action that extracts still frames from video files on disk.

The decision of still frame mode and source is made by the flattening algorithm based on the parallel in which each gap occur, and what video clips are present before or after. It favors using clips occurring after the gap. It only uses frames from clips before the gap in case there are no clips following it. To test this behaviour, I’ve defined three properties.

Property: Single Initial Video Clip

The following property checks that an initial single video clip, followed by one or more gaps, is used as the still frame source for those gaps.

hprop_flat_timeline_uses_still_frame_from_single_clip =  property $ do  -- 1. Generate a video track generator where the first  -- video part is always a clip  let genVideoTrack = do  v1 <- Gen.videoClip  vs <- Gen.list (Range.linear 1 5) Gen.videoGap  pure (VideoTrack () (v1 : vs))   -- 2. Generate a timeline with the custom video track  -- generator  timeline' <- forAll $ Gen.timeline  (Range.exponential 0 5)  (Parallel () <$> genVideoTrack <*> Gen.audioTrack)   -- 3. Flatten the timeline  let flat = Render.flattenTimeline timeline'   -- 4. Check that any video gaps will use the last frame  -- of a preceding video clip  flat  ^.. ( _Just  . Render.videoParts  . each  . Render._StillFramePart  . Render.stillFrameMode  )  & traverse_ (Render.LastFrame ===)

The custom video track generator (1) always produces tracks with an initial video clip followed by one or more video gaps. The generated timeline (2) can contain parallels with any audio track shape, which may result in a longer audio track and thus an implicit gap at the end of the video track. In either case, all video gaps should padded with the last frame of the initial video clip, which is checked in the assertion (4).

Property: Ending with a Video Clip

In case the video track ends with a video clip, and is longer than the audio track, all video gaps within the track should use the first frame of a following clip.

hprop_flat_timeline_uses_still_frames_from_subsequent_clips =  property $ do  -- 1. Generate a parallel where the video track ends with  -- a video clip, and where the audio track is shorter  let  genParallel = do  vt <-  VideoTrack ()  <$> ( snoc  <$> Gen.list (Range.linear 1 10) Gen.videoPart  <*> Gen.videoClip  )  at <- AudioTrack () . pure . AudioGap () <$> Gen.duration'  (Range.linearFrac  0  (durationToSeconds (durationOf AdjustedDuration vt) - 0.1)  )  pure (Parallel () vt at)   -- 2. Generate a timeline with the custom parallel generator  timeline' <- forAll $ Gen.timeline (Range.exponential 0 5) genParallel   -- 3. Flatten the timeline  let flat = Render.flattenTimeline timeline'   -- 4. Check that all gaps use the first frame of subsequent clips  flat  ^.. ( _Just  . Render.videoParts  . each  . Render._StillFramePart  . Render.stillFrameMode  )  & traverse_ (Render.FirstFrame ===)

The custom generator (1) produces parallels where the video track is guaranteed to end with a clip, and where the audio track is 100 ms shorter than the video track. This ensures that there’s no implicit video gap at the end of the video track. Generating (2) and flattening (3) is otherwise the same as before. The assertion (4) checks that all video gaps uses the first frame of a following clip.

Property: Ending with an Implicit Video Gap

The last property on still frame usage covers the case where the video track is shorter than the audio track. This leaves an implicit gap which, just like explicit gaps inserted by the user, are padded with still frames.

hprop_flat_timeline_uses_last_frame_for_automatic_video_padding =  property $ do  -- 1. Generate a parallel where the video track only contains a video  -- clip, and where the audio track is longer  let  genParallel = do  vt <- VideoTrack () . pure <$> Gen.videoClip  at <- AudioTrack () . pure . AudioGap () <$> Gen.duration'  (Range.linearFrac  (durationToSeconds (durationOf AdjustedDuration vt) + 0.1)  10  )  pure (Parallel () vt at)   -- 2. Generate a timeline with the custom parallel generator  timeline' <- forAll $ Gen.timeline (Range.exponential 0 5) genParallel   -- 3. Flatten the timeline  let flat = Render.flattenTimeline timeline'   -- 4. Check that video gaps (which should be a single gap at the  -- end of the video track) use the last frame of preceding clips  flat  ^.. ( _Just  . Render.videoParts  . each  . Render._StillFramePart  . Render.stillFrameMode  )  & traverse_ (Render.LastFrame ===)

The custom generator (1) generates a video track consisting of video clips only, and an audio track that is 100ms longer. Generating the timeline (2) and flattening (3) are again similar to the previous property tests. The assertion (4) checks that all video gaps use the last frame of preceding clips, even if we know that there should only be one at the end.

Properties: Flattening Equivalences

The last property I want to show in this case study checks flattening at the sequence and parallel levels. While rendering a full project always flattens at the timeline, the preview feature in Komposition can be used to render and preview a single sequence or parallel.

There should be no difference between flattening an entire timeline and flattening all of its sequences or parallels and folding those results into a single flat timeline. This is what the flattening equivalences properties are about.

hprop_flat_timeline_is_same_as_all_its_flat_sequences =  property $ do  -- 1. Generate a timeline  timeline' <- forAll $  Gen.timeline (Range.exponential 0 5) Gen.parallelWithClips   -- 2. Flatten all sequences and fold the resulting flat  -- timelines together  let flat = timeline' ^.. sequences . each  & foldMap Render.flattenSequence   -- 3. Make sure we successfully flattened the timeline  flat /== Nothing   -- 4. Flatten the entire timeline and compare to the  -- flattened sequences  Render.flattenTimeline timeline' === flat

The first property generates a timeline (1) where all parallels have at least one video clip. It flattens all sequences within the timeline and folds the results together (2). Folding flat timelines together means concatenating their video and audio tracks, resulting in a single flat timeline.

Before the final assertion, it checks that we got a result (3) and not Nothing. As it’s using the Gen.parallelWithClips generator there should always be video clips in each parallel, and we should always successfully flatten and get a result. The final assertion (4) checks that rendering the original timeline gives the same result as the folded-together results of rendering each sequence.

The other property is very similar, but operates on parallels rather than sequences:

hprop_flat_timeline_is_same_as_all_its_flat_parallels =  property $ do  -- 1. Generate a timeline  timeline' <- forAll $  Gen.timeline (Range.exponential 0 5) Gen.parallelWithClips   -- 2. Flatten all parallels and fold the resulting flat  -- timelines together  let flat = timeline' ^.. sequences . each . parallels . each  & foldMap Render.flattenParallel   -- 3. Make sure we successfully flattened the timeline  flat /== Nothing   -- 4. Flatten the entire timeline and compare to the  -- flattened parallels  Render.flattenTimeline timeline' === flat

The only difference is in the traversal (2), where we apply Render.flattenParallel to each parallel instead of applying Render.flattenSequence to each sequence.

Missing Properties

Whew! That was quite a lot of properties and code, especially for a warm-up. But timeline flattening could be tested more thoroughly! I haven’t yet written the following properties, but I’m hoping to find some time to add them:

• Clip playback timestamps are the same. The “clip occurrence” property only checks that the hierarchical timeline’s clips occur in the flat timeline. It doesn’t check when in the flat timeline they occur. One way to test this would be to first annotate each clip in original timeline with its playback timestamp, and transfer this information through to the flat timeline. Then the timestamps could be included in the assertion.

• Source assets used as still frame sources. The “still frames used” properties only check the still frame mode of gaps, not the still frame sources. The algorithm could have a bug where it always uses the first video clip’s asset as a frame source, and the current property tests would not catch it.

• Same flat result is produced regardless of sequence grouping. Sequences can be split or joined in any way without affecting the final rendered media. They are merely ways of organizing parallels in logical groups. A property could check that however you split or join sequences within a timeline, the flattened result is the same.

A Missing Feature

As pointed out earlier, parallels must have at least one video clip. The flattening algorithm can only locate still frame sources for video gaps from within the same parallel. This is an annoying limitation when working with Komposition, and the algorithm should be improved.

As the existing set of properties describe timeline flattening fairly well, changing the algorithm could be done with a TDD-like workflow:

1. Modify the property tests to capture the intended behaviour
2. Tests will fail, with the errors showing how the existing implementation fails to find still frame sources as expected
3. Change the implementation to make the tests pass

PBT is not only an after-the-fact testing technique. It can be used much like conventional example-based testing to drive development.

Obligatory Cliff-Hanger

In this post we’ve looked at timeline flattening, the simplest case study in the “Property-Based Testing in a Screencast Editor” series. The system under test was a module of pure functions, with complex enough behaviour to showcase PBT as a valuable tool. The tests are more closely related to the design choices and concrete representations of the implementation.

Coming case studies will dive deeper into the more complex subsystems of Komposition, and finally we’ll see how PBT can be used for integration testing. At that level, the property tests are less tied to the implementation, and focus on describing the higher-level outcomes of the interaction between subsystems.

Next up is property tests for the video classifier. It’s also implemented a pure function, but with slightly more complicated logic that is trickier to test. We’re going to look at an interesting technique where we generate the expected output instead of the input.

1. Introduction
2. Timeline Flattening
3. Video Scene Classification
4. Integration Testing

Thanks for reading! See you next time.

Credits

Thank you Chris Ford and Ulrik Sandberg for proof-reading and giving valuable feedback on drafts of this post.

Buy the Book

This series is now available as an ebook on Leanpub. While the content is mostly the same, there are few changes bringing it up-to-date. Also, if you’ve already enjoyed the articles, you might want support my work by purchasing this book. Finally, you might enjoy a nicely typeset PDF, or an EPUB book, over a web page.

Footnotes

Future posts will focus on individual case studies, covering increasingly complex components and how they are tested. I’ll reflect on what I’ve learned in each case, what bugs the tests have found, and what still remains to be improved.

For example, I’ll explain how using PBT helped me find and fix bugs in the specification and implementation of Komposition’s video classifier. Those were bugs that would be very hard to find using example-based tests or using a static type system!

This series is not a tutorial on PBT, but rather a collection of motivating examples. That said, you should be able to follow along without prior knowledge of PBT.

Komposition

In early 2018 I started producing Haskell screencasts. A majority of the work involved cutting and splicing video by hand in a non-linear editing system (NLE) like Premiere Pro or Kdenlive. I decided to write a screencast editor specialized for my needs, reducing the amount of manual labor needed to edit the recorded material. Komposition was born.

Komposition is a modal GUI application built for editing screencasts. Unlike most NLE systems, it features a hierarchical timeline, built out of sequences, parallels, tracks, clips, and gaps. To make the editing experience more efficient, it automatically classifies scenes in screen capture video, and sentences in recorded voice-over audio.

If you are curious about Komposition and want to learn more right away, check out its documentation.

Some of the most complex parts of Komposition include focus and timeline transformations, video classification, video rendering, and the main application logic. Those are the areas in which I’ve spent most effort writing tests, using a combination of example-based and property-based testing.

I’ve selected the four most interesting areas where I’ve applied PBT in Komposition, and I’ll cover one in each coming blog post:

1. Timeline flattening
2. Video scene classification
3. Focus and timeline consistency
4. Symmetry of undo/redo

I hope these case studies will be motivating, and that they will show the value of properties all the way from unit testing to integration testing.

Property-Based Testing

To get the most out of this series, you need a basic understanding of what PBT is, so let’s start there. For my take on a minimal definition, PBT is about:

1. Specifying your system under test in terms of properties, where properties describe invariants of the system based on its input and output.
2. Testing that those properties hold against a large variety of inputs.

It’s worth noting that PBT is not equal to QuickCheck, or any other specific tool, for that matter. The set of inputs doesn’t have to be randomly generated. You don’t have to use “shrinking”. You don’t have to use a static type system or a functional programming language. PBT is a general idea that can be applied in many ways.

The following resources are useful if you want to learn more about PBT:

• The introductory articles on Hypothesis, although specific to Python.
• “What is Property Based Testing?” by David R. MacIver is a definition of what PBT is, and particularly what it isn’t.

The code examples will be written in Haskell and using the Hedgehog testing system. You don’t have to know Haskell to follow this series, as I’ll explain the techniques primarily without code. But if you are interested in the Haskell specifics and in Hedgehog, check out “Property testing with Hedgehog” by Tim Humphries.

Properties of the Ugly Parts

When I started with PBT, I struggled with applying it to anything beyond simple functions. Examples online are often focused on the fundamentals. They cover concepts like reversing lists, algebraic laws, and symmetric encoders and decoders. Those are important properties to test, and they are good examples for teaching the foundations of PBT.

I wanted to take PBT beyond pure and simple functions, and leverage it on larger parts of my system. The “ugly” parts, if you will. In my experience, the complexity of a system often becomes much higher than the sum of its parts. The way subsystems are connected and form a larger graph of dependencies drives the need for integration testing at an application level.

Finding resources on integration testing using PBT is hard, and it might drive you to think that PBT is not suited for anything beyond the introductory examples. With the case studies in this blog series I hope to contribute to debunking such misconceptions.

Designing for Testability

In my case, it’s a desktop multimedia application. What if we’re working on a backend that connects to external systems and databases? Or if we’re writing a frontend application with a GUI driven by user input? In addition to these kinds of systems being hard to test at a high level due to their many connected subsystems, they usually have stateful components, side effects, and non-determinism. How do we make such systems testable with properties?

Well, the same way we would design our systems to be testable with examples. Going back to “Writing Testable Code” by Miško Hevery from 2008, and Kent Beck’s “Test-Driven Development by Example” from 2003, setting aside the OOP specifics, many of their guidelines apply equally well to code tested with properties:

• Determinism: Make it possible to run the “system under test” deterministically, such that your tests can be reliable. This does not mean the code has to be pure, but you might need to stub or otherwise control side effects during your tests.
• No global state: In order for tests to be repeatable and independent of execution order, you might have to rollback database transactions, use temporary directories for generated files, stub out effects, etc.
• High cohesion: Strive for modules of high cohesion, with smaller units each having a single responsibility. Spreading closely related responsibilities thin across multiple modules makes the implementation harder to maintain and test.
• Low coupling: Decrease coupling between interface and implementation. This makes it easier to write tests that don’t depend on implementation details. You may then modify the implementation without modifying the corresponding tests.

I find these guidelines universal for writing testable code in any programming language I’ve used professionally, regardless of paradigm or type system. They apply to both example-based and property-based testing.

Patterns for Properties

Great, so we know how to write testable code. But how do we write properties for more complex units, and even for integration testing? There’s not a lot of educational resources on this subject that I know of, but I can recommend the following starting points:

• “Choosing properties for property-based testing” by Scott Wlaschin, giving examples of properties within a set of common categories.
• The talk “Property-Based Testing for Better Code” by Jessica Kerr, with examples of generating valid inputs and dealing with timeouts.

Taking a step back, we might ask “Why it’s so hard to come up with these properties?” I’d argue that it’s because doing so forces us to understand our system in a way we’re not used to. It’s challenging understanding and expressing the general behavior of a system, rather than particular anecdotes that we’ve observed or come up with.

If you want to get better at writing properties, the only advice I can give you (in addition to studying whatever you can find in other projects) is to practice. Try it out on whatever you’re working on. Talk to your colleagues about using properties in addition to example-based tests at work. Begin at a smaller scale, testing simple functions, and progress towards testing larger parts of your system once you’re comfortable. It’s a long journey, filled with reward, surprise, and joy!

Testing Case Studies

With a basic understanding of PBT, how we can write testable code, and how to write properties for our system under test, we’re getting ready to dive into the case studies:

1. Introduction
2. Timeline Flattening
3. Video Scene Classification
4. Integration Testing

Credits

Thank you Chris Ford, Alejandro Serrano Mena, Tobias Pflug, Hillel Wayne, and Ulrik Sandberg for kindly providing your feedback on my drafts!

Buy the Book

This series is now available as an ebook on Leanpub. While the content is mostly the same, there are few changes bringing it up-to-date. Also, if you’ve already enjoyed the articles, you might want support my work by purchasing this book. Finally, you might enjoy a nicely typeset PDF, or an EPUB book, over a web page.

The Beginning

This journey started in January 2018. Having a wave of inspiration after watching some of Gary Bernhardt’s new videos, I decided to try making my own videos about practical Haskell programming. Not only producing high-quality content, but with high video and audio quality, was the goal. Haskell at Work was born, and the first video was surprisingly well-received by followers on Twitter.

With the subsequent episodes being published in rapid succession, a follower base on YouTube grew quickly. A thousand or so followers might not be exceptional for a programming screencast channel on YouTube, but to me this was exciting and unexpected. To be honest, Haskell is not exactly a mainstream programming language.

Early on, encouraged by some followers, and being eager to develop the concept, I decided to set up Patreon as a way of people to donate to Haskell at Work. Much like the follower count, the number of patrons and their monthly donations grew rapidly, beyond any hopes I had.

Fatigue Kicks In

The majority of screencasts were published between January and May. Then came the summer and my month-long vacation, in which I attended ZuriHac and spent three weeks in Bali with my wife and friends. Also, I had started getting side-tracked by my project to build a screencast video editor in Haskell. Working on Komposition also spawned the Haskell package gi-gtk-declarative, and my focus got swept away from screencasts. In all fairness, I’m not great at consistently doing one thing for an extended period. My creativity and energy comes in bursts, and it may not strike where and when I hope. Maybe this can be managed or controlled somehow, but I don’t know how.

With the lower publishing pace over the summer, a vicious circle of anxiety and low productivity grew. I had thoughts about shutting down the Patreon back then, but decided to instead pause it for a few months.

Regaining Energy

By October, I had recovered some energy. I got very good feedback and lots of encouragement from people at Haskell eXchange, and decided to throw myself back into the game. I published one screencast in November, but something was still there nagging me. I felt pressure and guilt. That I had not delivered on the promise given.

By this time, the Patreon donations had covered my recording equipment expenses, hosting costs over the year, and a few programming books I bought. The donations were still coming in, however, at around $160 per month, with me producing no obvious value for the patrons. The guilt was still there, even stronger than before.

I’m certain that this is all in my head. I do not blame any supporter for these feelings. You have all been great! With all the words of caution you hear about not reading the comments, having a YouTube channel filled with positive feedback, and almost exclusively thumbs-up ratings, I’m beyond thankful for the support I have received.

Trying Something Else

After Christmas this year, I had planned to record and publish a new screencast. Various personal events got in the way, though, and I had very little time to spend on working with it, resulting in the same kind of stress. I took a step back and thought about it carefully, and I’ve realized that money is not a good driver for the free material and open-source code work that I do, and that it’s time for a change.

I want to make screencasts because I love doing it, and I will do so when I have time and energy.

From the remaining funds in my PayPal account, I have allocated enough to keep the domain name and hosting costs covered for another year, and I have donated the remaining amount (USD 450) to Haskell.org.

Please keep giving me feedback and suggestions for future episodes. Your ideas are great! I’m looking forward to making more Haskell at Work videos in the future, and I’m toying around with ideas on how to bring in guests, and possibly trying out new formats. Stay tuned, and thank you all for your support!

Background

It all began with Haskell at Work, the series of screencasts focused on practical Haskell that I’ve been producing the last year. The screencasts are fast-paced tutorials centered around the terminal and text editor, complemented by a voice-over audio track.

My workflow for producing screencasts looks like this:

1. Write a very detailed script. The script usually gets uploaded to the website as-is, being used as the show notes for the screencast.
2. Record video separately, including all mistakes and retries, in a single video file. Each little part of editing or running commands in the terminal is separated by a rest, where I don’t type anything at all for a few seconds.
3. Record audio separately, where the voice-over audio is based on the script. A mistake or mispronunciation is corrected by taking a small break and then trying the same sentence or paragraph again.
4. Cut and arrange all parts of the screencast using a video editor. This is the most painful and time-consuming part of the process.

I’ve found this workflow beneficial for me, partly because of being a non-native English speaker and not being keen on coding and talking simultaneously, but also because I think it helps with organizing the content into a cohesive narrative. I can write the script almost as a text-based tutorial, read it through to make sure it flows well, and then go into recording. Having to redo the recording phase is very time-consuming, so I’m putting a lot of effort into the script phase to catch mistakes early.

Video Editors

I’ve used a bunch of video editors. First, I tried free software alternatives, including Kdenlive, OpenShot, and a few more. Unfortunately, the audio effects available were a bit disappointing. I use, at a minimum, normalization and a noise gate. Having a good compressor is a plus.

More importantly, these applications are built for general video editing, like film shot with a camera, and they are optimized for that purpose. I’m using a very small subset of their feature set, which is not suited to my editing workflow. It works, but the tasks are repetitive and time-consuming.

On the commercial side, there are applications like Premiere Pro and Final Cut Pro. These are proprietary and expensive systems, but they offer very good effects and editing capabilities. I used Premiere Pro for a while and enjoyed the stability and quality of the tools, but I still suffered from the repetitive workload of cutting and organizing the video and audio clips to form my screencasts.

What does a programmer do when faced with a repetitive task? Spend way more time on automating it! Thus, I started what came to be the greatest yak shave of my life.

Building a Screencast Video Editor

I decided to build a screencast video editor tailored to my workflow; an editor with a minimal feature set, doing only screencast editing, and doing that really well.

You might think “why not extend the free software editors to cover your needs?” That is a fair question. I wanted to rethink the editing experience, starting with a blank slate, and question the design choices made in the traditional systems. Also, to be honest, I’m not so keen on using my spare time to write C++.

I decided to write it in GHC Haskell, using GTK+ for the graphical user interface. Another option would be Electron and PureScript, but the various horror stories about Electron memory usage, in combination with it running on NodeJS, made me decide against it. As I expected my application to perform some performance-critical tasks around video and audio processing, Haskell seemed like the best choice of the two. There are many other languages and frameworks that could’ve been used, but this combination fit me well.

Komposition

About five months later, after many hours of hacking, Komposition was released open-source under the Mozilla Public License 2.0. The project working name was “FastCut,” but when making it open source I renamed it to the Swedish word for “composition.” It has nothing to do with KDE.

Komposition is a modal, cross-platform, GUI application. The modality means that you’re always in exactly one mode, and that only certain actions can be taken depending on the mode. It currently runs on Linux, Windows, and macOS, if you compile and install it from source.

At the heart of the editing model lies the hierarchical timeline, which we’ll dive into shortly. Other central features of Komposition include the automatic video and audio classification tools. They automate the tedious task of working through your recorded video and audio files to cut out the interesting parts. After importing, you’ll have a collection of classified video scenes, and a collection of classified audio parts. The audio parts are usually sentences, but it depends on how you take rests when recording your voice-over audio.

Keyboard-Driven Editing

Komposition is built for keyboard-driven editing, currently with Vim-like bindings, and commands transforming the hierarchical timeline, inspired by Paredit for Emacs.

There are corresponding menu items for most commands, and there’s limited support for using the mouse in the timeline. If you need help with keybindings, press the question mark key, and you will be presented with a help dialog showing the bindings available in the current mode.

The Hierarchical Timeline

The hierarchical timeline is a tree structure with a fixed depth. At the leaves of the tree there are clips. Clips are placed in the video and audio tracks of a parallel. It’s named that way because the video and audio tracks play in parallel. Clips within a track play in sequence.

If the audio track is longer than the video track, the remaining part of the video track is padded with still frames from an adjacent clip.

Explicit gaps can be added to the video and audio tracks. Video gaps are padded with still frames, and audio gaps are silent. When adding the gap, you specify its duration.

Parallels are put in sequences, and they are played in sequence. The first parallel is played until its end, then the next is played, and so on. Parallels and sequences are used to group cohesive parts of your screencast, and to synchronize the start of related video and audio clips.

When editing within a sequence or parallel, for example when deleting or adding clips, you will not affect the synchronization of other sequences or parallels. If there were only audio and video tracks in Komposition, deleting an audio clip would possibly shift many other audio clips, causing them to get out of sync with their related video clips. This is why the timeline structure is built up using sequences and parallels.

Finally, the timeline is the top-level structure that contains sequences. This is merely for organizing larger parts of a screencast. You can comfortably build up your screencast with a single sequence containing parallels.

Note that the timeline always contains at least one sequence, and that all sequences contain at least one parallel. The tracks within a parallel can be empty, though.

Documentation

The project website includes a user guide, covering the concepts of the application, along with recommendations on how to plan and record your screencasts to achieve the best results and experience using Komposition.

The landing page features a tutorial screencast, explaining how to import, edit, and render a screencast. It’s already a bit outdated, but I might get around to making an updated version when doing the next release. Be sure to check it out, it’ll give you a feel for how editing with Komposition works.

I can assure you, editing a screencast, that is about editing screencasts using your screencast editor, in your screencast editor, is quite the mind-bender. And I thought I had recursion down.

Implementation

I’ve striven to keep the core domain code in Komposition pure. That is, only pure function and data structures. Currently, the timeline and focus, command and event handling, key bindings, and the video classification algorithm are all pure. There are still impure parts, like audio and video import, audio classification, preview frame rendering, and the main application control flow.

Some parts are inherently effectful, so it doesn’t make sense to try writing them as pure functions, but as soon as the complexity increases, it’s worth considering what can be separated out as pure functions. The approach of “Functional core, imperative shell” describes this style very well. If you can do this for the complex parts of your program, you have a great starting point for automated testing, something I’ll cover later in this post.

GTK+

Komposition’s GUI is built with GTK+ 3 and the Haskell bindings from gi-gtk. I noticed early in the project that the conventional programming style and the APIs of GTK+ were imperative, callback-oriented, and all operating within IO, making it painful to use from Haskell, especially while trying to keep complex parts of the application free of effects.

To mitigate this issue, I started building a library (more yak shaving!) called gi-gtk-declarative, which is a declarative layer on top of gi-gtk. The previous post in this blog describes the project in detail. Using the declarative layer, rendering becomes a pure function (state -> Widget event), where state and event varies with the mode and view that’s being rendered. Event handling is based on values and pure functions.

There are cases where custom widgets are needed, calling the imperative APIs of gi-gtk, but they are well-isolated and few in numbers.

Type-Indexed State Machines

I had a curiosity itching when starting this project that I decided to scratch. Last year I worked on porting the Idris ST library, providing a way to encode type-indexed state machines in GHC Haskell. The library is called Motor. I wanted to try it in the context of a GUI application.

Just to give some short examples, the following type signatures, used in the main application control flow, operate on the application state machine that’s parameterized by its mode.

The start function takes a name and key maps, and creates a new application state machine associated with the name, and in the state of WelcomeScreenMode:

start  :: Name n  -> KeyMaps  -> Actions m '[ n !+ State m WelcomeScreenMode] r ()

The returnToTimeline function takes a name of an existing state machine and a TimelineModel, and transitions the application from the current mode to TimelineMode, given that the current mode instantiates the ReturnsToTimeline class:

returnToTimeline  :: ReturnsToTimeline mode  => Name n  -> TimelineModel  -> Actions m '[ n := State m mode !--> State m TimelineMode] r ()

The usage of Motor in Komposition is likely the most complicated aspect of the codebase, and I have been unsure if it is worth the cost. On the positive side, combining this with GADTs and the singleton pattern for mode-specific commands and events, GHC can really help out with pattern-matching and exhaustivity-checking. No nasty (error "this shouldn't happen") as the fall-through case when pattern matching!

I’m currently in the process of rewriting much of the state machine encoding in Komposition, using it more effectively for managing windows and modals in GTK+, and I think this warrants the use of Motor more clearly. Otherwise, it might be worth falling back to a less advanced encoding, like the one I described in Finite-State Machines, Part 2: Explicit Typed State Transitions.

Singleton Pattern

The singleton pattern is used in a few places in Komposition, as mentioned above. To show a concrete example, the encoding of mode-specific commands and events is based on the Mode data type.

data Mode  = WelcomeScreenMode  | TimelineMode  | LibraryMode  | ImportMode

This type is lifted to the kind level using the DataKinds language extension, and is used in the corresponding definition of the singleton SMode.

data SMode m where  SWelcomeScreenMode :: SMode WelcomeScreenMode  STimelineMode :: SMode TimelineMode  SLibraryMode :: SMode LibraryMode  SImportMode :: SMode ImportMode

The Command data type is parameterized by the mode in which the command is valid. Some commands are valid in all modes, like Cancel and Help, while others are specific to a certain mode. FocusCommand and JumpFocus are only valid in the timeline mode, as seen in their type signatures below.

data Command (mode :: Mode) where  Cancel :: Command mode  Help :: Command mode  FocusCommand :: FocusCommand -> Command TimelineMode  JumpFocus :: Focus SequenceFocusType -> Command TimelineMode  -- ...

Finally, by passing a singleton for a mode to the keymaps function, we get back a keymap specific to that mode. This is used to do event handling and key bindings generically for the entire application.

keymaps :: SMode m -> KeyMap (Command m) keymaps =  \case  SWelcomeScreenMode ->  [ ([KeyChar 'q'], Mapping Cancel)  , ([KeyEscape], Mapping Cancel)  , ([KeyChar '?'], Mapping Help)  ]  -- ...

In the spirit of calling out usage of advanced GHC features, I think singletons and GADTs are one more such instance. However, I find them very useful in this context, and worth the added cognitive load. You don’t have to go full “Dependent Haskell” or bring in the singletons library to leverage some of these techniques.

Automatic Scene Classification

The automatic classification of scenes in video is implemented using the Pipes and ffmpeg-light libraries, mainly. It begins with the readVideoFile, that given a video file path will give us a Pipes.Producer of timed frames, which are basically pixel arrays tagged with their time in the original video. The producer will stream the video file, and yield timed frames as they are consumed.

readVideoFile :: MonadIO m => FilePath -> Producer (Timed Frame) m ()

The frames are converted from JuicyPixels frames to massiv frames. Then, the producer is passed to the classifyMovement function together with a minimum segment duration (such that segments cannot be shorter than N seconds), which returns a producer of Classified frames, tagging each frame as being either moving or still.

classifyMovement  :: Monad m  => Time -- ^ Minimum segment duration  -> Producer (Timed RGB8Frame) m ()  -> Producer (Classified (Timed RGB8Frame)) m ()  data Classified f  = Moving f  | Still f  deriving (Eq, Functor, Show)

Finally, the classifyMovingScenes function, given a full duration of the original video and a producer of classified frames, returns a producer that yields ProgressUpdate values and returns a list of time spans.

classifyMovingScenes ::  Monad m  => Duration -- ^ Full length of video  -> Producer (Classified (Timed RGB8Frame)) m ()  -> Producer ProgressUpdate m [TimeSpan]

The time spans describe which parts of the original video are considered moving scenes, and the progress update values are used to render a progress bar in the GUI as the classification makes progress.

Automatic Sentence Classification

Similar to the video classification, Komposition also classifies audio files to find sentences or paragraphs in voice-over audio. The implementation relies on the sox tool, a separate executable that’s used to:

1. normalize the audio,
2. apply a noise gate, and
3. auto-split by silence.

One problem with sox is that it, as far as I can tell, can only write the split audio files to disk. I haven’t found a way to retrieve the time spans in the original audio file, so that information is unfortunately lost. This will become more apparent when Komposition supports editing the start and end position of clips, as it can’t be supported for audio clips produced by sox.

I hope to find some way around this, by extending or parsing output from sox somehow, by using libsox through FFI bindings, or by implementing the audio classification in Haskell. I’m trying to avoid the last alternative.

Rendering

The rendering pipeline begins with a pure function that converts the hierarchical timeline to a flat timeline representation. This representation consists of a video track and an audio track, where all gaps are made explicit, and where the tracks are of equal duration.

From the flat representation, an FFmpeg command is built up. This is based on a data type representation of the FFmpeg command-line syntax, and most importantly the filter graph DSL that’s used to build up the complex FFmpeg rendering command.

Having an intermediate data type representation when building up complex command invocations, instead of going directly to Text or String, is something I highly recommend.

Preview

The preview pipeline is very similar to the rendering pipeline. In fact, it’s using the same machinery, except for the output being a streaming HTTP server instead of a file, and that it’s passed the proxy media instead of the full-resolution original video. On the other side there’s a GStreamer widget embedded in the GTK+ user interface that plays back the HTTP video stream.

Using HTTP might seem like a strange choice for IPC between FFmpeg and GStreamer. Surprisingly, it’s the option that have worked most reliably across operating systems for me, but I’d like to find another IPC mechanism, eventually.

The HTTP solution is also somewhat unreliable, as I couldn’t find a way to ensure that the server is ready to stream, so there’s a race condition between the server and the GStreamer playback, silently “solved” with an ugly threadDelay.

Testing

Let’s talk a bit about testing in Komposition. I’ve used multiple techniques, but the one that stands out as unconventional, and specific to the domain of this application, is the color-tinting video classifier.

It uses the same classification functions as described before, but instead of returning time spans, it creates a new video file where moving frames are tinted green and still frames are tinted red. This tool made it much easier to tweak the classifier and test it on real recordings.

Property-Based Testing

I’ve used Hedgehog to test some of the most complex parts of the codebase. This has been incredibly valuable, and has found numerous errors and bad assumptions in the implementation. The functionality tested with Hedgehog and properties includes:

• Timeline commands and movement: It generates a sequence of commands, together with a consistent timeline and focus. It folds over the commands, applying each one to the current timeline and focus, and asserts that the resulting timeline and focus are still consistent. The tested property ensures that there’s no possibility of out-of-bounds movement, and that deleting or otherwise transforming the timeline doesn’t cause an inconsistent timeline and focus pair.
• Video scene classification: It generates known test scenes of random duration, that are either scenes of only still frames, or scenes with moving frames. It translates the test scenes, which are just descriptions, to real frames, and runs the classifier on the frames. Finally, it checks that the classified scenes are the same as the generated test scenes.
• Flattening of hierarchical timeline: The flattening process converts the hierarchical timeline to a flat representation. The tested property ensures that hierarchical and flat timelines are always of the same total duration. There are other properties that could be added, e.g. that all clips in the original timeline are present in the flat timeline.
• Round-trip properties of FFmpeg format printers and parsers: This is a conventional use of property-based tests. It ensures that parsing an FFmpeg-format timestamp string, produced by the FFmpeg-format timestamp printer, gives you back the same timestamp as you started with.

There are also cases of example-based testing, but I won’t cover them in this report.

Used Packages

Komposition depends on a fairly large collection of Haskell and non-Haskell tools and libraries to work with video, audio, and GUI rendering. I want to highlight some of them.

haskell-gi

The haskell-gi family of packages are used extensively, including:

• gi-gobject
• gi-glib
• gi-gst
• gi-gtk
• gi-gdk
• gi-gdkpixbuf
• gi-pango

They supply bindings to GTK+, GStreamer, and more, primarily generated from the GObject Introspection metadata. While GTK+ has been problematic to work with in Haskell, these bindings have been crucial to the development of Komposition.

massiv & massiv-io

The massiv package is an array library that uses function composition to accomplish a sort of fusion. It’s used to do parallel pixel comparison in the video classifier. Thank you, Alexey Kuleshevich (author and maintainer of the massiv packages) for helping me implement the first version!

Pipes

The Pipes library is used extensively in Komposition:

• The streaming video reader from ffmpeg-light is wrapped in a Pipes.Producer to provide composable streaming.
• In general, effectful operations with progress notifications are producers that yield ProgressUpdate values as they perform their work.
• pipes-safe is used for handling resources and processes.
• pipes-parse is used in stateful transformations in the video classifier.

A big thanks to Gabriel Gonzales, the author of Pipes and the related packages!

Others

To name a few more:

• I’ve used protolude as the basis for a custom prelude.
• The lens library is used for working with nested data structures, positional updates in lists, and monadic transformations.
• typed-process is used together with pipes-safe, in a situation where I couldn’t use the regular process package because of version constraint issues. The typed-process API turned out to be really nice, so I think it will be used more in the future.

Summary

Looking back at this project, the best part has been to first write it for my own use, and later find out that quite a lot of people are interested in how it’s built, and even in using it themselves. I’ve already received pull requests, bug reports, usability feedback, and many kind words and encouragements. It’s been great!

Also, it’s been fun to work on an application that can be considered outside of Haskell’s comfort zone, namely a multimedia and GUI application. Komposition is not the first application to explore this space — see Movie Monad and Gifcurry for other examples — but it is exciting, nonetheless.

Speaking of using Haskell, the effort to keep complex domain logic free of effects, and the use of property-based testing with Hedgehog to lure out nasty bugs, has been incredibly satisfactory and a great learning experience.

The Problematic Parts

It’s not been all fun and games, though. I’ve spent many hours struggling with FFmpeg, video and audio codecs, containers, and streaming. Executing external programs and parsing their output has been time-consuming and very hard to test. GTK+ has been very valuable, but also difficult to work with in Haskell. Finally, management of non-Haskell dependencies, in combination with trying to be cross-platform, is painful. Nix has helped with my own setup, but everyone will not install Komposition using Nix.

Next Steps

There are many features that I’d like to add in the near future.

• More commands to work with the timeline, e.g. yank, paste, and join.
• More Vim-like movement commands.
• Previewing of any timeline part. Currently you can only preview the entire timeline, a sequence, or a parallel.
• Adjustable clips, meaning that you can change the time span of a clip. This is useful if the automatic classification generated slightly incorrect clip time spans.
• Content-addressed project files, to enable reuse of generated files, and to avoid collision. This includes most files involved in importing, and generated preview frames.