Having an LLM write its own tests is like asking a dog whether it’s been a good boy. What does it mean to be a good boy? All your dog knows is that it yearns to say, yes, I have been a good boy.
If asked to formalize this, your dog might come back with:
Then, it will devise dozens of permutations of sitting, over and over again, unsure which aspect of sitting it is which upon crossing its threshold one has finally become Good but always looking up with doe eyes full of treats. “Yes,” it beckons you say.
But no! You are its master. You lay out what is and is not a good boy, and then beckon your dog to do trick after trick and see whether those parameters are met.
your tests should be data
What you want is for the LLM to produce test cases, not tests.
Fortunately, this is exactly where LLMs are at the peak of their code generating powers: Stamping out a ton of text which almost, but not quite, follows a pattern. They’re so good at this that it’s a problem, taking tiny mistakes as prophecies from a severe God and using them to creating new, more powerful mistakes. In software, this is called leverage.
But we also have a word for things which follow a pattern of structure but not content. In software, this is called “data”.
an example from spn
I am writing the missing everything tool for C, a Cargo-for-C if you will. It’s a build executor, package manager, a workspace manager. But the main thing it is is a gigantic pain in the ass to test. It’s exactly the worst kind of combinatorial explosion of OS, architecture, combinations of packages, dozens of disjoint features in TOML.
One such feature: spn lets you gate almost anything on the parameters of the build:
source = [
"main.c",
{ value = "windows.c", when = { os = "windows" } },
{ value = "linux.c", when = { os = "linux" } },
]
Internally, a pure function called apply_options takes a package plus the build parameters and produces the final list of values (in this case source files, but in general any field in TOML).
this fucking sucks
When I wrote it, I had Claude write some tests for it. I got a dozen tests that looked like this. Normally, I’d simplify the real code to trim the noise of prefixes and custom types and allocators, but here it’s important. The value of a test lies in your ability to understand its correctness at a glance.
Still, don’t read this too closely:
sp_test(options_apply, gated_source_linux) {
sp_mem_t mem = sp_test_arena(t);
// Set up a library with one plain source
spn_pkg_info_t info = sp_zero;
sp_str_om_insert(info.libs, sp_str_lit("mylib"), sp_zero_s(spn_target_info_t));
spn_target_info_t* lib = sp_str_om_at(info.libs, 0);
lib->source = sp_da_new(mem, spn_path_t);
sp_da_push(lib->source, ((spn_path_t) { .sub = sp_str_lit("main.c") }));
// Set up one source gated on `os = "linux"`
spn_when_t when = { .clauses = sp_da_new(mem, spn_when_clause_t) };
sp_da_push(when.clauses, ((spn_when_clause_t) {
.key = sp_str_lit("os"),
.value = spn_option_value_str(sp_str_lit("linux")),
}));
lib->gated.source = sp_da_new(mem, spn_gated_path_t);
sp_da_push(lib->gated.source, ((spn_gated_path_t) {
.path = sp_str_lit("nix.c"),
.tree = SPN_TREE_SOURCE,
.when = when,
}));
// Say that we're building on linux
spn_when_env_t env = sp_zero;
spn_when_env_init(mem, &env);
spn_when_env_set_facts(&env, (spn_when_facts_t) { .os = SPN_OS_LINUX });
// Make the call we want to test and verify
spn_path_roots_t roots = sp_zero;
spn_tree_roots_t trees = sp_zero;
spn_pkg_apply_options(mem, &info, &roots, trees, &env);
sp_must_eq(t, sp_da_size(lib->source), 2);
sp_expect_str_eq_c(t, lib->source[1].sub, "nix.c");
return SP_OK;
}
This fucking sucks. Every line of it looks plausible. Unfortunately, LLMs love to generate plausible-looking code which is nevertheless total fucking slop. The only thing they love more than that is seeing the aforementioned slop pass a test. It doesn’t matter if the test means anything.
And worst of all, there’s twelve of these fucking things. Let’s fix it.
turn it into data
describe your input
The state is a list of values, each with zero or more when clauses.
.source = {
{ .value = "main.c" },
{ .value = "win.c", .when = { { "os", "windows" } } },
{ .value = "linux.c", .when = { { "os", "linux" } } },
},
You don’t need to know C to understand the snippet, but all we’re doing is filling out a descriptor struct. It can be tempting to use the real type from your program, but real types aren’t meant to look pretty in a list of test cases. For example, in C, these are the bespoke types that the test case is filling in:
typedef struct {
const c8* key;
const c8* value;
bool negated;
} clause_t;
typedef struct {
const c8* value;
clause_t when [2];
} gated_t;
The function takes the gated values plus the build parameters. We’ll keep it simple and omit everything but what OS we’re building for. It’s important that in your test case type omitting a field is valid semantically, or you’ll find yourself slogging through boilerplate.
.facts = { .os = SPN_OS_LINUX },
describe how to verify it
Expectations. On Linux, we’d expect the function to spit out the following two source files after it’s all said and done:
.expect = { "main.c", "linux.c" },
I’m more or less terrified of shipping a giant piece of shit that doesn’t work, so there are a lot of tests. Despite the scale, and despite the wildly varying domains, every test looks exactly the same:
write ONE function
Now, we can strip the gross middle of Claude’s version and use the rest. I’m still leaving the messiness of reality, but you can see how you could factor this down to a couple functions trivially.
{
sp_mem_t mem = sp_test_arena(t);
spn_pkg_info_t info = sp_zero;
sp_str_om_insert(info.libs, sp_str_lit("mylib"), sp_zero_s(spn_target_info_t));
spn_target_info_t* lib = sp_str_om_at(info.libs, 0);
make_sources(mem, it->source, &lib->source, &lib->gated.source);
spn_when_env_t env = sp_zero;
spn_when_env_init(mem, &env);
spn_when_env_set_facts(&env, it->facts);
spn_path_roots_t roots = sp_zero;
spn_tree_roots_t trees = sp_zero;
spn_pkg_apply_options(mem, &info, &roots, trees, &env);
return expect_paths(t, lib->source, it->expect);
}
tests are free
Here’s part of the full suite from spn:
static const test_t tests [] = {
{
.name = "gated_on_os",
.facts = { .os = SPN_OS_LINUX },
.source = {
{ .value = "main.c" },
{ .value = "win.c", .when = { { "os", "windows" } } },
{ .value = "nix.c", .when = { { "os", "linux" } } },
},
.expect = { "main.c", "nix.c" },
},
{
.name = "negation",
.facts = { .os = SPN_OS_LINUX },
.source = {
{
.value = "posix.c",
.when = {
{ .key = "os", .value = "windows", .negated = true }
}
},
},
.expect = { "posix.c" },
},
{
.name = "every_clause_must_hold",
.facts = {
.os = SPN_OS_LINUX,
.mode = SPN_MODE_DEBUG
},
.source = {
{
.value = "a.c",
.when = {
{ .key = "os", .value = "linux" },
{ .key = "mode", .value = "release" }
}
},
{
.value = "b.c",
.when = {
{ .key = "os", .value = "linux" },
{ .key = "mode", .value = "debug" }
}
},
},
.expect = { "b.c" },
},
};
An LLM can’t fuck this up. It can’t. It’d have to change the imperative shell, and instead of being buried in a few hundred lines of imperative shell it lights up in bright green and screams out to you.
You can churn out dozens of these and know, beyond a shadow of a doubt, what they’re doing and whether they ought to exist. The LLM has generated a vast number of test cases, but no tests. You wrote the tests, because you are not a dog.
the end
The funny thing about all of this is that you should be doing this anyway. This has almost nothing to do with LLMs. In fact, you may begin to wonder why all parts of a program, not merely the tests, can’t be expressed by a small, fixed set of operations which transform data between known structures. Why debug code when debugging data is so much easier?
Well, ah…hm. Yeah. That’s a pretty good point! I guess the problem is that the real world exists, and our software is expected to do things instead of loop into its own bootstraps like some psychosexual fantasy of Gödel. Try that with data. But, hmm, TigerBeetle exists, and they move real money around with code that looks suspiciously like data when you squint hard enough.