My prompt couldn't add, and my eval didn't say why

On August 22 I started running jobfit with the threshold at 70. That day the queue had one job posting. The next day I lowered the number by hand until five showed up, and I didn't look at it again for fifteen days.

I didn't sit down to evaluate the scorer until September 7. And it turns out the threshold wasn't the only problem.

jobfit is a personal project that runs on an old Dell laptop I turned into a server (that's another post). It's a three-stage pipeline: stage 1 pulls remote job postings from five sources, stage 2 is a prefilter with deterministic rules that tosses what clearly doesn't fit, and in stage 3 an LLM scores each posting against my CV, with a single call and no agent. What makes the cut ends up in a Markdown queue. The jobfit README has the details.

The first day, I lowered the threshold by hand#

The threshold went into the repo on August 22, as a constant. I never measured it, and by the time I lowered it, it was already written in three files:

# src/jobfit/queue.py
DEFAULT_THRESHOLD = 70
 
# src/jobfit/evals.py
parser.add_argument("--threshold", type=int, default=70)
 
# src/jobfit/score.py
if score.why_not or score.fit_score < 70:

By the 23rd, jobfit had pulled in 846 postings and 135 made it to stage 3. What the prefilter rejected that day I can't reconstruct anymore, because the verdicts get overwritten on every run. The 135 score rows survive in a database backup, because September's re-score replaced 39 of them.

The August 23 queue has five postings: 78, 68, 60, 58 and 58. Four are below 70, so that day I lowered the threshold by hand until there was something to read, and what value I passed wasn't recorded anywhere. Then I didn't generate another queue until September 7: fifteen days without looking at the number, thinking the job market was just slow.

Evaluating an LLM starts with 39 blind labels#

On September 7 I labeled 39 postings, apply or skip, without seeing the model's score; the tool hides it. What it can't hide is that I'd already read the August 22 and 23 queues: Cosuno (78) and Sur (68) ended up in the set, and the only one that passes at 70 is one I'd already seen with its number.

At threshold 70, of the 22 I later marked apply, one would have surfaced. Precision 1 of 1, recall 1 of 22. And the model wasn't broken, it ranked well:

What was wrong was the scale. The rubric said 100 points, but across the 135 postings scores went from 3 to 78, with a median of 18. I'd put the cutoff at 70 on something that behaved like a 0 to 50 scale. That same day, twenty-three seconds after committing the eval, I dropped it to 25: precision 13 of 15 (87%), recall 13 of 22 (59%).

Correction: that day I wrote that 25 met the targets I'd set myself in the project's SPEC (precision above 0.8, recall above 0.6). It didn't, and in the same paragraph I wrote that the apply median was 34, when it's 33.

The other bug was in the prompt#

The rubric is a Markdown file that goes into the system prompt. The v1 rubric, the one that scored the August postings, ends with three worked examples. I added them up:

Two of the three added up wrong, both on the low side. Through the examples, I was teaching the model that the final score is the sum minus something. On top of that, the why_not rule told it to lower the score if it couldn't find anything wrong with the posting:

-you cannot find a genuine concern, the score is too high; lower it.
+you cannot find a genuine concern, you have not read the posting closely enough
+— look again. Do not lower the score to compensate for a thin `why_not`: the
+score is the sum of the components and the bullets are a separate obligation.

My hypothesis: between those two things, the model learned exactly what I asked for, add correctly at the top and shave points off everything else. The v2 rubric (the diff against v1) fixes the sums, writes the arithmetic into each example, changes that rule and spells out that the score is the sum, period:

-**Score: 24, confidence high.** This is the trap case. Shares AI vocabulary with
+**Score: 34, confidence high.** 8 + 15 + 4 + 2 + 5. This is the trap case. Shares AI vocabulary with
 
-**Score: 31, confidence medium.** Partial stack overlap, React yes but no Python
+**Score: 36, confidence medium.** 18 + 10 + 8 + 0 + 0. Partial stack overlap, React yes but no Python

Two numbers in a Markdown file. No linter checks them, no test runs them. There are tools for evaluating what the model spits out, even linters for prompts, but none I know of checks whether the math in the examples adds up. A prompt can have an arithmetic bug just like any other file, with one difference: nobody runs tests against a prompt.

What 39 labels can't tell you#

This section looks like it undercuts everything, and it's the most useful part:

• It's 38, not 39: one Huzzle posting shows up twice.
• Zero borderline: it measures the easy separation, not the hard middle.
• Inflated base rate: 22 of 39 are apply, so that 87% is a ceiling.
• 13 of the 17 skip are DevOps or infrastructure roles that stage 2 doesn't even let through anymore.

The prediction and the re-score#

Before running v2, I wrote down a prediction: scores will go up and threshold 25 will stop being the right one.

The re-score cost $0.77, and I compared it on the same 39 postings: 35 went up, 2 stayed the same and 2 went down. The median change was +6, but apply moved a lot more than skip: its median went from 33 to 47.5, while skip went from 18 to 22.

Open image full size

×Close image

The sweep with both rubrics:

With v1, only 19 and 20 met both SPEC targets. With v2, any threshold from 23 to 38 meets them: before, you had to hit one of two values; now there's a band.

Then I scored the remaining 123 postings with v2, for $1.04. Of the 163 that pass stage 2 today, 82 score 25 or above: half of them sit above the cutoff.

Evaluating an LLM means letting the data correct you#

The prediction half held, and which half failed depends on which SPEC rule I look at. By the numeric targets, 25 still works. But in the SPEC I also set myself a rule to favor precision over recall: a false positive costs me twenty minutes, and a false negative is one fewer posting among hundreds. By that rule, the cutoff is 35. With today's 53 labels, 35 gives precision 19 of 20 and recall 19 of 30, and 25 gives 23 of 28 and 23 of 30.

The threshold goes up to 35, as a separate change that doesn't touch the rubric. I almost left it at 25 with an argument that sounded prudent, don't move it without measuring how much a score varies between runs, but 25 came out of exactly that process. With 35, 59 postings sit above the cutoff instead of 82.

What's still unmeasured: no posting has been scored twice with the same rubric, so I don't know how much of that +6 is noise between runs, and the set doesn't have the ten borderline labels where a threshold earns its keep.

Three times I trusted what I believed, and three times a measurement corrected me. The first cost me fifteen days; the second, $0.77; the third, one query while I was writing this post. The difference is that the last two times, there was something written down that could contradict me.

If you have a threshold nobody has measured, tell me what it is. Mine are in the eval results, if you want to check them.