Ask twice: seven measurements from building with Jev

A coordinate lookup scored 3/24 where handing the model the naming job scored 22/24. Then two claims I had argued rather than measured turned out to be wrong.

Jev is a System One model from TypeSafe. It reads natural language like any LLM, and instead of writing a reply it returns a probability distribution over options you define. No prose, no reasoning trace, no JSON to repair.

I spent a day building waif, which reads a piece of text and names the feeling in it. That job has no right answer, which makes it an unusually honest test rig: nothing can be graded against a key, so every design decision has to be either argued or measured.

I measured five, argued two, and the two I argued were both wrong. Sections 1 and 7 are those, kept in place with the measurement underneath rather than quietly edited out.

1. A Choice splits its own vote between synonyms — but less than I claimed

Here is the argument I built on, and I am leaving it in its original words because the correction under it is the useful part.

Annoyed, irritated and frustrated are one feeling in three wordings. A Choice divides probability between them, so a text the model read perfectly clearly comes back split three ways, and confidence collapses for a reason that has nothing to do with the input. The distribution is telling you about your option list, not about the text. Therefore: never put sixty near-synonyms in one Choice.

Then I measured it, and the effect is real but far smaller than the argument needs. One Choice over all 62 words, same glosses, 23 texts:

The one-Choice version is more certain, not less. It picks the same word 21 times out of 23. On a plainly frustrated text it returns frustration at 0.98. Stripping the glosses and asking with 62 bare words barely moves it: 0.79.

So where does the splitting go? It shows up exactly where you would want it to:

shipped   pride 0.38  →  excitement 0.36
scan      worry 0.54  →  anxiety    0.39
sentit    guilt 0.60  →  embarrassment 0.26

Every one of those lost ground to a near-synonym — that is vote splitting. But the two-Choice design flattens on the same four texts (0.33, 0.53, 0.35, 0.32). The splitting is not caused by the option list being long. It is caused by the text genuinely sitting between two words, and both designs report it because both are calibrated. “I snapped at her in front of the kids” really is guilt and regret at once.

A long option list does not flatten a clear text. That is the part I had backwards, and the rule has to be narrower than I wrote it:

A Choice splits its vote when two options are the same answer in different wordings. The fix is criteria that separate them — not fewer options.

Sixty-two words each carrying a gloss that distinguishes it (annoyance: a small thing, quickly over against resentment: an old grievance still carried) are sixty-two alternatives, not sixty-two synonyms. The count was never the problem.

2. Ask twice anyway — for a reason that survived

Section 1 was the reason I split the question in two, and section 1 did not hold. The split stayed, and it is worth being precise about what is now holding it up, because “it scored the same and I had already built it” is not a reason.

Families are genuinely different answers — anger, fear, sadness and shame are not wordings of each other. And once the family is fixed, so are its shades: which shade of anger is a fair question, because the context has already ruled out the fifty-four words that were never in the running.

What that buys, and one Choice over 62 words cannot, is two separate uncertainty signals. Family confidence and word confidence are different doubts: “I do not know whether this is sadness or affection” is not the same failure as “it is clearly shame, but guilt or embarrassment?”. The page says different things in each case. One Choice gives you one number that cannot tell them apart.

The cost is honest too: a wobble in the family answer corrupts the word, because once family says sadness, nostalgia is not on the ballot. That cost me one text out of 23 — “drove past the old house, the tree we planted is taller than the roof” came back sorrow.

So the naming became two Choices in sequence: the first picks the family, the second picks the shade from that family alone. TypeSafe’s docs say a second request is warranted when an earlier answer determines the next question’s options. This is exactly that, and here is what it bought — scored on 24 texts against a list of acceptable answers for each:

Both of the two failures were the wrong family. Given the family, the shade was right every single time.

The alternative was speculative fan-out: ask all eleven within-family Choices in the first request, each stating its own family as a premise, and keep only the answer belonging to the family that won. That keeps a reading to one request, at roughly 3–4k input tokens against 1,250 + 444 for two. I reasoned about it and rejected it: two requests won on cost, and on a latency story I could explain.

That paragraph was wrong when I published it. It is still here because section 7 is the part of this post I would keep if I could only keep one.

3. Knowing which half to give the model is a decision you can measure

The job is: read a text, name the feeling. That job splits between the code I write and the model I call, and the only real decision is where the line falls — how much of the work do I hand over?

The first three rows of that table are me keeping most of it.

I had a reason. Human ratings exist for exactly the dimensions I was measuring: Warriner, Kuperman & Brysbaert scored 13,915 English words for valence, arousal and dominance by asking people. So the model’s job shrinks to placing the text on those three scales, and my code finishes the job — look up the nearest word in the table, return it. Research-backed coordinates instead of somebody’s guesses. It looked like the serious version.

It scored 3/24. Handing the model the whole naming job scored 22/24.

Three reasons, worth knowing before anyone else reaches for an emotion lexicon:

• Three scales were really about two. Across these emotion words, valence and dominance move together at +0.87 — a word that reads pleasant almost always reads in-control. The third scale is close to a copy of the first, so it separates far less than the theory promises.
• The unpleasant words are crammed into one corner. Fear, frustration, worry, terror, jealousy and embarrassment land in a ball small enough that taking the nearest one is close to picking at random. The coordinates are fuzzy to begin with: each is an average over about twenty raters who disagreed by around 1.7 on a 1–9 scale.
• Rating a word on its own is not the same measurement as reading a sentence. People rate the bare word “gratitude” as far more activated than an actual grateful message reads. The two sets of numbers were never on the same ruler.

I tried to fix that last mismatch by re-centring both sides against a sample of texts. It got worse, and instructively: the sample leaned negative, so the correction leaned negative, and a plainly warm text landed below the middle and got named from the sad half of the space. That is straightening a bent ruler with a bent ruler.

The lesson is not don’t use lexicons. It is that where the line falls between what code owns and what the model owns is a design decision with a number attached, and my intuition about it was wrong by a factor of seven.

The norms kept the one job they are good at. A Score is not given “rate this 1 to 5” — it is given a rubric, a written description of what each level means, the way a grading key spells out what a B looks like. Every level of mine now names words whose ratings were measured, so “as activated as rage or panic” is a claim a reader can argue with. “Very aroused” is only a word getting louder.

4. Confidence is peakedness — and peakedness misses a coin toss

A Score’s confidence measures how bunched together the answer is. It does not measure how likely the model is to be right, which is the trap everyone hits first.

Those two sound like the same thing until a reading like this one turns up. Confidence came back at 0.72, comfortably above any threshold I would set, and here is what was underneath it:

The 0.72 is not lying. Ninety-nine percent of the weight really is in two buckets and there is nothing anywhere else — that is about as bunched as an answer gets.

It is also a coin toss. Choosing between the top two is 50 against 49, and my page announced the winner in exactly the voice it uses at 0.98.

Confidence cannot catch this, because bunched into two neighbours is still bunched. What I actually wanted was the gap between first and second place, which is a different number entirely: 98 against 1 is a winner, 50 against 49 is a tie with a rounding error. Confidence says the same thing about both — and the gap is the one a reader cares about.

5. Audit your questions: one of mine fired on 21 of 24 inputs

A question set grows by accretion. Each addition looks free, and none of them announce that they have stopped saying anything.

So run them over a corpus and look at the spread. Mine had a Noul asking is more than one feeling present. It returned ≥0.6 on twenty-one of twenty-four texts. That is not a signal about the input, it is a property of writing — a question paying tokens to tell you something you already knew.

Two more went for a different reason. They were informative, but nothing downstream consumed them beyond appending a line to the output. A question whose entire effect is an occasional footnote costs a reader more attention than it returns.

The check is cheap: for every question, the min, max and spread of its answer across a representative corpus. Anything that barely moves is either a gate or a mistake.

6. Two small things that cost real time

A rubric level must not contain a word from another axis. My control rubric had a level reading “Overwhelmed: struggling to keep any grip on it”. That primes the model with a feeling while asking it about agency, and it labels the output with a word that is not a position on a control scale at all. Every level of an axis has to be a point on that axis.

Input is dominated by your rubrics, not by your input. Criteria are sent on every call, so a one-line text costs almost exactly what a paragraph does. At this size requests are the scarce resource and tokens are not — which is the whole argument for batching every independent question into one call.

I stopped one clause too early, and wrote that a dependent second call is therefore a real cost rather than a rounding error. Read it again: if requests are scarce and tokens are not, the conclusion goes the other way.

7. I reasoned where I should have measured

Someone read section 2, noticed I had talked myself out of the fan-out, and told me to go and time both. It took forty minutes and a dollar’s worth of nothing.

Run the same 24 texts through both designs, twice each, back to back on every text so neither gets the warmer connection:

One request was faster on 46 of 46 pairs. It named the same family 48 out of 48 times, and the same shade 45 of 48 — where all three misses were texts under 0.41 confidence that the page already reports as sitting between two words, and where the sequential design disagreed with itself between rounds on one of them. The eleven extra questions barely moved the six that were already there: mean drift of 0.022 on a 0–4 axis score, and identical intent 48 times out of 48.

So the fan-out is 1.9× faster for five hundredths of a cent a thousand readings, and it halves the request count against the limit that actually binds.

Two things I had, and did not put together:

Jev “ingests the state once and evaluates every question against it in parallel.” That sentence is in TypeSafe’s own model card, and I had quoted the half of it that suited me. Question count is nearly free; a round trip is not. Ten wasted questions cost less than one extra wait.

Jev charges for input only — output tokens are free. The fan-out triples the output, returning eleven probability distributions instead of one, and that is worth exactly nothing on the bill.

What I actually got wrong is narrower than “I didn’t measure it”, and more useful. The docs say a second request is warranted when the first answer determines the second question’s options. It does — and I read that as settling the matter. But determined is not unknown: there were only ever eleven option sets, all of them written down in my own source file, so every one of them could be asked on spec. The rule is about what the options are. It says nothing about when you are allowed to ask.

The tell was in my own sentence. “A latency story I could explain” is not a latency number. Anywhere a design note says presumably, roughly, or I could explain, there is a measurement someone is about to make for you, and it is cheaper to make it yourself.

waif is live at /waif. Every number in this post is on the page, under How it works, next to the vocabulary it was scored against.