After learning about Jev, I started revisiting a component of our EMNLP 2025 paper, KG-RAG. We used an LLM to score candidate GUI trajectories, then let a search algorithm decide which paths to explore. Jev also exposes bounded probabilistic judgments that ordinary code can act on.
The resemblance is meaningful. It is strongest at the boundary between a model’s judgment and a program’s control flow. Looking more closely, however, reveals differences in what is being searched, how scores are computed, and what training is intended to make reliable.
I wanted to use one example to understand when the scoring rules make the same choice, when they disagree, and what Jev might suggest for a future version of our trajectory scorer.
1 The shared idea: a model as a judge inside a program
KG-RAG addresses missing app-specific navigation knowledge. Its offline pipeline explores a UI Transition Graph (UTG), uses intent milestones to evaluate trajectories, and stores useful intent–trajectory pairs for online retrieval. Candidate paths are expanded in batches, scored, and filtered to a top-K frontier. The search is over UI paths; it is not a token-decoding tree. [Paper §3.2 and Algorithm 3]
For example, imagine two milestones: reach the settings screen, then open the privacy policy. We can ask how many milestones a trajectory has completed: zero, one, or two. Extracting the relevant LLM logits and applying softmax gives a distribution over those answers.
This is the useful connection to structured scoring: the model supplies numbers over an explicit answer space, and the surrounding program decides what those numbers mean for the task.
Jev makes that interface explicit through question types such as Noul for yes/no probability, Choice for categories, and Score for ordered levels. Choice and Score also return a separate confidence statistic. For our milestone example, the closest match is Score. [TypeSafe primitives]
2 One example, four different questions
Suppose the distribution for a candidate trajectory is:
- 0None completed
- 10%
- 1One completed
- 20%
- 2Both completed
- 70%
Our progress score: how large is the strongest probability?
There are two different numbers here: “2” is the model’s most likely completion count; “0.7” is the probability it assigns to that answer. The paper’s progress definition uses the latter. So 0.7 does not mean “70% of the task is done.” Keeping only the largest probability also loses the rest of the distribution—which is where the next score can help.
Our proximity score: does the ranking favor more progress?
Start with an intuitive question: does the model lean toward more rather than less completion? The ordering we want to favor is “two completed” first, “one completed” second, and “none completed” last: [2, 1, 0]. Our example has 70% > 20% > 10%, so it matches. Algorithm 2 compares the actual ordering with that target, squares the difference at each position, and puts a minus sign in front:
Same progress score, different proximity
Now compare two candidates. Both give “two milestones completed” a 70% probability, so their progress scores are tied:
Scroll sideways to compare all scores →
| Path | P(0), P(1), P(2) | Progress | Proximity |
|---|---|---|---|
| A | [0.1, 0.2, 0.7] | 0.7 | 0 |
| B | [0.2, 0.1, 0.7] | 0.7 | −2 |
Look at the remaining 30%. For A, the model’s next-best guess is “one milestone completed.” For B, it is “none completed.” Put another way: if the model is wrong about finishing both, A has the more promising fallback. That is useful information the shared progress score of 0.7 leaves out.
A ranks the outcomes [2, 1, 0], exactly matching the target ordering. B ranks them [2, 0, 1]: the last two positions are swapped. B therefore gets a penalty of −(0² + 1² + 1²) = −2, while A gets 0. With progress tied, KG-RAG prefers A.
Why prefer zero? It means zero ordering penalty, not zero progress. Squared differences can only be zero or positive; the minus sign turns them into zero or negative scores. Zero is the highest possible score because nothing disagrees with the target ordering. Think of starting at zero and deducting points for misplaced outcomes.
This also explains the name “proximity”: it measures closeness to an ideal ordering, not the number of UI steps left. A score of zero does not prove that the task is finished. For example, [0.32, 0.33, 0.35] has the same ordering as A and also gets zero, despite being much less decisive. Proximity uses ranks; it discards the sizes of the probability gaps. [Paper Algorithm 2]
The trajectory evaluator still has to judge whether the actions actually completed the milestones. Proximity only summarizes the ordering of that evaluator’s probabilities.
Jev’s Score: how much progress is expected?
This is the probability-weighted mean of the level numbers. It averages milestone counts, not logits. “1.6” is an expectation under the distribution; it does not mean the interface literally completed a fractional milestone. [Official Score documentation]
Jev’s Score confidence: how concentrated is the judgment?
Why have confidence as well as Score? “Definitely one milestone” and “a 50/50 chance of zero or two milestones” both average to one. But the second judgment is much more divided. Confidence summarizes that spread so a program can decide whether to seek more evidence. In BFS, one policy worth testing is to retain more candidate paths when the judgment is dispersed.
Levels are ordered answer positions; n is the number of positions.Here the three levels mean “0, 1, or 2 milestones completed.” Adjacent levels are one step apart, so 0 is farther from 2 than 1 is. “Cat, dog, bird” also has three classes but no such distance; that calls for Choice. The formula uses equally spaced level indices. For labels such as “low, medium, high,” equal spacing is a scoring convention, not a claim that real-world differences are equal.
Start with the same probabilities: P(0) = 0.1, P(1) = 0.2, and P(2) = 0.7. The most likely answer is 2 milestones. Confidence asks how closely the other possibilities cluster around that answer. This is different from Score, which averages all levels and gives 1.6.
Step 1: measure how far the probability spreads from the favourite
Imagine placing 100 counters on a number line: 10 at level 0, 20 at level 1, and 70 at level 2. Measure each counter’s distance from the favourite, level 2. Ten counters are two levels away; twenty are one level away; seventy are already there. The average distance is:
Call this distance D. Its unit is levels, not probability. A smaller D means the prediction is more concentrated around its most likely answer. For ordered answers, being one level away should count less than being two levels away.
Step 2: choose a reference for an even spread
To turn that distance into a normalized confidence, Jev needs a comparison scale. Its documented choice is a uniform distribution: equal probability on every level. For levels 0, 1, and 2, the middle position is 1.
That is where 2/3 comes from: the average of the distances 1, 0, and 1. It is not a learned parameter or a two-thirds chance of being correct. The denominator is fixed by the level scale. Five levels numbered 0–4 would instead give (2 + 1 + 0 + 1 + 2) / 5 = 6/5.
Notice the two reference points: the numerator measures distance from this prediction’s most likely level, here 2; the denominator measures the uniform reference from the middle of the scale, here 1. We do not move the denominator’s centre to 2. This is the normalization TypeSafe defines.
Step 3: turn spread into confidence
Divide D by the reference distance U, subtract from one, and stop the result from going below zero:
This construction gives confidence 1 when D is zero: all probability sits on one level. It reaches zero when D equals the reference spread. A larger D also receives zero, because the uniform baseline is a reference, not the largest possible spread. This is a chosen summary statistic, not a unique mathematical definition of confidence. [TypeSafe’s definition]
For our candidate, the average distance 0.4 is 60% of the reference distance 2/3. Subtracting that fraction from one gives:
The spread and final confidence both happen to be 0.4 in this example. That is a coincidence: for candidate B, [0.2, 0.1, 0.7], D is 0.5 and confidence is 0.25.
Confidence 0.4 does not mean a 40% chance that the answer is correct. The probability assigned to full completion is still 0.7. Confidence describes the distribution’s concentration around its favourite answer. A model can also be very concentrated on “zero milestones,” so high confidence does not make a path promising. This formula is for Score; unordered Choice answers use a different one.
Try it: the favourite stays fixed, but how far away is the disagreement?
Keep the probability of two milestones fixed at 70%. Move the remaining 30% between one and zero.
D = 0.40 · Confidence = 0.40
| Metric | Value | Question it answers |
|---|---|---|
| Progress | 0.7 | How large is the top probability? |
| Proximity | 0 | Does the probability order favor higher completion counts? |
| Jev Score | 1.6 | How many milestones are expected to be complete? |
| Jev Score confidence | 0.4 | How concentrated is the distribution near its mode? |
Try the same distribution yourself
Choose an example or adjust the relative weights. They are normalized to sum to 100%. All calculations run locally; no model is called.
- KG-RAG progress*
- 0.7 Largest probability
- KG-RAG proximity
- 0 0 is the best rank match
- Jev-style Score
- 1.6 Expected milestone count
- Jev-style Score confidence
- 0.4 Concentration, not correctness
Formula and tie conventions
*Progress follows the paper’s literal maximum-probability description. Proximity compares sorted indices with [2, 1, 0]. For equal probabilities, this demo puts the higher milestone first; Algorithm 2 does not specify ties. Confidence follows the official example code’s first-mode rule. Numbers are rounded to three decimals. No empirical performance claim is implied.
3 Can both scoring rules find the same path?
Yes. Even in the A/B example above, both prefer A: KG-RAG breaks the 0.7 tie using proximity, while Jev’s expected score is 1.6 for A and 1.5 for B. Different arithmetic can produce the same decision.
Let’s follow a whole search. The task is still to reach Settings and open the Privacy Policy. Use a tiny top-K search with K = 1: expand one UI action per round and keep one candidate to continue. This makes each selection visible; it is a pruned search, rather than exhaustive BFS.
We give both scoring rules the same hypothetical probabilities and the same final screen check. The second version uses Jev-style expected milestone scores inside our search loop. Jev supplies judgments; the search logic still lives in our code.
Round 1: from Home, choose Settings or Help
Neither candidate has passed the final screen check yet. The evaluator estimates:
Scroll sideways to compare all scores →
| Candidate | Probabilities | KG-RAGprogress / proximity | Jev-styleexpected count |
|---|---|---|---|
| Home → Settings | [0.20, 0.70, 0.10] | 0.70 / −6 | 0.90 |
| Home → Help | [0.70, 0.20, 0.10] | 0.70 / −8 | 0.40 |
KG-RAG keeps Settings: progress is tied at 0.70, but −6 is higher than −8. Settings ranks the outcomes [1, 0, 2], giving −(1² + 1² + 2²) = −6; Help ranks them [0, 1, 2], giving −(2² + 0² + 2²) = −8.
Jev-style scoring also keeps Settings: its expected completion is 0.70 + 2 × 0.10 = 0.90, versus 0.20 + 2 × 0.10 = 0.40 for Help. Intuitively, one candidate looks likely to have reached the first milestone; the other looks likely to have reached neither.
Round 2: from Settings, choose Privacy Policy or Notifications
Scroll sideways to compare all scores →
| Next screen | Probabilities | KG-RAGprogress / proximity | Jev-styleexpected count |
|---|---|---|---|
| Privacy Policy | [0.02, 0.08, 0.90] | 0.90 / 0 | 1.88 |
| Notifications | [0.20, 0.70, 0.10] | 0.70 / −6 | 0.90 |
Both choose Privacy Policy. KG-RAG prefers its larger progress score, 0.90 versus 0.70; proximity is not needed to break this decision. Jev-style scoring prefers its larger expected count, 0.08 + 2 × 0.90 = 1.88. The same final check then confirms that the Privacy Policy screen is open.
The useful lesson is that different scores need not mean different behavior. What matters for pruning is which candidate ranks higher. Here, the two rules agree at every selection. This is a worked example of agreement, not evidence that two trained models always produce the same probabilities or search results. The next example shows why that distinction matters.
4 What would make more sense for our BFS?
Now compare A with a path the evaluator is very sure has made little progress:
| Path | Distribution | Progress | Proximity | Expected count | Confidence |
|---|---|---|---|---|---|
| A | [0.1, 0.2, 0.7] | 0.7 | 0 | 1.6 | 0.4 |
| C | [0.9, 0.08, 0.02] | 0.9 | −8 | 0.12 | 0.82 |
C has higher confidence, but A is more promising. Being certain that a path fails does not make it worth expanding. If maximum probability is used as the primary ranking key, and proximity only breaks ties, C can outrank A. This is a consequence of that literal scoring rule, not a claim that we have reproduced a failure in the released implementation.
For a future ablation, I would separate three decisions:
- What to expand: try expected milestone count as the primary progress heuristic.
- What looks complete: inspect the probability of completing all milestones, with a task-appropriate success check.
- What needs more attention: use uncertainty to decide whether to rescore or keep a wider frontier. Automatically pruning uncertain paths may discard good options.
Expected progress is still a heuristic, not a guarantee of eventual success. A path with fewer completed milestones may have a better continuation. Equal weighting also treats each milestone as equally valuable. The meaningful experiment is to hold the evaluator, candidate graph, and search budget fixed, vary the scoring rule, and measure successful paths found and search cost.
5 Where Jev fits in a program
First, what do we actually ask Jev to do?
Think of a narrow function call inside a larger program. We provide the evidence to evaluate, called the state, along with a question and descriptions of the allowed answers. Jev returns a probabilistic judgment. Our code decides what to do with it.
Using the privacy-policy task, the three question types could look like this:
- Noul: “Does the supplied trajectory show that the privacy policy was opened?” Return a probability of yes.
- Choice: “Which screen does this description refer to: Settings, Help, Privacy Policy, or Other?” Return a category, its distribution, and confidence.
- Score: “How many milestones does this trajectory complete?” Define ordered levels 0, 1, and 2, then return their distribution, expected level, and confidence.
The word typed means that the answer has the declared shape and belongs to the permitted answer space. It does not mean the model understood the evidence correctly. Selecting “Privacy Policy” when the screen is actually Help is still a possible semantic error. The types constrain what the program receives; they do not verify the world. [Noul] [Choice] [Score]
Would Jev replace KG-RAG?
The natural experiment would replace or compare the trajectory evaluator. KG-RAG’s graph expansion would still propose paths, our ranking code would still choose a frontier, and the offline knowledge construction and online retrieval would still have their own jobs. Jev does not acquire those responsibilities simply because it can score a trajectory.
Suppose an evaluator returns [0.1, 0.2, 0.7]. Whether that distribution came from our LLM or Jev, ordinary code can calculate Score 1.6 and confidence 0.4. We can then test a policy such as “use expected progress for ranking, and retain more alternatives when the evaluator is uncertain.” That policy is our design choice; the confidence formula does not choose it for us.
Conversely, adding the same two formulas to our existing scorer does not reproduce Jev’s model. Matching the arithmetic is straightforward. Matching the quality and calibration of the underlying probabilities is an empirical question.
What does “parallel” mean here?
Our BFS evaluates several candidate paths in one round. Jev’s speculative fan-out concerns asking several questions about already available evidence before code knows which answers it will use.
Imagine a support ticket. A sequential workflow first asks “Is this a bug?” and, only after a yes, asks “How severe is it?” Fan-out asks both together. If it is a billing issue, code ignores the severity answer. The ticket already contained the evidence for both judgments, so speculation can save a round trip. [Official fan-out pattern]
That cannot conjure up new evidence. If judging a GUI action requires seeing a screen that has not yet been observed, we still need that observation. Batching known questions and eliminating dependencies are different things.
Why the speed baseline matters for our scorer
TypeSafe describes a model and parallel sampler designed for these structured judgments. Its comparisons with LLMs generating text or JSON motivate the interface, but our scorer already reads selected logits instead of generating a long explanation. We would need a direct comparison on the same trajectory judgments, at matched quality and comparable serving conditions, to establish the practical speed benefit for our BFS. [TypeSafe’s launch article]
6 Are the probabilities reliable? Checking calibration
“Predicted probabilities correspond to observed outcome frequencies” means: when the model says “about a 70% chance of completion” across many different cases, roughly seven out of ten should actually be complete.That is what calibration checks.
Collect 100 different held-out trajectories, each assigned about 70% probability of completing both milestones. Then inspect the actual action records and verify which ones really completed both:
100 paths, each predicted to have a 70% completion chance
Green: actually completed both milestones · Grey: did not complete both
Predicted 70% → observed 70/100: this group matches.
Switch to 40 completed paths: the model still predicts 70%, but only 40% actually complete. It overestimates completion in this group. Here 70% means the probability of completing both milestones, not “70% of milestones completed” and not “confidence = 0.7.”
One group makes the idea tangible. A real evaluation needs enough examples across different probability ranges; finite samples need not match exactly. TypeSafe describes improving this probability reliability as an objective of RLCD. [Official AI primer]
Confidence describes how concentrated a model’s answer distribution is; calibration compares predictions with actual outcomes.We can compute confidence from one path’s distribution alone. We cannot check calibration without ground truth. A model can be very certain and frequently wrong.
For our BFS, computing Jev-style Score and confidence is straightforward. What needs evaluation is whether the probabilities are reliable, and whether using them finds correct paths with less search.
Sources and scope
Based on the public paper and documentation reviewed on 22–23 October 2026. Numerical examples are illustrative calculations, not benchmark results or live Jev outputs. Progress follows the maximum-probability wording in §3.2, with proximity as a secondary ranking key; the implementation has not been audited here. The two-round search holds probabilities, K = 1, and the final screen check fixed to isolate the scoring rule.
- Guan et al., KG-RAG, EMNLP 2025 — especially §3.2 and Algorithms 1–3 in the paper.
- Aman Arora, Jev: A New Way to Make Probabilistic Decisions — the article that prompted this comparison.
- TypeSafe documentation: Score, Confidence, and Speculative fan-out.
- TypeSafe: Introducing System One Models & Jev and AI primer.