---
title: "Jev Othello benchmark: 900 games on prompting and splitting | UnlockJapan, Inc."
description: "TypeSafe's Jev played 900 Othello games: how prompting, decomposition and Python-computed checks change win rate, speed and cost, plus Claude Haiku 5.5."
url: https://unlock-japan.co.jp/en/blog/jev-othello-benchmark/
locale: en
date: 2026-10-21
author: "Takuma Yoshida"
---

2026-10-21

# Jev Othello benchmark: 900 games on prompting and splitting

[Takuma Yoshida](https://unlock-japan.co.jp/en/company/)

## Key points

- **Prompting**: Adding loosely chosen Othello principles made no clear difference (55–2–43). Seven principles drawn from expert teaching material did (71–2–27).
- **Decomposition and computation**: Splitting those principles into small yes/no questions did not beat asking them in one prompt (47–4–49). Computing the answers to those questions in Python and handing them to Jev did.
- **Against a general-purpose LLM**: Even the strongest Jev workflow lost to Claude Haiku 5.5, which got no strategy at all. Jev won clearly on speed and cost.

In our Jev Othello benchmark, a workflow that computes its checklist in Python and leaves only the final choice to Jev beat the same workflow with Jev answering the checklist 70–4–26 over 100 games. Against Claude Haiku 5.5 it still lost 30–3–67, while deciding each move about 28 times faster (0.24 s vs 6.85 s on average) at about a tenth of the cost.

## What we tested

We tested TypeSafe's decision model Jev at Othello, a game where the quality of each decision shows up in wins and losses. The end goal: can Jev, built into a workflow, decide better, faster and cheaper than a general-purpose LLM? Because Othello is fully observable and rule-bound, every answer Jev gives can be graded by a program.

The experiments are written up in Japanese as a series on note (links at the end; note offers an English machine translation). This page collects the numbers and conditions in one place.

## The methods in detail

Every method receives the board and the list of legal moves on each turn and picks one legal move. What differs is who prepares the inputs to that decision and who makes the final choice. We split the decision into three levels:

- Level 1 (strategy): decide the Othello principles
- Level 2 (checks): for each legal move, establish the facts those principles depend on
- Level 3 (final choice): pick one move from that material

![Grid showing which of Levels 1–3 each of the five methods uses, and whether Jev, Python, Claude Haiku or people handle each level](https://unlock-japan.co.jp/blog/jev-othello-methods-en.png)

### Basic

- **Model**: Jev 1.14.0 (`jev-1.14.0`), called through TypeSafe's HTTP API.
- **Input**: the board (eight strings; `B` black, `W` white, `.` empty), the side to move, the legal moves and the move number.
- **Question**: a single multiple-choice (Choice) question whose options are the legal moves: "Which move from `legal_moves` gives `current_player` the best chance of finishing the game with more discs than the opponent?"
- **Requests per move**: 1.
- **Move played**: Jev's `choice` (the option with the highest probability), as returned. Per-option probabilities and the confidence are logged but never change the move.

### Strategy

The Basic question with Othello principles added, still asked as one question. We tried two sets:

- **Loose principles (v1)**: take corners, reduce the opponent's legal moves, make your discs stable, and increase your final disc count, plus a line saying their importance depends on the position. No sources.
- **Seven principles (v2)** — : drawn mainly from the — [French Othello Federation (FFO) material](https://www.ffothello.org/othello/principes-strategiques/) — , with the — [British Othello Federation (BOF) material](https://www.britishothello.org/strategy/2-corners-stable-discs) — as a second source. The prompt adds that their weight depends on the position and the stage of the game and that none always comes first. — Take corners (a corner disc can never be flipped) / Avoid moves that let the opponent play a corner on their next turn / Avoid the squares next to an empty corner (X- and C-squares) unless there is a clear reason / Limit the opponent's moves while increasing your own / Prefer quiet moves, which flip no disc that touches an empty square / Keep few discs in the opening and the midgame / In the endgame, weigh finishing with more discs than the opponent

v2 also passes the stage of the game: opening with 41 or more empty squares, midgame with 21–40, endgame with 20 or fewer.

- **Model**: Jev 1.14.0
- **Requests per move**: 1
- **Move played**: Jev's `choice` (the option with the highest probability), as returned. Per-option probabilities and the confidence are logged but never change the move.

### Jev Atomic

The seven principles become yes/no questions about each legal move, answered by Jev (Level 2). Jev then picks the move from the table of answers (Level 3). This follows TypeSafe's own [guide to building with System One](https://docs.typesafe.ai/concepts/how-to-build-with-system-one): ask many narrow, independent questions and combine the answers.

**Level 2**: seven questions per legal move, each of the type that returns a probability of "yes" from 0 to 1 (Noul). Eight legal moves means 56 questions, all sent in one request. A probability of 0.5 or more reads as "yes".

| # | Question | Principle |
|---|---|---|
| 1 | Is the move a corner (A1, H1, A8 or H8)? | 1 |
| 2 | After the move, can the opponent play a corner on their next turn? | 2 |
| 3 | Is the move an X-square (B2, G2, B7, G7) whose corner is empty? | 3 |
| 4 | Is the move a C-square (A2, B1, G1, H2, A7, B8, H7, G8) whose corner is empty? | 3 |
| 5 | After the move, will the opponent have more legal moves than now? | 4 |
| 6 | Is it a quiet move (no flipped disc touches an empty square afterwards)? | 5 |
| 7 | Does it flip at least as many discs as every other legal move? | 6 and 7 |

**Level 3**: Jev does not see the board. It receives the side to move, the legal moves, the move number, the stage of the game, the seven short questions, the yes/no table for each legal move and the seven principles, and picks one move with a Choice question. The board is withheld so that Jev combines the table rather than re-deciding from scratch.

- **Model**: Jev 1.14.0 for both levels
- **Requests per move**: 2 (Level 2 and Level 3)
- **Move played**: Level 3's `choice`, as returned. Python only checks that the move is legal.

Early in the series we also tried a version that asked four principles as separate Choice questions and took a majority vote (Atomic v1). It is not part of the numbers on this page.

### Python Atomic

Python computes the seven Level 2 answers from the rules of Othello instead of asking Jev. Each question has exactly one correct answer given the board and the legal moves, so every answer is right. The computation took about 0.5 ms per move.

Level 3 is identical to Jev Atomic: for the same position and the same Level 2 answers, the request sent to Jev is byte-for-byte the same, and it does not say who answered Level 2.

- **Model**: Jev 1.14.0 for Level 3 only
- **Requests per move**: 1 (Level 3)
- **Move played**: Level 3's `choice`, as returned. Python never picks the move; if Jev chooses against every Python answer, that move is still played.

### Claude Haiku 5.5

- **Model**: `claude-haiku-5-5` at medium effort
- **Input**: the board, the side to move, the legal moves and the move number (the same information as Basic). The system prompt explains the board format and asks for the move most likely to finish with more discs than the opponent, returned as JSON. No principles are given.
- **Requests per move**: 1
- **Move played**: the returned move.
- **Timing and cost**: decision time is the API response time the CLI reports (CLI start-up excluded). Cost is the list-price figure the CLI returns.

### Router (Jev + Haiku)

![Sequence diagram: each move is first decided by Python Atomic, and only moves where Jev's confidence is below 0.5 are handed to Claude Haiku](https://unlock-japan.co.jp/blog/jev-othello-router-en.png)

Every move starts with Python Atomic. The confidence Jev returns at Level 3 decides who moves:

- **Confidence 0.5 or higher**: Jev's move is played (exactly 0.5 counts as Jev's).
- **Confidence below 0.5**: Claude Haiku receives the same board, side to move, legal moves (in the same order) and move number, and Haiku's move is played.

The 0.5 threshold is the value in [TypeSafe's documentation example](https://docs.typesafe.ai/confidence); no other value was tested. Jev is always asked first, so its time and cost are counted even on moves handed to Haiku.

- **Models**: Jev 1.14.0 and Claude Haiku 5.5, with the same settings as when they play alone
- **Requests per move**: 1–2

### Common conditions

- Each pairing played 100 games, with each side taking black in 50.
- The order of the legal moves was shuffled for every position. The order depends only on a random seed, the game number and the move number, and the same rule applies to both players. This keeps Jev 1.13's documented first-option bias out of the results.
- Every prompt sent to Jev is in English, which TypeSafe documents as the most accurate language.
- An illegal answer is never replaced with another move; the same input is retried up to three times.
- There is no time limit per move. Each request has a timeout as a safety net.
- Jev's API has no temperature or seed parameter.
- Jev's cost is estimated as input tokens × $0.042 per million tokens (output is free), not an invoiced amount.

## Results

Results are from the left-hand method's side. Every pairing is 100 games with colors alternated.

| Pairing | Wins | Draws | Losses | Win rate (decided games) | 95% interval | p-value |
|---|---:|---:|---:|---:|---:|---:|
| Jev 1.14.0 vs 1.13.0 (both Basic) | 63 | 2 | 35 | 64% | 54–74% | 0.006 |
| Strategy (loose principles) vs Basic | 55 | 2 | 43 | 56% | 46–66% | 0.27 |
| Strategy (seven principles) vs Basic | 71 | 2 | 27 | 72% | 63–81% | <0.001 |
| Jev Atomic vs Basic | 82 | 1 | 17 | 83% | 74–90% | <0.001 |
| Jev Atomic vs Strategy | 47 | 4 | 49 | 49% | 39–59% | 0.92 |
| Python Atomic vs Jev Atomic | 70 | 4 | 26 | 73% | 63–81% | <0.001 |
| Python Atomic vs Strategy | 76 | 5 | 19 | 80% | 71–88% | <0.001 |
| Python Atomic vs Claude Haiku 5.5 | 30 | 3 | 67 | 31% | 22–41% | <0.001 |
| Router vs Claude Haiku 5.5 | 40 | 6 | 54 | 43% | 32–53% | 0.18 |

The win rate counts decided games only. The 95% interval is the exact binomial (Clopper-Pearson) interval, and the p-value is a two-sided binomial test against equal strength. With 100 games per pairing, small differences may go undetected.

## Speed and cost

| Method | Time per move (mean) | Cost per game |
|---|---:|---:|
| Strategy | 0.19 s | ~$0.0014 |
| Jev Atomic | 0.42 s | ~$0.0081 |
| Python Atomic | 0.20–0.24 s | ~$0.0023–0.0024 |
| Router | ~3.7 s | ~$0.015 |
| Claude Haiku 5.5 | 6.85 s | ~$0.025 |

The Python Atomic range covers two runs against different opponents; the Python computation itself took about 0.5 ms. Costs are estimates from token counts at list API prices, not invoiced amounts.

The Router sent 1,490 of 2,996 moves (about half) to Haiku. Moves Jev kept took about 0.2 s and moves it handed off about 7.3 s, which averages to 3.7 s. Jev was least confident early: Haiku took 69% of opening moves, 52% in the midgame and 28% in the endgame.

## What we learned

![Scorecard of strength, speed and cost by method: Python Atomic beat Jev Atomic 70–4–26 and Strategy 76–5–19, at 0.20 s per move and $0.0024 per game](https://unlock-japan.co.jp/blog/jev-othello-scorecard-en.png)

### The quality of the principles decided the outcome

Four loosely chosen principles changed how Jev played without a clear change in results. When a corner was available, Basic took it 28% of the time (188 of 669 positions) and the principled prompt 100% (212 of 212). Rebuilt from expert material, the seven principles won 71–2–27.

### Splitting alone did not help; correct answers did

We graded Jev Atomic's yes/no answers against the rules after the games. Questions answerable from the square's position (is it a corner, is it next to one) were mostly right. Questions that required imagining the board after the move, or comparing how many discs flip, were close to chance. TypeSafe's [notes on Jev 1.13](https://docs.typesafe.ai/model-jaggedness/jev-1.13) list multi-step reasoning and counting as weak spots too.

Replacing those checks with Python made the workflow win 70–4–26 against Jev Atomic. How much each check should weigh in the final move is not something the rules compute, so that choice stayed with Jev.

### It lost to a general-purpose LLM, but won on speed and cost

The strongest Jev workflow still lost to Claude Haiku 5.5, which was given no strategy. It decided each move about 28 times faster at about a tenth of the cost. That trade-off looks useful where volume or latency matters more than the last bit of quality, such as high-volume, near-routine classification or decisions made while someone is waiting.

### 1.14.0 beat 1.13.0 and leans less on the first option

TypeSafe [documents](https://docs.typesafe.ai/model-jaggedness/jev-1.13#choice-option-order) that Jev 1.13 tends to pick the first option it is given. With options shuffled every move, 1.13.0 picked the first-listed move 51% of the time (22% expected by chance) and 1.14.0 33% (16% by chance). The bias is smaller but still there, so reordering options to check that the answer holds is still worth doing on 1.14.0.

## Conditions and limits

- Jev is 1.14.0 except in the version comparison. Claude Haiku is 5.5 at medium effort, given only the board and the legal moves. The Jev side carries principles and computed checks, so this is not a like-for-like comparison of equally engineered setups.
- These are results from Othello, a fully observable game with fixed rules. Every check could be computed and expert strategy material existed; real business decisions may not show the same gaps.
- The Router was tested only at the 0.5 confidence threshold from TypeSafe's documentation example.
- Timings reflect our environment and settings.

UnlockJapan has no partnership, payment or sponsorship relationship with TypeSafe.

Share this post

## Related reading

The series on note (Japanese, machine-translated)

note2026-10-15

### I had an AI (Jev) play Othello: old version vs. new version

note2026-10-19

### Asking an AI (Jev) better questions made it more accurate

note2026-10-19

### I split my questions to an AI (Jev) and scripted the fixed parts

note2026-10-19

### A tuned decision AI (Jev) vs. Claude Haiku at Othello

note2026-10-20

### What is Jev? Where it stands one month after release

[Blog](https://unlock-japan.co.jp/en/blog/)

---

- Site overview for agents: https://unlock-japan.co.jp/llms.txt
- Sitemap: https://unlock-japan.co.jp/sitemap.xml
- Other language: https://unlock-japan.co.jp/ja/blog/jev-othello-benchmark/
- Contact: contact@unlock-japan.co.jp