Clairic and Jev on 100 return emails: four experiments

Using Clairic, Jev and Claude Haiku, we ran four experiments sorting 100 return and refund emails, wiring Clairic and Jev together in different ways. Jev is a decision model from TypeSafe, which calls models of its kind System One models. Without Clairic, the six-step Jev + Python layout (experiment 4) was faster and cheaper than Claude Haiku 5.5 (0.3 s per case and 1.5 yen per 100 cases, against 3.0 s and 8.1 yen), but it got 78 right to Haiku's 91. The gap was in reading the email, not in deciding: in that layout Jev's decisions matched the policy's code in all 100 cases, so all 22 misses came from reading.

Models used: Claude Haiku 5.5 (claude-haiku-5-5), Jev (jev-1.14.0) and Clairic (@cf/cloudflare/clairic, the preview as of October 24, 2026).

What we tested

Can TypeSafe's decision model Jev, built into a workflow, decide better, faster and cheaper than a general-purpose LLM? We have been testing that question at Othello; this is the first test on something shaped like real work.

The task is sorting return and refund emails. The answer follows from the shop's policy, but before the policy can be applied, someone has to read from the email what happened and what the customer wants. For reading values out of text we used Clairic, Cloudflare's extraction model.

In its post introducing Clairic, Cloudflare wires three models together: Clairic extracts the values, Clef (a model that picks from fixed options) decides what to offer, and a general-purpose LLM writes the reply (Cloudflare's blog). Besides that layout with Jev in Clef's place, we tried a six-step layout of the author's design (explained under "How the layouts are described" below), and layouts with no Clairic at all.

The experiments are written up in Japanese as a series on note (links at the end; note offers an English machine translation). This page collects the numbers and conditions in one place.

In the note article on return emails, the “Clairic version” is Experiment 1 here (the layout from Cloudflare's blog, 68 right) and the “Jev+Python version” is Experiment 4 (78 right).

The task: 100 return emails to a fictional shop

We wrote 100 return and refund emails to a fictional online shop, Midori Kitchen (kitchenware and small appliances at about 1,500–60,000 yen; appliances carry a one-year warranty). For each email, the sorter reads the email and the order record from the shop's admin screen (order date, delivery date, item, quantity, unit price, the customer's returns in the past year, and the number of attachments), then picks one of six answers:

• Refund
• Exchange
• Repair
• Ask for photos or information
• Decline
• Pass to staff

The policy has nine rules, applied from the top; the first that applies wins (rule 8 alone is an exception to rules 5 and 6 when they lead to a refund). The person, Haiku and Jev all read the same text (in Japanese; this is a translation).

1. Injury or accident, a strong complaint, or circumstances this policy does not cover: pass to staff.
2. If it is unclear which order is meant or what happened: ask the customer for information.
3. Damage reported (cracked, chipped, dented) without photos: ask for photos first.
4. A customer with 3 or more returns in the past year: pass to staff.
5. A defect within 30 days of delivery (including a breakdown in normal use) or damage in shipping: refund or exchange as the customer wishes (exchange if no wish is stated). The shop pays shipping.
6. Change of mind (not as imagined, etc.): refund if within 14 days of delivery and unused. The customer pays shipping.
7. A warrantied appliance that breaks in normal use more than 30 days but within a year after delivery: repair.
8. If rule 5 or 6 leads to a refund of 30,000 yen or more: pass to staff.
9. Anything else (past the deadline, used change-of-mind returns, damage from misuse): decline.

Kind	Cases	What's in them
By the policy	40	6–7 cases for each of the six answers
Hard to read	25	No order number or one digit off, the item called something else ("the white kettle"), two items, "2 plates of a set of 6", "I attached photos" with no attachment, a long preamble, etc.
Gray	35	A regular a few days past the deadline, an unclear cause ("I may have dropped it, but it wobbled from the start"), a third return that was a defect every time, a gift or a discontinued item, a complaint of unclear strength, etc.

The correct answer is the one each case was designed with: for policy and hard-to-read cases, the answer it was built to have; for gray cases, the policy applied literally to the facts the email was written from. Claude (Anthropic's AI) fixed each answer first and then wrote the email and order record; Haiku, one of the sorters, wrote none of them. The cases were shuffled, and no sorter was told their kind.

How the layouts are described: six steps

This page describes each layout with a template of the author's that splits a decision into six steps by the form of the question. The ③ Extract, ④ Check and ⑤ Judge in each experiment's figure and text are these steps.

Step	Who	Question	What it does	In this task
① Prepare	Person + LLM	—	Write the policy and the possible answers	The nine rules and six answers above (written by a person with AI's help)
② Case	LLM	—	Bring the one case to decide	One email and its order record
③ Extract	Clairic / Jev / Haiku	"What?"	Copy the values that are written down	Delivery date, unit price, quantity, the customer's words, etc.
④ Check	Jev / Haiku	"Is it …?"	Check, one question at a time, whether the policy's conditions are met	Within 30 days? 30,000 yen or more? (settled by arithmetic) A defect? Does the customer want a refund? (settled by meaning)
⑤ Judge	Jev / Haiku	"Which?"	Pick one answer from what was checked	One of the six answers
⑥ Explain	Person / Haiku	—	Put the reason for the decision into words	A reply or an explanation

What changes from one experiment to the next is who takes ③ to ⑤ (Clairic, Jev or Python) and what ⑤ is shown. We call it the "six-step" layout when ④ goes as far as checking which of rules 1–9 apply and ⑤ is shown only the rules that apply.

All results

There are four experiments:

• Experiment 1: as in Cloudflare's blog. Clairic reads the values and Jev decides from the whole policy.
• Experiment 2: experiment 1 without Clairic. Jev reads the email through multiple choice and Python reads the order record; the decision is asked as in experiment 1.
• Experiment 3: six steps. Clairic at ③, Python and Jev at ④, and Jev at ⑤ shown only the rules that apply.
• Experiment 4: six steps with Clairic taken out of experiment 3. The reading is experiment 2's.

Setup	Correct (of 100)	By the policy (40)	Hard to read (25)	Gray (35)	Settled by the policy (62)	Time per case (mean)	Cost per 100 cases
Person	73	34	22	17	56	18.0 s	751 yen
Claude Haiku 5.5 (one call)	91	40	24	27	61	3.0 s	8.1 yen
Exp. 1: as in the blog	68	33	18	17	49	0.5 s	4.9 yen
Exp. 2: no Clairic, as in the blog	48	21	16	11	36	0.3 s	1.6 yen
Exp. 3: six steps (Clairic reads)	77	34	20	23	51	0.9 s	8.8 yen
Exp. 4: no Clairic, six steps	78	35	22	21	54	0.3 s	1.5 yen

"Settled by the policy" is the 65 policy and hard-to-read cases, less 3 whose email reads two ways. Of the person's 27 misses, 18 were gray cases, and in 13 of those the author passed the case to staff. The correct answer for a gray case is the policy applied literally, so passing a doubtful case to staff counts as a miss.

"The policy's code" is the nine rules written in Python. It takes what was checked up to ④, applies the rules from the top, and gives the policy's literal answer mechanically. In the six-step experiments 3 and 4 we also counted a version where the policy's code makes the final decision (⑤) instead of Jev, as a yardstick for whether ⑤'s decision departs from the policy. Given the true facts the emails were written from, the policy's code gets all 100 cases right.

Setup	⑤ by Jev	⑤ by the policy's code
Exp. 3: six steps (Clairic reads)	77	78
Exp. 4: no Clairic, six steps	78	78

Common conditions

• The policy and the possible answers were fixed before the 100 cases were written.
• Each experiment's questions were fixed after checking the wiring on 2 practice cases (separate from the 100) and were not changed after seeing that experiment's results. What practice changed is noted in each section.
• Fields that need the order line (delivery date, unit price, days since delivery, refund amount and so on) are scored on 99 cases: one case was written so that the email does not say which order it means.
• The text sent to Clairic and Jev is Japanese, because the task is a Japanese business process (Jev's documentation says it is most accurate in English).
• The models are Clairic (@cf/cloudflare/clairic, a preview model), Jev (jev-1.14.0) and Claude Haiku 5.5 (claude-haiku-5-5, effort medium, called through the Claude Code CLI, claude -p).
• Time is the sum of API response times up to the decision. For Haiku it is the API time the CLI reports, not including CLI start-up. For the person it is what the sorting page recorded while it was visible and in use (gaps of 3 minutes or more without input are left out).
• Cost is the API list price at 150 yen to the dollar: Clairic at $0.10 per million input tokens (Cloudflare's docs) and Jev at $0.042 (TypeSafe's docs); neither charges for output. Haiku ran inside a subscription, so its cost is a list-price estimate and nothing was billed. The person's time is costed at 1,500 yen an hour.
• Replies and written reasons (⑥) are not compared, since the person wrote none, and are left out of time and cost.

Person

The author (Takuma Yoshida) sorted the cases one at a time on a sorting page, picking one of the six answers, before seeing any machine results.

Claude Haiku 5.5 (one call)

• Input: the policy, the email and the order record in one call. It returns only the answer (one of the six), as JSON.
• Requests per case: 1

Experiment 1: as in Cloudflare's blog

This layout follows the damage-claim example in Cloudflare's blog (Clairic → Clef → general-purpose LLM, with Jev in Clef's place).

• ③ Extract (Clairic, 1 request): gets the order record and the email as they are, not the policy, and reads one typed field per fact (name, order number, item, days since delivery, the refund if one were given, attachments, returns in the past year, warranty, a list of defects or damage, whether it was used, the customer's wish, a list of other circumstances). Following the blog, Clairic does the arithmetic: days as a duration, and the refund as unit price × quantity.
• ④ (Python): only fills a fixed template with the values; nothing is checked. It does not work out thresholds such as "within 30 days" or "30,000 yen or more".
• ⑤ Judge (Jev, 1 request): reads that text and the whole policy only, and picks one of the six answers to "How should we handle this inquiry?" (a choice question). It never sees the email.
• Sent to a person: the blog advises using confidence to split automatic cases from those a person checks, but gives no rule. Before the run we set one: a case is marked for a person if Clairic's confidence in any value the decision uses (item, days, refund, attachments, returns) is below 0.5.
• Requests per case (up to the decision): Clairic 1, Jev 1
• Changed in practice: nothing. On the second practice case Clairic read the days as 340 (it was 4), but with confidence 0.38 the case was marked for a person, so we ran the layout as the blog has it.

It got 68 right.

Clairic field	Right
Order number	98 of 99
Item	95 of 100
Days since delivery	96 of 99
Refund if given	91 of 99
Attachments, past-year returns	100 of 100 each
Warranty	99 of 99

Of the 32 misses, Clairic's values (days, refund, item) were wrong in 7. In the other 25 the values were right and Jev went wrong applying the policy text (a refund where the customer wanted an exchange in 5, a refund where the answer was to decline in 4, and so on). All 4 cases passed to staff for a refund of 30,000 yen or more (rule 8) were right.

11 cases were marked for a person; of the other 89, 63 (71%) were right. Of the 4.9 yen per 100 cases, 4.2 yen was Clairic (2,796 input tokens per case on average).

Experiment 2: no Clairic, deciding as in the blog

We gave Clairic's job in experiment 1 to Jev and Python, and kept ⑤ asked as in experiment 1. Jev answers only with options and yes/no, never a number or the customer's words, so nothing is extracted.

• ③ Extract: none. Python reads the order record as it is (delivery date, warranty, unit price, past returns, attachments). Which line is decided by the item Jev picks.
• ④ (Jev, 1 request): sees the email and the customer's orders, and answers six questions from options: which item, what happened, whether it was used, the wish, whether damage a photo would show was reported, and whether rule 1's circumstances apply. Rule 1 is asked about in the policy's own words, but Jev does not see the whole policy. How many items have a problem is asked as options from 1 up, only in the 3 cases where an item was ordered more than once.
• ④ (Python): Jev's answers and the values from the order record go into a fixed template. No thresholds such as "within 30 days" or "30,000 yen or more" are worked out (as in experiment 1).
• ⑤ Judge (Jev, 1 request): reads that text and the whole policy, and picks one of the six answers to "How should we handle this inquiry?". It never sees the email.
• Requests per case (up to the decision): Jev 2
• Changed in practice: nothing.

It got 48, down from experiment 1's 68: taking Clairic out did not raise the score. Jev picked "ask for photos or information" 41 times (the correct answer in 10 cases); 40 of those had no attachment, and only 3 reported damage. It may have been pulled by the template's "0 photos attached" and "damage a photo would show: none" (a guess made afterwards). The reading was right on the item in 98 of 99 cases, the order in 99 of 100, and the number of items with a problem in 98 of 99.

Experiment 3: splitting the decision into six steps

In experiments 1 and 2, ④ checked none of the policy's conditions, and Jev at ⑤ applied the policy itself, from the filled-in text and the whole policy. In experiment 3, following the six-step template above, ④ checks which of rules 1–9 apply, and Jev at ⑤ is shown only the rules that apply. Clairic reads, as in experiment 1.

Take the coffee grinder case: a defect 5 days after delivery with a refund wish meets rule 5, and 4 returns in the past year meet rule 4. Rules apply from the top, so the correct answer is rule 4's "pass to staff". At ⑤, Jev weighed the two rules and gave "pass to staff" 0.93 and "refund" 0.06.

• ③ Extract (Clairic, 1 request): gets the order record and the email. It copies the order number as written in the email; the attachments, past-year returns and the date the email arrived from the record; and the one order line the email points to (item, delivery date, quantity, unit price, warranty). It does no arithmetic. The email itself is copied field by field in the customer's own words, not mapped to the policy's terms (what happened, the trigger, whether it was used, the wish, damage, how many items have a problem, other circumstances).
• ④ Check (Python and Jev, 1 request): Jev sees the customer's words from ③ and the policy, not the email, and answers four questions: what happened (defect / shipping damage / change of mind / damage from misuse / unclear), whether it was used, the wish, and whether rule 1's circumstances apply. Python works out from ③'s dates and numbers the days since delivery, the deadlines, the warranty and the refund amount, and which of rules 1–9 apply.
• ⑤ Judge (Jev, 1 request): sees only the policy and "the rules that apply" (e.g. rule 5 → refund, rule 8 → pass to staff), and picks one answer (with a probability for each).
• ⑥ Explain (Claude Haiku 5.5): writes the reason in up to 150 characters. We compare only up to ⑤.
• Requests per case (up to the decision): Clairic 1, Jev 2 (in sequence, since ④ uses ③'s answers)
• Changed in practice: ③ had copied "one plate was broken in the box", yet Jev at ④ answered that no damage was reported. So Python now decides whether damage was reported, from whether ③ copied any description of damage.

It got 77 right; with ⑤ done by the policy's code, 78. Jev's ⑤ matched what the code gives for the same "rules that apply" in 99 of 100 cases.

Step	Field	Right
③ Clairic	Order number (as written in the email)	96 of 100
③ Clairic	Attachments, past-year returns, date received	100 of 100 each
③ Clairic	The order line in the record	98 of 100
③ Clairic	Delivery date, unit price, warranty	99 of 99 each
③ Clairic	Quantity ordered, items with a problem	95 of 99 each
④ Jev	What happened	92 of 100
④ Jev	Whether it was used	75 of 100
④ Jev	The wish	95 of 100
④ Jev	Rule 1's circumstances	86 of 100
④ Python	Damage reported	90 of 100
④ Python	Days since delivery	99 of 99
④ Python	Refund amount	98 of 99

Most of the 23 misses came from reading at ③ and ④. On rule 1, Jev said a circumstance existed when none did in 11 cases. On damage, the practice-time rule, "damage was reported if ③ copied any description of damage", proved too broad: peeling coating, discoloration and wobbling counted too (10 false reports, 6 of which went wrong as "ask for photos or information").

Clairic reads the whole order record and email, an average of 4,737 input tokens per case, so 7.1 of the 8.8 yen per 100 cases was Clairic.

Experiment 4: six steps, no Clairic

We gave Clairic's job in experiment 3 to Jev and Python, reading as in experiment 2. ④'s rule checks and ⑤ are the same as in experiment 3.

• ③ Extract: none (as in experiment 2).
• ④ Check (Jev, 1 request): uses the answers to experiment 2's six questions.
• ④ Check (Python): applies the rules one by one (as in experiment 3's ④).
• ⑤ Judge (Jev, 1 request): picks one answer from the rules that apply and the policy (as in experiment 3's ⑤).
• Requests per case (up to the decision): Jev 2 (the reading request is the same one as in experiment 2)
• Changed in practice: nothing. On the second practice case (a plate broken in the box) Jev read no damage report, and the answer was wrong. It is the same weakness experiment 3 moved to Python, but with no Clairic text there was nowhere to move it, so we ran the questions unchanged.

It got 78, about the same as experiment 3 (77), while the cost per 100 cases fell from 8.8 to 1.5 yen (a sixth) and the time per case from 0.9 to 0.3 s. Jev's ⑤ matched the policy's code in all 100 cases. The set of cases it got right shifted a little: 64 right in both, 14 only in experiment 4, 13 only in experiment 3.

The reading is exactly experiment 2's, so changing how ⑤ was asked moved the score from 48 to 78 on its own.

The misses shifted too. On damage, experiment 3's 10 false reports fell to 0, but Jev now missed 5 real ones. On rule 1, false "yes" answers rose from 11 to 19; 19 of the 22 misses include a misread of rule 1.

Reading accuracy (③ and ④)

The fields read from the email in the two six-step experiments. In experiment 3 (Clairic reads), damage is decided by Python. Experiment 2 (no Clairic, as in the blog) shares experiment 4's reading.

Field	Exp. 3: six steps (③ Clairic, ④ Jev)	Exp. 4: no Clairic, six steps (Jev)
What happened	92/100	95/100
Whether it was used	75/100	92/100
The wish	95/100	98/100
Damage reported	90/100	95/100
Rule 1's circumstances	86/100	79/100
Items with a problem	95/99	98/99

Routing by confidence (thresholds drawn afterwards)

Jev returns a probability for each answer, and Clairic a confidence for each value. Jev's confidence is the highest answer probability; Clairic's is the lowest confidence among the five values the decision uses (item, days since delivery, refund amount, attachments, past returns). The idea is to leave only the cases at or above a threshold to Jev (Jev's answer at ⑤ goes out as is, with no person looking) and pass the rest to a person. When the split uses Clairic's confidence, the answer used is still Jev's at ⑤.

First, how the confidence of the 100 cases was distributed.

The same Jev gave very different confidence depending on the layout. In the blog layout (experiment 1), Jev's confidence was spread out: 39% of cases were at 0.8 or higher. At ⑤ in the six-step layouts (experiments 3 and 4), it leaned high: 71% and 73% of cases were at 0.9 or higher, and 28% each were exactly 1.0. Every case at 0.99 or higher was one where exactly one rule applied. Clairic's confidence put 55% of cases between 0.8 and 0.99, and none at 1.0. With the same line at 0.8, Jev gets 39% of cases in the blog layout and 79 to 81% in the six-step layouts.

Next, the accuracy on the cases at or above each threshold, the ones left to Jev.

In the blog layout (experiment 1), Jev's accuracy rose with the threshold: 95% at 0.8 or higher and 100% at 0.97 or higher. The two misses at 0.8 or higher were both gray cases, and of the 61 below 0.8, 31 were right. Splitting by Clairic's confidence, raising the threshold did not lift the accuracy on the cases left to Jev above about 70%: most misses happened not in reading the values but afterwards, in Jev's decision.

In the six-step layouts (experiments 3 and 4), accuracy passed 90% from 0.9 up, but some cases were wrong even at confidence 1.0 (2 in experiment 3, 1 in experiment 4). ⑤ sees only the rules that apply, so it cannot notice a misreading at ③ or ④. In experiment 3, splitting by the lowest probability among ④'s four answers instead, all 22 cases at 0.85 or higher were right.

Here are the cases left to Jev at each threshold, and how many of them were right.

Threshold	Exp. 1 (as in the blog): Jev's confidence	Exp. 1 (as in the blog): Clairic's confidence	Exp. 3 (six steps, Clairic reads): confidence of Jev at ⑤	Exp. 4 (no Clairic, six steps): confidence of Jev at ⑤
≥ 0.5	58 of 80	63 of 89	77 of 99	78 of 100
≥ 0.7	44 of 50	51 of 71	75 of 91	75 of 89
≥ 0.8	37 of 39	45 of 60	69 of 81	72 of 79
≥ 0.9	24 of 25	30 of 41	65 of 71	68 of 73
≥ 0.97	12 of 12	15 of 20	47 of 50	46 of 48
≥ 0.99	6 of 6	2 of 5	41 of 43	43 of 45
1.0	4 of 4	none	26 of 28	27 of 28

Taking Jev's confidence of 0.8 or higher in the blog layout (experiment 1) as "sure", here is what changes with where the rest go. The last row alone takes Jev's confidence of 0.8 or higher at ⑤ in experiment 4 (no Clairic, six steps) as "sure" instead. "Unseen misses" counts wrong answers that went out without any person looking at them.

Routing	Correct	Unseen misses	Cost per 100 cases	Time per case (mean)	Person's time (100 cases)
Person only	73	0	751 yen	18.0 s	30.0 min
Claude Haiku 5.5 only	91	9	8.1 yen	3.0 s	0 min
Exp. 1 (as in the blog) only	68	32	4.9 yen	0.5 s	0 min
Sure: Jev; rest: person	80	2	524 yen	12.9 s	20.8 min
Sure: Jev; rest: Haiku	92	8	10.1 yen	2.4 s	0 min
Jev and Haiku agree: automatic; otherwise person	86	2	309 yen	10.6 s	11.9 min
Exp. 4: sure: Jev; rest: person	83	7	128 yen	3.4 s	5.1 min

Here is how "Sure: Jev; rest: person" reaches 80. For the 39 cases at 0.8 or higher, Jev's answer went out as is, and 37 were right. For the other 61, the author's own sorting was used, and 43 were right: 37 and 43 make 80. The whole difference from the person alone (73) comes from the 39 cases left to Jev. The author got 30 of those 39 right, and the 7 that Jev got right and the author got wrong are exactly the gain (there was no case the author got right and Jev got wrong). By kind, the 7 were 3 by-the-policy, 2 hard-to-read and 2 gray cases; in 4 of them the author had passed the case to staff. The person's 20.8 minutes is the time the author spent on those 61 cases, and most of the 524 yen per 100 is that time (at 1,500 yen an hour).

The last row applies the same 0.8 line to experiment 4, the six-step layout without Clairic. Jev's confidence at ⑤ there leans high, so 79 cases were at 0.8 or higher, and 72 of them were right. The other 21 used the author's sorting, and 11 were right, for 83 in all. That is 3 more than the split in experiment 1, and since the person looks at only 21 cases, the person's time was 5.1 minutes and the cost 128 yen per 100. In exchange, wrong answers that go out without a person looking rise from 2 in experiment 1 to 7. In all 7, Jev's decision at ⑤ followed the policy; the miss came earlier, in reading the email. ⑤ decides from the rules that apply alone, so a misreading before it does not show in its confidence.

These thresholds were drawn after seeing the 100 results.

What we learned

On speed and cost, the Jev layouts beat Haiku

Experiment 4, six steps without Clairic, took 0.3 s per case and 1.5 yen per 100 cases, faster and cheaper than Claude Haiku 5.5 at 3.0 s and 8.1 yen. The layouts with Clairic were also fast, at 0.5–0.9 s, but at 4.9–8.8 yen per 100 cases they cost about what Haiku did. Most of that cost is Clairic reading the order record and the email.

On accuracy, no Jev layout reached Haiku

The best Jev layout got 78 right; Haiku got 91. On the 62 cases the policy settles, the best Jev layout got 54 and Haiku 61.

Shown only the rules that apply, Jev decided as the policy's code did

The same Jev (jev-1.14.0) scored anywhere from 48 to 78 depending on the layout, and the difference lay in what Jev was handed at ⑤. Experiments 2 and 4 share exactly the same reading, yet handing Jev the filled-in text and the whole policy gave 48, and handing it only the rules that apply gave 78. In the blog layout (experiment 1), 25 of the 32 misses had the values right and went wrong applying the policy text. Once each "Is it …?" had been checked at ④ and Jev weighed only the rules that applied, Jev's answer at ⑤ matched the policy's code in 99 of 100 cases in experiment 3 and in all 100 in experiment 4. That these layouts still got only 77–78 right is because most misses happened before ⑤, in the reading at ③ and ④.

The gap to Haiku was in reading, mostly rule 1

Reading here means taking what the email says and reading it in the policy's terms (a defect? used or not? rule 1's circumstances?). Given the true facts, the policy's code gets all 100 cases right, and in experiment 4 Jev's answer at ⑤ matched the code in all 100. So all 22 misses in experiment 4 came from Jev's reading of the email. In 19 of them, Jev read injury, a strong complaint or circumstances the policy does not cover (rule 1) into an email that had none. Haiku got 15 of those 19 right. Jev at ④ does not see the whole policy, so it cannot tell where "circumstances the policy does not cover" begin; Haiku in one call decides with the policy in front of it. Even Jev in experiment 3, shown the policy, said such a circumstance existed when none did in 11 cases.

By kind of email, only 2 of the 13-case gap between experiment 4 (no Clairic, six steps) and Haiku were in the 25 hard-to-read cases; 6 were in the 35 gray cases and 5 in the 40 by-the-policy cases. The gap is less in confusingly written emails than in the gray and by-the-policy ones.

Arithmetic was more accurate in code, and dropping Clairic kept the score

In the blog layout (experiment 1), Clairic's days since delivery were right in 96 of 99 cases and its refund amounts in 91 of 99. In experiment 3, where Clairic only copies dates, quantities and prices and Python does the arithmetic, the days were right in 99 of 99 and the refund in 98 of 99. Going further, with Python reading the order record as it is and Jev reading the email through multiple choice (experiment 4), the score stayed about the same as reading with Clairic (78 against 77) at a sixth of the cost.

Jev's confidence separated right from wrong in the blog layout

In the blog layout (experiment 1), 37 of the 39 cases at Jev's confidence 0.8 or higher were right. Splitting by Clairic's confidence, raising the threshold kept accuracy on the cases left to Jev at about 70%. In the six-step layouts (experiments 3 and 4), ⑤ sees only the rules that apply, and some cases were wrong even at confidence 1.0.

Conditions and limits

• The shop, emails and order records are fictional, written by Claude. They are not real customers' emails.
• The person is one person, the author. The correct answer for gray cases is the policy applied literally, which is a different yardstick from a person's judgment.
• The layouts were chosen after seeing other layouts' results on the 100 cases. No experiment's questions were changed after seeing its own 100 results, but the layouts were chosen on the same 100 cases, so on new emails the gaps may narrow.
• The confidence thresholds (0.8, 0.97 and so on) were drawn after seeing the 100 results. On new emails, accuracy on the cases left to Jev may be lower.
• With 100 cases, differences of a few cases can arise by chance. Haiku's first run, which also wrote a reply alongside the answer, got 89 (the 91 on this page is the run that returned the answer alone).
• Clairic is a preview model.
• The text sent to Clairic and Jev was Japanese.
• Times are API times in our setup. The person's time is time with the page on screen.

UnlockJapan has no partnership with, payment from or sponsorship by TypeSafe or Cloudflare.