Short version: on 37 UK and US finance questions, OpenAI Luna got 29 right, JEV-27B-VL 28, Jev 24, Clef Flash 22, Kev-4B 20 and Julia 1 8. We ran the full set three times, and the scores came out the same each time (Jev scored 23 once).
Contents
- The test
- Demo video
- The six models
- Results
- Results by finance area
- Yes/no vs multiple choice, UK vs US
- The hardest cases
- Strengths and weaknesses
- How they differ
- Which one to use
- Limits of this test
Demo video
The test
- 37 cases: 18 UK and 19 US, covering tax, accounting, bookkeeping, auditing, advisory and sales.
- Two question types:
- 22 multiple choice: pick one of several options.
- 15 yes/no: called noul in Jev's API.
- Each case has one correct answer fixed in advance from an official source, such as GOV.UK/HMRC, the IRS, the FRC, FASB, ACCA, the AICPA or the SEC. The source and its exact wording are stored with the case.
- Every model got exactly the same facts and question.
- Option names were hidden. Options were sent as
option_1,option_2and so on, so a label likeby_30sep_eff_1octcouldn't give the answer away. - We ran the full set three times. The scores came out the same each time, except for Jev, which scored 24, 24 and 23.
A case counts as right only if the model's top answer matches the fixed answer.
The six models
Jev 1.13 (TypeSafe) - TypeSafe's hosted decision model, the one the others are compared against. - Text only. - You send the facts and typed questions (choice, score, yes/no), and you get back an answer, a probability for every option, and a confidence value. - Price: $0.042 per million input tokens; output is free.
OpenAI Luna (gpt-6-luna, OpenAI Decisions API)
- OpenAI's hosted decision model.
- Same idea as Jev, with a different request format: yes/no is called a predicate, and choices and levels are sent as lists.
- Reads text and images.
- OpenAI has not published a price for the Decisions API yet.
JEV-27B-VL (autotrust) - An open-weight 27-billion-parameter vision-language model, run locally in a 4-bit build. - It's a chat model, so it answers with the letter of an option. Its probabilities come from how likely each letter was. - Reads text, images and video. - No per-call fee.
Clef Flash (Cloudflare) - An open-weight 9-billion-parameter decision model, built on Qwen3.5-9B, run locally. - Same question types as Jev, with probabilities for every option. - Reads text, JSON, images and video. - No per-call fee.
Kev-4B (open source, jaredpalmer/kev) - A 4-billion-parameter open model built on Qwen3.5 to work like Jev. - Its API matches Jev's, so code written for Jev works with it. - Text only. - No per-call fee.
Julia 1 (Supersonic Labs) - A very small 144-million-parameter decision model, built on mmBERT-small. - Same question types as Jev. - Text only. - No per-call fee.
Results
| Rank | Model | Right (of 37) | Multiple choice (of 22) | Yes/no (of 15) | Time per case (median) | Runs as |
|---|---|---|---|---|---|---|
| 1 | OpenAI Luna | 29 (78%) | 18 | 11 | 0.24 s | Hosted API |
| 2 | JEV-27B-VL | 28 (76%) | 18 | 10 | 1.01 s | Open weights, local |
| 3 | Jev 1.13 | 24 (65%) | 17 | 7 | 0.19 s | Hosted API |
| 4 | Clef Flash | 22 (59%) | 15 | 7 | 0.73 s | Open weights, local |
| 5 | Kev-4B | 20 (54%) | 12 | 8 | 0.15 s | Open weights, local |
| 6 | Julia 1 | 8 (22%) | 2 | 6 | 0.04 s | Open weights, local |
How to read the times: - They're the median time per case. For the hosted models (Jev, Luna) this includes the internet round trip. - Jev is the speed baseline. Kev takes about 0.8× as long as Jev, Luna about 1.3×, Clef about 4× and JEV-27B about 5×. Julia takes about a fifth as long.
Results by finance area
| Area (cases) | Luna | JEV-27B | Jev | Clef | Kev | Julia |
|---|---|---|---|---|---|---|
| Tax (13) | 8 | 5 | 5 | 5 | 4 | 4 |
| Accounting (6) | 5 | 6 | 6 | 5 | 4 | 2 |
| Bookkeeping (6) | 6 | 6 | 6 | 5 | 5 | 0 |
| Auditing (5) | 5 | 5 | 4 | 4 | 5 | 0 |
| Advisory (4) | 3 | 3 | 2 | 2 | 1 | 0 |
| Sales (3) | 2 | 3 | 1 | 1 | 1 | 2 |
Tax is the biggest weak spot: with 13 cases it's the largest area, and no model got more than 8 right. Bookkeeping is the easiest: Luna, JEV-27B and Jev got all 6.
Yes/no vs multiple choice, UK vs US
| Luna | JEV-27B | Jev | Clef | Kev | Julia | |
|---|---|---|---|---|---|---|
| Multiple choice (22) | 18 | 18 | 17 | 15 | 12 | 2 |
| Yes/no (15) | 11 | 10 | 7 | 7 | 8 | 6 |
| UK cases (18) | 15 | 14 | 11 | 12 | 11 | 4 |
| US cases (19) | 14 | 14 | 13 | 10 | 9 | 4 |
- Yes/no questions are harder than multiple choice for every model except Julia.
- Jev and Clef got under half of them (7 of 15).
- Most of these questions turn on one exact rule, such as a threshold, a date or an exception. A model that misses the rule gets it wrong.
- Julia is the opposite. It got only 2 of 22 multiple-choice cases, but 6 of 15 yes/no cases.
The hardest cases
Only 2 cases were answered correctly by all six models: - capitalising a press service over the threshold - a warehouse fire after year end
One case beat every model:
| Case | Correct answer | What the models said |
|---|---|---|
| Which contractors need a Form 1099-NEC (US, 2026 payments) | Only Lopez | Luna, JEV-27B, Jev and Clef: Lopez and Reyes. Kev and Julia: all three. |
Cases only one model got right:
| Case | Correct answer | Only right | Everyone else said |
|---|---|---|---|
| UK Making Tax Digital: does Dana have to use it from April 2026? (qualifying income is turnover, not profit) | No | Luna | Yes |
| UK VAT registration: rolling 12-month test vs tax year | Register by 30 Sep, effective 1 Oct | Julia | Register by 30 Sep, effective 1 Sep |
| US FBAR with two small foreign accounts (balances count together) | Yes, file | Julia | No |
Julia's two lone wins sit next to 29 misses, so they don't make it reliable. They do show that the strongest models can all make the same mistake on rules about thresholds and combining amounts.
Strengths and weaknesses
OpenAI Luna - ✅ Most right overall (29/37) and best at tax (8/13) and yes/no (11/15). - ✅ Only model to get the Making Tax Digital case right. - ✅ Fast (0.24 s) and reads images. - ⚠️ Hosted only, and the Decisions API has no published price yet. - ⚠️ Different request format from Jev, so it isn't a drop-in swap.
JEV-27B-VL - ✅ 28/37, one behind Luna. Top or tied-top in accounting, bookkeeping, auditing and sales. - ✅ Open weights, runs locally, no per-call fee, reads images and video. - ⚠️ Slowest model (about 1 s per case, 5× Jev). - ⚠️ A chat model, so you ask with lettered options and read the letter back. It doesn't use Jev's question format. - ⚠️ Weak on tax (5/13).
Jev 1.13 - ✅ 24/37, fast (0.19 s), cheap, and the simplest interface: typed questions in, typed answers with probabilities and confidence out. - ✅ Perfect on accounting and bookkeeping (6/6 each). - ⚠️ Weak on yes/no questions (7/15, tied with Clef) and on tax (5/13). - ⚠️ Text only, and its score moved by one case between runs (24, 24, 23).
Clef Flash - ✅ 22/37 with Jev's question format, and reads images and video. - ✅ Open weights, runs locally. - ⚠️ Below Jev on most areas and on yes/no (7/15). About 4× slower than Jev.
Kev-4B - ✅ Drop-in replacement for Jev (same API) and the fastest model that's still useful (0.15 s). - ✅ Open weights, small, 5/5 on auditing. - ⚠️ 20/37: 4 behind Jev, and weak on US tax (1/7) and multiple choice (12/22).
Julia 1 - ✅ Fastest by far (0.04 s) and tiny. - ⚠️ 8/37 is not usable for these finance questions. It got 0 in bookkeeping, auditing and advisory, and 2 of 22 multiple-choice questions.
How they differ
| Jev 1.13 | OpenAI Luna | JEV-27B-VL | Clef Flash | Kev-4B | Julia 1 | |
|---|---|---|---|---|---|---|
| Made by | TypeSafe | OpenAI | autotrust | Cloudflare | open source (jaredpalmer) | Supersonic Labs |
| Size | not disclosed | not disclosed | 27B | 9B | 4B | 144M |
| Runs as | hosted API | hosted API | open weights, local | open weights, local | open weights, local | open weights, local |
| Reads | text | text, images | text, images, video | text, JSON, images, video | text | text |
| How you ask | typed questions | typed questions (own format) | chat, lettered options | typed questions | typed questions (Jev's API) | typed questions |
| Returns probabilities | yes, plus confidence | yes | yes, via the letter's probability | yes | yes, plus confidence | yes |
| Price per call | $0.042 per 1M input tokens | not published | none | none | none | none |
| Right on 37 finance cases | 24 | 29 | 28 | 22 | 20 | 8 |
Which one to use
These recommendations are based only on the results above:
- Most correct finance answers: OpenAI Luna (29/37), with JEV-27B-VL right behind (28/37) if you want open weights and no per-call fee, and can accept about 1 s per question.
- Fast, cheap and simple to build with: Jev (24/37, 0.19 s).
- Jev's API without the hosted service: Kev-4B works with code written for Jev, at the cost of 4 fewer right answers.
- Images or video as well: JEV-27B-VL, Clef Flash or Luna.
- Not for finance questions like these: Julia 1.
- Tax questions: every model got at least 5 of 13 wrong (Luna missed 5, the rest missed 8 or more). Have a person check tax answers, whichever model you use.
Limits of this test
- 37 cases is a small set. One case moves a score by about 3 percentage points, so models 1–2 cases apart are close.
- Rules and thresholds are as of October 2026, for the UK and US only.
- Options were sent in their written order, with their names hidden.
- JEV-27B-VL ran as a 4-bit local build. The full-precision model may score differently.
- A right answer is a match with the source-based answer, not a judgment of the model's reasoning. Check the source on each case before relying on it for client work.