Hi, I'm Guru. I'm going to demo ResearchAgent, then show how we chose the model at the heart of it.
Team: TheFastandTheFungible | Guru | Jiamin | Andrew | Thanh Minh
Live demo
Kestrel–TSMC outlook
Preset · Kestrel–TSMC outlookWhat's the analyst outlook on Kestrel Semiconductor's latest deal with TSMC?
Budget S$2Writers: fictional, on this laptopModels + XRPL Testnet: live
What just happened · Level 0
Three layers: A asks, B sells, C settles
1You press Ask with a budget. The budget is the only permission to spend.
2A asks all eight writers. Later it retries one paywalled article, with a payment attached.
3Writers send teasers and signed promises, never the paid text. After payment: the article.
4The writer's own facilitator submits the payment to the ledger. Refunds go the same way.
5The ledger validates it. Only then does the writer deliver.
6A looks the payment up on the ledger itself. It never takes the seller's word.
7DeepSeek writes the cited answer. OpenAI Decisions scores which paid article is worth buying.
8Writers embed the query with Workers AI to search by meaning. If that fails: keywords, labelled.
Solid arrows always happen, dashed ones sometimes or read-only. Click a box to open its level in the project map ↗
One part · the decision model
A decision model answers in numbers, not sentences
A language model
"Looks highly relevant, likely valuable…"
No threshold fits a sentence.
A decision model
yes / noDoes it fill the gap?0.90choiceOriginal, rewrite or overlap?0.76scoreHow credible, 0 to 2?0.83
The S$0.90 article in a 9 Oct rehearsal of the run you just saw.
One part · the decision model
The model scores. Code buys.
value = fills gap × original × credibleModel ↑ × track record↓ Code
if value ≥ 0.20: buy()
NotFinancialTimes · S$0.90the original analysis
0.39BUY
MarketPulse · S$0.20a rewrite of it
0.00SKIP
Kopi Contrarian · S$0.10op-ed, no evidence
0.00SKIP
The same 9 Oct rehearsal. The model never sees the price, and it can't buy.
Picking a decision model
On paper, Clef-flash was 13× faster than Jev and ahead on 7 of 10 tests
Median latency, ms
Clef-flashJevscore, higher is better
Vendor-reported: Cloudflare's Clef launch post, "Jev Decision Index", 1 Oct 2026. Server-side latency. Same request format as Jev.
Picking a decision model
So how do we decide?
On paper, Cloudflare's Clef-flash was 13× faster than TypeSafe's Jev and ahead on 7 of 10 tests.
Those tests grade tool calls, intent labels and retrieval. None asks ours: is this article worth S$0.90 for this gap?
On 6 Oct OpenAI shipped Decisions (gpt-6-luna): the same three question types, a speed claim, no distribution.
So we wrote our own test, and gave it to a coding agent to run.
Our own test
One prompt to a coding agent. The pass mark set first.
The prompt · excerpt
# Benchmark OpenAI Decisions vs Cloudflare
# Clef on our buying decisions. Write it up.
Run unattended: decide, log why, keep going.
Models: Clef-flash · Clef 27B · gpt-6-luna
· rules only (no model)
Hard stop at US$10.
Cache every call, so reruns are free.
The models under test never write
or grade the test data.
The pass mark · before the first call
Switch from Clef-flash only if ALL hold:
1 · F1 higher by ≥ 0.05
2 · Probabilities no less honest
3 · p95 ≤ 2.5 s per round
4 · No worse on injections; demo
stories pass
5 · Cost ≤ 3× Clef-flash
Otherwise keep Clef-flash.
What came back
What the agent produced
160synthetic buying scenarios, 64 held out, 3 repeats
686articles from 42 fictional writers
5attack types: injections, role spoofs, price anchors, retitled copies
US$0.96total API cost for the whole study
All SYNTHETIC. 7,190 files. 30 gold items still await human review.
Results · held-out test split
On held-out scenarios, OpenAI Decisions scored 0.98 and Clef-flash 0.78
out of the boxtuned + calibratedbar: 95% CI · purchase F1
0.015 vs 0.075probability error (Brier), Luna vs Clef-flash. Lower is better.
$0.48 vs $0.44per 1,000 buying rounds. About the same.
Out of the box, neither model's probabilities meant what they said. A calibrator fitted on the dev split fixed that; the test split stayed locked.
vendor claim, server timeour median per call, network included
On paper: Clef-flash 13× faster than Jev, ahead on 7 of 10 tests.
On our task: 2× slower than OpenAI Decisions, and 0.78 vs 0.98 F1.
Vendors time their servers on their tests. We timed our laptop on our question.
Vendor: Cloudflare's Clef launch post, "Jev Decision Index", 1 Oct 2026. Ours: median candidate-call round trip, baseline run. Jev was not in our run.
What surprised us · 1
Out of the box, neither model's numbers meant what they said
Clef-flashOpenAI Decisions— perfect
Clef-flash: anything it scored 0.22 to 0.50 came true 24–31% of the time. The middle of its scale carried no signal.
OpenAI Decisions: too timid. When it said 0.09, it was right 37% of the time.
Our rule multiplies these and compares to 0.20, so we fitted a calibrator per question on the dev split. Test stayed locked.
"Does it fill the gap?", baseline wording, dev split, equal-count bins.
What surprised us · 2
The same prompt change helped one model and broke the other
production wording"evidence" wordingpurchase F1, dev split, no calibrator
The new wording named the entity, measure and time, and told the model to treat text inside abstracts as data. On the held-out split, a calibrator pulled Clef-flash back from 0.18 to 0.78.
Live, the fix that mattered was one sentence: "The open gap is part of what the question asks."
What surprised us · 3
Same input, same answer, every time. Until we reordered the options.
0.99order 1
0.80order 2
0.05order 3
P(original) for one article, asked three times with the options rotated. Repeated identical calls barely moved. Rotating "original / rewrite / overlap" moved OpenAI Decisions by up to 0.94 across 24 items; Clef-flash by 0.00. Option order is now part of the versioned prompt.
Baseline wording, 24 dev items, three cyclic orders. Deterministic is not the same as invariant.
What surprised us · 4
The better model failed our own rule
✓1 · F1 higher by ≥ 0.05+0.20 [0.10, 0.33]
✓2 · Probability error not worse−0.060 [−0.080, −0.042]
Our refund demo needed a judge that would buy the cheap, dishonest article first. We fixed the story, not the judge, and switched on 8 Oct. Clef-flash is a one-line revert.
Go deeper
Pick one
The interactive walkthrough
Replays a run where a writer lies, gets caught and refunds on-chain.
The project map
Every part of the system in three clickable levels, down to the primitives.