Can an AI swarm on Claude Opus 5.5 beat Polymarket? Not on these four markets. On October 24 and 25 we locked every forecast from the open-source MiroFish swarm, three Claude models asked directly and OrcaLayer Smart Money, and published them in a public repository before any market resolved (the Claude Opus vs Polymarket forecasts). Scored against how the markets resolved, the Polymarket price had a mean Brier score of 0.167, the best AI forecast 0.203 and Opus asked directly 0.218. Lower is better.
TL;DR
- Nothing beat the market. The Polymarket price at our snapshot had a mean Brier score of 0.167. The best AI forecast, the swarm on Claude Opus 5.5, scored 0.203, and a coin flip scores 0.250. By absolute error, the second measure part one promised, the top of the table is the same: price 0.351, swarm 0.419.
- The swarm was not measurably better than asking Opus directly. Swarm 0.203, Opus asked directly 0.218. Their forecasts differed by 1.5 to 3 points on three markets and matched on the fourth, and one swarm run cost 500 to 900 times as much as one direct answer.
- The ranking hangs on one market. The Bank of Canada raised rates, as most economists in a Reuters poll expected. The price gave a hold 18.7%, the models and swarms 45% to 55%, MiroFish's report agents 89% to 93% and Smart Money's dollar share 96.5%. Without that market, three sources in the table score better than the price.
- Smart Money's dollar share had the best and the worst single scores. It was nearly perfect on Toronto and Iran and badly wrong on the Bank of Canada.
- A permutation test across both rounds cannot separate skill from luck yet. With one NO among seven resolved markets, the best possible result only says a source gave its lowest forecast to the NO market.
How the four Polymarket markets resolved
| Market | Market question | Outcome |
|---|---|---|
| Toronto | Will Olivia Chow win the 2026 Toronto mayoral election? | YES: Chow re-elected with 54.2% of the vote (unofficial count, all polls reporting) |
| Likud | Will Likud win the most seats in the 2026 Israeli legislative election? | YES: Likud 27 seats, Yashar 21 (final count) |
| Bank of Canada | Will the Bank of Canada make no change to the target for the overnight rate at the October interest rate announcement? | NO: raised by 25 bp to 2.5%, its first hike since July 2023 |
| US x Iran | US x Iran ceasefire continues through October 31? (NO only on a US air or missile strike on Iranian territory) | YES: no qualifying strike reported; resolved November 7 after an UMA dispute vote |
AI vs Polymarket: Brier scores on four resolved markets
Each market column is the Brier score, (p โ outcome)ยฒ, with p as a probability from 0 to 1. Mean Brier averages the four. Mean absolute error averages |p โ outcome|. Mean log score averages โln(p given to what happened), with a floor of 0.001. The last column is the source's mean Brier minus the price's, so a plus means worse than the market. For all three scores, lower is better.
Part one promised the Brier score and the absolute error. After it was published, a reader on r/algotrading asked for the log score and a comparison with the price, so we added both. The top three are the same on every measure.
| Source | Toronto | Likud | Bank of Canada | US x Iran | Mean Brier | Mean absolute error | Mean log score | Mean Brier minus price |
|---|---|---|---|---|---|---|---|---|
| Polymarket price at the snapshot | 0.119 | 0.483 | 0.035 | 0.031 | 0.167 | 0.351 | 0.502 | 0 |
| Swarm on Opus 5.5 | 0.230 | 0.360 | 0.203 | 0.021 | 0.203 | 0.419 | 0.581 | +0.037 |
| Opus 5.5 asked directly | 0.260 | 0.384 | 0.203 | 0.026 | 0.218 | 0.435 | 0.613 | +0.051 |
| Haiku 5.5 asked directly | 0.270 | 0.449 | 0.203 | 0.040 | 0.240 | 0.460 | 0.666 | +0.074 |
| Coin flip, 50% | 0.250 | 0.250 | 0.250 | 0.250 | 0.250 | 0.500 | 0.693 | +0.083 |
| Sonnet 5.5 asked directly | 0.230 | 0.490 | 0.250 | 0.040 | 0.253 | 0.470 | 0.694 | +0.086 |
| Smart Money, wallets on YES | 0.274 | 0.564 | 0.154 | 0.108 | 0.275 | 0.499 | 0.757 | +0.108 |
| Swarm on Haiku 5.5 | 0.250 | 0.563 | 0.303 | 0.048 | 0.291 | 0.505 | 0.782 | +0.124 |
| MiroFish report agent (Opus swarm) | 0.160 | 0.360 | 0.792 | 0.068 | 0.345 | 0.538 | 0.984 | +0.178 |
| Smart Money, money on YES | 0.017 | 0.498 | 0.931 | 0.004 | 0.363 | 0.466 | 1.196 | +0.196 |
Direct figures are the median of five answers per model. The swarm on Sonnet 5.5 ran on two markets only (Toronto 0.250, Bank of Canada 0.245, mean 0.248, vs 0.077 for the price on the same two), so it is not ranked. The report agents of the Haiku and Sonnet swarms are in the data file: Haiku's averaged 0.365 over four markets and Sonnet's 0.521 over two. Every source, market and round is in scoreboard.csv.
Finding 1: no AI forecast beat the Polymarket price
The Polymarket price had the lowest mean Brier score, the lowest absolute error and the lowest log score. Every language model, every swarm and both Smart Money measures did worse on all three.
The models gained most on the price where the price was furthest from the outcome. On Likud the market gave 30.5%. Opus and Haiku asked directly said 38% and 33%, and the swarm on Opus said 40%, which scored 0.360 compared with the price's 0.483. Sonnet said 30% and did slightly worse than the market (0.490). On Toronto the price, at 65.5%, was on the right side of 50%, while Opus and Haiku asked directly (49% and 48%) were not.
Finding 2: the ranking hangs on the Bank of Canada
A hike was what most economists in the Reuters poll expected (19 of 34), and the price gave it 81%. The models had the same poll in the news we gave them and stayed at 45% to 55% for a hold. Smart Money's dollar share sat on a hold at 96.5%.
Leave out one market at a time and only this one changes who is first. Without the Bank of Canada, the price (0.211) falls behind Smart Money's dollar share (0.173), the Opus report agent (0.196) and the swarm on Opus (0.204), and outside the table the Haiku report agent (0.204). Without Toronto, Likud or Iran, the price stays first, and without Iran no model or swarm beats a 50% coin flip on the other three.
Finding 3: MiroFish's report agents scored worse than their swarms
MiroFish ends each run with a report agent that writes a summary and the number a user sees. In our runs it never saw the agents' answers. The simulation had already shut down, so it wrote from the knowledge graph built from the seed files (part one).
All three report agents said 89 to 93 on the Bank of Canada and scored worse than their own swarms there: Opus 0.792 vs 0.203, Haiku 0.846 vs 0.303, Sonnet 0.865 vs 0.245. That one market made each of them worse than its swarm over the four. On average over the other markets each did better: Opus 0.196 vs 0.204, Haiku 0.204 vs 0.287, Sonnet (Toronto only) 0.176 vs 0.250. Compared with their swarms, they moved toward the outcome five times, away from it four times (the three Bank of Canada calls and the Opus agent on Iran) and stayed the same once.
Finding 4: an AI swarm on Claude Opus cost hundreds of times more for a 0.015 gain
The swarm on Opus and Opus asked directly differ by 0.015 on the mean. Toronto (52 vs 49), Likud (40 vs 38) and Iran (85.5 vs 84) went slightly the swarm's way, and the Bank of Canada was a tie at 45. Four markets cannot tell a gap that small apart from luck.
One swarm run on Opus cost $13.63 to $24.00 in API fees. That is 500 to 900 times one direct answer (about 2.5 cents) and about 100 to 170 times the five direct answers we scored.
Finding 5: Smart Money measures where the money sits
Part one said Smart Money is not a probability and that we would score it anyway, like every other source. The scores show why it is not one. Smart Money's dollar share on YES was 86.8% on Toronto and 93.7% on Iran, the best single scores in the table (0.017 and 0.004), and 96.5% on a Bank of Canada hold, the worst (0.931).
The dollar share is driven by a few large positions, so it reaches 90% and more when the biggest wallets agree. That shows where they stand. As an estimate of how likely the event is, it swings too far. By Brier score it is last in the table (0.363). By absolute error, which punishes one confident miss less, it is fifth (0.466). The share of smart wallets, which stays closer to 50% (25% to 67% on these markets), scored 0.275.
Skill or luck? A permutation test across both rounds
Mean Brier depends on how hard the markets were. A permutation test asks a narrower question: did a source put its high forecasts on the markets that resolved YES and its low ones on the markets that resolved NO, more than chance would? We keep each source's forecasts, reassign the outcomes to the markets in every possible order, and count how often the reshuffled score is at least as good as the real one. Lower means better than chance.
Round one (four markets in early October, cheaper models, three resolved so far) and this round give seven resolved markets for the Polymarket price and Smart Money, with one NO among them. The models changed between rounds, so they have four.
| Source | Markets | Share of reshuffles at least as good as the real result | Smallest possible |
|---|---|---|---|
| Polymarket price | 7 | 0.143 | 0.143 |
| Smart Money, wallets on YES | 7 | 0.429 | 0.143 |
| Smart Money, money on YES | 7 | 1.000 | 0.143 |
| Swarm on Opus 5.5 | 4 | 0.500 | 0.250 |
| Opus 5.5 asked directly | 4 | 0.500 | 0.250 |
The price reached the smallest possible share because its lowest forecast was on the only NO market. Any source that does that reaches it, so this does not show skill. Smart Money's dollar share got the worst possible result: its highest forecast was on that NO market. The models, on four markets, are at chance. permutation.csv has every source, and each new round adds rows to the same file.
What this doesn't show
Four markets, one of which decided most of the ranking, are not enough to rank forecasters. A different mix of markets could reorder the middle of the table.
Four markets can still show two things. A swarm on a frontier model stayed within 3 points of that model's own answer. And on the Bank of Canada, MiroFish's report agents moved 37 to 44 points away from their swarms toward the wrong outcome, which outweighed what they gained on the other markets.
Round one's last market, on British Columbia's next premier, has not resolved yet. We will add it to the scoreboard when it does.
The takeaway
The market price was the best forecast on these four markets, and the AI swarm was not measurably better than Opus asked directly. If you build on MiroFish, take the forecast from the agents' own answers and skip the report agent's number, which in our runs was written without them.
Smart Money on OrcaLayer is worth reading next to the price, as a map of where the money sits: the Smart Money leaderboard and each market's Smart Money consensus.
FAQ
Did the AI swarm on Claude Opus beat Polymarket? No. On four resolved markets the Polymarket price had a mean Brier score of 0.167. The best AI forecast, the MiroFish swarm on Claude Opus 5.5, scored 0.203, and Opus asked directly (median of five answers) scored 0.218. Lower is better. The price was also first by absolute error and by log score.
Did the swarm beat asking Claude Opus directly? Barely. The swarm scored 0.203 and Opus asked directly 0.218, a gap four markets cannot tell apart from luck. Their forecasts were 1.5 to 3 percentage points apart on three markets, and a swarm run cost 500 to 900 times as much as one direct answer.
How accurate were Claude Opus, Sonnet and Haiku at forecasting these Polymarket markets? Asked directly, Opus 5.5 had a mean Brier score of 0.218, Haiku 5.5 0.240 and Sonnet 5.5 0.253, against 0.250 for a 50% coin flip and 0.167 for the Polymarket price. None beat the price.
How did OrcaLayer Smart Money do? Smart Money's dollar share on YES scored 0.363, the worst Brier score in the table, because 96.5% of smart-wallet dollars sat on a Bank of Canada hold and the bank raised rates. By absolute error it ranked fifth. The share of smart wallets scored 0.275. Neither is a probability. Part one said so, and also said we would score them like every other source, so we did.
What is a Brier score? The squared distance between the forecast probability and what happened: (p โ outcome)ยฒ, with outcome 1 for YES and 0 for NO. 0 is perfect, 0.25 is what a 50% coin flip always gets, 1 is confidently wrong.