LLM paper trading
Nine model families traded paper books in crypto and US stocks under hypotheses registered before the first trade. Each arm ran for a fixed period, from three weeks to two months. This is what the data shows.
The question
Most public demonstrations of language models trading are single runs, reported on gross returns, against whichever benchmark flatters them. We tried the opposite: a harness that counts every cost, including tokens priced at list rate for models we ran on subscriptions, that crashes rather than invent a decision when a model fails, and that commits its hypotheses to git before each arm's first trade.
Three questions were registered. Do benchmark scores predict trading results under identical inputs? Is a model trader net-profitable against doing nothing, after costs? And how do temperament and protocol compliance differ between model families given byte-identical prompts?
Four arms, four periods
The programme grew one arm at a time. A crypto fleet of 26 independent $500 books tested risk gates, web access, shorts and prompt framing. An equities fleet of 17 $10,000 books traded a pinned list of 20 US stocks. A crypto panel put the same frozen data in front of eight or nine models at each of 378 four-hour boundaries. investbench replicated an external multi-agent equity pipeline weekly.
Each arm is reported over the period it ran: the crypto fleet from 11 to 31 July, stocks from 27 July to 28 August, the panel from 31 July to 2 October, and investbench for four weekly runs in September. The hypotheses were registered for longer windows, so the results below are readings over these periods rather than scores against those registrations. investbench ran one replicate.
How a decision was made
At each wake the harness built a numbers-only context pack in code: moving averages, ATR, trailing returns, the last 20 candles, and for stocks fundamentals and an optional earnings calendar. One model received it with a fixed prompt and returned strict JSON: action, symbol, size, stop, thesis and confidence. A deterministic risk kernel could veto the order. Fills were simulated with spread, slippage and fees, and every wake, fill and veto went into an append-only ledger pinned to the experiment's config.
Stocks: 0 of 17 beat the benchmark
Over 32 days the equal-weight benchmark of the 20 stocks rose 7.82%. The 17 seats averaged −4.59% and none finished above the benchmark; two finished above starting capital. The fleet lost $7,808 on $170,000. Costs were not the reason: tokens came to $310 and exchange fees to $7.44 across 476 fills. Protocol compliance was near perfect, with 1,157 clean decisions in 1,164 wakes.
The loss was in entry timing. Five sessions after a model bought, the stock was down 1.27% on average; a random entry in the same stock over the same window was up 2.45%. Of 209 closed lots, 14 made money. Theses converged too: 89% cite the 20-day moving average.
Equal-weight total return of the 20 pinned stocks. Thin lines: each seat. f-gemini stops at its 15% drawdown kill on 19 August.
f- baseline, fe- with an earnings calendar, fb- with tickers blinded, g- with looser risk limits.
Crypto fleet: a lead that was variance
Measured from each arm's own start, the seats with web access, looser gates and a profit-directed prompt looked ahead. But the arms started on different days into a falling market. Over the five days when every seat was live, the gated arms averaged −0.17% and the freer arms +0.16%, a difference of t = 0.26. The freer arms differed in spread, about thirteen times wider, not in return; two seats carried the positive mean.
84% of wakes never reached a model. A deterministic triage step answered them, so a seat in cash called a model about once a day and tended to stay in cash. Activity was set by the harness more than by the models.
26 to 31 July. The vertical stroke is each group's mean; the freer arms' mean leaves out e-opus, added late and unpaired (with it, −0.44%).
Crypto panel: same pack, nine models
Every member answered the same frozen pack at every boundary, so differences between members are differences between models. The pairing held across all 378 boundaries. Members bought 51% (Opus) to 99% (Mercury) of the time, and at a boundary with two or more buyers, 62% of them on average picked the same coin.
Their buys beat the equal-weight menu by 1.35 points over seven days. That market rose 62% over the window, with ZEC tripling and NEAR close behind. The panel shows a shared disposition; it does not show skill.
A disposition to buy what had just risen was rewarded here. The same disposition lost money in July and in August stocks.
The one stable trait: buying after a rise
Across both asset classes, every harness and every model, entries followed a positive trailing move far more often than chance. In the crypto fleet 83.5% of buys did, against a base rate of 48.5%; in the panel 76% to 95% per member, against 53%; in stocks 61% against 50%.
Whether that paid depended on the market. It lost money in the falling July crypto window and in August stocks, which mean-reverted over five sessions, and earned money in the August to October crypto rally. It is the one finding a successor should test, with the random-timing null the panel registered and never ran.
The tick on each bar is how often a rise came before any symbol-boundary at all. Stocks: entries after a positive prior session, without look-ahead.
Base rate 53.0% for every member, since all of them saw the same data.
investbench: a corrected lead, still not skill
The live board paired each week's book with the benchmark from a week later. Aligned, the final lead is +$3,110, not the +$790 the board showed. That number still says little. Each weekly benchmark leg averaged only the 3 to 7 pool names that happened to be priced again, and the book sat about 19% invested for three of the four weeks. A mostly-cash book beat a falling subset of names, once.
Light: as the live board paired it, a week out. Dark: aligned. Share of the book invested: W37 21%, W38 20%, W39 16%, W40 82%.
What the audit found
Recomputing every figure from the archived ledgers changed some of them. The pack builder rounded every price to two decimals, so coins under a dollar reached the models as flat lines. What the project's record called a stale DOGE feed is this rounding. A published entry-timing statistic, 81.4% of stock entries on an up day, used the entry session's own close, which comes after the entry; without the look-ahead it is 61.4% against a base rate of 49.6%. The experiment's registered scorer counted the operator's halt as a drawdown kill.
The panel stored each pack only as a hash. The 14 packs that survive in model-session records match their hashes exactly. The other 364 cannot be rebuilt, because every pack contained a candle still forming at the boundary and the archive keeps only finished ones.
The ATR beside them, 1.96%, was computed from unrounded prices. Six of the panel's 20 coins were under a dollar.
What a successor needs
The infrastructure built to keep the experiment honest mostly worked: the config pin, the refusal to fabricate decisions, the pairing hash and the append-only ledgers are what made this audit possible. What failed sat where no guard looked, an off-by-one week, an unmeasured rounding step, a halt read as a kill.
The next step is fewer, longer arms, several replicates of each configuration, and the random-timing null computed as the data arrives, not after.
References
- PAPER.mdFull write-up, with every table, test and threat to validity
- analysis/paperScripts that recompute each number from the archive branches
- HYPOTHESES.mdRegistrations H1 to H11, appendices A to K
- METHODS.mdHarness, cost model and dated changelog