This article is a statement of how we think about evaluating a trading system, including our own. It is educational and general, not personalised investment advice, and not a recommendation to buy, sell, or hold any security.
A single trade tells you almost nothing. It could have been skill, and it could have been luck, and from one result you cannot tell the two apart. So the honest way to judge a system is not by asking what it did last, but by asking whether the way it decides is any good.
The short version
Process, not outcome: judge a trading system by its rules, its risk discipline, and its edge measured across many trades, not by any one result. A single outcome is dominated by luck, so it is nearly uninformative on its own. A good decision can have a bad outcome and a bad decision a good one. We publish the rules and the whole record precisely so the process, and not one lucky or unlucky call, is what you get to evaluate.
Process and outcome are not the same thing
The distinction is old and it is not ours. Decision theory has always separated the quality of a decision from the quality of its result, because at the moment you decide, the outcome has not happened yet. Annie Duke, drawing on her years as a poker player, gave the everyday version a name: “resulting,” the habit of grading a decision by how it turned out rather than by whether it was sound given what was knowable at the time. In any domain with real randomness, and markets are such a domain, the two come apart constantly.
A good decision can have a bad outcome. You can follow a rule with a genuine long-run edge, size the position sensibly, and still lose on that particular trade, because a single trade is one draw from a wide distribution. A bad decision can have a good outcome just as easily. You can buy on a whim, ignore every risk consideration, and get paid, because luck does not check your reasoning first. Neither result, taken alone, tells you which kind of decision it was. That is the whole problem with judging on outcomes: the outcome is exactly the piece of evidence that randomness corrupts.
Why one result is almost uninformative
The reason a single outcome carries so little information is variance. In trading, the spread of possible results around any given decision is enormous relative to the edge itself. A strategy with a real, positive expectancy still loses on a large share of its individual trades, and can string several losers together by chance alone. Run the same sound process a hundred times and the good decisions win out. Run it once and the coin can land anywhere.
This is why hit rate, the fraction of calls that worked, is such a treacherous headline. A system can be right well under half the time and still compound, if its winners are larger than its losers, and a system can be right most of the time and still bleed, if the occasional loss is big enough. We work through exactly that arithmetic in how accurate are stock signals. The point for now is narrower: because the edge is small next to the noise, you need many results before the signal separates from the luck. One call, however dramatic, is mostly noise. A short winning streak is mostly noise too. Sample size is not a technicality here; it is the difference between evidence and an anecdote.
A published process can be judged directly
Here is where a systematic approach has an advantage that a discretionary one structurally cannot match. If the process is a fixed set of rules, written down in advance, then you can evaluate the process itself, not just its results. You can ask whether the rules have a plausible reason to work, whether risk is controlled before return is chased, whether the edge holds up across many trades and across different market conditions rather than resting on a handful of lucky ones.
A discretionary hot streak offers none of that. When the decisions live in one person’s head and change from day to day, there is no stable process to inspect. All you can see is the outcomes, which is precisely the layer that luck contaminates. A run of good calls from a discretionary operator is indistinguishable, from the outside, from a run of good luck, because there is no fixed method to check the calls against. The results are all you are given, and the results are the least reliable thing to judge on. A written, repeatable process turns an unfalsifiable story into something you can actually examine.
How we try to make the process judgeable
This is the reasoning behind how Shishin is built and shown. The board is produced by a fixed model: four engines (Genbu, Suzaku, Byakko, and Seiryu), routed by a regime read, applied the same way every day. We describe how those pieces fit together, and how the whole thing grew out of a single trading bot, in from bot to system. The rules do not bend because a recent trade went badly, and they do not get more aggressive because a recent trade went well. That consistency is the point: it is what makes the process, rather than the mood of an operator, the thing on trial, which is the same reason we argue rules beat conviction.
And we show the whole record, not the flattering slice. Winners and losers sit side by side, with no email gate, so you are looking at the distribution the process actually produces and not a curated set of good outcomes. The live results are paper-traded in public and the daily signals and NAV are committed to a tamper-evident, timestamped log at the independent-verification page, so a claim cannot be quietly edited after the fact once an outcome is known. The five-year backtest is a hypothesis about the process, honestly a hypothesis and not a promise, with its own failure modes laid out in why backtests lie. None of this asks you to trust a result. It asks you to judge a method, across a full sample, and gives you the material to do it.
Judge the whole distribution, not the endpoint
Judging a process well also means refusing to reduce it to a single number, even a number computed over many trades. Return alone can be bought with risk you would never knowingly take; a good-looking average can hide a ruinous worst case. The honest reading looks at the shape of the whole distribution at once, return alongside risk-adjusted return alongside the depth of the worst loss, which is the argument we make in the trinity. A good process is one whose entire distribution of outcomes is acceptable, not one that happened to end on a high note. The endpoint is an outcome. The distribution is the process showing its work.
The honest limit
Everything above cuts against us too, and it should. A good process still loses, and it loses often. Judging by process rather than outcome does not mean the losses stop coming; it means an individual loss is not, by itself, evidence that the process is broken, just as an individual win is not evidence that it works. That symmetry is uncomfortable, because it removes the reassurance of a recent good result and the alarm of a recent bad one, and leaves you with the slower work of assessing a method over time.
We will not dress that up. Our own record is honest but young: a five-year backtest, which is a hypothesis, and a live public track that is still short, which is a small sample by the very standard this article argues for. A process being sound in principle does not make it right, and a rule can be wrong too. The most we claim is that we have tried to make the process visible and checkable, so that when you decide what to make of it, you are judging the thing that actually matters and not the coin flip of the last result. That is the standard we ask to be held to. It is the standard we try to hold ourselves to.
Sources & further reading
- Duke, A. (2018). Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts. Portfolio.
- How accurate are stock signals, on why hit rate misleads and expectancy is the honest measure.
- Why backtests lie, on treating a backtest as a hypothesis about the process, not a promise.
- The trinity, on judging the whole distribution rather than a single endpoint.
- Why rules beat conviction, on the consistency that lets a written process be judged in the first place.
- The public track record and the independent-verification page, where the process shows its full, unedited output.