Everyone loves a hit rate. “78% accurate” sounds authoritative until you ask: accurate on what horizon? Against which benchmark? With what universe of forecasts included or excluded? Without definitions, track records become marketing. With definitions, they become a useful—but still imperfect—tool for comparing transparency and consistency.
Before accepting any score, translate the claim into a sentence a careful friend would agree with. For example: “When this source published a directional call on a large-cap U.S. equity, the price was higher 90 trading days later in 60% of cases, measured from the close of publication day.” If you cannot write that sentence, you do not yet understand the metric.
The same forecast can look brilliant at 30 days and wrong at 180. Good track records specify horizons (or a small, justified set of horizons) and stick to them. Switching horizons after the fact is one of the easiest ways to accidentally fool yourself—or others.
A source can be “often right” while still being economically useless if the calls are timid or late. Conversely, bold calls can look mediocre by hit rate but valuable by tail outcomes. This is why serious evaluation often pairs hit rates with magnitude metrics and drawdown awareness—not because retail investors need PhD statistics, but because honesty requires more than one number.
Meet fictional newsletter “Northstar Notes.” On twelve non-overlapping dates in 2024–2025 it published: “Large-cap Issuer ORBIT will outperform the sector ETF XLK over the next 90 trading days.” We score only that claim family. Horizon is fixed at 90 trading days from the publication-day close. Benchmark is ORBIT total return minus XLK total return over the same window. A hit means excess return > 0. Miss means excess return ≤ 0. Delisted names, if any, remain in the sample as misses or unresolved per a pre-written rule—here, none delisted.
Fictional outcomes after 90 trading days (excess vs XLK, percentage points): +4.2, −1.1, +2.0, +0.3, −3.5, +1.8, −0.4, +5.1, −2.2, +0.9, −1.7, +3.0. Count of hits (positive excess): 7. Count of misses: 5. Hit rate: 7/12 = 58.3%. Mean excess: about +0.7 percentage points. Median excess: about +0.6. Worst miss: −3.5. Best hit: +5.1.
What a careful notebook says: “Under a pre-registered 90-trading-day excess-vs-XLK rule, Northstar’s ORBIT-style calls hit 7 of 12 (58%). Sample is small; one or two outcomes flip the story. Absolute price-up hit rate is a different metric and is not reported here.” What a marketing blurb might wrongly say: “Nearly 60% accurate—proven edge!” The second sentence invents proof and erases the benchmark.
Failure-mode drill on the same numbers: (1) If we had switched to 30 days after seeing results, suppose nine of twelve would have been hits—that is horizon shopping. (2) If we dropped the two worst misses because “the thesis changed,” hit rate jumps to 7/10—that is silent exclusion. (3) If we mixed in eight crypto calls without segmentation, the blended score becomes uninterpretable. (4) If two calls were edited post-publication and we only scored final text, we must disclose the edit rule or we are not measuring the same object readers saw live.
Copy the template fields for your own journal: source, exact claim sentence, asset, direction, anchor timestamp, horizon (trading days), benchmark definition, hit/miss rule, inclusions/exclusions, sample size, hit rate, optional mean/median excess, and known failure modes. The personal forecast journal guide expands the template.
If you cannot tick these boxes, treat the number as undefined—not as secretly bullish.
Even a clean scoreboard is a partial view of forecasting quality. It may omit magnitude, path dependency, correlation across calls, costs, and changing market regimes. Public education about scoring reduces false certainty; it does not manufacture guaranteed foresight.
This guide is not investment advice and not an endorsement of any newsletter, including fictional ones used as examples. Soft product note: prefer methodology that stays visible so headline numbers remain traceable—optional dashboards never replace your own definitions.
Transparent research products—InsightMeter included—prefer visible rules: what was measured, when, and under what constraints. Summaries should be traceable. If a user cannot trace a headline number back to a definition, the product is asking for mistrust—and careful readers notice instantly.
Forecast scoring & backtests · Building a personal forecast journal · Backtest vs live forecast scoring pitfalls · Sample research walkthrough · Read a forecast scorecard without overclaiming · Forecast calibration: does stated confidence mean anything?
A forecast track record is only as honest as its definitions. Ask for the sentence-length claim, lock horizons, disclose exclusions, practice on worked examples, and treat metrics as a starting point for questions—not a finale. Nothing here is a recommendation to buy or sell any security.
Learn by topic: Form 4 / insider literacy · Form 8-K & fundamentals · Form D / exempt offerings · 13F / institutional snapshots