A Performance Metric Isn’t a Fixed Number, It’s a Random Variable
A performance metric (return or risk metric) isn't a fixed number—it's a random variable. Stop comparing strategies as if it weren't.
This post builds on my earlier post, where I discussed comparing trading strategies from a distributional perspective. Here I show a simple implementation using Pyfolio Reloaded's bootstrap tools and seaborn. For readers interested in a more rigorous statistical treatment—modeling the full conditional distribution of the performance metric—I discuss the approach using GAMLSS in https://arxiv.org/abs/2606.31251
Using daily S&P 500 data (2024–2025), I applied this approach to two return streams:
- Buy & Hold;
- A strategy that goes long, flat, or short each day with equal (1/3) probability, with at most one open position and no transaction costs.
Rather than reporting one Sharpe ratio or one maximum drawdown, bootstrapping estimates the sampling distribution of each metric, making the uncertainty visible. Below are 1,000 bootstrapped Sharpe ratios and 1,000 bootstrapped maximum drawdowns for each strategy, shown as box plots. The line inside each box marks the median; wider boxes and longer whiskers mean more uncertainty in that metric.
Comparing medians (or means) here isn't the same as comparing two single-run numbers — each one already summarizes 1,000 resamples of that metric, a stable characteristic of the strategy's performance rather than one noisy realization. The overlap and spread of the two distributions show how much confidence to place in that ranking.
