Case study: is this Sharpe ratio significant?

The data

The returns are invented for illustration, not market data. They are 60 monthly returns drawn with a fixed seed from a normal distribution with mean 0.75% and volatility $\frac{12\%}{\sqrt{12}} \approx 3.4641\%$ a month. The bill pays $r_f = 3\%$ a year, 0.25% a month, as in lesson 4.6.2.

So the model behind the data has an excess return of 0.5% a month, or 6% a year. Its annual volatility is 12% and its true Sharpe ratio 0.5. A desk never knows the truth; here, as in case study 4.6.5, each estimate can be checked against it.

The excess returns $x_t = R_t - r_f$ are what the Sharpe ratio is built from, with $R_t$ the strategy's return in month $t$. The 60 months include 20 losing months; the worst is $-5.94\%$ and the best $+7.86\%$. The mean excess return is 1.049% a month and the sample standard deviation $s$ is 3.539%: 12.59% and 12.26% a year.

The Sharpe ratio and its standard error

Lesson 4.6.2 defined the Sharpe ratio as the excess return per unit of volatility. Here it is written $\text{SR}$, and its estimate from data $\widehat{\text{SR}}$, so that $s$ stays the sample standard deviation.

Here $\widehat{\text{SR}}_{\text{month}} = 0.2964$, and the annual $\widehat{\text{SR}} = 1.0268$. The truth is 0.1443 a month and 0.5 a year: the estimate is twice the true value.

Is that unusual? It is a ratio of two estimates, so it has a standard error of its own.

With the sample's $\widehat{\text{SR}}_{\text{month}}$ in the formula, the monthly standard error is 0.1319 and the annual one 0.4569, near $\frac{1}{\sqrt{5}} \approx 0.4472$. A 95% interval for the annual Sharpe ratio is $1.0268 \pm 1.96 \times 0.4569$: from 0.1313 to 1.9224 (lesson 5.8.2). It holds the true 0.5, and it nearly reaches 0.

A simulation checks the formula. Over 100,000 seeded five-year samples from the model, the sample Sharpe ratio averages 0.5045. Its standard deviation is 0.4601, close to the formula's 0.4495 at the true 0.5.

In 12.65% of those samples it is 1.03 or more, and in 13.44% it is negative. So a strategy with a true Sharpe ratio of 0.5 shows 1.03 in about one five-year backtest in eight. It trails the bill in about one in seven.

A t test of no skill

"No skill" means a mean excess return of 0. That is lesson 5.8.3's t test with $\mu_0 = 0$, and its statistic is the Sharpe ratio in disguise.

Here $t = 1.0268 \times \sqrt{5} \approx 2.2961$ with 59 degrees of freedom, so the two-sided p-value is 0.0252. That is significant at the 5% level but not at 1%, where $t$ would need 2.6618. The one-sided p-value, for "the strategy beats the bill", is 0.0126.

A bootstrap interval

The formula assumes normal, independent months. The bootstrap of lesson 5.8.2 drops the first assumption: resample the 60 months with replacement, recompute the annual Sharpe ratio, and repeat 10,000 times.

The 2.5% and 97.5% quantiles give the interval 0.1423 to 2.0546, against the formula's 0.1313 to 1.9224. The bootstrap Sharpe ratios have standard deviation 0.4871, and 1.21% of them are 0 or below.

The interval leans a little to the right, but the two methods agree. The data rule out 0 at the 5% level only just, and they cannot tell 0.5 from 1.5.

How many years does a Sharpe ratio need?

With $t \approx \widehat{\text{SR}}\sqrt{T}$, a sample Sharpe ratio is significant at the 5% level once $\widehat{\text{SR}}\sqrt{T} > 1.96$, that is after $T = \left(\frac{1.96}{\widehat{\text{SR}}}\right)^2$ years. Lesson 5.8.3's sample-size formula, with $z_{0.8} = 0.8416$, gives the years for an 80% chance of detecting a true Sharpe ratio.

A Sharpe ratio of 0.5, respectable for a fund, needs 15.4 years of data to be told apart from 0. It needs 31.4 years to be found reliably. The small $\frac{\text{SR}^2}{2}$ term in the standard error changes this little: 15.5 years instead of 15.4.

The mean is the uncertain part

Case study 5.4.6 noted that a volatility can be estimated far better than a mean return. These five years show how much better.

• The mean. The annual mean excess return is 12.59%, with standard error $\frac{12.26\%}{\sqrt{5}} \approx 5.48$ percentage points: the $\frac{\sigma}{\sqrt{T}}$ of case study 4.6.5 and lesson 5.8.2. Its 95% t interval runs from 1.62% to 23.56%, around a true 6%.
• The volatility. The annual volatility is 12.26%. Lesson 5.8.2's chi-squared interval runs from 10.39% to 14.95%, around a true 12%.

The mean's standard error is 44% of the estimate itself; the volatility's is about 9%, roughly $\frac{1}{\sqrt{2(n - 1)}}$ for normal data. So the uncertainty of a Sharpe ratio is almost all in its numerator.

More frequent data helps only the volatility. With 1,260 daily returns over the same five years, the volatility's relative error falls to about 2%. The mean's standard error stays $\frac{12\%}{\sqrt{5}} \approx 5.37$ points: only more years shrink it.

Twenty strategies tried

The strategy was the best of 20 ideas backtested on the same five years. Suppose none of them has any skill: lesson 5.8.3 warned what happens.

In 5,000 seeded batches of 20 independent strategies with no skill, the best Sharpe ratio has median 0.83. It reaches the backtest's 1.03 in 21.18% of batches, against 1.22% for a single strategy. In 64% of batches at least one of the 20 is significant at the 5% level.

The exact version agrees. One strategy with no skill reaches $t \ge 2.2961$ with probability 0.0126, the one-sided p-value, so the best of 20 does with probability $1 - (1 - 0.0126)^{20} \approx 0.2243$.

A Bonferroni correction for 20 tries asks each for $p \le 0.0025$, two-sided: $t \ge 3.1585$, an annual Sharpe ratio of 1.41 over five years. The backtest's 1.03 does not pass.

Desks therefore record how many strategies were tried, and keep data aside to test the winner on, as case study 4.6.5 advised. Adjusted measures such as the "deflated Sharpe ratio" build the count in.

import numpy as np
from scipy.stats import ttest_1samp

R = np.random.default_rng(598).normal(0.0075, 0.12 / np.sqrt(12), 60)
RF = 0.03 / 12
DAILY = np.random.default_rng(252).normal(0.0004, 0.01, 756)


def _raises(f, *args, **kwargs):
    try:
        f(*args, **kwargs)
    except ValueError:
        return True
    return False


def _stats(*args, **kwargs):
    got = sharpe_stats(*args, **kwargs)
    assert got is not None, "sharpe_stats returned None (did you forget to return?)"
    return got


def test_case_study():
    """The case study's 60 months: Sharpe ratio about 1.0268, standard error 0.4569, t 2.2961, p 0.0252"""
    got = _stats(R, RF)
    if abs(got[0] - 1.0355134955) < 1e-6:
        raise AssertionError("got a Sharpe ratio of 1.0355: that divides by n. Use the sample standard deviation with ddof=1")
    if abs(got[0] - 0.2964254726) < 1e-6:
        raise AssertionError("got 0.2964: that is the monthly Sharpe ratio. Multiply by the square root of periods")
    expect_close(got, (1.026847958263304, 0.45693194906708584, 2.296101837233615, 0.02523944563138756), what="sharpe_stats(r, rf)", rel=1e-9)


def test_risk_free_rate():
    """Without subtracting the bill the ratio is about 1.2716: rf matters"""
    expect_close(_stats(R)[0], 1.2715658855220144, what="sharpe_stats(r) with rf = 0", rel=1e-9)


def test_matches_t_test():
    """The t statistic and p-value equal SciPy's one-sample t test of a zero mean excess return"""
    res = ttest_1samp(R - RF, 0)
    got = _stats(R, RF)
    expect_close(got[2:], (float(res.statistic), float(res.pvalue)), what="t_stat and p_value", rel=1e-9)


def test_losing_strategy():
    """A losing strategy has a negative Sharpe ratio and t, and a two-sided p-value of about 0.00125"""
    expect_close(_stats(-R, RF), (-1.5162838127807248, 0.4681445343133967, -3.390513678560265, 0.0012496623447016545), what="sharpe_stats(-r, rf)", rel=1e-9)


def test_daily():
    """Three years of daily returns, periods = 252: Sharpe ratio about 0.1141, standard error about 0.5774"""
    expect_close(_stats(DAILY, 0.03 / 252, 252), (0.11414190728482858, 0.577357731379311, 0.19769958269013926, 0.8433333305780878), what="sharpe_stats(daily, rf, 252)", rel=1e-9)


def test_bootstrap_case_study():
    """With seed 5852 the bootstrap interval is about 0.1423 to 2.0546, as in the case study"""
    got = bootstrap_interval(R, RF, seed=5852)
    assert got is not None, "bootstrap_interval returned None (did you forget to return?)"
    expect_close(got, (0.14227536992923157, 2.054623742026796), what="bootstrap_interval(r, rf, seed=5852)", rel=1e-9)


def test_bootstrap_defaults():
    """With the default seed 0 the interval is about 0.1551 to 2.0358"""
    expect_close(bootstrap_interval(R, RF), (0.15505232661625515, 2.0358276113556757), what="bootstrap_interval(r, rf)", rel=1e-9)


def test_bootstrap_level():
    """A 90% interval with seed 1 is about 0.2917 to 1.8604, narrower than the 95% one"""
    expect_close(bootstrap_interval(R, RF, level=0.9, seed=1), (0.29172254684002374, 1.8603905606518771), what="bootstrap_interval(r, rf, level=0.9, seed=1)", rel=1e-9)


def test_bootstrap_daily():
    """Daily returns, periods = 252, 2,000 resamples with seed 3: about −0.9979 to 1.2302"""
    got = bootstrap_interval(DAILY, 0.03 / 252, 252, n_boot=2000, seed=3)
    expect_close(got, (-0.9978506995692057, 1.2302088480070756), what="bootstrap_interval(daily, rf, 252, n_boot=2000, seed=3)", rel=1e-9)


def test_bad_inputs():
    """Fewer than 3 returns, no volatility, or periods of 0 raise ValueError in both functions"""
    for f in (sharpe_stats, bootstrap_interval):
        assert _raises(f, [0.01, 0.02]), f"{f.__name__} with 2 returns should raise ValueError"
        assert _raises(f, [0.01] * 12), f"{f.__name__} with every return 0.01 should raise ValueError"
        assert _raises(f, R, RF, 0), f"{f.__name__} with periods = 0 should raise ValueError"


def test_bad_level():
    """A level of 0, 1 or 1.5 raises ValueError in bootstrap_interval"""
    for level in (0, 1, 1.5):
        assert _raises(bootstrap_interval, R, RF, level=level), f"bootstrap_interval(level={level}) should raise ValueError"