Case study: the tangency portfolio from data

The sample

The returns are invented for illustration, not market data. They are 120 months drawn with a fixed seed from the model of case study 3.2.5: expected returns $\boldsymbol{\mu} = (0.06, 0.08, 0.11)$ a year and covariance matrix $\Sigma$. Case study 3.2.5 drew such a sample with seed 7; this one uses seed 13.

Because we invented the model, we can compare each estimate with the truth, which a desk never can. The sample means times 12 estimate the annual expected returns, and 12 times the sample covariance matrix estimates $\Sigma$ (case study 3.2.5):

The estimated correlations are 0.302, 0.008 and 0.469 for AB, AC and BC, against 0.3, 0.1 and 0.5. The volatilities are close; the means are not.

B had a poor decade, 2.49% a year against a true 8%, and that is not unusual. An annual mean estimated from $T$ years has a typical error, its standard error, of about $\frac{\sigma_i}{\sqrt{T}}$ (lesson 5.8.2 explains why).

With the true volatilities and ten years, that is 3.79, 5.69 and 7.91 percentage points: as large as the excess returns themselves. B's shortfall is only 0.97 standard errors.

The tangency portfolio from the sample

Put the estimates $\hat{\boldsymbol{\mu}}$ and $\hat{\Sigma}$ into lesson 4.6.2's formula, $\hat{\mathbf{w}} = \frac{1}{k}\hat{\Sigma}^{-1}(\hat{\boldsymbol{\mu}} - r_f\mathbf{1})$ with $r_f = 0.03$. The hat marks an estimate. First check that $k$, the sum of the entries of $\hat{\Sigma}^{-1}(\hat{\boldsymbol{\mu}} - r_f\mathbf{1})$, is positive: here $k = 4.7006$.

The weights are $\hat{\mathbf{w}} = (0.9911, -0.5234, 0.5323)$. B's poor decade earned it a short sale of 52%. Measured on the sample itself, the portfolio expects 13.83% with a volatility of 15.18%, a Sharpe ratio of 0.7136.

To confirm, scipy.optimize.minimize searches directly for the weights with the highest sample Sharpe ratio, adding up to 1. It minimises the negative of the ratio with SLSQP, the method case study 4.5.5 used for the frontier. It finds the same weights, to two decimal places, and the same 0.7136.

Units. The formula works in any period, as long as every input uses the same one. With monthly means, the monthly covariance matrix and $\frac{r_f}{12}$, the weights come out identical. Monthly means against the annual 3% give $k = -22.06$: a false "no tangency portfolio".

How much would another decade change it?

A desk has one sample, not the model. To see how much the answer depends on the particular months, it uses the bootstrap.

Draw 120 months at random from the 120 observed, with replacement, so some months appear twice and others not at all. Each draw is a decade that could have happened, built from months that did.

Repeat 1,000 times and recompute the tangency weights each time. In 49 of the resamples $k \le 0$: there is no tangency portfolio at all. For the other 951, the middle 90% of the weights runs over:

The weight in B could as easily be $-365\%$ as $+18\%$. The ranges are centred on this sample's estimate, not on the truth.

So the bootstrap measures how noisy the answer is, not where the true answer lies. Here each true weight is inside its range, near one end.

The means do the damage

Which estimate causes the trouble? Swap them one at a time.

• True means, sample covariance: the weights are $(0.5603, 0.0622, 0.3774)$, close to the true $(0.5185, 0.1601, 0.3214)$.
• Sample means, true covariance: $(1.0855, -0.6395, 0.554)$, as far off as before.

The covariance matrix is estimated well from 120 months; the means are not. And the formula $\Sigma^{-1}(\boldsymbol{\mu} - r_f\mathbf{1})$ turns small differences between means into large positions, more so when assets are correlated.

Raise B's estimated mean by one percentage point, well inside its standard error of 5.69 points. B's weight moves from $-52.34\%$ to $-44.11\%$.

The sample flatters itself

In the sample, $\hat{\mathbf{w}}$ always looks best: it was chosen to have the highest sample Sharpe ratio, noise included. Judged with the model's true $\boldsymbol{\mu}$ and $\Sigma$, it expects 7.61% with a volatility of 15.7%, a Sharpe ratio of 0.294. An equal split, $(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$, which needs no estimates at all, has 0.3844; the true best is 0.3951.

This sample is typical.

Over 1,000 fresh ten-year samples from the model, 138 have no tangency portfolio. For the rest, the median in-sample Sharpe ratio is 0.6424, while the median true Sharpe ratio of the estimated portfolio is 0.2935. In 93.2% of them the estimated portfolio does worse than the equal split.

So a Sharpe ratio measured on the data used to choose the weights overstates what the weights will earn. Judge a method on data it has not seen. Case study 5.8.5 asks when a Sharpe ratio is large enough to be more than luck.

Constraints and shrinkage

Two families of fixes tame the noise, and desks often use both.

Constraints. Forbid short sales, as case study 4.5.5 did for the frontier: give minimize bounds of 0 and 1 on each weight.

On this sample the long-only tangency portfolio is $(0.689, 0, 0.311)$. Its sample Sharpe ratio falls to 0.5855, but its true one rises to 0.3827, almost the equal split's. Caps on each weight work the same way.

Shrinkage. Pull each estimated mean part of the way towards the average of the three means, $\bar{\mu}$:

$\delta$ (delta), between 0 and 1, is how far to pull. With $\delta = 0.5$ the weights become $(0.8858, -0.254, 0.3682)$, with a true Sharpe ratio of 0.3353. With $\delta = 1$ every mean is the same, so the formula returns the sample minimum-variance portfolio $(0.7609, 0.0655, 0.1736)$, with a true 0.3604.

Shrinking gives up a little of what the sample says in exchange for much less noise. In the bootstrap, $\delta = 0.5$ shortens every range in the figure: B's runs from $-173\%$ to 14%.

Three more fixes are common. Desks shrink $\hat{\Sigma}$ towards a simpler matrix, start from the means CAPM implies (lesson 4.6.3), or average the weights over resamples.

On the desk

import numpy as np

_MU = np.array([0.06, 0.08, 0.11])
_SIGMA = np.array([0.12, 0.18, 0.25])
_R = np.array([[1.0, 0.3, 0.1], [0.3, 1.0, 0.5], [0.1, 0.5, 1.0]])
_COV = np.diag(_SIGMA) @ _R @ np.diag(_SIGMA)


def _sample(seed):
    return np.random.default_rng(seed).multivariate_normal(_MU / 12, _COV / 12, size=120, method="cholesky")


def _raises(f, *args, **kwargs):
    try:
        f(*args, **kwargs)
    except ValueError:
        return True
    return False


def test_sample_weights():
    """The seed-13 sample gives tangency weights of about (0.9911, -0.5234, 0.5323)"""
    w = sample_tangency(_sample(13), 0.03)
    expect_close(w, [0.9911295770835862, -0.5234319223429728, 0.5323023452593866], what="sample_tangency(X, 0.03)", rel=1e-8)


def test_weights_add_up():
    """The weights add up to 1"""
    expect_close(float(np.sum(sample_tangency(_sample(13), 0.03))), 1.0, what="sum of the weights")


def test_periods():
    """Monthly inputs with a monthly rate give the same weights as annualised ones"""
    X = _sample(13)
    expect_close(sample_tangency(X, 0.0025, periods=1), sample_tangency(X, 0.03, periods=12), what="weights from monthly inputs", rel=1e-8)


def test_shrink_half():
    """Shrinking the means halfway gives about (0.8858, -0.2540, 0.3682)"""
    w = sample_tangency(_sample(13), 0.03, shrink=0.5)
    expect_close(w, [0.8858113300671483, -0.2540155180408047, 0.36820418797365634], what="sample_tangency(X, 0.03, shrink=0.5)", rel=1e-8)


def test_full_shrink_is_min_variance():
    """Full shrinkage gives the sample minimum-variance portfolio"""
    X = _sample(13)
    S = np.cov(X, rowvar=False)
    a = np.linalg.solve(S, np.ones(3))
    expect_close(sample_tangency(X, 0.03, shrink=1.0), a / a.sum(), what="sample_tangency(X, 0.03, shrink=1)", rel=1e-8)


def test_no_tangency():
    """Case study 3.2.5's seed-7 sample has all means below 3%: ValueError"""
    assert _raises(sample_tangency, _sample(7), 0.03), "expected ValueError: k is negative, so there is no tangency portfolio"


def test_too_few_rows():
    """Three months of three assets are not enough: ValueError"""
    assert _raises(sample_tangency, _sample(13)[:3], 0.03), "expected ValueError for 3 rows and 3 columns"


def test_stale_price():
    """An asset whose returns are all 0 makes the covariance matrix singular: ValueError"""
    X = _sample(13).copy()
    X[:, 1] = 0.0
    assert _raises(sample_tangency, X, 0.03), "expected ValueError for a singular covariance matrix"


def test_nan():
    """A missing return (NaN) raises ValueError"""
    X = _sample(13).copy()
    X[5, 2] = np.nan
    assert _raises(sample_tangency, X, 0.03), "expected ValueError when X contains NaN"


def test_bad_shape_and_shrink():
    """A one-dimensional X, or shrink outside [0, 1], raises ValueError"""
    X = _sample(13)
    assert _raises(sample_tangency, X[:, 0], 0.03), "expected ValueError for a one-dimensional X"
    assert _raises(sample_tangency, X, 0.03, shrink=1.5), "expected ValueError for shrink = 1.5"
    assert _raises(sample_tangency, X, 0.03, shrink=-0.1), "expected ValueError for shrink = -0.1"


def test_bootstrap_counts():
    """1,000 resamples with seed 1: 951 tangency portfolios and 49 skipped"""
    weights, skipped = bootstrap_tangency(_sample(13), 0.03, n_boot=1000, seed=1)
    weights = np.asarray(weights)
    expect_equal(weights.shape, (951, 3), what="shape of weights")
    expect_equal(skipped, 49, what="skipped")


def test_bootstrap_median():
    """The median bootstrap weights are about (0.9871, -0.5164, 0.5428)"""
    weights, _ = bootstrap_tangency(_sample(13), 0.03, n_boot=1000, seed=1)
    expect_close(np.median(np.asarray(weights), axis=0), [0.9871146764558107, -0.5163783830856097, 0.5428365007967242], what="median weights", rel=1e-8)


def test_bootstrap_repeatable():
    """The same seed gives the same resamples; each row adds up to 1"""
    X = _sample(13)
    a, _ = bootstrap_tangency(X, 0.03, n_boot=50, seed=3)
    b, _ = bootstrap_tangency(X, 0.03, n_boot=50, seed=3)
    expect_close(np.asarray(a), np.asarray(b), what="two runs with seed 3")
    expect_close(np.asarray(a).sum(axis=1), np.ones(len(a)), what="row sums")


def test_shrinking_narrows():
    """Shrinking the means halfway narrows the spread of B's bootstrap weight"""
    X = _sample(13)
    plain, _ = bootstrap_tangency(X, 0.03, n_boot=1000, seed=1)
    shrunk, skipped = bootstrap_tangency(X, 0.03, n_boot=1000, seed=1, shrink=0.5)
    wide = np.ptp(np.percentile(np.asarray(plain)[:, 1], [5, 95]))
    narrow = np.ptp(np.percentile(np.asarray(shrunk)[:, 1], [5, 95]))
    assert narrow < wide, f"expected a narrower 90% range for B with shrink=0.5, got {narrow:.4f} against {wide:.4f}"
    expect_equal(skipped, 88, what="resamples skipped with shrink=0.5")