Repair a covariance matrix that is not positive definite

A risk manager stressed the desk's covariance matrix by hand. To model a sell-off, she raised the covariance of US equities and credit to a correlation of 0.95, and left every other entry alone.

Every variance is positive, and each pair of assets looks possible on its own. Yet the value-at-risk engine rejects the matrix, and the portfolio optimiser returns nonsense. The desk needs a diagnosis, a repair, and a choice between two repairs.

Your task

1. Write diagnose(cov, tol=1e-10): the first problem found, as one of the strings below, checked in this order. Write $s$ for tol times the largest entry of cov in size, a margin for rounding.
   • "not square"
   • "not symmetric": some entry differs from its mirror image $c_{ji}$ by more than $s$.
   • "negative variance": some diagonal entry is below $-s$.
   • "correlation outside [-1, 1]": some $\lvert c_{ij} \rvert$ exceeds $\sqrt{c_{ii}\,c_{jj}}$ by more than $s$.
   • "not positive semidefinite": the smallest eigenvalue is negative by more than tol times the largest eigenvalue in size.
   • "valid", when none of these applies.
2. Write worst_portfolio(cov): a pair (w, variance). Here w is the vector of length 1 that makes $\mathbf{w}^\top\Sigma\mathbf{w}$ smallest, with $\Sigma$ for cov, and variance is that value. A negative value is a position the risk engine cannot price.
3. Write nearest_psd(cov): the positive semidefinite matrix nearest to cov. Distance is the Frobenius norm, $\lVert A - B \rVert_F = \sqrt{\textstyle\sum_{i,j}(a_{ij} - b_{ij})^2}$, which is np.linalg.norm(A - B). For a symmetric matrix, the nearest one keeps the eigenvectors and sets each negative eigenvalue to 0.
4. Write repair_keep_variances(cov, floor=1e-3): a repair that keeps every variance. Turn cov into a correlation matrix and raise its eigenvalues below floor to floor. Rescale to 1 on the diagonal, then turn it back into a covariance matrix with the original volatilities.

Functions 2 to 4 raise ValueError unless cov is square and symmetric, with the margin of task 1. The fourth also raises it unless every variance is positive.

The data

COV holds the stressed annual covariance matrix of five asset classes, in the order of ASSETS. Before the edit, US equities and credit had a covariance of 0.00792, a correlation of 0.55.

Hint: Where to start

A covariance matrix must be positive semidefinite, because $\mathbf{w}^\top\Sigma\mathbf{w}$ is a variance (lesson 3.7.1). That lesson tests a symmetric matrix by its eigenvalues, with a tolerance for rounding. np.linalg.eigvalsh returns them smallest first.

Hint: The worst portfolio

On vectors of length 1, $\mathbf{w}^\top\Sigma\mathbf{w}$ runs from the smallest eigenvalue to the largest (lesson 3.7.1). The eigenvector of the smallest eigenvalue reaches the bottom; np.linalg.eigh gives it (lesson 3.6.3).

Hint: Two repairs

Lesson 3.6.3 writes $\Sigma$ as a sum of pieces $\lambda_i\mathbf{q}_i\mathbf{q}_i^\top$. Dropping the pieces with $\lambda_i < 0$ moves $\Sigma$ by the square root of the sum of their $\lambda_i^2$, the least any repair can. Case study 3.7.4 explains the floor and the rescaling, and why a floor of 0 leaves a singular matrix.