Standard Error of Coeficient V/S Variance of Coefficient of Multiple Linear Regression V/S Var-Covar Matrix

0
votes

I am in the process of understanding standard errors of regression coeficient with my knowledge of definitions of each terms.

  • As I understand the definition of Standard Error is a measure of the statistical accuracy of an estimate, equal to the standard deviation of the theoretical distribution of a large population of such estimates.
  • Now standard deviation is square root of variance. So if we get variance of coefficients, we will get standard error. Now Variance of Coefficients as I understand is V[b], where b is a matrix of all the estimated coefficients, where X is the Dependent Variable matrix including X0=1.
  • But when I search for the equation for Var[b], I get a equation for Var[b] saying it is actually Var-Covariance matrix, and variance is found in diagnol of this matrix and standard error by taking square root of diagnol of this matrix.
  • That puzzles me as if diagnol is variance of coefficients, then why variance-covariance matrix is defined as V[b]? I assume somewhere I lost in understanding the terms properly. Any help here? I am a novice in stat. Please help me with details.
1
matrixregressionlinear-regressioncoefficientsstandard-error

1 Answers

0
votes

Found answer in this excellent material which actually solves lots of mathematical questions you might be having on Multiple Linear Regression and origin of assumptions: https://web.stanford.edu/~mrosenfe/soc_meth_proj3/matrix_OLS_NYU_notes.pdf Page 8. To answer my own question:

  • It might give wrong to represent Variance-Covariance Matrices as V[b] as it is not! It must be represented as variance Covariance Matrix of β^ as E[(βˆ − β)(βˆ − β)].

  • Now following formula will make sense. Variation of Residualsinverse(Transpose(X Matrix)%%X Matrix), where Variation of Residuals is defined as (Transpose of Residual Matrix%*%Residual Matrix)/(Number of Rows - Number of Columns).

  • This makes it clear that by definition of Variance Covariance matrices, the Diagnol of this Matrix defines Variance of each coefficient and Square root of the same as Standard Error, which is nothing but Standard Error.