> For the complete documentation index, see [llms.txt](https://mc.hohoweiya.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mc.hohoweiya.xyz/gibbs/linear.md).

# Linear Regression

The variable selection problem has a natural Bayesian solution: Any collection of models having different sets of regressors can be computed via their Bayes factors.

## A semiconjugate prior distribution

Let $$\beta \sim MN(\boldsymbol\beta\_0, \Sigma\_0)$$, then

$$
\beta\mid \mathbf y,\mathbf X, \sigma^2 \sim N((\Sigma\_0^{-1}+\mathbf X^T\mathbf X/\sigma^2)^{-1}(\Sigma\_0^{-1}\boldsymbol\beta\_0+\mathbf X^T\mathbf y/\sigma^2), (\Sigma\_0^{-1}+\mathbf X^T\mathbf X/\sigma^2)^{-1})
$$

and let $$1/\sigma^2\sim Ga(\nu\_0/2,\nu\_0\sigma\_0^2/2)$$, then

$$
1/\sigma^2\mid \mathbf y,\mathbf X,\boldsymbol\beta \sim Ga(\[\nu\_0+n]/2, \[\nu\_0\sigma\_0^2+SSR(\boldsymbol \beta)]/2)
$$

Then we can construct the following Gibbs sampler:

![](/files/-LNYSZXGeWm4T8qThu7y)

## Weakly informative prior distributions

1. unit information prior
2. the parameter estimation should be invariant to changes in the scale of the regressors.

For the second case, we can derive a Monte Carlo approximation:

![](/files/-LNYSZXIPe2a37Zgo7fu)

since

$$
p(\sigma^2,\boldsymbol\beta\mid \mathbf y, \mathbf X)\propto p(\boldsymbol\beta\mid \mathbf y,\mathbf X,\sigma^2)p(\sigma^2\mid \mathbf y,\mathbf X)
$$