Monte-Carlo
  • Introduction
  • AIS
  • Generate R.V.
    • Special Distribution
    • Copulas
    • Minimum of Two Exponential
  • Gibbs
    • Comparing two groups
    • Linear Regression
    • Simulation of Exp-Abs-xy
  • Markov Chain
  • MC Approximation
  • MC Integration
    • Rao-Blackwellization
  • MC Optimization
  • MCMC
    • MCMC diagnostics
  • Metropolis-Hastings
    • Metropolis
    • Independent MH
    • Random Walk MH
    • ARMS MH
  • PBMCMC
  • RJMCMC
  • Diagnosing Convergence
  • SMCTC
  • Tempering
    • Parallel Tempering
  • Misc
    • R vs. Julia
  • References
Powered by GitBook
On this page
  • A semiconjugate prior distribution
  • Weakly informative prior distributions
  1. Gibbs

Linear Regression

PreviousComparing two groupsNextSimulation of Exp-Abs-xy

Last updated 6 years ago

The variable selection problem has a natural Bayesian solution: Any collection of models having different sets of regressors can be computed via their Bayes factors.

A semiconjugate prior distribution

Let β∼MN(β0,Σ0)\beta \sim MN(\boldsymbol\beta_0, \Sigma_0)β∼MN(β0​,Σ0​), then

β∣y,X,σ2∼N((Σ0−1+XTX/σ2)−1(Σ0−1β0+XTy/σ2),(Σ0−1+XTX/σ2)−1)\beta\mid \mathbf y,\mathbf X, \sigma^2 \sim N((\Sigma_0^{-1}+\mathbf X^T\mathbf X/\sigma^2)^{-1}(\Sigma_0^{-1}\boldsymbol\beta_0+\mathbf X^T\mathbf y/\sigma^2), (\Sigma_0^{-1}+\mathbf X^T\mathbf X/\sigma^2)^{-1})β∣y,X,σ2∼N((Σ0−1​+XTX/σ2)−1(Σ0−1​β0​+XTy/σ2),(Σ0−1​+XTX/σ2)−1)

and let 1/σ2∼Ga(ν0/2,ν0σ02/2)1/\sigma^2\sim Ga(\nu_0/2,\nu_0\sigma_0^2/2)1/σ2∼Ga(ν0​/2,ν0​σ02​/2), then

1/σ2∣y,X,β∼Ga([ν0+n]/2,[ν0σ02+SSR(β)]/2)1/\sigma^2\mid \mathbf y,\mathbf X,\boldsymbol\beta \sim Ga([\nu_0+n]/2, [\nu_0\sigma_0^2+SSR(\boldsymbol \beta)]/2)1/σ2∣y,X,β∼Ga([ν0​+n]/2,[ν0​σ02​+SSR(β)]/2)

Then we can construct the following Gibbs sampler:

Weakly informative prior distributions

  1. unit information prior

  2. the parameter estimation should be invariant to changes in the scale of the regressors.

For the second case, we can derive a Monte Carlo approximation:

since

p(σ2,β∣y,X)∝p(β∣y,X,σ2)p(σ2∣y,X)p(\sigma^2,\boldsymbol\beta\mid \mathbf y, \mathbf X)\propto p(\boldsymbol\beta\mid \mathbf y,\mathbf X,\sigma^2)p(\sigma^2\mid \mathbf y,\mathbf X)p(σ2,β∣y,X)∝p(β∣y,X,σ2)p(σ2∣y,X)