README.md

Advanced_Bayesian_Methods

Advanced Bayesian Methods (STA9105)

Each file has a theoretical and computational problem about advance bayesian methods. All analytical derivations are in pdf files and the computational works are contained in ipynb files.

1. Bayesian Linear regression

2. Hierarchical Linear models

2-1. Model outline

• The mixed linear model (=hierarchical linear model) with multidimensional random effects
  • Let $y_j \in \mathbb{R^{n^j}}$ and $X_j \in \mathbb{R^{n_j \times d}}$ be the observation vector and the design matrix for subject $j = 1, \dots, m$, respectively. Using subject-specific random effects $\beta_j \in \mathbb{R}^d$, we can write a mixed effect model as

2-2. Drive conditional posterior

• Full derivation is describe in 'Solution for analytical questions.pdf'

2-3. Gibbs sampling and posterior check

a. prior

b. full conditional distribution

c. Posterior predictive check

3. Generalized Linear models

3-1. Probit model outline

$$\begin{aligned} p(y|X, \beta, \pi) = \text{exp} \left( \sum_{i=1}^{n} L(y_i | \eta_i, \pi)\right) \end{aligned}$$

3-2. Normal Approximation

• The exponential family is easy to approximate to the normal distribution

• Likelihood is approximated as

  • $p(y | X, \beta, \phi) \propto \text{N}(\beta | \hat{\beta}, V)$, where
  • $\hat{\beta}$ is MLE, $V = \left[ X' \text{diag}(-L''(y_i | \hat{\eta_i})) X \right]^{-1}$
  • $L''(y_i | \hat{\eta_i}) = y_i \frac{(-(X_i\hat{\beta}) \phi(X_i\hat{\beta})\Phi(X_i\hat{\beta}) - \phi(X_i\hat{\beta})^2)}{\Phi(X_i\hat{\beta})^2} + (n_i - y_i) \frac{\phi(X_i\hat{\beta})^2 + (X_i\hat{\beta}) \phi(X_i\hat{\beta})(1 - \Phi(X_i\hat{\beta}))}{(1 - \Phi(X_i\hat{\beta}))^2} $

• Posterior is approximated as

3-3. Methods to obtain posterior distribution of $\beta$

a. Independent Metropolis-Hastings

b. Random walk Metropolis

c. Data augmentation