Nonconjugate Bayesian models often break the clean coordinate updates that make mean-field variational inference attractive. A common practical response is to use local Gaussian approximations and Taylor expansions inside a coordinate-ascent routine. This note uses the label “Laplace-delta” for that practical hybrid, but the terminology needs care. It is not the same as ordinary Laplace approximation, it is not identical to Laplace variational inference, and it is not expectation propagation.

The distinctions matter because each method optimizes or approximates a different object.

Name Basic object
Ordinary Laplace approximation A local Gaussian approximation to a posterior or integral near a mode.
Laplace variational inference A variational update that approximates an intractable coordinate by a local Gaussian form.
Delta-method variational inference Taylor approximations to expectations of nonlinear functions under a variational distribution.
Practical Laplace-delta hybrid A coordinate-ascent scheme that uses Laplace updates for nonconjugate blocks and delta approximations for moments needed elsewhere.
Expectation propagation A separate approximation framework based on local factor approximations and moment matching.

Calling a Taylor expansion “expectation propagation” is imprecise. EP has its own objective and update structure. A delta approximation may be used in a broader approximate-inference workflow, but it is not EP by itself.

Coordinate target

Suppose the parameter is partitioned as \(\theta=(\eta,\psi)\) and the variational approximation factorizes as

\[q(\eta,\psi)=q_\eta(\eta)q_\psi(\psi).\]

The unrestricted coordinate optimum for \(q_\eta\) satisfies

\[\log q_\eta^\star(\eta) = \mathbb E_{q_\psi}\{\log p(y,\eta,\psi)\} + \text{constant}.\]

Define

\[f(\eta)=\mathbb E_{q_\psi}\{\log p(y,\eta,\psi)\}.\]

If \(f\) is not in a recognizable conjugate form, a Laplace variational update approximates this coordinate by a Gaussian centered at the maximizer:

\[q_\eta(\eta)\approx N(\hat\eta,H^{-1}), \quad \hat\eta=\arg\max_\eta f(\eta), \quad H=-\nabla^2 f(\hat\eta).\]

This approximation is local. It works best when the coordinate target is close to Gaussian around a dominant mode and the negative Hessian is positive definite.

Delta approximations

Other coordinate updates may require expectations such as \(\mathbb E_q[g(\eta)]\). When exact Gaussian moments are unavailable, a second-order delta approximation gives

\[\mathbb E_q[g(\eta)] \approx g(m) + \frac{1}{2}\operatorname{tr}\{\nabla^2 g(m)V\},\]

where \(m=\mathbb E_q(\eta)\) and \(V=\operatorname{Var}_q(\eta)\). Exact moments should be used when they are available. The delta method is a local approximation and can be poor for strongly nonlinear functions, high variance, boundary behavior, or tail-sensitive quantities.

Logistic-regression example

For Bayesian logistic regression with prior \(\beta\sim N(0,\tau^2I)\) and outcomes \(y_i\in\{0,1\}\),

\[\log p(\beta\mid y) = \sum_i y_i x_i^\top\beta - \sum_i \log\{1+\exp(x_i^\top\beta)\} - \frac{1}{2\tau^2}\beta^\top\beta + \text{constant}.\]

The gradient and negative Hessian are

\[\nabla \ell(\beta)=X^\top(y-p)-\beta/\tau^2,\]

and

\[H(\beta)=X^\top W X + I/\tau^2,\]

where \(p_i=\operatorname{logit}^{-1}(x_i^\top\beta)\) and \(W_{ii}=p_i(1-p_i)\). A Laplace approximation uses the posterior mode and precision \(H(\hat\beta)\).

stable_logistic <- function(eta) {
  ifelse(eta >= 0, 1 / (1 + exp(-eta)), exp(eta) / (1 + exp(eta)))
}

laplace_logistic <- function(X, y, tau = 5, tol = 1e-8, maxit = 100) {
  X <- as.matrix(X)
  y <- as.numeric(y)
  p <- ncol(X)
  beta <- rep(0, p)

  logpost <- function(beta) {
    eta <- drop(X %*% beta)
    sum(y * eta - log1p(exp(eta))) - sum(beta^2) / (2 * tau^2)
  }

  for (iter in seq_len(maxit)) {
    eta <- drop(X %*% beta)
    prob <- stable_logistic(eta)
    grad <- crossprod(X, y - prob) - beta / tau^2
    w <- prob * (1 - prob)
    precision <- crossprod(X, X * w) + diag(1 / tau^2, p)
    step <- backsolve(chol(precision), forwardsolve(t(chol(precision)), grad))

    step_scale <- 1
    current <- logpost(beta)
    repeat {
      proposal <- beta + step_scale * drop(step)
      if (logpost(proposal) >= current || step_scale < 1e-6) break
      step_scale <- step_scale / 2
    }

    beta <- proposal
    if (max(abs(step_scale * step)) < tol) break
  }

  eta <- drop(X %*% beta)
  prob <- stable_logistic(eta)
  w <- prob * (1 - prob)
  precision <- crossprod(X, X * w) + diag(1 / tau^2, p)
  covariance <- chol2inv(chol(precision))
  gradient <- drop(crossprod(X, y - prob) - beta / tau^2)

  list(mode = beta, precision = precision, covariance = covariance,
       gradient = gradient, iterations = iter)
}

This is an ordinary Laplace approximation for one nonconjugate block. In a larger variational algorithm, the same calculation could be used as the coordinate update for a nonconjugate factor after taking expectations over the other factors.

A full coordinate-ascent algorithm would repeat this calculation with the other factors held at their current variational distributions, update the Gaussian approximation for the nonconjugate block, update any conjugate factors using the moments implied by that block, and continue until the evidence lower bound or a stable surrogate stops changing materially. The important point is that every approximation enters at a named location. The local Gaussian approximation is used for the coordinate target. The delta method is used only for expectations that are not available in closed form.

This separation also helps with debugging. If the local mode is unstable, the Laplace step is the problem. If the mode is stable but a downstream moment is inaccurate, the delta approximation may be the problem. If both local pieces look reasonable but uncertainty is too small, the mean-field factorization may be the problem.

The block below provides a small executable check. It simulates a logistic-regression dataset, computes the local Gaussian approximation, verifies that the gradient is small at the returned mode, and checks that the local precision matrix is positive definite.

set.seed(42)
n <- 120
X <- cbind(1, rnorm(n))
beta_true <- c(-0.4, 1.1)
prob <- stable_logistic(drop(X %*% beta_true))
y <- rbinom(n, size = 1, prob = prob)

fit <- laplace_logistic(X, y, tau = 5)
stopifnot(max(abs(fit$gradient)) < 1e-6)
stopifnot(min(eigen(fit$precision, symmetric = TRUE, only.values = TRUE)$values) > 0)

Diagnostics

A Laplace-delta routine should report more than the final mean and variance. Useful diagnostics include:

  • maximum absolute gradient at the mode;
  • positive definiteness of the local precision matrix;
  • number of Newton or quasi-Newton iterations;
  • line-search reductions;
  • sensitivity to initialization;
  • comparison with MCMC or direct quadrature on small cases;
  • simulation-based calibration when the method is used repeatedly.

The approximation can fail under multimodality, skewness, weak curvature, boundary modes, heavy tails, strong posterior dependence, or nonlinear moments evaluated far from the local mean. Mean-field factorization can also remove dependence that is essential for uncertainty in downstream summaries.

A useful implementation should therefore keep the approximation modest. It should start with a small example where direct optimization or MCMC is available, compare the mode and uncertainty summaries, and only then move to the larger model. The method is most defensible when the reported summaries are controlled by the local posterior mass near one mode. It is least defensible when the analysis depends on tails, rare events, or model components that are weakly identified.

What to report

When using this kind of approximation, report the factorization, the coordinate target, the local Gaussian construction, which expectations were approximated by the delta method, and which diagnostics were checked. Avoid presenting the method as exact posterior inference. It is a useful approximation when its local assumptions are reasonable and when the reported summaries are not dominated by features the approximation cannot capture.

References