Antonio De Leon

Variational Bayes for Non-Conjugate Models: A Laplace-Delta Tutorial

2025-03-20T00:00:00+00:00

Introduction

This tutorial presents a comprehensive guide to the Variational Bayes with Laplace-Delta (VB-LD) method, a framework for approximate Bayesian inference in models containing non-conjugate parameter relationships. VB-LD addresses the challenge of efficiently approximating posterior distributions when conditional conjugacy is absent without resorting to computationally intensive or unstable alternatives, such as Importance Sampling.

Methodology Overview

The VB-LD approach combines two techniques:

Laplace Approximation for Non-Conjugate Updates:
- Provides a Gaussian approximation to intractable variational distributions by matching the mode and curvature of the log-posterior
- Particularly effective when the true posterior is unimodal and approximately Gaussian near its peak
- Avoids sampling-based approaches that may suffer from high variance or particle degeneracy
Delta Method for Expectation Propagation:
- Enables closed-form approximation of necessary moments through second-order Taylor expansion
- Crucial for maintaining the mean-field factorization when updating conjugate parameters

1. Preliminaries

1.1 Mean-Field Variational Bayes (MFVB)

Consider the observed data $ D \in \mathcal{D} $ and a parameter vector $ \boldsymbol{\theta} = (\boldsymbol{\eta}, \boldsymbol{\psi}) \in \Theta $ where:

$ \boldsymbol{\eta} $: non-conjugate parameters
$ \boldsymbol{\psi} $: conjugate parameters

MFVB posits a structured variational family $ \mathcal{Q} $ of distributions that factorize as:

\[\mathcal{Q} = \{ q : q(\boldsymbol{\theta}) = q_{\boldsymbol{\eta}}(\boldsymbol{\eta})q_{\boldsymbol{\psi}}(\boldsymbol{\psi}), q_{\boldsymbol{\eta}} \in \mathcal{Q}_{\boldsymbol{\eta}}, q_{\boldsymbol{\psi}} \in \mathcal{Q}_{\boldsymbol{\psi}} \}\]

The optimal approximation $ q^* \in \mathcal{Q} $ minimizes the Kullback-Leibler divergence:

\[q^* = \text{arg} min_{q \in \mathcal{Q}} \text{KL}(q\|p) = \text{arg} min_{q \in \mathcal{Q}} \int q(\boldsymbol{\theta}) \log \frac{q(\boldsymbol{\theta})}{p(\boldsymbol{\theta}|D)} d\boldsymbol{\theta}\]

Equivalently, we maximize the Evidence Lower Bound (ELBO):

\[ \text{ELBO}(q) = \mathbb{E}_q[\log p(D, \boldsymbol{\theta})] - \mathbb{E}_q[\log q(\boldsymbol{\theta})] \]

where:

\[ \mathbb{E}_q[\log p(D, \boldsymbol{\theta})] := \text{Expected complete-data log-likelihood} \]

\[ \mathbb{E}_q[\log q(\boldsymbol{\theta})] := \text{Entropy of } q \]

Optimality Conditions

The stationary points of the ELBO satisfy:

$$ \log q^*(\boldsymbol{\eta}) = \mathbb{E}_{q^*(\boldsymbol{\psi})}[\log p(D, \boldsymbol{\eta}, \boldsymbol{\psi})] + \text{const.} $$ $$ \log q^*(\boldsymbol{\psi}) = \mathbb{E}_{q^*(\boldsymbol{\eta})}[\log p(D, \boldsymbol{\eta}, \boldsymbol{\psi})] + \text{const.} $$

1.2 Coordinate Ascent Variational Inference (CAVI)

The CAVI algorithm iteratively optimizes each variational factor while holding others fixed:

Algorithm 1: CAVI for MFVB

Initialize $ q^{(0)}(\boldsymbol{\eta}) $, $ q^{(0)}(\boldsymbol{\psi}) $ by their setting hyperparameters
While stopping criteria not met:
1. Update non-conjugate parameters: $$ q^{(k)}(\boldsymbol{\eta}) \propto \exp\left(\mathbb{E}_{q^{(k-1)}(\boldsymbol{\psi})}[\log p(D, \boldsymbol{\eta}, \boldsymbol{\psi})]\right) $$
2. Update conjugate parameters: $$ q^{(k)}(\boldsymbol{\psi}) \propto \exp\left(\mathbb{E}_{q^{(k)}(\boldsymbol{\eta})}[\log p(D, \boldsymbol{\eta}, \boldsymbol{\psi})]\right) $$

1.3 Non-Conjugate Challenges

When $ \boldsymbol{\eta} $ is non-conjugate, the update:

\[ q(\boldsymbol{\eta}) \propto \exp\left(\mathbb{E}_{q(\boldsymbol{\psi})}[\log p(D, \boldsymbol{\eta}, \boldsymbol{\psi})]\right) \equiv \exp(f(\boldsymbol{\eta})) \]

faces three fundamental difficulties:

The normalizing constant $ Z = \int \exp(f(\boldsymbol{\eta})) d\boldsymbol{\eta} $ is typically intractable
Required expectations $ \mathbb{E}_{q(\boldsymbol{\eta})}[g(\boldsymbol{\eta})] $ lack closed forms
The ELBO cannot be evaluated exactly

Some Solutions

Importance Sampling... but suffers from high variance in high dimensions and is prone to particle collapse
Quadrature Methods... but exponential complexity in $ k = \dim(\boldsymbol{\eta}) $
Conjugate Modifications... but restrictive modeling assumptions

2. VB-LD Algorithm

2.1 Laplace Approximation Step

Given the variational distribution:

\[ q(\boldsymbol{\eta}) \propto \exp\left(\mathbb{E}_{q(\boldsymbol{\psi})}[\log p(D,\boldsymbol{\eta},\boldsymbol{\psi})]\right) \equiv \exp(f(\boldsymbol{\eta})) \]

Mode Finding:
- Compute gradient $ \nabla f(\boldsymbol{\eta}) $
- Solve $ \nabla f(\hat{\boldsymbol{\eta}}) = 0 $ via Newton-Raphson: \[ \boldsymbol{\eta}^{(k+1)} = \boldsymbol{\eta}^{(k)} - [\nabla^2 f(\boldsymbol{\eta}^{(k)})]^{-1}\nabla f(\boldsymbol{\eta}^{(k)}) \]
- Terminate when $ |\boldsymbol{\eta}^{(k+1)} - \boldsymbol{\eta}^{(k)}| < \epsilon $
Covariance Approximation: \[ \Sigma = -\left[\nabla^2 f(\hat{\boldsymbol{\eta}})\right]^{-1} \] where the Hessian is computed via: \[ [\nabla^2 f(\boldsymbol{\eta})]_{ij} = \frac{\partial^2 f}{\partial \eta_i \partial \eta_j} \]

Laplacian approximation

$$ q(\boldsymbol{\eta}) \approx \mathcal{N}(\hat{\boldsymbol{\eta}}, \Sigma) $$

2.2 Delta Method for Expectation Approximation

When updating $ q(\boldsymbol{\psi}) $, we are required to compute expectations of the form:

\[ \mathbb{E}_{q(\boldsymbol{\eta})}[g(\boldsymbol{\eta})] \]

To do so, we approximate such quantities as follows,

Delta method (second-order Taylor approximation)

$$ \mathbb{E}[g(\boldsymbol{\eta})] \approx g(\hat{\boldsymbol{\eta}}) + \frac{1}{2}\text{tr}\left[\nabla^2 g(\hat{\boldsymbol{\eta}})\Sigma\right] $$

2.3 Complete VB-LD Algorithm

Algorithm 2: VB-LD Implementation

Initialize $ q^{(0)}(\boldsymbol{\psi}) $
While stopping criteria not met:
1. Laplace Step:
  - Compute $ \hat{\boldsymbol{\eta}}^{(k)} = \text{arg} max f(\boldsymbol{\eta}) $
  - Evaluate $ \Sigma^{(k)} = -[\nabla^2 f(\hat{\boldsymbol{\eta}}^{(k)})]^{-1} $
  - Non-Conjugate Update: $ q^{(k)}(\boldsymbol{\eta}) = \mathcal{N}(\hat{\boldsymbol{\eta}}^{(k)}, \Sigma^{(k)}) $
2. Delta Step:
  - For all required $ g $, compute: $\mathbb{E}[g(\boldsymbol{\eta})] \approx g(\hat{\boldsymbol{\eta}}^{(k)}) + \frac{1}{2} \text{tr}[ \nabla^2 g(\hat{\boldsymbol{\eta}}^{(k)}) \Sigma^{(k)} ] $
  - Conjugate Update: Use the previous quantity to update $ q^{(k)}(\boldsymbol{\psi}) $

Statistical Computing: A tutorial for UCSC Graduate Students

2025-02-16T00:00:00+00:00

Hi, grads! Whether you’re running Bayesian models, tuning machine learning algorithms, or analyzing massive datasets, this guide aims to help you take advantage of the full potential of UCSC’s Hummingbird (HB) clusters.

What Are the Hummingbird Clusters?

The HB clusters are UCSC’s shared high-performance computing (HPC) systems, built to handle tasks that would overwhelm personal computers. Designed for researchers across disciplines, they offer:

Parallel computing: Run thousands of simulations or MCMC chains simultaneously
High-speed storage: Process TB-scale datasets (e.g., genomics, climate models)
Specialized hardware: Access GPU nodes for deep learning
Scalability: Transition seamlessly from small interactive tests to large batch jobs

Who Should Use HB?

These clusters are ideal for statistical work requiring:

Computational intensity: Hierarchical models, bootstrapping, or optimization
Reproducibility: Version-controlled environments for collaborative projects
Speed: Accelerate workflows that take days on a laptop

What’s in This Guide?

You’ll learn to:

Test code interactively (e.g., debug an RStan model)
Submit batch jobs (e.g., parallelize 100 MCMC chains)
Optimize resources (avoid memory crashes, leverage GPUs)
Manage workflows (from data storage to result analysis)

Introduction to Cluster Computing for Stats Grads

Brief Glossary:

Node: Dedicated server (128 cores/256GB RAM typical) - Think of it as a powerful workstation
Core: Individual processing unit (Like a CPU thread) - Your basic computation unit
GPU Node: Specialized nodes with 4 NVIDIA A100 GPUs (80GB VRAM each) for deep learning
Scratch Space: 1PB high-speed temporary storage (Auto-cleaned every 14 days) - Perfect for intermediate results

I. Interactive Development Sessions

When to Use Interactive:

Debugging code
Exploratory analysis
Small simulations
Model prototyping
Visualization

Example 1: Debugging Bayesian Models in R

# Request interactive resources: 4 cores, 8GB RAM for 2 hours
srun --pty --mem=8G --cpus-per-task=4 --time=02:00:00 bash

# Load R environment with Bayesian stack
module load R/4.3.0

# Start R session with debugging capabilities
R
> library(rstan)          # Load STAN interface
> debug(fit_model)        # Set breakpoint in function
> source("hierarchical_bayes.R")  # Run script until breakpoint
> where                  # Show call stack when breakpoint hits

Example 2: Interactive ML Development with Jupyter

# Request heavier resources for data e ploration: 8 cores, 16GB RAM
srun --pty --mem=16G --cpus-per-task=8 --time=04:00:00 bash

# Load Python environment
module load python/3.11

# Start Jupyter Lab on cluster (no local browser)
python -m jupyter lab --no-browser --port=8889

# On your local machine, create SSH tunnel:
ssh -L 8889:localhost:8889 cruzid@hb.ucsc.edu
# Now access via http://localhost:8889 in local browser

II. Batch Processing for Production Workloads

When to Use Batch Processing:

Long-running computations (>4 hours)
Parameter sweeps/optimization runs
Production model training/inference
Final analyses requiring full resources
Reproducible pipeline executions

Example 1: Large-Scale Bayesian Inference

#!/bin/bash
#SBATCH --job-name=stan_meta          # Job identifier
#SBATCH --output=mcmc_%A_%a.log       # Log file template (JobID_ArrayID)
#SBATCH --array=1-100                 # Parallelize 100 independent chains
#SBATCH --cpus-per-task=4             # 4 cores per chain (for within-chain parallel)
#SBATCH --mem=16G                     # 16GB RAM per chain
#SBATCH --time=24:00:00               # 24hr ma  runtime

# Load environment
module load R/4.3.0

# Run STAN model with chain-specific data
Rscript run_stan.R --model hierarchical \
                  --data ${SLURM_ARRAY_TASK_ID} \  
                  --iter 5000

Example 2: Distributed ML Training

#!/bin/bash
#SBATCH --job-name= gb_ensemble       # Job name
#SBATCH --nodes=2                     # Use 2 physical servers
#SBATCH --ntasks-per-node=16          # 32 total tasks (16 per node)
#SBATCH --mem=128G                    # 128GB total RAM (64GB/node)
#SBATCH --time=48:00:00               # 2-day ma  runtime
#SBATCH --gres=gpu:2                  # Request 2 GPUs per node

# Load ML environment
module load python/3.11

# Train  GBoost ensemble with cross-validation
python train_ensemble.py --n-estimators 1000 \  # 1000 trees
                        --depth-range 3-10 \   # Search depth 3-10
                        --gpu                  # Enable GPU acceleration

Reproducible Environment Setup

Statistical Computing Environments

# R: Create project-specific environment with renv
module load R/4.3.0
R -e "renv::init()"            # Initialize project
R -e "renv::install('brms')"   # Install Bayesian regression models

# Python: Lock dependencies with conda-lock
module load miniconda3
conda create -n stats_proj python=3.11  # New environment
conda install -n stats_proj numpy pandas scikit-learn  # Core stack
conda-lock lock --file environment.yml --platform linu -64  # Create reproducible lockfile

Big Data Best Practices

Chunked Processing: Use dask.dataframe.read_csv(chunksize=1e6) for memory-efficient ETL
Memory Mapping: numpy.memmap('large_array.npy') for out-of-core 100GB+ arrays
Columnar Storage: pd.read_parquet('data.parquet') for fast I/O of structured data

Pro Tip: Always test workflows interactively before submitting batch jobs!

Validate data loading in small sessions
Profile memory usage with RStudio/python -m memory_profiler
Test single array job element before full submission

Performance Optimization Guide

Understanding Cluster Resources

All compute happens on remote servers - your laptop just submits jobs
Storage paths like /hb/home are network-mounted - accessible from all nodes
Always test scripts with small resources first!

Memory Management Essentials

# For R: Profile memory usage with Valgrind
# This creates detailed memory usage reports
module load R/4.3.0
R -d "valgrind --tool=massif" -f bayesian_analysis.R
# After running, analyze with:
ms_print massif.out.* > memory_report.txt

# For Python: Track memory allocation
# First install memory profiler in your environment
pip install memray
# Run profiling and generate report
python -m memray run -o profile.bin ml_pipeline.py
python -m memray stats --json profile.bin > memory_stats.json

Why Memory Matters:

Jobs exceeding requested memory get automatically killed
Use 10-20% less than node maximums for safety
Monitor memory during runs with seff JOBID

Parallel Computing Patterns Explained

Method	SLURM Directives	When to Use	Example Command
Embarrassing Parallel	--array=1-100	Independent tasks (Bootstrap/permutation tests)	`sbatch --array=1-100 job.sh`
MPI (Message Passing)	--nodes=4 --ntasks-per-node=16	Inter-process communication (Gibbs sampling)	`mpirun -np 64 ./model`
Multithreading	--cpus-per-task=32	Shared-memory tasks (XGBoost/CV tuning)	`export OMP_NUM_THREADS=32`

Key Concept: Always match parallel method to your algorithm:

Embarrassing Parallel: No data sharing between tasks
MPI: Needs data exchange between processes
Multithreading: Single process with multiple threads

Getting Help with HPC

Cluster Support Channels

Email Support: hummmingbird@ucsc.edu
Slack Channel: Link
Documentation at https://hummingbird.ucsc.edu/docs

# Check job efficiency - run this while job is active
seff $(squeue -u $USER -h -o %i)
# Look for:
# - CPU Utilization: Should be >90% for good efficiency
# - Memory Usage: Should be < requested amount

Avoid These Common Mistakes

Memory Overallocation:
Bad: --mem=256G (max is 256GB/node)
Good: --mem=230G (leave 10% margin)
Ignoring Error Logs:
Always check slurm-JOBID.err after failures
Local Installs:
Never use pip install --user - it can break cluster environments

The Energy Score for Multivariate Normal Distributions

2025-02-12T00:00:00+00:00

Introduction

The Energy Score serves as the multivariate extension of the Continuous Ranked Probability Score (CRPS), maintaining its desirable properties while handling vector-valued forecasts. This post presents the closed-form expression for multivariate normal distributions and its connection to classical CRPS results.

Insight:

The Energy Score generalizes CRPS through expectations of pairwise distances, avoiding direct integration over multivariate CDFs

Special function relationships enable closed-form solutions for normal distributions

Energy Score: Definition

For a $d$-dimensional forecast distribution $P$ and observation vector $\mathbf{y} \in \mathbb{R}^d$:

\[\text{ES}(P, \mathbf{y}) = \mathbb{E}\left[\|\mathbf{X} - \mathbf{y}\|\right] - \frac{1}{2} \mathbb{E}\left[\|\mathbf{X} - \mathbf{X'}\|\right]\]

where $\mathbf{X}, \mathbf{X’} \sim P$ are independent random vectors.

The Multivariate Normal Case

For $P = \mathcal{N}_d(\boldsymbol{\mu}, \mathbf{\Sigma})$ and observation $\mathbf{y}$:

\[\text{ES}(P, \mathbf{y}) = \sqrt{2\text{tr}(\mathbf{\Sigma})} \cdot \frac{\Gamma\left(\frac{d+1}{2}\right)}{\Gamma\left(\frac{d}{2}\right)} \left[ {}_1F_1\left(-\frac{1}{2}; \frac{d}{2}; -\frac{\delta^2}{2}\right) - 1 \right]\]

Components:

$\delta^2 = (\boldsymbol{\mu} - \mathbf{y})^\top \mathbf{\Sigma}^{-1} (\boldsymbol{\mu} - \mathbf{y})$ (Mahalanobis distance squared)
${}_1F_1$: Confluent hypergeometric function
$\Gamma$: Gamma function

Practical Note:

Requires special function implementations (e.g., scipy.special.hyp1f1)

Reduces to familiar CRPS when $d=1$

Special Cases

Isotropic Covariance

When $\mathbf{\Sigma} = \sigma^2\mathbf{I}_d$:

\[\text{ES}(P, \mathbf{y}) = \sqrt{2d}\sigma \cdot \frac{\Gamma\left(\frac{d+1}{2}\right)}{\Gamma\left(\frac{d}{2}\right)} \left[ {}_1F_1\left(-\frac{1}{2}; \frac{d}{2}; -\frac{\|\boldsymbol{\mu} - \mathbf{y}\|^2}{2\sigma^2}\right) - 1 \right]\]

Univariate Case (Classical CRPS)

For $P = \mathcal{N}(\mu, \sigma^2)$ and observation $y$:

\[\text{CRPS}(P, y) = \sigma \left[ z(2\Phi(z) - 1) + \sqrt{\frac{2}{\pi}}(e^{-z^2/2} - 1) \right]\]

where $z = \frac{y - \mu}{\sigma}$ and $\Phi$ is the standard normal CDF.

Computational Approaches

Component	Implementation Strategy
${}_1F_1$ function	scipy.special.hyp1f1
Gamma functions	Standard math libraries
High-dimensional cases	Monte Carlo sampling

Implementation Suggestion:

For d > 3, consider Monte Carlo approximation:

$$ \text{ES}(P, \mathbf{y}) \approx \frac{1}{N}\sum_{i=1}^N \|\mathbf{X}_i - \mathbf{y}\| - \frac{1}{2N^2}\sum_{i,j=1}^N \|\mathbf{X}_i - \mathbf{X}_j\| $$

Proof Sketch

Key Steps

Non-central Chi Distribution: $|\mathbf{X}-\mathbf{y}|$ follows a non-central chi distribution
Central Chi Moments: $\mathbb{E}[|\mathbf{X}-\mathbf{X’}|]$ uses central chi properties
Hypergeometric Connection: Relate Bessel functions to ${}_1F_1$ through series expansions

Technical Lemma

For $\mathbf{V} \sim \mathcal{N}_d(\boldsymbol{\nu}, \mathbf{\Omega})$:

\[\mathbb{E}[\|\mathbf{V}\|] = \sqrt{2\text{tr}(\mathbf{\Omega})} \cdot \frac{\Gamma\left(\frac{d+1}{2}\right)}{\Gamma\left(\frac{d}{2}\right)} \cdot {}_1F_1\left(-\frac{1}{2}; \frac{d}{2}; -\frac{\boldsymbol{\nu}^\top \mathbf{\Omega}^{-1} \boldsymbol{\nu}}{2}\right)\]

Comments

The Energy Score’s extension of CRPS to multivariate settings addresses a critical need in modern probabilistic forecasting. Where the univariate CRPS revolutionized verification of scalar forecasts [1], complex systems increasingly demand joint calibration assessments of vector-valued predictions - from weather models (temperature-pressure-wind vectors) to financial risk (correlated asset returns).

Some Motivations:

Dependency Awareness: Unlike marginal CRPS averaging, the Energy Score's pairwise distance terms (𝔼‖𝐗−𝐗'‖) directly penalize misrepresented correlations [2]

Properness Preservation: Maintains the CRPS' crucial property of being strictly proper - forecasters can't game the system by misrepresenting uncertainties [1]

Consistency: Reduces exactly to CRPS when 𝑑=1, ensuring backward compatibility

While computation requires special functions (${}_1F_1$, Γ) or Monte Carlo methods, this cost reflects the intrinsic complexity of multivariate dependence structures. As shown in [3], alternative scores like the variogram score make different tradeoffs, but the Energy Score remains uniquely tied to the CRPS.

Implementation Insight:

For high-𝑑 systems, the hypergeometric term ${}_1F_1(−1/2;𝑑/2;−𝛿^2/2)$ approaches $e^{−𝛿^2/(2𝑑)}$ - suggestion some interesting connections to Gaussian kernels in RKHS theory [2]

As multivariate probabilistic AI/ML systems proliferate, the Energy Score provides a principled verification framework - one that honors the CRPS’ “properness” philosophy.

References

On Bayesian Methodology. Part 6/6

2024-12-19T00:00:00+00:00

Final Comments

Different Perspectives on Statistical Modeling and Prediction

Statistical modeling and prediction can be approached from several perspectives, each with unique goals and implications:

Traditional Statistical Perspective:
- Models are typically predefined, and the goal is to accurately summarize the posterior distribution.
- Computation continues until approximate convergence is achieved, with approximations used sparingly.
Machine Learning Perspective:
- The primary focus is on prediction rather than parameter estimation.
- Computation halts once cross-validation accuracy plateaus, emphasizing scalability and efficiency within a fixed computational budget.
Model Exploration Perspective:
- Applied statistical work often involves trying out many models, some of which may exhibit poor data fit, low predictive performance, or slow convergence.
- Approximations are more acceptable here but must faithfully reproduce key posterior features.

Key Insight:

The distinction here is not between inference vs. prediction or exploratory vs. confirmatory analysis, but rather how much trust is placed in a model and how computation approximates it.

Model Trust:

The process of iterative model building highlights how much computation and model development rely on trust in individual models and their approximations.

These differing perspectives influence how statistical methods evolve as new challenges emerge in applied settings.

Justification of Iterative Model Building

The iterative model-building process is central to modern Bayesian Workflows and represents the next transformative step in data science:

Historical Progression:
- Data Summarization: The foundation of statistics up to 1900.
- Modeling: Began with Gauss and Laplace, continuing to this day.
- Computation: The current focus, enabling iterative workflows and complex modeling.
Real-World Considerations:
- A Bayesian Workflow acknowledges the limitations of human and computational resources.
- The goal is to simplify processes for humans, even in idealized settings where exact computation is automated.
- Fully automated computation yielding perfect results remains unattainable.

Simplifying Challenges:

Computational challenges are easier to address when fewer "moving parts" exist in the modeling process.

Model Selection and Overfitting

An iterative workflow risks overfitting, as model improvement often involves conditioning on data discrepancies:

Double Dipping: Using data multiple times during model iteration can compromise the frequency properties of inferences.

Warning:

Double dipping and post-selection inference can lead to overfitting. Model improvement conditioned on discrepancies must be carefully managed.

Garden of Forking Paths:

The model-building process often involves paths that depend on the specific data observed.
Instead of selecting the best-fitting model, a Bayesian Workflow emphasizes iterative improvements, ensuring each step is justified.

Example:

Suppose model $M_1$ fails a posterior predictive check, leading to the development of $M_2$, which incorporates more prior information and better fits the data. Had the data differed, $M_1$ might have sufficed. This highlights the iterative nature of a Bayesian Workflow.

To mitigate post-selection inference issues:

Embed multiple models in a larger framework.
Use predictive model averaging or incorporate all models simultaneously.
Perform severe tests of the assumptions underlying each model.

Bigger Datasets Demand Bigger Models

Larger datasets enable the fitting of complex models, such as hierarchical Bayesian models and deep learning approaches:

These models facilitate information aggregation and partial pooling across diverse data sources.
Effective modeling requires:
1. Regularization: To stabilize estimates.
2. Latent Variable Modeling: To address missingness and measurement errors.

Insight:

A model does not exist in isolation; it emerges from engagement with the application and available data.

Prediction, Generalization, and Poststratification

Statistical tasks often involve generalization, which Bayesian methods address effectively:

Generalizing from Sample to Population: Using hierarchical models and partial pooling.
Generalizing from Control to Treatment Groups: Leveraging regularization to handle large nonparametric models.
Generalizing from Observed Data to Underlying Constructs: Applying multilevel modeling for latent variables.

Key Principle:

Just as priors are understood in the context of the likelihood, models should be understood in light of their intended use.

Methods in Bayesian Framework:

Hierarchical modeling and transportability via Bayesian graph models.
Regularization to handle complex, large-scale models.

The iterative nature of Bayesian Workflow, with its emphasis on model trust, computational efficiency, and careful navigation of overfitting risks, reflects the dynamic and evolving nature of modern statistical practice. By embedding models within a larger framework and embracing the iterative process, a Bayesian Workflow ensures robust and insightful statistical analyses.

On Bayesian Methodology. Part 5/6

2024-12-02T00:00:00+00:00

VII. Understanding and Comparing Multiple Models

Visualizing Models in Relation to Each Other

A key aspect of a Bayesian Workflow is fitting multiple models for a single problem. This process is not about selecting the best model or averaging models, but rather using a series of fitted models to gain a deeper understanding of each one.

Remark:

Model comparison in this context is not about selecting or averaging but about exploring the process of fitting multiple models to understand them better.

Considerations When Fitting Multiple Models:

Be aware of researcher degrees of freedom:
- Overfitting can occur if a single “best” model is chosen.
- A set of fitted models might appear to bracket total uncertainty, but other unconsidered models could still contribute.
Multiverse analysis: Evaluate situations where multiple models pass all checks to ensure robustness.

Cross-validation and Model Averaging

Cross-validation (CV) is a powerful tool for evaluating models, but it requires careful interpretation:

Key Principle:

If there is significant uncertainty in model comparisons, avoid selecting the single model with the best cross-validation result. This discards the uncertainty from the CV process.

Alternative Approaches:

Stacking:
Combines inferences by weighting models to minimize cross-validation error. Stacking can sometimes be seen as pointwise model selection.
Scaffold Models:
Include models that are deliberately simple (e.g., for comparison), experimental, or flawed (e.g., with coding errors). These models serve as benchmarks or stepping stones but are not included in final predictions.

While a Bayesian Workflow emphasizes continuous model expansion over averaging, there are cases where averaging predictions over competing Bayesian models is reasonable.

Comparing a Large Number of Models

When faced with many candidate models, the goal is often to find a simpler model with comparable predictive performance to a more complex, expanded model.

Warning:

Selecting one model based solely on minimizing cross-validation error risks overfitting and suboptimal choices.

Projection Predictive Variable Selection:

A stable and reliable method to identify smaller models with strong predictive performance.
Avoids overfitting by:
- Projecting submodels based on the expanded model’s predictions.
- Not fitting each model independently to the data.

This approach enables efficient comparison of a large number of models while maintaining robust predictive performance.

Understanding and comparing multiple models is an integral part of Bayesian Workflow. By focusing on the process of model fitting rather than rigid selection or averaging, practitioners can better navigate uncertainty and extract meaningful insights from their analyses. Tools like stacking and projection predictive variable selection ensure that model comparison remains both rigorous and practical.

On Bayesian Methodology. Part 4/6

2024-11-13T00:00:00+00:00

IV. Addressing Computational Problems

The Folk Theorem of Statistical Computing:

When computational problems arise, they often indicate an issue with the model itself.

Many cases of poor convergence are tied to regions of parameter space that are either irrelevant or nonsensical.

Instead of throwing more computational resources at the problem, the first step should be to identify and address potential model pathologies.

Starting at Simple and Complex Models and Meeting in the Middle

Diagnosing computational problems often requires a two-pronged approach:

Simplify the problematic model step by step until it works reliably.
Start with a simple, well-understood model and gradually add features until the issue reappears.

This process helps isolate the root cause of the problem.

Getting a Handle on Models That Take a Long Time to Fit

Slow computation is often symptomatic of deeper issues, such as poorly performing Hamiltonian Monte Carlo (HMC). However, debugging becomes harder as computation times increase.
Key strategies include:

Viewing model choices as provisional.
Fitting multiple models to understand computational and inferential behavior in the applied problem.

Monitoring Intermediate Quantities

Saving and plotting intermediate quantities during computation can reveal hidden issues with the model or algorithm. These visualizations often provide valuable clues for debugging.

Stacking to Reweight Poorly Mixing Chains

In situations where multiple chains mix slowly but remain within reasonable ranges, stacking can be used:

Combine simulations by assigning weights to chains through cross-validation.
Particularly useful during model exploration when diagnostics suggest some progress but full convergence remains elusive.

Posterior Distributions with Multimodality and Difficult Geometry

Multimodality and complex posterior geometries pose significant challenges:

Disjoint Posterior Volumes:
- Near-zero mass for all but one mode.
- Symmetric volumes, such as label switching in mixture models.
- Distinct volumes with significant probability mass.
Unstable Tails:
A single posterior volume with arithmetically unstable regions.

Each scenario requires tailored strategies for efficient computation.

Reparameterization

HMC-based samplers perform best when:

The mass matrix is well-tuned.
The posterior geometry is smooth, with no sharp corners or irregularities.

For many classical models, results like the Bernstein-von Mises theorem simplify posterior geometry with sufficient data. When this is not the case, reparameterization can significantly improve computational performance by simplifying posterior geometry.

Marginalization

Challenging geometries in posterior distributions often stem from parameter interactions. Marginalizing over certain parameters can simplify computations:

Approximations like the Laplace method can be particularly effective for latent Gaussian models.
Exploiting the structure of the problem can lead to substantial improvements.

Adding Prior Information

Many computational issues can be mitigated by incorporating prior information:

Priors help address weakly informative data regions, improving model behavior without sacrificing inference quality.
While the primary purpose of priors is not to fix fitting problems, their inclusion often resolves computational challenges.

Identifiability?

“Identification” is an asymptotic property in classical statistics

Bayesian inference prioritizes inference with finite data.

If data are insufficient for certain aspects of the model, priors can supply the necessary information.

Ladder of Abstraction:

Poor mixing of MCMC.
Difficult geometry as a mathematical explanation.
Weakly informative data as a statistical explanation.
Substantive prior information as a solution.

Addressing computational issues can start at either end of this ladder, transitioning from troubleshooting to workflow optimization.

Adding Data

Similar to priors, additional data can constrain models and resolve computational problems:

Incorporate new data sources into the model.
Models that work well with larger datasets may struggle in small data regimes; expanding the dataset can improve performance.

Addressing computational problems in Bayesian modeling involves a combination of simplifying models, leveraging prior information, and refining computational techniques. A systematic approach, starting with diagnostics and iterative improvements, ensures both model reliability and computational efficiency.

V. Evaluating and Using a Fitted Model

Evaluating a fitted model involves multiple checks, each tailored to the specific goals of the analysis. The aspects of the model that require evaluation depend on the application and the intended users of the statistical methods.

Posterior Predictive Checking

Posterior predictive checking involves simulations from the posterior distribution to evaluate model performance:

While there’s no universal guide for which checks to perform, conducting a few direct checks can safeguard against gross misspecification.
Similarly, there’s no definitive rule for deciding when a failed check necessitates adjustments to the model.
The choice of checks depends on the analysis goals and the costs and benefits of adjustments.

Key Principle:

Seek “severe tests”—checks likely to fail if the model produces misleading answers to critical questions.

Cross-Validation and Influence of Individual Data Points

Cross-validation (CV) enhances predictive diagnostics, especially for flexible models, by providing insights into model fit and data influence:

Calibration Checks: Use the cross-validation predictive distribution to assess calibration.
Difficult Observations: Identify observations or groups that are hard to predict.
Influence Diagnostics: Examine the additional information provided by individual observations.

Leave-one-out cross-validation (LOO-CV) is a popular method, though it doesn’t always align with inferential goals for multilevel (hierarchical) models.
Calibration insights:
- Posterior predictive checking compares marginal prediction distributions to data.
- LOO-CV predictive checking evaluates the calibration of conditional predictive distributions.
- Probability integral transformations (PIT) under good calibration are uniform.

Practical Tip:

Cross-validation for multilevel models requires thoughtful implementation to ensure alignment with inferential goals.

Influence of Prior Information

Understanding how prior information affects posterior inferences is essential for a robust evaluation:

A statistical model can be understood in two ways:
1. Generatively: Explore how parameters map to data using prior predictive simulations.
2. Inferentially: Examine the path from inputs (data and priors) to outputs (estimates and uncertainties).
Sensitivity Analysis:
- Measure shrinkage between prior and posterior distributions, such as comparing posterior standard deviations or quantiles.
- Use importance sampling to approximate and compare posteriors across models.
- Conduct static sensitivity analysis to study posterior sensitivity to prior perturbations without re-fitting the model.

Practical Insight:

Sensitivity analysis highlights the balance between prior information and data, offering valuable diagnostic insights without excessive computation.

Summarizing Inference and Propagating Uncertainty

Traditional methods of summarizing Bayesian inference often fail to fully represent the complexity of variation and uncertainty:

Tables and graphs of parameter estimates and uncertainties only capture one-dimensional margins.
Marginal posterior distribution graphs become unwieldy for models with many parameters and fail to illustrate the interplay of hierarchical model uncertainties.

Tools for Advanced Summaries:

Use visualization tools like the bayesplot R package to effectively summarize and explore Bayesian inference results.

Evaluating a fitted model is a multifaceted process. It involves a combination of diagnostic checks, sensitivity analyses, and advanced visualization techniques. The goal is not just to identify potential misfits but to refine the model for better inference and predictive accuracy.

VI. Modifying a Model

Constructing a Model for the Data

Model construction is a creative process where the modeler combines existing components to account for new data, enhance features of existing data, or establish links to underlying processes.

Model Building as a Task:

Model construction is akin to a language-like task, where components are assembled to encompass new data, existing data features, and links to underlying processes.

Reasons for Model Expansion:

Response to new data.

Failures of models fit to existing data.

Computational challenges with current fitting procedures.

Incorporating Additional Data

Expanding a model to include more data is a critical step in a Bayesian Workflow.
It’s often said that the value of a statistical method lies not just in how it handles data but in the choice of what data to use.

Working with Prior Distributions

Traditionally, Bayesian statistics refers to noninformative or fully informative priors, but in practice, these rarely exist:

Uniform Prior: Depends on parameterization, carrying implicit information.
Reference Prior: Based on asymptotic regimes and fictional data assumptions.
Informative Prior: Rarely encompasses all available knowledge.

Ladder of Priors:
Think of prior distributions as existing on a continuum:

Improper flat prior.
Super-vague but proper prior.
Very weakly informative prior.
Generic weakly informative prior.
Specific informative prior.

Priors also act as constraints, shrinking estimates toward simpler models. However, the need for prior information varies based on:

The role of the parameter in the model.
The parameter’s position in the hierarchy.

Critical Insight: Priors must be specified for each model in the workflow. Expanded models often require additional thought regarding parameterization and joint priors to avoid unintended effects like cancellations or overly stabilizing priors.

When introducing new parameters:

Consider tightening priors on overarching metrics like means and standard deviations.
Be cautious of the concentration of measure in higher-dimensional spaces.

A Topology of Models

Models within a framework can be thought of as forming a topology or network structure. This structure reflects connections and partial orderings rather than probabilities assigned to individual models.

Model Topology?:

A topology of models refers to their connections and relationships, not a probability space. The goal is not to average over models but to navigate among them effectively.

Examples of Model Navigation Tools:

Automatic Statistician: Explores models in specified but open-ended classes, like time series or regression models, using inference and model criticism.
Prophet: A time series forecasting tool allowing users to build models from predefined building blocks.

Model Operations: Models, treated as probabilistic random variables, can be combined in multiple ways:

Additive, multiplicative, linear mixing, log-linear mixing, pointwise mixing, and more.
Each model has its internal structure, with parameters estimated from data, and parameters across models can interact (e.g., shared parameters).

Applications:

Forecasting: Using interconnected models to predict future outcomes.

Causal Inference: Exploring relationships between variables using networked model structures.

On Bayesian Methodology. Part 3/6

2024-11-01T00:00:00+00:00

III. Using Constructed Data to Find and Understand Problems

Fake-Data Simulation

The core idea of fake-data simulation is to test whether a procedure can recover correct parameter values when applied to simulated data. This involves the following steps:

Simulate Fake Data:
Choose reasonable parameter values and generate a fake dataset that matches the size, structure, and shape of the original data.
Evaluate the Procedure:
- Information Beyond the Prior: Simulate fake data from the model using fixed, known parameters and check if the observed data adds meaningful information. Analyze point estimates and the coverage of posterior intervals.
- Parameter Recovery: Assess whether the true parameters are recovered within the uncertainty range implied by the fitted posterior distribution.
- Behavior Across Parameter Space: Explore how the model behaves in different regions of the parameter space, revealing the various “stories” the model encodes about data generation.
Two-Step Procedure:
Fit the model to real data, draw parameters from the resulting posterior distribution, and use these parameters for fake-data checking.

Key Insight:

If a model cannot make reliable inferences on fake data generated from itself, it’s unlikely to provide reasonable inferences on real data.

While fake-data simulations help evaluate a model’s ability to recover parameters, they also highlight potential weaknesses. For example:

Creating fake data that causes the procedure to fail can deepen understanding of an inference method.
Overparameterized models may yield comparable predictions despite wildly different parameter estimates, limiting the usefulness of predictive checks.

Simulation-Based Calibration (SBC)

SBC provides a more comprehensive approach than truth-point benchmarking by fitting the model multiple times and comparing posterior distributions to simulated data. However, SBC has its challenges:

Computational Cost: Requires significant resources to fit the model repeatedly.
Priors and Modeler Bias:
- Weakly informative priors, often chosen conservatively, can lead to extreme datasets during SBC.
- This mismatch can obscure insights about calibration and posterior behavior.

Open Research Question: How effective is SBC with a limited number of simulations?

Simulation-based calibration and truth-point benchmarking are complementary, with SBC offering broader insights but at a higher computational expense.

Experimentation Using Constructed Data

Simulating data from different scenarios provides valuable insights into models and inference methods. This experimentation allows practitioners to:

Understand how a model performs under varying conditions.
Explore the limits of inferences by fitting the model to data generated from challenging scenarios.
Gain a deeper understanding of both computational issues and the underlying data.

Important Consideration: Testing a model with a single fake dataset is not sufficient.

Even if the computational algorithm works, there’s a 5% chance that a random draw will fall outside a 95% uncertainty interval.

Bayesian inference is calibrated only when averaging over the prior.

Parameter recovery can fail not because of algorithmic errors but due to insufficient information in the observed data.

Simulation of statistical systems under diverse conditions not only addresses computational challenges but also enhances our understanding of data and inference.

On Bayesian Methodology Part. 2/6

2024-10-11T00:00:00+00:00

I. Before Fitting a Model

Choosing an Initial Model
The starting point of almost all analyses is to adapt what has been done before. Drawing from existing methods provides a foundation for further refinement.

Modular Construction
A Bayesian model is built from modules rather than whole models. Naming and constructing models in a modular fashion makes it easier to:

Identify connections between seemingly different models.
Adapt models to the specific requirements of an analysis project.

Modules often act as placeholders, which can be replaced or expanded as necessary.

Scaling and Transforming the Parameters
Parameters should be interpretable for both practical and ethical reasons. Transformations can make parameters more meaningful, aiding in:

Clear communication of results.
Effective use of prior information.

Prior Predictive Checking
Prior predictive checks are invaluable for understanding the implications of prior distributions within generative models.

Key Approaches

Considering priors on outcomes and deriving corresponding joint priors on parameters.
Using simulations to elicit expert knowledge on measurable quantities, which is often easier than soliciting opinions on unobservable model parameters.

Generative and Partially Generative Models
Fully Bayesian data analysis typically requires a generative model, a joint probability distribution for all data and parameters.

Insights

Bayesian inference does not require the generative model. It only needs the likelihood derived from the data, and different generative models can share the same likelihood.
Bayesian data analysis, however, depends on the generative model to enable predictive simulation and model checking.
The Bayesian workflow evaluates a series of generative models to improve understanding and predictions.

Prior and posterior predictive checks can vary under different generative models while satisfying the likelihood principle.
Some models commonly used in Bayesian analysis are not fully generative:

Regression models and survival data with censoring.
Models with improper priors, which lack a joint distribution for data and parameters and cannot sample from the prior predictive distribution. Improper priors often serve as placeholders on the path to a fully Bayesian model.

In applied Bayesian work, complexity often arises from incorporating multiple data sources. This necessitates balancing simplicity with flexibility in modeling.

Progression of Generative Models

At one extreme, non-generative methods consist solely of data summaries with no model for the data.
Classical statistical models provide probability distributions $ p(y; \theta) $ for data $ y $ given parameters $ \theta $, but no distribution for $ \theta $.
Partially generative Bayesian models include unmodeled data $ x $, such as sample sizes, design settings, or hyperparameters, represented as $ p(y, \theta \mid x) $.
Fully generative models encompass everything, represented as $ p(y, \theta, x) $, leaving no data “left out.”

II. Fitting a Model

Bayesian computation has evolved significantly over time. Early methods relied on analytic calculations and normal approximations. In the 1990s, the advent of advanced algorithms expanded the landscape for exploring posterior distributions:

Gibbs and Metropolis Algorithms: Enabled Bayesian inference for many models.
Sequential Monte Carlo (SMC): A generalization of Metropolis applicable to broader Bayesian computations.
Variational Inference (VI): A fast but potentially inaccurate approximation, building on the expectation-maximization (EM) algorithm.
Hamiltonian Monte Carlo (HMC): Uses gradient computations to navigate continuous probability spaces efficiently.

These innovations have revolutionized Bayesian workflow, but safe usage requires strong diagnostics to flag unreliable computations.

Initial Values, Adaptation, and Warmup

The initial phase of inference algorithms is critical:

Initial Values: While theoretically irrelevant in the asymptotic limit, poor initial values can bias results.
Warmup Phase: Moves simulations from unrepresentative initial values toward the typical set, a concept in information theory. This phase serves three purposes:
1. Reduces bias from initial values.
2. Provides information to tune algorithm parameters.
3. Flags computational issues early.

How Long to Run an Iterative Algorithm

The duration of iterative algorithms like MCMC impacts result reliability and computational efficiency:

Convergence Diagnostics:
- Standard practice: Run until a mixing measure, Vehtari et al. (2020), is below 1.01 for all parameters.
- Monitor the multivariate mixing statistic, Lambert and Vehtari (2020).
Balancing Accuracy and Speed:
While increasing adequate sample size or reducing Monte Carlo error improves accuracy, it limits model exploration if computation is too slow.
Leveraging Parallelism:
Instead of increasing iterations, variance reduction can be achieved by increasing parallel chains.

Approximate Algorithms and Models

Markov chain simulation is an approximation where theoretical error reduces with more iterations. However, scalability challenges arise as models and datasets grow larger, necessitating faster alternatives:

Techniques for Approximation:
- Empirical Bayes
- Linearization
- Laplace approximation
- Nested approximations (e.g., INLA)
- Data-splitting (e.g., expectation propagation)
- Mode-finding (e.g., variational inference)
- Penalized maximum likelihood
Diagnostics for Approximate Algorithms:
Use diagnostic tools to ensure the algorithm reproduces key posterior features for the specific model.

Insights: Approximate algorithms can be viewed as exact algorithms for approximate models.

Empirical Bayes: Replaces prior distributions with data-dependent point-mass priors.
strong>Laplace Approximation:</strong> Data-dependent linearization of the model.
Nested Laplace Approximation: Linearizes conditional posterior distributions.

Fit Fast, Fail Fast

Efficient Bayesian workflow emphasizes quickly identifying and discarding flawed models:

Failing Fast: Saves time by avoiding perfect inference for fundamentally flawed models.

Fast Algorithms vs. Early Failure There is extensive literature on fast approximation algorithms to fit desired models, but less focus on algorithms designed to minimize time spent on models that will ultimately be abandoned.

On Bayesian Methodology. Part 1/6

2024-09-29T00:00:00+00:00

A Bayesian Methodology Workflow. Towards an all-encompassing framework.

In today’s data-driven world, the need for a robust and systematic framework for applied Bayesian Statistics has never been more critical. As scientific challenges grow increasingly complex, spanning diverse fields such as environmental science, medicine, economics, and machine learning, the ability to manage uncertainty and interpret data meaningfully is paramount. A workflow that eases the navigation of model building, inference, and validation processes.

Statistical analysis serves as a cornerstone of the scientific method, acting as a bridge between theoretical models and empirical evidence. Within this broader context, Bayesian analysis stands out for its flexibility in incorporating prior knowledge and capacity to provide probabilistic interpretations of uncertainty. A Bayesian workflow framework would improve this process by unifying data exploration, model development, and post-inference critique stages, ensuring that each step aligns with the ultimate goals of robust and transparent decision-making.

This review builds upon the foundational work of Gelman et al. (2020). As science increasingly relies on interdisciplinary collaboration and complex data, frameworks like the one described by Gelman et al. (2020) are indispensable for ensuring methodological rigor, fostering reproducibility, and advancing discovery.

What is a Bayesian Workflow?

A Bayesian Workflow provides a structured approach to managing the dynamic and iterative nature of applied Bayesian Statistics. To better understand it, it’s helpful to differentiate three closely related concepts:

Bayesian Inference:
Whose primary goal is to compute the conditional probabilities or posterior distributions.
Bayesian Data Analysis:
Which refers to the application of Bayesian inference to specific datasets as described in Gelman et al. (2017)
Bayesian Workflow:
A broader framework that includes:
- Model building
- Inference
- Model checking and improvement
- Model comparison (not limited to selection or averaging)

The workflow emphasizes the value of analyzing models that may initially seem flawed or suboptimal, as they often provide important insights to understand the model and potential extensions.

Why Is a Bayesian Workflow Important?

A well-defined Bayesian Workflow addresses several challenges in statistical modeling:

Organized Model Understanding:
It alleviates computational burdens by structuring model evaluation and refinement into manageable steps.
Better Decision-Making:
It helps practitioners identify which assumptions to relax and which to retain, enabling more thoughtful model selection.
Model-Data Relationships:
Understanding how models fit the data, even when models reflect poor performance, can provide useful insights.
Diverse Conclusions:
Different models often lead to different conclusions; A Bayesian Workflow encourages practitioners to explore and understand these variations.

The Big Picture

% ### ToDo: Include Diagram

Integrating Statistical Innovations

Many statistical methodologies began as informal techniques or “hacks” and have since been formalized within Bayesian frameworks in recent years. Key examples include:

Multilevel Modeling:
Extends empirical Bayes methods by incorporating prior inference directly into Bayesian models.
Exploratory Data Analysis (EDA):
Functions as a form of predictive model checking.
Regularization Techniques:
Replace ad hoc variable selection methods in regression, offering more robust solutions.
Non-parametric Models:
They serve as Bayesian replacements for techniques like kernel smoothing and broadening their application.

These methods have expanded statistical frameworks, making them more modular and adaptable while retaining their core principles.

Statistics and Uncertainty

At its heart, statistics is about managing uncertainty. A Bayesian Workflow recognizes and aims to address uncertainty in several forms:

Data and Model Parameters:
Variability and imprecision are inherent in the data and parameters.
Model Fitting:
Challenges in determining whether a model is appropriately specified.
Model Setup and Expansion:
Difficulty in deciding how to structure and extend models to capture key dynamics.
Interpretation:
Ambiguity in translating statistical results into meaningful conclusions.

Workflows often become disorderly when moving beyond predefined designs and analyses. Practical constraints—such as limited time, computational resources, and the consequences of incorrect decisions—frequently necessitate shortcuts. A structured workflow mitigates these challenges by offering a systematic approach to balancing rigor with practicality.

A Bayesian Workflow provides a comprehensive framework for organizing applied Bayesian Statistics. It embraces the iterative nature of statistical modeling, acknowledges the importance of learning from flawed models, and prioritizes the exploration of uncertainty. By following these structured principles, practitioners can make informed, defensible decisions while navigating the complexities of statistical analysis.

For those interested in exploring this further, I would recommend you review Gelman et al. (2020) and consider how these principles can be applied in your work.

Antonio De Leon

Variational Bayes for Non-Conjugate Models: A Laplace-Delta Tutorial

Introduction

Methodology Overview

1. Preliminaries

1.1 Mean-Field Variational Bayes (MFVB)

1.2 Coordinate Ascent Variational Inference (CAVI)

1.3 Non-Conjugate Challenges

2. VB-LD Algorithm

2.1 Laplace Approximation Step

2.2 Delta Method for Expectation Approximation

2.3 Complete VB-LD Algorithm

Further Reading

Statistical Computing: A tutorial for UCSC Graduate Students

What Are the Hummingbird Clusters?

Who Should Use HB?

What’s in This Guide?

Introduction to Cluster Computing for Stats Grads

Brief Glossary:

I. Interactive Development Sessions

Example 1: Debugging Bayesian Models in R

Example 2: Interactive ML Development with Jupyter

II. Batch Processing for Production Workloads

Example 1: Large-Scale Bayesian Inference

Example 2: Distributed ML Training

Reproducible Environment Setup

Statistical Computing Environments

Performance Optimization Guide

Memory Management Essentials

Parallel Computing Patterns Explained

Getting Help with HPC

The Energy Score for Multivariate Normal Distributions

Introduction

Energy Score: Definition

The Multivariate Normal Case

Special Cases

Isotropic Covariance

Univariate Case (Classical CRPS)

Computational Approaches

Proof Sketch

Key Steps

Technical Lemma

Comments

On Bayesian Methodology. Part 6/6

Final Comments

Different Perspectives on Statistical Modeling and Prediction

Justification of Iterative Model Building

Model Selection and Overfitting

Bigger Datasets Demand Bigger Models

Prediction, Generalization, and Poststratification

On Bayesian Methodology. Part 5/6

VII. Understanding and Comparing Multiple Models

Visualizing Models in Relation to Each Other

Cross-validation and Model Averaging

Comparing a Large Number of Models

On Bayesian Methodology. Part 4/6

IV. Addressing Computational Problems

Starting at Simple and Complex Models and Meeting in the Middle

Getting a Handle on Models That Take a Long Time to Fit

Monitoring Intermediate Quantities

Stacking to Reweight Poorly Mixing Chains

Posterior Distributions with Multimodality and Difficult Geometry

Reparameterization

Marginalization

Adding Prior Information

Adding Data

V. Evaluating and Using a Fitted Model

Posterior Predictive Checking

Cross-Validation and Influence of Individual Data Points

Influence of Prior Information

Summarizing Inference and Propagating Uncertainty

VI. Modifying a Model

Constructing a Model for the Data

Incorporating Additional Data

Working with Prior Distributions

A Topology of Models

On Bayesian Methodology. Part 3/6

III. Using Constructed Data to Find and Understand Problems

Fake-Data Simulation

Simulation-Based Calibration (SBC)