# Simulation based calibration for OncoBayes2

####

#### Mon Mar 17 10:52:33 2025

This report documents the results of a simulation based calibration (SBC) run for `OncoBayes2`. TODO

The calibration data presented here has been generated at and with the `OncoBayes` git version as:

    ## Created:  2025-03-14 18:24:03 UTC
    ## git hash: 93931c92e5ab1b5d4e91a56633979bda3a3a6036
    ## MD5:      3b0d717520a54320f4f7b35a0dbdeb3d

The MD5 hash of the calibration data file presented here must match the above listed MD5:

    ## /Users/weberse2/rwork/OncoBayes2/inst/sbc/calibration.rds
    ##                        "3b0d717520a54320f4f7b35a0dbdeb3d"

# Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

## Model description TODO

The fake data simulation function returns … TODO. Please refer to the `sbc_tools.R` and `make_reference_rankhist.R` R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

# SBC results

## Sampler Diagnostics Overview

data_scenario | N | total_divergent | min_ess_bulk | min_ess_tail | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail
--- | --- | --- | --- | --- | --- | --- | --- | ---
combo2_EX | 10000 | 0 | 137.142 | 41.504 | 1.019 | 0 | 396.903 | 388.595
combo2_EXNEX | 10000 | 0 | 44.494 | 34.822 | 1.058 | 0 | 345.659 | 284.837
combo3_EXNEX | 10000 | 0 | 10.145 | 36.984 | 1.153 | 1 | 300.542 | 396.677
log2bayes_EX | 10000 | 0 | 901.675 | 705.767 | 1.019 | 0 | 325.927 | 463.457
log2bayes_EXNEX | 10000 | 0 | 255.437 | 385.463 | 1.016 | 0 | 410.029 | 571.638

Large Rhat is defined as exceeding \(1.1\).

## Sampler Adaptation & Performance Overview

ESS speed is in units of ESS per second.

## \(\chi^2\) Statistic: Single-agent logistic regression, stratified

param | statistic | df | p.value
--- | --- | --- | ---
beta_group[A,I(log(drug_A/1)),intercept] | 25.811 | 31 | 0.730
beta_group[A,I(log(drug_A/1)),log_slope] | 41.958 | 31 | 0.091
beta_group[B,I(log(drug_A/1)),intercept] | 21.178 | 31 | 0.907
beta_group[B,I(log(drug_A/1)),log_slope] | 29.222 | 31 | 0.558
beta_group[C,I(log(drug_A/1)),intercept] | 17.754 | 31 | 0.973
beta_group[C,I(log(drug_A/1)),log_slope] | 28.614 | 31 | 0.589
beta_group[D,I(log(drug_A/1)),intercept] | 28.269 | 31 | 0.607
beta_group[D,I(log(drug_A/1)),log_slope] | 36.307 | 31 | 0.235
beta_group[E,I(log(drug_A/1)),intercept] | 41.325 | 31 | 0.102
beta_group[E,I(log(drug_A/1)),log_slope] | 24.128 | 31 | 0.805
beta_group[F,I(log(drug_A/1)),intercept] | 26.643 | 31 | 0.690
beta_group[F,I(log(drug_A/1)),log_slope] | 35.533 | 31 | 0.263
mu_log_beta[I(log(drug_A/1)),intercept] | 44.339 | 31 | 0.057
mu_log_beta[I(log(drug_A/1)),log_slope] | 27.782 | 31 | 0.632
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] | 25.504 | 31 | 0.745
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] | 24.755 | 31 | 0.779
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_intercept] | 23.731 | 31 | 0.821
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_log_slope] | 23.718 | 31 | 0.822

## \(\chi^2\) Statistic: Single-agent logistic regression, EXchangeable/NonEXchangeable

param | statistic | df | p.value
--- | --- | --- | ---
beta_group[A,I(log(drug_A/1)),intercept] | 23.744 | 31 | 0.821
beta_group[A,I(log(drug_A/1)),log_slope] | 30.227 | 31 | 0.506
beta_group[B,I(log(drug_A/1)),intercept] | 23.635 | 31 | 0.825
beta_group[B,I(log(drug_A/1)),log_slope] | 25.869 | 31 | 0.728
beta_group[C,I(log(drug_A/1)),intercept] | 18.944 | 31 | 0.956
beta_group[C,I(log(drug_A/1)),log_slope] | 31.386 | 31 | 0.447
mu_log_beta[I(log(drug_A/1)),intercept] | 28.083 | 31 | 0.617
mu_log_beta[I(log(drug_A/1)),log_slope] | 26.874 | 31 | 0.679
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] | 38.496 | 31 | 0.167
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] | 25.210 | 31 | 0.758

## \(\chi^2\) Statistic: Double combination, fully exchangeable, stratified

param | statistic | df | p.value
--- | --- | --- | ---
beta_group[A,I(log(drug_A/1)),intercept] | 37.011 | 31 | 0.211
beta_group[A,I(log(drug_A/1)),log_slope] | 21.811 | 31 | 0.889
beta_group[A,I(log(drug_B/2)),intercept] | 39.789 | 31 | 0.134
beta_group[A,I(log(drug_B/2)),log_slope] | 23.456 | 31 | 0.832
beta_group[B,I(log(drug_A/1)),intercept] | 22.317 | 31 | 0.873
beta_group[B,I(log(drug_A/1)),log_slope] | 27.066 | 31 | 0.669
beta_group[B,I(log(drug_B/2)),intercept] | 43.053 | 31 | 0.073
beta_group[B,I(log(drug_B/2)),log_slope] | 32.736 | 31 | 0.382
beta_group[C,I(log(drug_A/1)),intercept] | 27.034 | 31 | 0.670
beta_group[C,I(log(drug_A/1)),log_slope] | 20.493 | 31 | 0.925
beta_group[C,I(log(drug_B/2)),intercept] | 33.933 | 31 | 0.328
beta_group[C,I(log(drug_B/2)),log_slope] | 23.206 | 31 | 0.842
beta_group[D,I(log(drug_A/1)),intercept] | 28.915 | 31 | 0.574
beta_group[D,I(log(drug_A/1)),log_slope] | 16.461 | 31 | 0.985
beta_group[D,I(log(drug_B/2)),intercept] | 42.400 | 31 | 0.083
beta_group[D,I(log(drug_B/2)),log_slope] | 44.774 | 31 | 0.052
eta_group[A,I(drug_A/1 * drug_B/2)] | 36.128 | 31 | 0.241
eta_group[B,I(drug_A/1 * drug_B/2)] | 28.154 | 31 | 0.613
eta_group[C,I(drug_A/1 * drug_B/2)] | 35.904 | 31 | 0.249
eta_group[D,I(drug_A/1 * drug_B/2)] | 33.549 | 31 | 0.345
mu_eta[m[1]] | 32.954 | 31 | 0.372
mu_log_beta[I(log(drug_A/1)),intercept] | 23.642 | 31 | 0.825
mu_log_beta[I(log(drug_A/1)),log_slope] | 20.877 | 31 | 0.915
mu_log_beta[I(log(drug_B/2)),intercept] | 52.454 | 31 | 0.009
mu_log_beta[I(log(drug_B/2)),log_slope] | 35.859 | 31 | 0.251
tau_eta[stratum_1,m[1]] | 31.539 | 31 | 0.439
tau_eta[stratum_2,m[1]] | 26.470 | 31 | 0.698
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] | 24.141 | 31 | 0.805
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] | 27.366 | 31 | 0.654
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] | 30.080 | 31 | 0.513
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] | 36.685 | 31 | 0.222
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_intercept] | 26.515 | 31 | 0.696
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_log_slope] | 25.638 | 31 | 0.738
tau_log_beta[stratum_2,I(log(drug_B/2)),tau_intercept] | 30.310 | 31 | 0.501
tau_log_beta[stratum_2,I(log(drug_B/2)),tau_log_slope] | 38.931 | 31 | 0.155

## \(\chi^2\) Statistic: Double combination, EXchangeable/NonEXchangeable

param | statistic | df | p.value
--- | --- | --- | ---
beta_group[A,I(log(drug_A/1)),intercept] | 30.010 | 31 | 0.517
beta_group[A,I(log(drug_A/1)),log_slope] | 28.704 | 31 | 0.585
beta_group[A,I(log(drug_B/2)),intercept] | 55.987 | 31 | 0.004
beta_group[A,I(log(drug_B/2)),log_slope] | 35.859 | 31 | 0.251
beta_group[B,I(log(drug_A/1)),intercept] | 28.032 | 31 | 0.620
beta_group[B,I(log(drug_A/1)),log_slope] | 29.773 | 31 | 0.529
beta_group[B,I(log(drug_B/2)),intercept] | 31.514 | 31 | 0.441
beta_group[B,I(log(drug_B/2)),log_slope] | 22.880 | 31 | 0.853
beta_group[C,I(log(drug_A/1)),intercept] | 34.157 | 31 | 0.318
beta_group[C,I(log(drug_A/1)),log_slope] | 32.730 | 31 | 0.382
beta_group[C,I(log(drug_B/2)),intercept] | 36.954 | 31 | 0.213
beta_group[C,I(log(drug_B/2)),log_slope] | 22.662 | 31 | 0.861
eta_group[A,I(drug_A/1 * drug_B/2)] | 30.803 | 31 | 0.476
eta_group[B,I(drug_A/1 * drug_B/2)] | 16.941 | 31 | 0.981
eta_group[C,I(drug_A/1 * drug_B/2)] | 40.141 | 31 | 0.126
mu_eta[m[1]] | 30.374 | 31 | 0.498
mu_log_beta[I(log(drug_A/1)),intercept] | 31.200 | 31 | 0.456
mu_log_beta[I(log(drug_A/1)),log_slope] | 38.246 | 31 | 0.174
mu_log_beta[I(log(drug_B/2)),intercept] | 35.872 | 31 | 0.251
mu_log_beta[I(log(drug_B/2)),log_slope] | 22.042 | 31 | 0.882
tau_eta[stratum_1,m[1]] | 23.347 | 31 | 0.836
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] | 26.074 | 31 | 0.718
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] | 31.277 | 31 | 0.452
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] | 35.789 | 31 | 0.254
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] | 27.834 | 31 | 0.630

## \(\chi^2\) Statistic: Triple combination, EXchangeable/NonEXchangeable

param | statistic | df | p.value
--- | --- | --- | ---
beta_group[A,I(log(drug_A/1)),intercept] | 32.678 | 31 | 0.384
beta_group[A,I(log(drug_A/1)),log_slope] | 19.565 | 31 | 0.945
beta_group[A,I(log(drug_B/2)),intercept] | 34.298 | 31 | 0.312
beta_group[A,I(log(drug_B/2)),log_slope] | 27.168 | 31 | 0.664
beta_group[A,I(log(drug_C/4)),intercept] | 35.750 | 31 | 0.255
beta_group[A,I(log(drug_C/4)),log_slope] | 21.235 | 31 | 0.906
beta_group[B,I(log(drug_A/1)),intercept] | 30.573 | 31 | 0.488
beta_group[B,I(log(drug_A/1)),log_slope] | 32.429 | 31 | 0.396
beta_group[B,I(log(drug_B/2)),intercept] | 40.211 | 31 | 0.124
beta_group[B,I(log(drug_B/2)),log_slope] | 27.136 | 31 | 0.665
beta_group[B,I(log(drug_C/4)),intercept] | 26.752 | 31 | 0.685
beta_group[B,I(log(drug_C/4)),log_slope] | 32.486 | 31 | 0.393
beta_group[C,I(log(drug_A/1)),intercept] | 39.846 | 31 | 0.133
beta_group[C,I(log(drug_A/1)),log_slope] | 41.907 | 31 | 0.091
beta_group[C,I(log(drug_B/2)),intercept] | 47.046 | 31 | 0.032
beta_group[C,I(log(drug_B/2)),log_slope] | 16.160 | 31 | 0.987
beta_group[C,I(log(drug_C/4)),intercept] | 29.664 | 31 | 0.535
beta_group[C,I(log(drug_C/4)),log_slope] | 25.882 | 31 | 0.727
eta_group[A,I(drug_A/1 * drug_B/2 * drug_C/4)] | 15.398 | 31 | 0.991
eta_group[A,I(drug_A/1 * drug_B/2)] | 27.686 | 31 | 0.637
eta_group[A,I(drug_A/1 * drug_C/4)] | 22.899 | 31 | 0.853
eta_group[A,I(drug_B/2 * drug_C/4)] | 38.829 | 31 | 0.158
eta_group[B,I(drug_A/1 * drug_B/2 * drug_C/4)] | 27.264 | 31 | 0.659
eta_group[B,I(drug_A/1 * drug_B/2)] | 36.986 | 31 | 0.212
eta_group[B,I(drug_A/1 * drug_C/4)] | 28.256 | 31 | 0.608
eta_group[B,I(drug_B/2 * drug_C/4)] | 32.838 | 31 | 0.377
eta_group[C,I(drug_A/1 * drug_B/2 * drug_C/4)] | 21.888 | 31 | 0.886
eta_group[C,I(drug_A/1 * drug_B/2)] | 39.770 | 31 | 0.134
eta_group[C,I(drug_A/1 * drug_C/4)] | 40.499 | 31 | 0.118
eta_group[C,I(drug_B/2 * drug_C/4)] | 21.338 | 31 | 0.903
mu_eta[m[1]] | 33.357 | 31 | 0.353
mu_eta[m[2]] | 31.917 | 31 | 0.421
mu_eta[m[3]] | 29.562 | 31 | 0.540
mu_eta[m[4]] | 47.590 | 31 | 0.029
mu_log_beta[I(log(drug_A/1)),intercept] | 32.288 | 31 | 0.403
mu_log_beta[I(log(drug_A/1)),log_slope] | 38.899 | 31 | 0.156
mu_log_beta[I(log(drug_B/2)),intercept] | 22.259 | 31 | 0.875
mu_log_beta[I(log(drug_B/2)),log_slope] | 29.869 | 31 | 0.524
mu_log_beta[I(log(drug_C/4)),intercept] | 50.182 | 31 | 0.016
mu_log_beta[I(log(drug_C/4)),log_slope] | 32.858 | 31 | 0.376
tau_eta[stratum_1,m[1]] | 34.381 | 31 | 0.309
tau_eta[stratum_1,m[2]] | 29.786 | 31 | 0.528
tau_eta[stratum_1,m[3]] | 29.030 | 31 | 0.568
tau_eta[stratum_1,m[4]] | 25.248 | 31 | 0.757
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] | 25.990 | 31 | 0.722
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] | 34.573 | 31 | 0.301
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] | 37.050 | 31 | 0.210
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] | 44.774 | 31 | 0.052
tau_log_beta[stratum_1,I(log(drug_C/4)),tau_intercept] | 43.821 | 31 | 0.063
tau_log_beta[stratum_1,I(log(drug_C/4)),tau_log_slope] | 31.763 | 31 | 0.428

## Session Info

    ## R version 4.4.2 (2024-10-31)
    ## Platform: aarch64-apple-darwin20
    ## Running under: macOS Sequoia 15.3.1
    ##
    ## Matrix products: default
    ## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib
    ## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
    ##
    ## locale:
    ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
    ##
    ## time zone: Europe/Zurich
    ## tzcode source: internal
    ##
    ## attached base packages:
    ## [1] tools     stats     graphics  grDevices utils     datasets  methods
    ## [8] base
    ##
    ## other attached packages:
    ## [1] ggplot2_3.5.1    broom_1.0.6      tidyr_1.3.1      dplyr_1.1.4
    ## [5] assertthat_0.2.1 knitr_1.47       here_1.0.1
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] gtable_0.3.5      jsonlite_1.8.8    compiler_4.4.2    tidyselect_1.2.1
    ##  [5] jquerylib_0.1.4   scales_1.3.0      yaml_2.3.8        fastmap_1.2.0
    ##  [9] R6_2.5.1          generics_0.1.3    backports_1.5.0   tibble_3.2.1
    ## [13] munsell_0.5.1     rprojroot_2.0.4   bslib_0.7.0       pillar_1.9.0
    ## [17] rlang_1.1.4       utf8_1.2.4        cachem_1.1.0      xfun_0.45
    ## [21] sass_0.4.9        cli_3.6.2         withr_3.0.0       magrittr_2.0.3
    ## [25] digest_0.6.35     grid_4.4.2        lifecycle_1.0.4   vctrs_0.6.5
    ## [29] evaluate_0.24.0   glue_1.7.0        colorspace_2.1-0  fansi_1.0.6
    ## [33] rmarkdown_2.27    purrr_1.0.2       pkgconfig_2.0.3   htmltools_0.5.8.1