Running a Closure Test

In this tutorial, we will demonstrate how to perform a closure test using Colibri. We will do so by performing a closure test with the Les Houches parametrisation model, which was implemented in this tutorial.

Step 1: runcard

In the colibri/examples/les_houches_example/runcards directory you will find an example runcard called lh_fit_closure_test.yaml, which looks like this:

################################################################################
# Les Houches model: Bayesian fit with UltraNest (closure test)
#
# Run with:      les_houches_exe runcards/lh_fit_closure_test.yaml
# Output folder (created in the current directory): lh_fit_closure_test/
# Evolve with:   evolve_fit lh_fit_closure_test
################################################################################

meta: 'Les Houches model: UltraNest nested sampling fit, level 0 closure test on a reduced DIS dataset.'

#######################
# Data and theory specs
#######################

dataset_inputs:
# DIS
- {dataset: SLAC_NC_NOTFIXED_P_EM-F2, variant: legacy_dw}
- {dataset: SLAC_NC_NOTFIXED_D_EM-F2, variant: legacy_dw}
- {dataset: BCDMS_NC_NOTFIXED_P_EM-F2, variant: legacy_dw}
- {dataset: BCDMS_NC_NOTFIXED_D_EM-F2, variant: legacy_dw}
# - {dataset: CHORUS_CC_NOTFIXED_PB_NU-SIGMARED, variant: legacy_dw}
# - {dataset: CHORUS_CC_NOTFIXED_PB_NB-SIGMARED, variant: legacy_dw}
# - {dataset: NUTEV_CC_NOTFIXED_FE_NU-SIGMARED, cfac: [MAS], variant: legacy_dw}
# - {dataset: NUTEV_CC_NOTFIXED_FE_NB-SIGMARED, cfac: [MAS], variant: legacy_dw}
# - {dataset: HERA_NC_318GEV_EM-SIGMARED, variant: legacy}
# - {dataset: HERA_NC_225GEV_EP-SIGMARED, variant: legacy}
# - {dataset: HERA_NC_251GEV_EP-SIGMARED, variant: legacy}
# - {dataset: HERA_NC_300GEV_EP-SIGMARED, variant: legacy}
# - {dataset: HERA_NC_318GEV_EP-SIGMARED, variant: legacy}
# - {dataset: HERA_CC_318GEV_EM-SIGMARED, variant: legacy}
# - {dataset: HERA_CC_318GEV_EP-SIGMARED, variant: legacy}
# - {dataset: HERA_NC_318GEV_EAVG_CHARM-SIGMARED, variant: legacy}
# - {dataset: HERA_NC_318GEV_EAVG_BOTTOM-SIGMARED, variant: legacy}
# - {dataset: NMC_NC_NOTFIXED_EM-F2, variant: legacy_dw}
# - {dataset: NMC_NC_NOTFIXED_P_EM-SIGMARED, variant: legacy}

theoryid: 40000000                     # The theory from which the predictions are drawn.
use_cuts: internal                     # The kinematic cuts to be applied to the data.

closure_test_level: 0                  # False: experimental data; 0: pseudodata without noise;
                                       # 1: pseudodata with noise.
closure_test_pdf: LH_PARAM_20250519    # PDF used to generate the pseudodata (ignored if
                                       # closure_test_level is False).

# flavour_mapping: ["\\Sigma", "g", "V", "V3"]  # Restrict the fit to these evolution-basis flavours.

#####################
# Loss function specs
#####################

positivity:                            # Positivity datasets, used in the positivity penalty.
  posdatasets:
  - {dataset: NNPDF_POS_2P24GEV_F2U, variant: None, maxlambda: 1e6}

positivity_penalty_settings:
  positivity_penalty: false
  alpha: 1e-7
  lambda_positivity: 0

integrability_settings:
  integrability: false

use_t0_covmat: true                    # Use the t0 covariance matrix in the chi2.
t0pdfset: NNPDF40_nnlo_as_01180        # The PDF used to build the t0 covariance matrix.

###################
# Methodology specs
###################

prior_settings:
  prior_distribution: uniform_parameter_prior
  prior_distribution_specs:
    bounds:
      alpha_gluon: [-0.1, 1]
      beta_gluon: [9, 13]
      alpha_up: [0.4, 0.9]
      beta_up: [3, 4.5]
      epsilon_up: [-3, 3]
      gamma_up: [1, 6]
      alpha_down: [1, 2]
      beta_down: [8, 12]
      epsilon_down: [-4.5, -3]
      gamma_down: [3.8, 5.8]
      norm_sigma: [0.1, 0.5]
      alpha_sigma: [-0.2, 0.1]
      beta_sigma: [1.2, 3]

ultranest_settings:
  ultranest_seed: 123456
  sampler_plot: true
  n_posterior_samples: 100             # Number of posterior samples (replicas) written out.
  posterior_resampling_seed: 123456
  ReactiveNS_settings:                 # Any ultranest.ReactiveNestedSampler option.
    vectorized: false
    ndraw_max: 500                     # Maximum number of points to propose at once.
  Run_settings:                        # Any ReactiveNestedSampler.run option.
    min_num_live_points: 200           # Minimum number of live points throughout the run.
    min_ess: 50                        # Target number of effective posterior samples.
    frac_remain: 0.3                   # Stop when this fraction of the integral is left.
  SliceSampler_settings:
    nsteps: 106                        # Accepted steps before a sample counts as independent.

actions_:
- run_ultranest_fit

Points to note:

  • Underlying law: We will be using the PDF grid LH_PARAM_20250519, which has been produced by computing the relevant PDFs for the Les Houches model with the best-fit values for the parameters, taken from Ref. A+05.

  • Fitting method: By choosing the action run_ultranest_fit, we are running a Bayesian fit. (You can read more about how to run a Bayesian fit in this tutorial.) If you instead want to perform a closure test with the monte carlo replica method, you can find out how to do in this tutorial.

  • Closure test level: To run a Level 1 closure test with this runcard, you can simply change closure_test_level: 0 to 1. To run a model-specific closure, test, see this section.

  • Flavour mapping: This should only be used in closure tests. It masks other flavours, such that closure tests can be run on desired flavours only.

Step 2: producing the fit

If you have enabled the executable for this model, you are ready to run a fit by following command from the colibri/les_houches_example directory:

les_houches_exe runcards/lh_fit_closure_test.yaml

A directory called lh_fit_closure_test, containing the output of the fit, should have been created. You can read more about the fit folders here.

You are now ready to evovle your fit, which you can learn more about in this section.