Subgroup Calibration Under Distribution Shift

Research Question

Design a post-hoc calibration method that remains reliable when subgroup composition shifts between calibration and test time.

Background

Many calibration methods look good on average but fail on protected or operational subgroups once the test distribution shifts. This task isolates that failure mode. The fixed pipeline trains a tabular classifier, then applies a user-defined calibration mapping on held-out calibration data before evaluation on shifted test data.

Classical baselines include:

• Temperature scaling: one global temperature for all samples
• Isotonic regression: non-parametric monotone calibration
• Beta calibration: a richer parametric mapping on probabilities
• Group-wise temperature scaling: separate temperatures per subgroup

Task

Modify the CalibrationMethod class in custom_subgroup_calibration.py. The fixed code loads data, creates a shifted split, trains the base classifier, and computes metrics. Your method only controls the post-hoc calibration mapping.

class CalibrationMethod:
    def fit(self, probs, labels, groups=None):
        ...

    def predict_proba(self, probs, groups=None):
        ...

Inputs are positive-class probabilities from the base classifier. groups contains subgroup IDs when available and may be ignored by group-agnostic methods.

Evaluation

This benchmark uses three lightweight tabular proxies that are already available in the current scikit-learn package setup. We would normally prefer Adult, ACSIncome, COMPAS, and Law School Admissions, but those require package-level data changes that are outside this task directory. To keep the benchmark runnable offline, we use cached scikit-learn datasets with similar calibration and subgroup-shift behavior:

• breast_cancer: binary classification on the scikit-learn breast cancer dataset
• california_housing: binary high-value/low-value decision built from California housing
• diabetes: binary high-risk/low-risk decision built from the diabetes target

For each dataset, the split is intentionally shifted:

• a domain score determines the held-out test tail
• subgroup labels are quartiles of a separate proxy feature
• calibration is fit on the source region and evaluated on the shifted region

Metrics

Lower is better for:

• worst_group_ece
• brier
• max_subgroup_gap

Higher is better for:

• subgroup_auroc

Notes

• The task is deliberately low compute and should run with a small tabular classifier.
• If you need the exact Adult/ACSIncome/COMPAS/Law School datasets, they should be added through a package-level data change, not inside this task directory.