Collaborative Research: Calibrated Hypothesis Testing

Scientific findings should come with error rates that mean what they say: among findings assigned a 5 percent chance of error, about 5 in 100 should turn out to be wrong. This standard, called calibration, underlies trusted probability claims from weather forecasting to machine learning (ML), but it is not yet a routine part of the statistical…