Probing the Limits of Atmospheric Predictability with a Deep Learning Framework

Weather forecasts are believed to be inherently limited by the growth of small errors present at the initial time, with all skill lost by about two weeks. This long-held belief derives from experiments with traditional models that represent the laws of physics for the atmosphere. Recently, forecast models based on machine learning (ML) have emerged with skill comparable to the physics-based models. Since the ML models do not solve to solve physical equations but only learn from data, they provide a new way to explore the limit of weather forecast skill. In this research, the two-week limit of forecast skill is tested by making small changes to the initial state using ML tools that reduce forecast errors far in the future. The main hypothesis is that the long-lead limit of weather forecast skill is much longer than currently believed, perhaps 3-4 weeks or longer.

Current estimates of the limit of atmospheric predictability, absent influences from slowly-varying boundary conditions, is about two weeks. The relative contribution of nonlinear upscale error growth from the mesoscale relative to error growth within synoptic to planetary scales is unclear, in part because it is estimated from physics-based models which parameterize small-scale processes that strongly influence predictability. Machine learning (ML) enables a transformative new approach to predictability research that warrants reconsideration of the two-week limit. ML models have forecast skill comparable to physics-based models at a fraction of the computational cost, and they are built using tools that can take derivatives of all components of the forecast. These tools are used with a nonlinear gradient-descent approach to find initial conditions that minimize long-lead forecasts errors. Two models, GraphCast and NeuralGCM, and two reanalysis products, ERA5 and MERRA2, are used to test the hypothesis that the deterministic atmospheric predictability limit is greater than 3 weeks. A second hypothesis is that errors are dominated by synoptic and planetary scales rather than the smaller mesoscale. In that case skillful weather forecasts with lead times longer than two weeks could be achievable in practice.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Award Number: 2501400
Principal Investigator: Gregory Hakim
Funds Obligated: $337,147
State: WA