Developing user-centric training in rigorous research: post-selection inference, publication bias, and critical evaluation of statistical claims.

Project Summary / Abstract
As scientific practice evolves in response to exponential increases in data volume, availability of
rapid computational statistics, and the so-called “reproducibility crisis”, researchers are
developing new methods for collecting and analyze data in rigorous and responsible fashion.
The aim of this proposal is to develop three training units for researchers in the neurosciences
that will bring learners up to speed on these developments, improving the rigor and quality of
their scientific research by deepening their understanding of the role of statistics in biomedical
research. Each unit, developed iteratively in a cycle of testing, evaluation, and revision will be
designed for online or classroom use suitable for diverse learning styles. Units will comprise a
series of short video segments and interactive exercises that lead learners in a process of
guided discovery and self-reflection as they move toward a set of well-specified learning goals.
Our units will teach neuroscientists to avoid common pitfalls in designing and analyzing data. In
the first unit, we address a set of easy-to-make mistakes wherein a researcher alters her plans
midway through the process of data analysis. The practice of HARKing—hypothesizing after the
results are known—involves testing hypotheses that are formulated after viewing research
outcomes. Outcome switching occurs when a study yields negative results based on the pre-
specified outcome measures, but other measures are reported instead. The Garden of Forking
Paths refers to the latitude that researchers have in shaping a statistical analysis as they go
along. The second unit addresses the problem of publication bias, which arises when authors
and journals prefer to publish positive results in favor of negative one, and can lead researchers
to reduplicate efforts or draw mistaken inferences from published data. The aim of this unit is to
make students aware of problem, teach them how to adjust when reading the literature, and
suggest strategies for avoiding publication bias in their own work. The third unit will train
students how to figure out whether when a statistical analysis rigorous and reliable. Students
will learn how to ask “Are the data appropriate what we want to learn?” “Is the choice of
statistical test reasonable?” “Are the inferences supported by the evidence?”
By developing this set of units, to be included in a broader neuroscience curriculum, we can
train a new generation of biomedical scientists who are well-equipped to work with the vast
datasets that are becoming available thanks to new research tools and technologies. These
scientists will be able to work more accurately, make new discoveries more efficiently, and
advance our knowledge in the health and life sciences at a faster rate than ever before.

Grant Number: 5UE5NS132949-03
NIH Institute/Center: NIH
Principal Investigator: Carl Bergstrom