PANEL: Experimental design and the reproducibility crisis: pitfalls, power, and practical best practice

The justification of sample size is one of the hardest sections of a proposal an IACUC (ethical review board) has to assess, yet is arguably one of the most important (1). This is a key opportunity for the IACUC to assess whether the experiment is well designed, whether animals are being used in the most efficient and productive way possible (2), and even whether the experiment is worth pursuing in its proposed design (3).

Poor experimental design and poor statistics have two consequences of concern to an IACUC: 1) they increase sample size; and 2) they generate false positives which may then commit animals to further unnecessary experiments. To provide perspective on this issue, consider the fact that only 10% of compounds that ‘work’ in an animal model actually ‘work’ in human trials (4) – that is the false positive rate in terms of human outcomes of animal models is 90%. Poor experimental design plays a huge role in this highly wasteful (and invasive) use of animals (5-7). Similarly, although estimates vary, a very large proportion of published papers contain basic statistical errors (8). For example 38% of papers in a recent review of work published in Nature Medicine contained basic statistical errors (9). Similarly a review of top-tier neuroscience journals revealed basic errors in 50% of behavioral neuroscience papers, and 100% of molecular neuroscience papers surveyed (10).

Learning Objectives:

1. This talk will go over in everyday terms, the reasons why good experimental design is so important, what it should look like in an IACUC proposal, how to identify reasonable and unreasonable sample sizes, and how to identify when improved experimental design may allow a PI to perform the same work better with fewer animals.

2. This talk will be user friendly (it won’t use equations), and the goal is to provide a set of rules of thumb for quick assessment of experimental design and sample size so that IACUC members can triage out protocols where they might want to seek input from a statistician.