Animal cognition: a reanalysis highlights studies’ statistical fragility

A meta-analysis brings studies together, but does not make their weaknesses disappear. A preregistered reanalysis of 28 publications on animal cognition finds low statistical power in primary studies. The findings mainly concern experiments on learning and memory in rodents, not animal cognition as a whole.

Illustration of findings from rodent cognition studies pooled in a meta-analysis.
MindsetEditorial illustration, not study data.

An analysis of previously published syntheses

The researchers conducted a second-order meta-analysis: their starting point was not a new animal experiment, but existing meta-analyses. Their dataset includes 28 publications, 48 eligible models and 5,740 effect-size observations from primary studies. That last figure therefore represents neither a count of animals nor necessarily that many distinct studies.

They assessed statistical power, as well as errors in the magnitude or direction of estimated effects. Here, power means an analysis’s ability to detect an effect under certain assumptions. It depends, among other things, on the magnitude of the effect assumed to be real.

Low power, but conditional on assumptions

At the primary-study level, estimated power reached 17.4% using uncorrected pooled effects, 9.0% after a regression-based bias correction and 13.4% using another method in a sensitivity analysis. All three estimates were low, but they are not measurements independent of the assumptions used.

When the researchers assumed larger effects, set independently of the previous estimates, power increased. It would therefore be incorrect to read these percentages as the proportion of reliable studies or true conclusions.

Estimates of errors in effect magnitude and direction also varied depending on the assumed effect and the correction method. At the level of the meta-analyses themselves, results were generally more favorable, while remaining sensitive to analytical choices.

What this reanalysis changes about how we read the findings

This study encourages looking beyond the “meta-analysis” label. Pooling results can improve an analysis without making its conclusions independent of the quality of the included studies or the statistical methods chosen.

The scope of the findings remains limited: the dataset is concentrated on learning and memory experiments in rodents. It does not support the conclusion that all animal cognition research has the same weaknesses, let alone any inference of an effect in humans.

The authors advocate improvements in the design and reporting of the studies represented, along with broader coverage of species and cognitive domains. This methodological work does not test any intervention intended to improve memory.

Limitations to keep in mind

Only the PubMed abstract was consulted, not the full text. The details of the included studies and analyses therefore cannot be examined here. Funding and conflicts of interest were not checked. The findings depend on assumed effects and mainly concern experimental work in rodents.

What we can take away

A meta-analysis is not an automatic guarantee of robustness. In this animal cognition dataset, the low power of primary studies and sensitivity to methods call for measured conclusions.

Sources and provenance

Partager cet article

X f in

An error or a source to report? Contact the editorial team.