On bayesian data analysis and bayes factors

There’s a lot of buzz around bayesian data analysis (BDA) in psychology blogs, social media, and journal articles. For instance, in 2015 the APS Observer ran three columns dedicated to BDA in consecutive issues of the journal (Gallistel, 2015a, b, & c), and browsing the latest issues of Psychonomic Bulletin and Review gives an impression of increased interest in the topic.

Bayesian data analysis is more than bayes factors

However, it appears that there is an imbalance in what many beginning bayesian data analysts think about BDA. From casual observation and discussions, I’ve noticed a tendency for people to equate bayesian methods with computing bayes factors; that is, testing (usually null) hypotheses using bayesian model comparison.

I don’t have good data on people’s impressions of what BDA is, but here’s another anecdote. At a recent conference on Bayesian statistics Mark Andrews summarized his experiences teaching BDA workshops for social scientists. The talk was quite interesting, and what particularly picked my curiosity was his comment that for many–if not most–workshop participants, bayesian data analysis meant hypothesis testing with bayes factors (at 20 minutes in the linked video). (As an aside, he also noted that Stan has now superseded JAGS and BUGS as the preferred choice for a probabilistic modeling language. Go Stan!)

This imbalance or conflation of bayes factors and bayesian data analysis (if it is real and not merely bias in my observations!) is quite disappointing because BDA is a vast field of awesome methods, and bayes factors (BF) are only one thing that you can do with it1. In fact, many textbooks on BDA mention BFs only in footnotes (Gelman et al., 2013; McElreath, 2016). I’ve also written about BDA on my blog about a half-dozen times, but only once about bayes factors (Vuorre, 2016).

Further, it is really only this one method that people bicker over on social media: the bayes vs. frequentism argument usually turns into a p-values vs. bayes factors argument. The tedium of this argument (there really aren’t good reasons to prefer p-values) may even give out the impression that BDA is tedious and limited to model comparison and hypothesis testing problems. It isn’t! Bayes has benefits over frequentism that reach far beyond the p-vs-bf issue.

Here’s a practical example: It is well known that estimating generalized linear mixed models is kind of difficult. In fact, maximum likelihood methods routinely fail, especially when data are sparse of parameters are plenty (you’ve heard of multilevel models not converging, right? That’s the issue.). However, bayesian methods (via MCMC for example) usually have no problem estimating these models in situations when maximum likelihood fails! This benefit of bayes over frequentism (only the first thing to come to mind) doesn’t usually appear in the tedious p-vs-bf arguments, although one could argue that its practical implications are greater.

Reasons for thinking that BDA is BF

I suspect that one factor contributing to the apparent conflation of BDA and BFs is that there are vocal groups of psychological scientists doing interesting and important work promoting the use of Bayes Factors for hypothesis testing, and bayesian methods more generally (eg. Dienes, 2015; Ly, Verhagen, & Wagenmakers, 2015; Morey, Romeijn, & Rouder, 2016; Rouder, Morey, Verhagen, Province, & Wagenmakers, 2016). A quick reading of some of these texts may give the (false) impression that bayes factors are a larger portion of BDA than what they actually are. I think the same goes with the APS Observer columns mentioned above. To be completely clear, these papers are not about, nor do they say, that BDA is BF. I’m merely pointing out that in the psychological literature, (important!) papers about bayesian methods focusing on BF vastly overshadow in number papers that discuss other features of BDA.

But the real root cause for this conflation is probably the fetish-like desire of hypothesis tests in psychological science.

A modest proposal

If now is the time to move toward a new statistical paradigm in psychology, we could take the opportunity to emphasize not only bayesian hypothesis testing, but the importance of modeling, estimation and bayesian methods more generally. As Dr. Andrews noted, BDA could instead be thought of as flexible probabilistic modeling.

References

Dienes, Z. (2015). How Bayes factors change scientific practice. Journal of Mathematical Psychology. https://doi.org/10.1016/j.jmp.2015.10.003
Gallistel, C. R. (2015a). Bayes for Beginners 2: The Prior. APS Observer, 28(8). Retrieved from https://www.psychologicalscience.org/observer/bayes-for-beginners-2-the-prior
Gallistel, C. R. (2015b). Bayes for Beginners 3: The Prior in Probabilistic Inference. APS Observer, 28(9). Retrieved from https://www.psychologicalscience.org/observer/bayes-for-beginners-3-the-prior-in-probabilistic-inference
Gallistel, C. R. (2015c). Bayes for Beginners: Probability and Likelihood. APS Observer, 28(7). Retrieved from https://www.psychologicalscience.org/observer/bayes-for-beginners-probability-and-likelihood
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis, Third Edition. Boca Raton: Chapman and Hall/CRC.
Ly, A., Verhagen, J., & Wagenmakers, E.-J. (2015). Harold Jeffreys’s default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology. https://doi.org/10.1016/j.jmp.2015.06.004
McElreath, R. (2016). Statistical Rethinking: A Bayesian Course with Examples in R and Stan. CRC Press.
Morey, R. D., Romeijn, J.-W., & Rouder, J. N. (2016). The philosophy of Bayes factors and the quantification of statistical evidence. Journal of Mathematical Psychology. https://doi.org/10.1016/j.jmp.2015.11.001
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E.-J. (2016). Is There a Free Lunch in Inference? Topics in Cognitive Science, 8(3), 520–547. https://doi.org/10.1111/tops.12214
Vuorre, M. (2016, August 23). Statistical inference: Prix fixe or à la carte? Retrieved 4 May 2017, from https://mvuorre.github.io/post/2016/2016-08-23-free-lunch-in-inference/
Vuorre, M., & Bolger, N. (2017). Within-subject mediation analysis for experimental data in cognitive psychology and neuroscience. OSF Preprint. https://doi.org/http://dx.doi.org/10.17605/OSF.IO/6JHPF


  1. To be sure, I think a bayes factor can be a truly great tool when the competing models are well specified. I haven’t yet implemented a bayes factor method for my bayesian multilevel mediation package (Vuorre & Bolger, 2017), but might include one in the future. One difficulty here is specifying the competing models in a meaningful way.

Links to scientific practices workshop materials

We recently ran a Scientific Practices Workshop, and one of us later collected several links for follow-up materials for the interested. I thought the list of links was a fantastic source of materials, so I post it here:

Why this is important? (New)

Would you like to learn more about research integrity?

Would you like to learn more about reproducibility?

Would you like to pre-register a study?

Would you like to start using github?

Would you like to learn more about Reproducible Reports?

Or as Matti puts it: “Using RStudio and RMarkdown to achieve happiness and life satisfaction”:

Would you like start using a style guide for your code?

  • R: Hadley Wickham style guide (one, two)
  • Answer: Yes.

Would you like to join the Peer Review Openness Initiative or learn more about the Open Science Framework?

Would you like to watch scientists try to define a p-value?

Updated 2016-01-14

All Posts by Category or Tags.