Quantitative research methodology, which is characterized by positivism, measurement, and statistics,1, 2 has dominated the scientific literatures in many disciplines. Clinical research, in particular, often relies on quantitative data to describe, predict, and explain the complex phenomena at work.3, 4, 5, 6 Other disciplines sometimes turn to medical paradigms as learning models and benchmark standards for improving scientific rigor and assessing intellectual merit.7, 8
During the past few decades, quantitative research methodology has undergone many scientific breakthroughs. Paradigmic shifts and complex statistical models, coupled with technological advancements, open up a whole new horizon for quantitative researchers.9, 10, 11 Unfortunately, despite these exciting new developments, many fallacies and misconceptions continue to infiltrate the way quantitative researchers conceive, collect, and analyze their data.12, 13, 14 In fact, all too often, researchers associate “quantitative research” with “statistics,” failing to realize that a lot needs to happen before a statistical procedure can be applied to analyze the data. Without a rigorous research design, a sound sampling scheme, a reliable and valid instrument, and a meticulous data cleaning mechanism in place, no sophisticated statistic procedures can evade the Garbage In, Garbage Out fallacy.15
Unique in its multidisciplinary approach, this special issue brings together renowned and aspiring scholars from nursing, psychology, and education in their collective effort to shape the future of quantitative research methodology. By offering a platform for critically rethinking the current practices of quantitative research, this issue focuses on finding new ways of addressing major methodological challenges. It is my contention that with a little imagination and creativity, researchers from different disciplines can actually “talk” to each other because we are unified by the methodology we use to investigate a phenomenon. In a very real sense, we speak the same language of methodology that is boundary-free.
This special issue addresses eight major methodological challenges in the various stages of a research process. The eight articles are organized in such a way that they generally follow the logical sequence of the three main stages: research design, data collection, and data analysis.
The first article, “Potential Pitfalls in Collecting and Analyzing Longitudinal Data From Chronically Ill Populations” by Diane Holditch-Davis and Janet Levy, describes four common pitfalls in designing, collecting, and analyzing longitudinal data from preterm infants and their mothers: selection of time points, measurement, choosing appropriate statistical procedures, and missing values. Dr Holditch-Davis is Marcus Hobbs Distinguished Professor of Nursing and Associate Dean for Research Affairs at Duke University School of Nursing. Dr Levy is Assistant Research Professor at Duke University School of Nursing. They bring many years of quantitative research experience working with at-risk populations to this insightful study.
The second article, “A Primer on Propensity Score Analysis” by William R. Shadish and Peter M. Steiner, discusses the role that propensity score analysis can play in designing an intervention study when traditional alternatives such as randomized trial or regression discontinuity are impossible. Dr Shadish is Professor and Founding Faculty at University of California, Merced, where he is also Chair of Psychological Sciences. His scholarly works include the highly influential book Experimental and Quasi-Experimental Designs for Generalized Causal Inference16 with Tom D. Cook and Donald. T. Campbell. Dr Steiner is Senior Research Associate at the Institute for Policy Research, Northwestern University. Their contributions to the science of causal inference in human subject research are widely recognized in the social and behavioral fields.
The third article, “Why Sum Scores May Not Tell Us All About Test Takers” by Matthias von Davier, demonstrates the importance of going beyond the observed sum of positive or correct responses when analyzing responses from rating scales or tests and discusses how modern psychometric methods can be used to provide in-depth analysis about respondents. Dr von Davier is Principal Research Scientist at Center for Global Assessment, Educational Testing Service. Although his works have dealt with highly technical aspects of mathematical modeling of response patterns,17, 18 I think the readers will find his article in this special issue very accessible and informative.
The fourth article, “Data Cleaning Basics: Best Practices in Dealing With Extreme Scores” by Jason W. Osborne, discusses how extreme scores can seriously bias the results from quantitative research and summarizes best practices in dealing with them. Dr Osborne is Associate Professor of Educational Psychology at North Carolina State University and Senior Research Fellow of Friday Institute. His most recent contribution to the quantitative research field is his edited book Best Practices in Quantitative Methods,15 which has inspired this special issue.
The fifth article, “The Importance of Attending to Underlying Statistical Assumptions” by Hongwei Yang and Schuyler W. Huck, discusses how applied researchers have handled the underlying assumptions of statistical tools and how failure to do so may impact their findings. Dr Yang is Assistant Professor of Educational Policy Studies and Evaluation at University of Kentucky. Dr Huck is Distinguished Professor of Educational Psychology and Counseling and Chancellor's Teaching Scholar at University of Tennessee. Their analysis of how the authors of this journal have handled the statistical assumptions of their quantitative studies is highly illuminating and sobering. Dr Huck's recent book Statistical Misconceptions12 was also a source of inspiration for this special issue.
The sixth article, “Interpreting Significance: The Difference Between Statistical Significance, Effect Size, and Practical Importance” by Pav Kalinowski and Fiona Fidler, clarifies the difference between statistical significance, effect size, and practical importance and explains why confidence intervals should be used to disambiguate these concepts. Mr Kalinowski is a PhD candidate at School of Psychological Science, La Trobe University, in Australia. Dr Fidler is Postdoctoral Fellow at School of Psychological Science, La Trobe University, and Honorary Associate at Environmental Science, School of Botany, University of Melbourne. Dr Fidler's advocacy on “the new stats”19 provides a refreshing vision in the future of quantitative research.
The seventh article, “Disattenuation of Correlations due to Fallible Measurement” by myself, discusses the impact of failure to correct for measurement error on correlations and the controversies over adjusting for such attenuation in applied research. The last article, “Retrospective Statistical Power: Fallacies and Recommendations” also by myself, discusses the controversies associated with computing the retrospective power after a study has been concluded and using the observed effect size for calculating the ad hoc power. Both articles are based on my many years of teaching quantitative methods and my observations of potential pitfalls in using them.
I believe that using a sound methodology in a research endeavor is not only an intellectual pursuit but also a moral imperative.20 As Osborne21 elegantly puts it, researchers are “romantic fools” who believe in the magic of research (p. ix). The challenge to search for “the best practices” in quantitative research may sometimes seem insurmountable because of the intricacies embedded in the perplexing arrays of mathematical symbols and statistical formulas. Without recognizing the “unfounded lore” that permeates the literatures, quantitative researchers will continue to live in their fantasy world of “statistical and methodological myths and urban legends” (p. xv).14 As Huck12 contends, we need to “undo” (p. xiii) the misconceptions so deeply ingrained in our human intuition and research training. Without this conscientious effort, much of our research literatures will continue to be infiltrated with unsubstantiated findings and invalid conclusions.
It is my hope that this special issue will serve as a guidepost for those aspiring quantitative researchers who are navigating through these uncharted territories. Let the expedition begin.
FULL TEXT