Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals:: Science Publishing Group

Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals

Kory Floyd¹, Nathan T. Woo², Jeannette Maré¹, Kaylin L. Duncan³

¹Department of Communication, University of Arizona, Tucson, United States

²Department of Communication Studies, California State University, Sacramento, United States

³Department of Communication Studies, University of Alabama, Tuscaloosa, United States

Communication and Linguistics Studies, Volume 9, Issue 2, June 2023

Pages: 27-41

Received: May 22, 2023

Accepted: Jun. 08, 2023

Published: Jun. 15, 2023

DOI: 10.11648/j.cls.20230902.12

Downloads:

Views:

Download PDF: Download PDF

Export Citation: Export citation to RIS Export citation to BibTeX

Abstract

Background: The ability to draw accurate inferences from research depends heavily on the quality and representativeness of research samples. Research samples in the social sciences, including communication, are frequently criticized for being small and unrepresentative, yet there is substantial variation in sample characteristics. Objective: The objective of this project was to undertake a systematic examination of the characteristics of human samples used in communication research in major communication journals, in order to respond to the criticisms that such samples are small, underpowered, and lacking in external validity. Method: To ascertain the status of human samples in communication research, this project examined every empirical study published between 2010 and 2019 in four top communication journals—Communication Monographs, Communication Research, Human Communication Research, and Journal of Communication—that reported data from human subjects. The data set included 1,264 individual studies and a total sample size of 932,060 participants. Results and Conclusion: Sample sizes ranged from 10 to 57,847 participants, with an average of 740.12 participants, and were larger for non-experiments than experiments, quantitative than qualitative studies, and secondary than primary data analyses. Ninety-four countries were represented in the samples, although more than 70% of samples were recruited exclusively from the United States. Compared to U. S. demographics, such studies oversampled younger participants, female participants, and white participants.

Keywords

Samples, Representativeness, Statistical Power

To cite this article

Kory Floyd, Nathan T. Woo, Jeannette Maré, Kaylin L. Duncan. (2023). Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals. Communication and Linguistics Studies, 9(2), 27-41. https://doi.org/10.11648/j.cls.20230902.12

Copyright © 2023 Authors retain the copyright of this article.
This article is an open access article distributed under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

References

1.	Afifi, W. A., & Cornejo, M. (2020). #CommSoWEIRD: The question of sample representativeness in interpersonal communication research. In M. L. Doerfel & J. L. Gibbs (Eds.), Organizing inclusion: Moving diversity from demographics to communication processes (pp. 238–259). Routledge. https://doi.org/10.4324/9780429450495
2.	Altman, D. G., & Bland, J. M. (2005). Standard deviations and standard errors. British Medical Journal, 331 (7521), 903. https://doi.org/10.1136/bmj.331.7521.903
3.	Andrew, S. (2020, June 30). The US has 4% of the world’s population but 25% of its coronavirus cases. CNN Health. https://www.cnn.com/2020/06/30/health/us-coronavirus-toll-in-numbers-june-trnd/index.html
4.	Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63 (7), 602–614. https://doi.org/10.1037/0033-066X.63.7.602
5.	Bates, B. (2021). Making communication scholarship less WEIRD. Southern Communication Journal, 86 (1), 1–4. https://doi.org/10.1080/1041794X.2020.1861078
6.	Carminati, L. (2018). Generalizability in qualitative research: A tale of two traditions. Qualitative Health Research, 28 (13), 2094–2101. https://doi.org/10.1177/1049732318788379
7.	Centers for Disease Control and Prevention. (2022). National Health and Nutrition Examination Survey. National Center for Health Statistics. https://www.cdc.gov/nchs/nhanes/index.htm
8.	Chakravartty, P., Kuo, R., Grubbs, V., & McIlwain, C. (2018). #CommunicationSoWhite. Journal of Communication, 68, 254–266. https://doi.org/10.1093/joc/jqy003
9.	Coleman, R. (2018). Designing experiments for the social sciences. Sage.
10.	Coles, R. L. (2002). Black single fathers: Choosing to parent full-time. Journal of Contemporary Ethnography, 31 (4), 411–439. https://doi.org/10.1177/0891241602031004002
11.	Combs, J. G. (2010). Big samples and small effects: Let’s not trade relevance and rigor for power. Academy of Management Journal, 53 (1), 9–13. https://doi.org/10.5465/amj.2010.48036305
12.	Davern, M. E. (2008). Representative sample. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (pp. 720–722). Sage.
13.	Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013). Influence of trial sample size on treatment effect estimates: Meta-epidemiological study. British Medical Journal, 346. https://doi.org/10.1136/bmj.f2304
14.	De Winter, J. C. (2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research, and Evaluation, 18 (1), Article 10. https://doi.org/10.7275/e4r6-dj05
15.	Emerson, R. W. (2015). Convenience sampling, random sampling, and snowball sampling: How does sampling affect the validity of research? Journal of Visual Impairment & Blindness, 109 (2), 164–168. https://doi.org/10.1177/0145482X151900215
16.	Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
17.	Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26 (3), 213–224. https://doi.org/10.1002/bdm.1753
18.	Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13 (3), e1002106. https://doi.org/10.1371/journal.pbio.1002106
19.	Hedt, B. L., & Pagano, M. (2011). Health indicators: Eliminating bias from convenience sampling estimators. Statistics in Medicine, 30 (5), 560–568. https://doi.org/10.1002/sim.3920
20.	Hendriks, T., Warren, M. A., Schotanus-Dijkstra, M., Hassankhan, A., Graafsma, T. Bohlmeijer, E., & de Jong, J. (2019). How WEIRD are positive psychology interventions? A bibliometric analysis of randomized controlled trials on the science of well-being. The Journal of Positive Psychology, 14 (4), 489–501. https://doi.org/10.1080/17439760.2018.1484941
21.	Hennink, M., & Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science & Medicine, 292, 114523. https://doi.org/10.1016/j.socscimed.2021.114523
22.	Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33 (2/3), 61–83. https://doi.org/10.1017/S0140525X0999152X
23.	International Monetary Fund (2018). IMF annual report. https://www.imf.org/external/pubs/ft/ar/2018/eng/
24.	Johnston, M. P. (2017). Secondary data analysis: A method of which the time has come. Qualitative and Quantitative Methods in Libraries, 3 (3), 619–626.
25.	Kaufmann, R., & Tatum, N. T, (2017). Do we know what we think we know? On the importance of replication in instructional communication research. Communication Education, 66 (4), 479–481. https://doi.org/10.1080/03634523.2017.1342849
26.	Kraemer, H. C., & Blasey, C. (2016). How many subjects: Statistical power analysis in research. Sage.
27.	Lannutti, P. J., Butauski, M., Rubinsky, V., & Hudak, N. (2021). Setting the agenda: LGBTQ+ and SGM family communication. Journal of Family Communication, 21 (2), 138–143. https://doi.org/10.1080/15267431.2021.1912048
28.	Levine, T. R., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings. Communication Monographs, 76 (3), 286–302. https://doi.org/10.1080/03637750903074685
29.	Maleki, F., Ovens, K., McQuillan, I., & Kusalik, A. J. (2019). Size matters: How sample size affects the reproducibility and specificity of gene set analysis. Human Genomics, 13 (Suppl 1), 42. https://doi.org/10.1086/s40246-019-0226-2
30.	Martino, M. L., Lemmo, D., & Gargiulo, A. (2021). A review of psychological impact of breast cancer in women below 50 years old. Health Care for Women International, 42 (7–9), 1066–1085. https://doi.org/10.1080/07399332.2021.1901901
31.	McEwan, B., Carpenter, C. J., & Westerman, D. (2018). On replication in communication science. Communication Studies, 69 (3), 235–241. https://doi.org/10.1080/10510974.2018.1464938
32.	Minarik, T., Berger, B., Althaus, L., Bader, V., Biebl, B., Brotzeller, F., Fusban, T., Hegemann, J., Jesteadt, L., Kalweit, L., Leitner, M., Linke, F., Nabielska, N., Reiter, T., Schmitt, D., Spraetz, A., & Sauseng, P. (2016). The importance of sample size for reproducibility of tDCS effects. Frontiers in Human Neuroscience, 10, 453. https://doi.org/10.3389/fnhum.2016.00453
33.	Moher, D., Dulberg, C. S., & Wells, G. A. (1994). Statistical power, sample size, and their reporting in randomized controlled trials. Journal of the American Medical Association, 272 (2), 122–124. https://doi.org/10.1001/jama.1994.03520020048013
34.	Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6 (7), Article e1000097. https://doi.org/10.1371/journal/pmed1000097
35.	Murphy, K. R., & Aguinis, H. (2019). HARKing: How badly can cherry-picking and question trolling produce bias in published results? Journal of Business and Psychology, 34 (1), 1–17. https://doi.org/10.1007/s10869-017-9524-7
36.	Muthukrisha, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., & Thue, B. (2020). Beyond Western, educated, industrial, rich, and democratic (WEIRD) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological Science, 31 (6), 678–701. https://doi.org/10.1177/0956797620916782
37.	Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716. https://doi.org/10.1126/science.aac4716
38.	Parsons, A., Turner, R., Ingleton, H., Dubrow-Marshall, L., Kefalogianni, M., Omylinska-Thurston, J., Karkou, V., & Thurston, S. (2021). Flowing towards freedom with multimodal creative therapy: The healing power of therapeutic arts for ex cult-members. The Arts in Psychotherapy, 72, Article 101743. https://doi.org/10.1016/j.aip.2020.101743
39.	Peterson, R. A., & Merunka, D. R. (2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67 (5), 1035–1041. https://doi.org/10.1016/j.busres.2013.08.010
40.	Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo Sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences, 1145 (45), 11401–11405. https://doi.org/10.1073/pnas.1721165115
41.	Radler, B. T. (2014). The Midlife in the United States (MIDUS) series: A national longitudinal study of health and well-being. Open Health Data, 2 (1), e3. https://doi.org/10.5334/ohd.ai
42.	Rains, S. A., Levine, T. R., & Weber, R. (2018). Sixty years of quantitative communication research summarized: Lessons from 149 meta-analyses. Annals of the International Communication Association, 42 (2), 105–124. https://doi.org/10.1080/23808985.2018.1446350
43.	Roser, M., & Ortiz-Ospina, E. (2013, May 25). Global extreme poverty. Our World in Data. https://ourworldindata.org/extreme-poverty
44.	Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 2 (2), 107–114. https://doi.org/10.1177/2515245919838781
45.	Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96 (5), 1055–1064. https://doi.org/10.1037/a0023322
46.	Soliz, J., & Phillips, K. E. (2018). Toward a more expansive understanding of family communication: Considerations for inclusion of ethnic-racial and global diversity. Journal of Family Communication, 18 (1), 5–12. https://doi.org/10.1080/15267431.2017.1399890
47.	Statista. (2021, January 20). Median age of the resident population of the United States from 1960 to 2019. https://www.statista.com/statistics/241494/median-age-of-the-us-population/
48.	Thalmayer, A. G., Toscanelli, C., & Arnett, J. J. (2021). The neglected 95% revisited: Is American psychology becoming less American? American Psychologist, 76 (1), 116–129. https://doi.org/10.1037/amp0000622
49.	The Economist Intelligence Unit. (2020). Democracy index 2020 in sickness and in health? https://pages.eiu.com/rs/753-RIQ-438/images/democracy-index-2020
50.	UNESCO (2021). Access. Global Education Monitoring Report. https://www.education-progress.org/en/articles/access
51.	United Nations. (2019). Population. United Nations. https://www.un.org/en/global-issues/population
52.	U.S. Census Bureau. (2019, December 3). Census Bureau reports nearly 77 million students enrolled in U.S. schools. https://www.census.gov/newsroom/press-releases/2019/school-enrollment.html#:~:text=Enrolled%20students%20make%20up%2025%25%20of%20the%20U.S.,Hispanic%2C%2015%25%20black%20alone%20and%205%25%20Asian%20alone.
53.	U. S. Census Bureau. (2019). Quick Facts/United States. https://www.census.gov/quickfacts/fact/table/US/PST045219
54.	U. S. Census Bureau. (2020, December 15). Census Bureau releases 2020 demographic analysis estimates. https://www.census.gov/newsroom/press-releases/2020/2020-demographic-analysis-estimates.html
55.	U. S. Census Bureau. (2022). Datasets. https://www.census.gov/data/datasets.html
56.	Wang, H. J., Kjellstrand, C. M., Cockfield, S. M., & Solez, K. (1998). On the influence of sample size on the prognostic accuracy and reproducibility of renal transplant biopsy. Nephrology Dialysis Transplantation, 13 (1), 165–172. https://doi.org/10.1093/ndt/13.1.165
57.	Zhang, Z., Xu, X., & Ni, H. (2013). Small studies may overestimate the effect sizes in critical care meta-analyses: A meta-epidemiological study. Critical Care, 17 (1), R2. https://doi.org/10.1186/cc11919