Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.
Lloyd Sealy Library
John Jay College of Criminal Justice

Faculty Scholarship Resources: Caveats of Bibliometrics

Resources to assist faculty with understanding, locating, and collecting bibliometric measures related to their scholarly output. An overview of emerging alternative methods of research assessment.

Journal impact factor: limitations

An informative explanation of journal impact factor (with permission from its author, Dr. Nick Talley): Journal Impact: Can We Do Better? (2017).

Bohannan, J. (2016). Hate journal impact factors? New study gives you one more reasonScience.

Here's a brief explainer on impact factors by a John Jay librarian.

Understanding the Strengths and Weaknesses of Bibliometric Measures: A Bibliography

Many academic institutions partially rely on bibliometric measures to assess the research productivity and quality of their faculty. Yet, even though tenure and promotion decisions often refer to impact factor, citation counts, h-index and other metrics, assessment committees, as well as academics in general, continue to debate the usefulness and objectivity of bibliometrics.

The following bibliography lists just a few recent examples from the ongoing discussion about the suitability of bibliometrics for assessing the quality and impact of a researcher's work. Becoming familiar with some of bibliometrics' caveats may be helpful for tenure/promotion candidates and those evaluating their work.

Editorial (2005). Not so deep impact. Nature 435, 1003-1004.

Adler, J., & Taylor, P. (2008). Citation statistics: A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS)Berlin: International Mathematical Union.       

Alberts, B. (2013). Impact factor distortions. Science 340, 787.

Althouse, B. M., West, J. D., Bergstrom, C. T., & Bergstrom, T. (2009). Differences in impact factor across fields and over time. Journal Of The American Society For Information Science & Technology, 60(1), 27-34.

Bladek, M. (2013). DORA: San Francisco Declaration on Research Assessment. College & Research Libraries News 75(4), 191-196.

Bornmann, L., & Daniel, H. (2009). The state of h index research. Is the h index the ideal way to measure research performance?. EMBO Reports, 10(1), 2-6.

Bornmann, L., & Daniel, H. (2007). What do we know about the h index?. Journal Of The American Society For Information Science & Technology, 58(9), 1381-1385.

Browman, H. I., & Stergiou, K. I. (2008). Use and misuse of bibliometric indices in evaluating scholarly performance. Oldendorf/Luhe, Germany: Inter-Research Science Center.

Council for the Humanities, Arts & Social Sciences (CHASS). (2005). Measures of quality and impact of publicly funded humanities, arts and social sciences research. Canberra, ACT: Council for Humanities, Arts and Social Sciences (CHASS).

Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2012). Manipulating Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting.

Esposito, J. (2017). The Measure of All Things: Some Notes on CiteScore. The Scholarly Kitchen, Jan. 11.

Gorman, G. (2013). How to Leave Your Journal Impact Factor. Online Information Review, 37(2).

Howard, J. (2013) Rise of 'Altmetrics' Revives Questions About How to Measure Impact of Research. Chronicle of Higher Education, June 3.

Jarwal, S. D., Brion, A. M., & King, M. L. (2009). Measuring research quality using the journal impact factor, citations and 'Ranked Journals': blunt instruments or inspired metrics?. Journal Of Higher Education Policy & Management, 31(4), 289-300.

Johnston, R. (2009). Where there are data ... Quantifying the unquantifiable. Political Studies Review, 7(1), 50-62.

Pendlebury, D. (2009). The use and misuse of journal metrics and other citation indicators. Archivum Immunologiae et Therapiae Experimentalis, 57 (1), 1-11. 

Roemer, R. C., & Borchardt, R. (2012). From bibliometrics to altmetrics A changing scholarly landscape. College & Research Libraries News, 73(10), 596-600.

San Francisco Declaration on Research Assessment. Hosted by the American Society for Cell Biology.

Smith, K. M., E. Crookes, and P. A. Crookes. (2013). Measuring research ‘impact’ for academic promotion: issues from the literature. Journal of Higher Education Policy and Management 35(4): 410-420.

Varki, A. (2017). Scientific journals: Rename the impact factor. Nature, 548, 7668.
Williams, G. (2007). Should we ditch impact factors?. BMJ: British Medical Journal, 334(7593), 568-569.


Subject Guide