THE USES, BENEFITS AND PITFALLS OF BIBLIOMETRICS JULIA LANE AMERICAN INSTITUTES FOR RESEARCH TO DESCRIBE SCIENTIFIC COMMUNICATION OUTLINE • Context • Scientific Communication • Bibliometrics • Uses, Benefits and Pitfalls • Some other ideas and opportunities • What can be done SCIENTIFIC COMMUNICATION Knowledge • Creation • Transmission • Adoption OPEN ACCESS CONTEXT • Authors: of such articles, who will see their papers more read, more cited, and better integrated into the structure of science • Academic readers: in general at institutions that cannot afford the journal, or where the journal is out of scope • Researchers: at smaller institutions, where their library cannot afford the journal • Readers: in general, who may be interested in the subject matter • The general public: who will have the opportunity to see what scientific research is about • Taxpayers: who will see the results of the research they pay for • Patients: and those caring for them, who will be able to keep abreast of medical research Source: http://en.wikipedia.org/wiki/Open_access_journal accessed May 26, 2012 Scientific problems Priorities for scientific problems to be solved Public sector funding Funding for R&D Fund R&D Private sector funding Funding for research 1 Funding for research communication Perform the research Existing Scientific Knowledge New scientific knowledge 2 Communicate the results 3 Disseminated scientific knowledge Apply the Knowledge Publication Research funders Scientists TITLE:Do NODE: Funding for industrial development Publishers and infomediaries Readers Companies research, communicate and apply the results A0 Bo-Christer Björk, 2007 4 Better quality of life NUMBER: Government SCIENTIFIC COMMUNICATION • Is about people BIBLIOMETRICS: USES • …a set of methods to quantitatively analyze scientific and technological literature..used. • …in library and information sciences • …to explore the impact of • their field, • a set of researchers, or • a particular paper. • …. in quantitative research assessment exercises of academic output which is starting to threaten practice based research • Paraphrased from http://en.wikipedia.org/wiki/Bibliometrics BIBLIOMETRICS • Is about documents BENEFITS Benefits • Have focused attention on quantitative measures of impact • Attracted some smart people to think about hard problems • Identified some interesting patterns PITFALLS (ILLUSTRATIVE NOT EXHAUSTIVE) Scientific validity • Limited in its behavioral micro-foundations • Based on suspect scientific frame (unit of analysis, currency, coverage etc.) • Not generalizable or replicable Inferential validity • Generates spurious results • Subject to misuse and gaming Value for evaluation • Creates perverse incentive structure ILLUSTRATIVE CRITICISM Evaluators often rely on numerically–based shortcuts drawn from the closely related fields (Hood and Wilson, 2001) of bibliometrics and scientometrics — in particular, Thompson Scientific’s Journal Impact Factor (JIF). However, despite the popularity of this measure, it is slow (Brody and Harnad, 2005); narrow (Anderson, 2009); secretive and irreproducible (Rosner, et al., 2007); open to gaming (Falagas and Alexiou, 2008); and based on journals, not the articles they contain. Priem and Hemminger, 2010 ILLUSTRATIVE EXAMPLE WHY GETTING IT RIGHT MATTERS MEASURING COMMUNICATION: SOME IDEAS AND OPPORTUNITIES Focus on people (scientists) • How to start a movement Make use of scientific advances • New theories (graph theory; RCT) • New applications (social networks) • New ways of communicating knowledge • New application (graph oriented databases) • New data (natural language processing; computational linguistics) New opportunities => Potential for new science, new scientific field and theoretically grounded, metrics CONCEPTUAL FRAMEWORK Source: Ian Foster University of Chicago SCIENTIFIC ADVANCES • Graph Theory applied to social networks • E.g. Jason Owen Smith • dyadic measures of the strength of individual ties; • structural measures of cohesion in the overall network; and • node demographic measures that highlight the degree of heterogeneity of academic researchers • Randomized Controlled Trials • E.g John Willinsky • Physicians • Community Health Organizations NEW WAYS OF COMMUNICATING KNOWLEDGE Table 1: A partial list of popular Web 2.0 tools, and similar tools aimed at scholars. Description General–use application Scholarship–specific application Social bookmarking Delicious (http://delicious.com/) CiteULike(http://www.citeulike.org/, Connotea(http://www.connotea.org/) Social collection management iTunes(http://www.apple.com/itunes/) Mendeley(http://www.mendeley.com/, Zotero(http://www.zotero.org/) [reference managers] Digg(http://digg.com/), Social news/recommendations Reddit(http://www.reddit.com/), FriendFeed(http://friendfeed.com/) Faculty of 1000 (http://facultyof1000.com/), [similar, but curated] Publisher–hosted comment spaces (e.g., blog comments) Most Web 2.0 applications British Medical Journal http://www.bmj.com/), PloS(http://www.plos.org/), BioMed Central (http://www.biomedcentral.com/), Bioinformatics (Oxford University Press journal) (http://bioinformatics.oxfordjournals.org/) Microblogging Twitter(http://twitter.com/) Encyclopedia of Life(http://www.eol.org/), Scholarpedia(http://www.scholarpedia.org/), Citizendium(http://en.citizendium.org/) User–edited reference Wikipedia(http://www.wikipedia.org/) Blogs Wordpress.com(http://wordpress.com/), Research Blogging(http://researchblogging.org/), Blogger(https://www.blogger.com) Blogger(https://www.blogger.com) Social networks Data repositories Social video Facebook (http://www.facebook.com/), MySpace (http://www.myspace.com/), Orkut (http://www.orkut.com/) DBPedia(http://dbpedia.org/About) YouTube(http://www.youtube.com/), Vimeo (http://www.vimeo.com/) Nature Networks(http://network.nature.com/), VIVOweb (http://vivoweb.com/); GenBank(http://www.ncbi.nlm.nih.gov/genbank/) SciVee(http://www.scivee.tv/) Source M. Jensen (2007) Taraborelli (2008) Anderson (2009) Neylon and Wu (2009) Table 2: Calls for Web 2.0 metrics of scholarship. Suggested Web 2.0 sources for metrics Tags, “discussions in blogspace, comments in posts, reclarification, and continued discussion.” Social bookmarking: CiteULike, Connotea Twitter, blogs, video and “Wikipedia, or any of the special ‘–pedias’ out there” Zotero, Mendeley, CiteULike, Connotea, Faculty of 1000, article comments Main use Establishing scholars’ authority Augmenting or replacing peer review Broadening the scope of the JIF Filtering articles Norman in Cheverie, et al. (2009) “scholastic bookmarking, and tagging (e.g., the ‘Slashdot index’) … Tenure and promotion academic networks like LinkedIn” Patterson (2009) “… social bookmarks; blog coverage; and the Comments, Notes and ‘Star’ “[A]ssessing research articles on their ratings that have been made on the own merits …” article.” Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web by Jason Priem and Bradley M. Hemminger. First Monday, Volume 15, Number 7 - 5 July 2010 http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/ NEW DATA • Natural language processing (=> mine actual text and identify topics) • Information extraction to capture data on people and institutions WHAT CAN BE DONE • Build on current advances to develop and automatically extract de-duplicated crossreferenced database of • • • • • • papers (and references) Topics People Grants publication venues Institutions • Build a common data infrastructure • STAR METRICS WHAT CAN BE DONE TO CREATE NEW SCIENCE • Agree on handful of people based metrics • Engage social scientists to develop open and transparent data and standardized measures • Use them REFERENCES (PLUS THE NEW YORKER) • McCabe, Mark J. and Snyder, Christopher M., The Economics of Open-Access Journals (July 9, 2010). Available at SSRN: http://ssrn.com/abstract=914525 or http://dx.doi.org/10.2139/ssrn.914525 • McCabe, Mark J. and Snyder, Christopher M., Did Online Access to Journals Change the Economics Literature? (January 23, 2011). Available at SSRN: http://ssrn.com/abstract=1746243 or http://dx.doi.org/10.2139/ssrn.174624 • Philip M. Davis Open access, readership, citations: a randomized controlled trial of scientific journal publishing FASEB J July 2011 25:2129-2134; published ahead of print March 30, 2011, doi:10.1096/fj.11-183988 • Evans, James and Jacob Reimer (2009) “Open Access and Global Participation in Science,” Science 323: 1025. • Leydesdorff, L. (2008), Caveats for the use of citation indicators in research and journal evaluations. J. Am. Soc. Inf. Sci., 59: 278–287. doi: 10.1002/asi.20743 • Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web by Jason Priem and Bradley M. Hemminger. First Monday, Volume 15, Number 7 - 5 July 2010 http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/ • Kaye Husbands-Fealing, Julia Lane, Jack Marburger, Stephanie Shipp, and Bill Valdez The Handbook of Science of Science Policy,, Stanford University Press, 2011. • Julia Lane and Stefano Bertuzzi “Measuring the Results of Science Investments” Science, Volume 331, pages 678-680, February 11, 2011. • Julia Lane “Let’s Make Science Metrics More Scientific” Nature, Volume 464, pages 488– 489, March 25, 2010. • Julia Lane“Assessing the Impact of Science Funding” Science, Volume 324. no. 5932, pp. 1273 – 1275, 5 June 2009.