Bibliometrics in higher education: the impact on libraries Keith Webster University of Queensland, Australia Dr Berenika M. Webster CIBER, Univ. College London Outline • Bibliometrics and university libraries – Structure of disciplines – Collection management – Research evaluation • Bibliometric techniques • Sample projects Structure of disciplines • Domain analysis • Patterns of scholarship Collection management • Patterns of use • Core journals in discipline • International comparison Research evaluation • Approaches to research assessment: experiences from UK, NZ and Australia • Peer assessment and metrics • Bibliometrics – Assessment tools – Challenges of bibliometric approach – How to increase the quality and impact of published outputs What do governments want for their money? • Economic outcomes – increase wealth creation & prosperity – improve nation’s health, environment & quality of life • Innovation • R&D from private sector • Improved competitiveness • Less “curiosity-driven” activity Research Assessment Exercise (UK) • Started in mid-1980s to determine size of research funding from HEFCEs to universities. Early 1990s brought shift from quantity to quality • 60 subject panels examine, for quality, selected outputs of UK academics. Also, previous funding, evidence of esteem, PhD completions and res. environment are taken into consideration • Grades for submission units; correlates with bibliometrics measures (Charles Oppenheim’s research) • It will be scraped in 2008. New systems will be a hybrid of a “basket of metrics” and peer-review (???) • Changed behaviour of scholars – Increased volume of publications (salami slicing?) – Increased quality of outputs (numbers of 5 and 5* departments increase) Performance-Based Research Fund (PBRF) New Zealand • Introduced in early 2003 • 12 panels grades individuals in 41 subject categories • Portfolio has three components – Publications (4 best and the rest) – Peer esteem – Contribution to research environment • 2003 round produced lower scores than expected and great variability between subjects (from 4.7 in philosophy to 0.3 in nursing). 40% of submitted staff were judged research inactive (grade R) • Scores for institutions are calculated from these and used together with PhD completion and external funding metrics to calculate awards to institutions • 2006 round incorporates “breaks” for new researchers Research Quality Framework (RQF) - Australia • To be introduced in 2007 (?) • Quality includes the intrinsic merit of original research and academic impact. This relates to recognition of the originality of research by peers and its impact on the development of the same or related discipline areas within community of peers. Assessment: peer review and metrics • Impact or use of the original research outside of the peer community that will typically not be reported in traditional peer review literature. Relates to the recognition by qualified end-users that quality research has been successfully applied to achieve social, cultural, economic and/or environmental outcomes. Assessment: expert opinion and narratives or metrics? RQF evidence portfolio • Four “best” outputs (with rationale for selection) • List of all published outputs in the last six years • Statement of “early impact” as assessed by qualified end-users (description of outcome; identification of beneficiaries; metrics illustrating benefits; linkage between claimant and beneficiary) • Context statement for the research group Evaluation: peer vs. metrics • Evaluate/reward past performance • Measures short to medium term impacts • Established use for the evaluation of academic impact (within the group of peers) • No established/accepted procedures for evaluation of impact outside academe Peer review • “Traditional” assessment is an outputs-based peer-review – – – – – – – – – – a panel of experts “reads” submissions or outputs not always transparent (selection; evaluation criteria) subjective (concept of quality is very difficult to objectivise) relies on proxies (e.g. ranking/prestige of publishing journal; inst. affiliation of authors, etc.) composition of panels will affect their judgements lacks comparability between panels “punishes” innovative or multidisciplinary research “old boys” club disadvantaging young researchers assesses what already has been assessed by journals’ peer-review processes time and resource intensive Metrics • Range of unobtrusive and transparent measures can be: – Input-based • Income • Success rates – Output-based • Long-term impact (patents, IP, etc.) • Bibliometric measures (volume and impact/quality) Problems with metrics (1) • Input-based – Retrospective – Creates “Matthew effect” – Disadvantages younger researchers (without previous track record) – Undervalues research done “on the string” Problems with metrics (2) • Output based – – – – – – – – Citations as a measure of intellectual influence? Comparing like with like (benchmarking) Inadequate tools Social science and humanities (in search for new metrics) Research on the periphery (local citation indices) New models of communication (capturing it) Impacts on wider community (“tangible” outcomes) Lack of expertise in analysis and interpretation (education) Indicators of quality as measured using published outputs • Number of publications • Citation counts to these publications (adjusted for self-citations) • Citations per publication • Percentage of uncited papers • Impact factors (of publishing journals) • Diffusion factor (of citing journals) – profile of users of research (who, where, when and what) • “Impact factor” of a scholar - Hirsh index (h index) -what “window” should be used? 4, 5, 10 years? – – (numbers of papers with this number of citations). Your h index =75 if you wrote at least 75 papers with 75 citations each. Note: These should not be seen as “absolute” numbers but always seen in the context of the discipline, research type, institution profile, seniority of a researcher, etc. Compare like with like! • Applied research attracts fewer citations than basic research. • Differences in citation behaviour between disciplines (e.g. papers in organisational behaviour attract 5 times as many citations as papers in accounting). • Highest IF journal in immunology is Ann Rev Immun (IF 47.3) Mean for cat. 4.02; and in health care and services category is Milbank Q. (IF of 3.8). Mean for cat. 1.09. • Matthew effect. Benchmarking must be done using comparable variables! Tools available • Publication and citation data – – – – Web of Science SCOPUS Google Scholar Local/national databases • Other bibliometric indicators – Journal Citation Reports (JCR) – Other indicators databases (national, essential, university, institutional) – ISIHighlyCited.com WoS and Scopus: subject coverage (% of total records) WoS Social Sciences, 14 SCOPUS Arts & Humanities, 9 Social Sciences, 2 Physical Sciences, 25 Science, 77 Biological & Environmental Sciences, 13 Health & Life Sciences, 60 Google Scholar ? Jacso, 2005 Web of Science • Covers around 9,000 journal titles and 200 book • • • • • • series divided between SCI, SSCI and A&HCI. Electronic back files available to 1900 for SCI and mid- 50s for SSCI and mid-70s for A&HCI. Very good coverage of sciences; patchy on “softer” sciences, social sciences and arts and humanities. US and English-language biased. Full coverage of citations. Name disambiguation tool. Limited downloading options. Scopus • Positioning itself as an alternative to ISI • More journals from smaller publishers and open access (13,000 • • • • • • • • journal titles; 600 open access journals; 750 conf proceedings; 600 trade publications, 2.5 mil “quality” web pages) Source data back to 1960. Excellent for physical and biological sciences; poor for social sciences; does not cover humanities or arts. Better international coverage (60% of titles are non-US) Not much of a back file (e.g. citation data for the last decade only) Not “cover to cover” and not up to date Easy to use in searching for source publications; clumsy in searching cited publications. Citation tracker works up to 1000 records only. Limited downloading options. Google Scholar • Coverage and scope? • Inclusion criteria? • Very limited search options • No separate cited author search • Free! Taiwan in the three sources,1996-2005 70000 Scopus (1.23%) 60000 WoS (0.98%) Google Scholar 50000 40000 30000 20000 10000 0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Social sciences • Communication in social sciences – – – – Books National journals International journals Non-scholarly outputs (culture-creating role) – – – – – Chinese Social Sciences Citation Index Polish Sociology Citation Index Serbian Social Sciences Index Polish Citation Index for Humanities European Humanities Citation index • “International” tools inadequate • Creation of “national” and “regional” tools Science on the periphery • Poor coverage in international databases • Intl. benchmarking difficult • Impossible for local evaluations • Development of national sources of data • While volume is growing, citation rates lag behind Science on the periphery Figure 1: Polish scientific papers in SCI (in Polish and foreign journals), three-year running means and Polish SCI journals, 19801999 30 Polish SCI journals (right scale) 7000 Papers per year 6000 25 20 5000 4000 15 3000 10 Papers in foreign journals 2000 5 1000 Papers in Polish journals 0 journals 1981 1983 1985 0 1987 1989 1991 1993 1995 1997 Number of Polish journals in SCI 8000 Capturing new forms of communication • Use of online journals (deep log analysis) • Traffic on websites and downloads • In-links to websites Impact of research: example of biomedicine • Informing policies (citations on guidelines, govt. policy, development of medicines) • Building capacity (training; development) • Relationship between research and health outcomes and cost savings • Healthier workforce Citations on clinical guidelines • 47 Clinical Guidelines and 64 Health Technology Appraisals prepared by the UK’s National Institute for Clinical Excellence (NICE) were analysed for citations – cited 25% of UK papers (2.5 more than expected) – Majority were clinical papers – Median lag time between publication of res. paper and its citation was 3.3 years Relative commitment to research and burden of disease 0.8 Percent of biomed research Prostate cancer SE NL 0.6 US y = 0.1112x + 0.1645 R 2 = 0.087 0.4 IT CA UK DE ES FR JP 0.2 AU CH 0.0 0.0 0.5 1.0 1.5 Percent of all deaths 2.0 2.5 3.0 How to increase “quality” of your publications? • Publish in the right journals (prestige; importance to the discipline; impact factor vs. diffusion factor) • Publish in English • Write review articles • Engage in basic research • Become a journal editor (Lange, 1997) • Acquire a co-author (preferably from US or UK) • Get external funding (from different sources) • Make your outputs available in open access (own website, institutional and subject repositories) (Antelman, 2004; Harnard various) • “Advertise” your publications on listservs and discussion groups • Make and maintain professional/social contacts with others in your research area (Rowlands, 2000)