KSR Ltd
How to search for diagnostic test
accuracy (DTA) studies:
what works and what doesn’t!
Kate Misso,
Information Specialist Manager,
Kleijnen Systematic Reviews (KSR) Ltd, York.
Acknowledgements
• Anne Eisinga: Cochrane UK Support Unit
• Ruth Mitchell, Cochrane Renal Group
• Julie Glanville, York Health Economics Consortium (YHEC)
Eisinga A, Mitchell R, Glanville J. Searching for studies for systematic reviews of
diagnostic test accuracy: challenges in searching for DTA studies [PowerPoint
presentation]. Birmingham: UK Support Unit for Systematic Reviews of Diagnostic Test
Accuracy, University of Birmingham, 2012 [cited 11.5.12].
Eisinga A, Mitchell R, Glanville J. Searching for studies for systematic reviews of
diagnostic test accuracy: sources of DTA studies [PowerPoint presentation].
Birmingham: UK Support Unit for Systematic Reviews of Diagnostic Test Accuracy,
University of Birmingham, 2012 [cited 11.5.12].
• Marie Westwood, Kleijnen Systematic Reviews Ltd.
KSR Ltd
A search is a search is a search...
KSR Ltd
Intervention SR vs. DTA SR - What’s the same ?
• Clear research question
• PICO Structure & filters
• Planning terms
• Resources
• Quality assurance/peer review
• Managing references
• Documentation & reporting
Kleijnen Systematic Reviews Ltd , 6 Escrick Business Park, Riccall Road, Escrick, York, YO19 6FD, United Kingdom Tel: +44 (1904) 727980 www.systematic-reviews.com
A search is a search is a search...
What’s so different about DTA searching ?
Some principles are the same, but
• Research question
• Strategy structure & filters ...?
• Planning terms
• Resources
• Quality assurance/peer review
• Managing references
• Documentation & reporting
KSR Ltd
What‘s the problem ?
• Incomplete & inconsistent reporting by authors of DTA studies
• Incomplete & inconsistent indexing in databases
• Methodological search filter performance impaired by
inconsistencies in reporting & indexing
Searches for DTA reviews likely to retrieve large numbers of
records to scan for potentially relevant studies (Eisinga, 2008)
Searches need to be comprehensive and sensitive, but may lack
precision (NNR)
KSR Ltd
Where to start ?
KSR Ltd
Clearly defined answerable research question
e.g.
Effectiveness of new generation cardiac CT (NGCCT), using CT750
HD, Brilliance iCT, Somatom Definition Flash, or Aquilion ONE
for:
• diagnosis of clinically significant coronary artery disease (CAD)
in patients who are difficult or impossible to image accurately
using 64-slice CT technology.
• treatment planning in babies, infants, children and adults
diagnosed with complex congenital heart defects
How to start ?
• Understanding the question...
Intervention PICO = Pop, Intervention, Comparison, Outcomes (+ SD ?)
Diagnostic PICRO =
Population/condition (P)
Index test (I)
Comparator test (C)
Reference Standard (R)
Outcomes (O)
• Type of review ? Rapid review or comprehensive DTA SR ?
• Iterative approach, possibly step-wise searching
• Collaborative input from whole team & advice from clinical
experts
KSR Ltd
Structuring strategy
KSR Ltd
Applying PICRO to our NGCCT question
e.g.
P – Difficult to image: obese, irregular heart rate, stent/CABG ?
Target Condition – CAD or specific complex congenital heart defects
Index test – NGCCT High definition/Dual energy imaging, scanning,
named technologies
Reference standard - Invasive coronary angiography (ICA) for CAD
Reference standard – 64 slice CT for paediatric pop
O – detection or diagnosis or treatment planning
Multi-strand strategy structure (Glanville, 2008)
• May not need to combine all PICRO concepts
• Depending on research question, could search multi-strand
combination:
1 - Condition SUBJ/diagnostic SH
2 - Condition SUBJ AND Index test SUBJ
3 - Condition SUBJ AND Diagnostic Measure SUBJ
4 - Condition.ti,ab. AND Index test.ti,ab.
5 - 1 OR 2 OR 3 OR 4
SUBJ = subject heading
SH = subheading
.ti,ab. = title & abstract fields
KSR Ltd
Multi-strand strategy structure: example
1.
2.
3.
4.
Urinary Tract Infections/diagnosis
Urinary Tract Infections/ AND “Diagnostic Tests and Procedures”/
Urinary Tract Infections/ AND “Sensitivity and Specificity”/
Urinary Tract/ AND Bacteriuria/ AND “Diagnostic Tests and
Procedures”/
5. (Urinary tract adj3 infect$ adj3 diagnos$).ti,ab.
6. (urinary tract AND bacteriuria AND diagnos$).ti,ab.
7. 1 OR 2 OR 3 OR 4 OR 5 OR 6
(Glanville, 2008)
KSR Ltd
Identifying terms
•
•
•
•
•
•
•
•
Scoping document or briefing note ?
Medline MeSH or Embase EMTREE Thesauri
Manufacturers’ websites for brand names
Quick PubMed search
Any background papers (seed and target references)
Ask an expert
Wikipedia ??
Don’t re-invent the wheel...
KSR Ltd
“Here’s one someone else prepared earlier”
Check existing SRs & protocols
• PROSPERO (CRD)
• Cochrane Library - CDSR
• DARE & HTA
• NETSCC – completed and ongoing projects
• NICE – DAR programme
• MEDION - http://www.mediondatabase.nl/
• ARIF reviews database - http://www.arif.bham.ac.uk/
• Look for references, databases & relevant search terms
KSR Ltd
Lack of consistency in terminology & keywords
KSR Ltd
• Combination of text terms and subject indexing/thesaurus terms
• Often text term heavy with limited indexing for new technologies
• NGCCT DAR: all text terms, no relevant MeSH/Emtree
Where to search ?
• Embase, Medline, CENTRAL, DARE, NHS EED, HTA
• Other d/bs: SCI, SSCI, Biosis, PsycINFO, CINAHL, PASCAL,
IBSS.... ?
• Trials Registers: Clinicaltrials.gov, mRCT, WHO ICTRP
• Conference abstracts ?
• Scanning reference lists
• Citation searching & PubMed “related citations” function
• eTOCs/ejournals ?
• Guidelines & guidance ?
• DTA resources: MEDION & ARIF
• Coming soon! Cochrane Register of DTA studies (CRDTAS)
KSR Ltd
To filter or not to filter ?
KSR Ltd
• Routine use of DTA search filters should generally be avoided
• Performance varies across different topic areas (Cochrane DTA
Handbook, de Vet, 2008)
• Even most sensitive DTA filters miss relevant studies (Whiting, 2011)
• Typical RCT filter performance when combined with subject
search in MEDLINE is high 90s%
• Best DTA filter performance when combined with subject search
in MEDLINE was high 80s% (Eisinga, 2012)
Use of DTA filters
Can a DTA filter enhance a search?
• as part of multi-strand search approach can increase overall
sensitivity
• can be helpful in initial scoping search or to gather useful
candidate search terms for complex multi-strand strategy
Limiting a search, without a DTA filter:
• Restrict to human studies
• Exclude publication types e.g. Case reports
• Make use of proximity operators for phrase searching
• Think about field limits e.g. Cardiac.ti,ab.
KSR Ltd
What to expect ?
•
•
•
•
•
More sensitive search with large numbers retrieved to screen
Lower precision (NNR)
Iterative strategy development & searching
Multi-strand approach
Importance of supplementary methods (ref checking, conf
abs, handsearching etc)
NGCCT DAR: 24 included studies (26 publications)
13 from Medline
5 from Embase
5 from SCI (WoK)
2 conference abstracts
1 from reference checking
26 includes; 3986 records screened (0.65% relevance)
KSR Ltd
Searching for cost-effectiveness of DTAs
• Step-wise approach
e.g. NGCCT DAR
• First attempt @ comparator search = +30 000 refs !
• Focussed search with economics filter + NGCCT terms
-Medline, Embase, NHS EED, HEED & Paediatric Economic
Database Evaluations (PEDE)
• Guidelines & guidance search:
- NICE Guidance, www.guidelines.gov, International
Guidelines Library (GIN), TRIP database, HTA database
• Be flexible, test out options
KSR Ltd
Quality assurance & peer review
Feedback is essential !
Collaborative approach with review team
Ask another Info Specialist
Peer review of search strategies (PRESS-EBC, McGowan, 2010)
PRESS wiki (http://pressforum.pbworks.com)
Cochrane TSCs: Peer review of strategy as part of DTA
protocol peer review
• Additional (earlier) peer review on TSC PRESS site (via Archie)
• Analysis of missed references !
• Post-hoc analysis of database yield ?
•
•
•
•
•
•
KSR Ltd
Documentation & reporting
•
•
•
•
•
Documenting – internal record keeping
Reporting – transparent, comprehensive & reproducible
PRISMA statement: reporting for SRs & MAs
Word limit ? Present at least one strategy, preferably all
“All search strategies are available upon request from the
author”
KSR Ltd
Further developments to improve DTA retrieval
KSR Ltd
• Improved reporting of DTA studies – Standards for Reporting of
Diagnostic Accuracy (STARD) initiative (Bossuyt, 2003)
• Greater consistency in use of available DTA subject terms still needed
• Standardising terms for Index tests would improve searching (CRDTAS)
• Cochrane Register of Diagnostic Test Accuracy Studies (CRDTAS)
A single register of DTA studies (Cochrane Renal Group)
• Embase DTA indexing- new EMTREE term (from 1/01/2011)
Diagnostic test accuracy study/
7415 records indexed (as of 3.5.12)
Sources of support
• Cochrane Handbook for DTA reviews
http://srdta.cochrane.org/handbook-dta-reviews
• UK Support Unit (UKSU)
[email protected]
• Other Info Specialists ?
o Cochrane Information Retrieval Methods Group (IRMG)
o HTAi Information Retrieval Group (IRG)
o PRESS (wiki or Archie for TSCs)
KSR Ltd
KSR Ltd
Kate Misso
Information Specialist Manager
Kleijnen Systematic Reviews (KSR) Ltd.
01904 727989
[email protected]
www.systematic-reviews.com