Chapter 6: Qualitative Evaluation*
Study (year) . | Aim . | Method . | Setting . | Participants . |
---|---|---|---|---|
Ahmed . (2019) | To explore the views of clinician–scientists and quality improvement experts regarding proposed domains of PCC, and to gain an understanding of current practices and opportunities for measurement of PCC at a healthcare system level. | Semi-structured interviews ( = 16) | Canada, USA, UK | Clinician–scientists ( = 4), Quality improvement experts ( = 12) |
Benn . (2015) Chapter 6: Qualitative Evaluation* | To conduct a quasi-experimental evaluation of the feedback initiative and its effect on quality of anaesthetic care and perioperative efficiency. | Interviews ( = 35) | Teaching hospital in London, UK | Consultant anaesthetists ( = 24), surgical nursing leads ( = 6), perioperative service leads ( = 5) |
Breidenbach . (2021) | To identify factors that inhibit of facilitate the usage of PROs for clinical decision-making and monitoring patients in existing structures for oncological care, certified colorectal cancer centres in Germany. | Semi-structured interviews ( = 12) | Cancer centres participating in EDIUM study in Germany | Physicians ( = 7), psycho-oncologist ( = 1), nurses ( = 3), physician assistant ( = 1) |
D’Lima . (2017)* | To report the experience of anaesthetists participating in a long-term initiative to provide comprehensive personalized feedback to consultants on patient-reported quality of recovery indicators in a large London teaching hospital. | Semi-structured interviews ( = 21) | Teaching hospital in London, UK | Consultant anaesthetists ( = 13), surgical nursing leads ( = 6), theatre manager ( = 1), clinical coordinator for recovery ( = 1) |
Exworthy . (2003) | To review qualitative findings from an empirical study within one English primary care group on the response to a set of clinical performance indicators relating to general practitioners in terms of the effect upon their clinical autonomy. | Semi-structured interviews ( = 52) | Primary care group in southern England, UK | GPs ( = 29), practice nurses ( = 12), practice managers ( = 11) |
Gagliardi . (2008) | To explore patient, nurse, physician, and manager preferences for cancer care quality indicators. | Interviews ( = 30) | Two teaching hospitals, Canada | Surgeons ( = 2), radiation oncologists ( = 2), medical oncologist ( = 1), nurses ( = 5), managers ( = 5), patients ( = 15) |
Gill . (2012) | To explore the perspectives of general practitioners on the introduction of child-specific quality markers to the UK’s Quality Outcomes Framework. | Semi-structured interviews ( = 20) | Five Primary Care Trusts, England | GPs ( = 20) |
Gray . (2018) | To explore the role that metrics and measurement play in a wide-reaching ‘Lean’-based continuous quality improvement effort carried out in the primary care departments of a large, ambulatory care healthcare organization. | Semi-structured interviews ( = 130) | Large, multispecialty, ambulatory care organization, USA | Primary care physicians (# of participants not disclosed) |
Hicks . (2021) | To identify all available patient-reported outcome measures relevant to diseases treated by vascular surgeons and to evaluate vascular surgeon perceptions, barriers to widespread implementation, and concerns regarding PROs. | Focus groups (# of focus groups not disclosed) | Society for Vascular Surgery, USA | Society for Vascular Surgery members (# of participants not disclosed) |
Litvin . (2015) | To systematically solicit recommendations from Meaningful Use exemplars to inform Stage 3 Meaningful Use clinical quality measure requirements. | Focus groups ( = 3) | A national Electronic Health Record-based primary care practice-based research network, USA | General internists ( = 5) internal medicine/paediatric physicians ( = 2), family medicine physicians ( = 16) |
Maxwell . (2002) | To investigate the acceptability among general practitioners of a patient-completed post-consultation measure of outcome and its use in conjunction with two further quality indicators: time spent in consultation and patients reporting knowing the doctor well. | Focus groups ( = 7) | Oxford, Coventry, London, and Edinburgh, UK | GPs ( = 46) |
Rasooly . (2022) | To understand the current state of quality and performance measurement in primary diabetes care, and the facilitators and barriers to their implementation. | Interviews ( = 26) | Tertiary hospitals CHCs in Shanghai, China | Patients ( = 12), family doctors ( = 3), endocrinologists ( = 2), CHC managers ( = 4), policymakers ( = 5) |
Van den Heuvel . (2010) | To describe and explore the views of German general practitioners on the clinical indicators of the Quality and Outcomes Framework | Focus groups ( = 7) | North-western part of Germany | GPs ( = 54) |
Wilkinson . (2000) | To investigate reactions to the use of evidence-based cardiovascular and stroke performance indicators within one primary care group. | Semi-structured interviews ( = 29) | Fifteen practices from a primary care group in southern England | GPs ( = 29) |
CHC, community healthcare centre, GP, General Practitioners, PCC, patient-centred care, PRO, patient-reported outcome, *Articles report on the same study.
Articles report on the same study but retained to incorporate potential differing interpretations of the data.
The articles used qualitative methodology (either interviews or focus groups). The CASP quality assessment revealed that most included studies met most quality criteria ( Supplementary Appendix S2 ). All studies provided clear description of findings. However, there were some methodological limitations with several studies. Nine of 14 articles did not discuss ethical issues with many failing to report ethics approval of the data collection. The relationship between researchers and participants was not adequately discussed in 11 studies, 2 articles did not specify the research aim, and several others failed to report the participant recruitment strategy and only included a limited discussion of how data were analysed.
Data from included studies addressed the first three objectives but no articles addressed the fourth objective.The themes for each objective are described below, and summarized in Table 2 .
Objectives and themes.
Objective . | Themes . |
---|---|
What is the role of clinical indicators in supporting quality improvement? | Show where changes need to be made |
Motivate physicians to improve quality of care | |
Increase physicians’ accountability | |
Can encourage myopic quality improvement | |
Should be used by physicians, not government or the public | |
Should not be used punitively | |
What is needed to strengthen the ability of indicators to drive improvements in quality? | Support and participation of physicians in their development |
Recording data should be straightforward | |
Feedback delivered in a way that is helpful for physicians | |
Availability of sufficient resource for quality improvement | |
Quality improvement requires working together | |
Incentives have advantages and disadvantages | |
Key attributes of effective indicators | Target the most important areas for quality improvement |
Consistent with good medical care | |
Within physicians’ control | |
Reliable | |
Consider patient-reported measures alongside |
Objective . | Themes . |
---|---|
What is the role of clinical indicators in supporting quality improvement? | Show where changes need to be made |
Motivate physicians to improve quality of care | |
Increase physicians’ accountability | |
Can encourage myopic quality improvement | |
Should be used by physicians, not government or the public | |
Should not be used punitively | |
What is needed to strengthen the ability of indicators to drive improvements in quality? | Support and participation of physicians in their development |
Recording data should be straightforward | |
Feedback delivered in a way that is helpful for physicians | |
Availability of sufficient resource for quality improvement | |
Quality improvement requires working together | |
Incentives have advantages and disadvantages | |
Key attributes of effective indicators | Target the most important areas for quality improvement |
Consistent with good medical care | |
Within physicians’ control | |
Reliable | |
Consider patient-reported measures alongside |
Shows where changes need to be made.
Physicians noted that a key role of clinical indicators was their ability to illuminate specific areas of care requiring change. In many cases, physicians stated that it was only through clinical indicators that they received regular feedback on the quality of their care. Physicians appreciated the objective assessment of quality that clinical indicators provided, as opposed to intuiting where care may require improvement. Physicians also thought that clinical indicators could facilitate up-to-date, evidence-based care, provided that the indicators were based on best practice.
Physicians commented on two ways in which clinical indicators motivated efforts to improve quality of care: first, seeing the clinical indicator feedback was often a prompt for physicians to take action on quality improvement. Physicians expressed that it was difficult to ignore this type of objective feedback. Second, clinical indicator feedback showing improvements in care motivated physicians, as it demonstrated tangible evidence of how quality improvement could translate into improved outcomes. Many physicians also thought that engaging in quality improvement was part of being a ‘good’ physician.
Physicians thought that measuring quality using clinical indicators would make them more accountable for the quality of their care. Some were concerned however that clinical indicators could be used by their organization for performance management, and they feared a loss of autonomy in their practice.
Physicians were concerned that clinical indicators could lead to a myopic view and produce unintended consequences. They commented that many of the ‘softer’ aspects of quality were difficult to quantify using indicators and risked being side-lined in favour of areas of care more easily quantified. Physicians were concerned that using clinical indicators may distract them from providing more holistic, patient-centred care. Overall, physicians stressed that clinical indicators should be a means to good care, not an end in themselves.
Physicians stressed that clinical indicators should be used by physicians for the purpose of quality improvement, not by government or the public. They emphasized the potential for indicators to be misinterpreted by those outside the profession and were worried about being held accountable for measures they could not influence. Physicians also highlighted the tensions between their own priorities for quality improvement and the priorities of government or their organization. They thought that government or organization management were more likely to prioritize productivity and efficiency over the quality of patient care, and were worried that clinical indicators could entrench these priorities.
Physicians thought that clinical indicators could either be employed in a ‘soft’ manner to encourage quality improvement or a ‘hard’ manner where poor performance would be criticized or punished. They stressed that this punitive approach would only isolate physicians and was unlikely to improve the quality of care.
Support and participation of physicians in their development.
Physicians thought that clinical indicators were more likely to drive improvements in quality if they had the support of clinicians. Physicians were more inclined to use the indicators to make changes to their practice if they understood their purpose and agreed with the measures. They suggested that one way of ensuring their buy-in was to involve them in the development of clinical indicators.
Physicians thought that recording data for clinical indicators could lead to an unmanageable increase in their workload and may require additional support staff. They suggested that recording indicator data should be integrated into their workflow and automated where possible.
Physicians had several suggestions for useful ways to deliver clinical indicator feedback. They wanted indicator feedback delivered in a manner that was visually appealing and easy to interpret—most suggested the use of charts rather than tables. Comparison feedback between departments, practices, or individual physicians was also considered useful. Physicians found it helpful to see patterns over time in their feedback. They also highlighted that the timing of feedback was important and should be aligned with appropriate interventions to improve quality.
Physicians stated that sufficient resources were required for both the use of clinical indicators and subsequent improvements in quality. Physicians also emphasized that they needed sufficient time and resources to reflect on their practice and makes any changes to respond to indicator feedback and improve quality.
Physicians emphasized that measuring quality of care was not enough to improve quality—it was also crucial that they had support to translate feedback into quality improvement. Most importantly, physicians wanted clinical indicator feedback to be linked to a clear action for improvement. They also suggested that quality improvement needed to happen as a team.
Physicians thought that while tying incentives to clinical indicators could accelerate quality improvement, there was also the potential for unintended consequences and ‘gaming’ the system.
Target the most important areas of care for quality improvement.
Physicians thought that the number of clinical indicators should be limited and only cover the most important areas of care. In particular, physicians suggested a focus on diseases where improved care can have a substantial impact, or a focus on especially high-risk patients. Technical process indicators were also suggested as an important aspect of care to measure. Physicians were generally resistant to productivity-oriented indicators.
Physicians thought it was important for clinical indicators to be evidence-based and to reflect best practice. They felt that indicators must be consistent with other policies and guidelines, and indicators should not contradict each other.
Physicians thought it was important that clinical indicators measured aspects of care that were within their control. This was particularly important if indicators were tied to incentives or used punitively. Despite many physicians agreeing that outcome indicators measured what was ultimately important, they also expressed concern that outcomes were often affected by factors outside of physicians’ control.
Physicians commented on several important attributes that would increase their trust in clinical indicators being able to drive improvements in quality. Physicians thought there were several important attributes that made a clinical indicator reliable and hence trustworthy. They stated that clinical indicators should be of high quality, valid, precise, technically specific, clearly defined, and only require information that could be measured accurately.
Physicians agreed that there was a role for patient-reported outcome measures in driving quality improvement. Patient-reported outcome measures and patient experience indicators were seen as representing one aspect of quality that was important to consider. However, physicians also recommended that such measures should be considered alongside other clinical indicators. They also thought that some aspects of patient experience are subjective and therefore less helpful for quality improvement.
This systematic review found overall agreement that indicators could play a clear role in motivating physicians to improve the quality of care and showing where changes needed to be made. While it was felt that indicators increased physicians’ accountability, it was clear that they should be used by physicians themselves, rather than by the government or the public, and should not be used punitively. There was concern that an overreliance on indicators might lead to myopic quality improvement at the expense of more holistic care. In order to strengthen the ability of indicators to drive improvements in quality, physicians need to support and participate in the process of indicator development, recording relevant data should be straightforward, indicator feedback needs to be meaningful, and physicians and their teams need to be adequately resourced to act on findings.
While it was recognized that incentives might accelerate quality improvement, there was also the risk of unintended consequences and ‘gaming’. Key attributes of effective indicators were a focus on the most important areas for quality improvement, consistency with good medical care, measurement of aspects of care that were within the control of physicians and reliability. While there was support for the use of patient-reported outcome measures alongside clinical indicators, there was a potential disconnect between the supposed subjectivity of these measures and the desire for indicators to be ‘accurate’ or objective.
This thematic synthesis of data identified from a systematic review of the literature was focused on physicians’ views regarding the utility of clinical indicators in practice. This is important to understand given the increasing use of clinical indicators and expectations that physicians will use and act on clinical indicator data. As we did not have access to the raw data from primary studies, our findings represent a synthesis of selected data included in the primary studies as well as the authors’ interpretations of that data. The literature search and coding were performed by one reviewer, which may have resulted in bias in the selection of articles. Lastly, texts in languages other than English were excluded.
There were also several limitations in the literature included in this systematic review. As noted, most (9/14) articles did not discuss ethical issues associated with their research with many failing to report ethics approval of the data collection. Generalizability of the results to all physicians is difficult to ascertain because most participants were primary care physicians. Generalizability of the results may also depend on when the data were obtained (given that perspectives are likely to change over time) and the specific health systems examined in each study. Unfortunately, it was not feasible to disaggregate themes according to study context due to the limited number of included studies.
While our literature search did not return results for physicians’ perspectives on the best tools for appraising the quality of clinical indicators, Jones et al . [ 8 ] have previously developed the quality improvement critical appraisal tool (QICA), to provide an explicit basis for clinical indicator selection. The findings of our review are consistent with key aspects of the QICA tool, including the need for indicators to measure the most important aspects of medical care and to be evidence-based, acceptable, concordant with other measures of the issue, reliable and to consider the potential for unintended effects, such as bias as well as the resource implications of measurement itself [ 12 ]. There were several technical characteristics listed in the QICA tool that were not explored in our systematic review, including the need for a well-defined target population, exclusions and measurement systems, need for indicators to reflect differing cultural values, the power and precision of an indicator to detect clinically important changes beyond random variation, and potential ethical issues involved with data gathering and reporting of results [ 12 ].
The final part of the QICA tool addresses the practical implications of indicator implementation in both data collection and data analysis [ 12 ]. There was significant overlap here between the characteristics included in the tool and those that physicians thought were important in the systematic review. Similar findings included the importance of limiting extra work in collecting data for clinical indicators, ensuring that technology is sufficient, ensuring that indicator feedback is actionable and that the results are understandable by physicians so they can be used to improve the quality of care.
This review found that indicators can play an important motivating role for physicians to improve the quality of care and show where changes need to be made. For indicators to be effective, physicians should be involved in indicator development, recording relevant data should be straightforward, indicator feedback must be meaningful to physicians, and clinical teams need to be adequately resourced to act on findings. Effective indicators need to focus on the most important areas for quality improvement, be consistent with good medical care, and measure aspects of care within the control of physicians. Studies cautioned against using indicators primarily as punitive measures, and there were concerns that an overreliance on indicators could lead to a narrowed perspective on quality of care.
In this systematic review, we found that physicians believe that they should participate in the development of indicators and control the use of those indicators. However, it is worth noting that there are other legitimate groups and stakeholders that also have an interest in the development and use of indicators. Physicians form one professional group among a broader range of multi-disciplinary health providers, as well as patients themselves, whose perspectives need to be engaged in indicator development. It has also been argued that a key impediment faced by collaborative healthcare teams working towards quality improvement is the ‘structured embeddedness of medical dominance’ [ 13 ]. Balancing the perspectives of multiple professional groups as well as patients while avoiding the tendency for physicians to disengage from the process entirely is one of the challenges for the use of clinical indicators for driving quality improvements in policy as well as practice, and would be a valuable area for future research.
This review identified facilitators and barriers to meaningfully engaging physicians in developing and using clinical indicators to improve the quality of healthcare. Such information will help maximize the extent to which the potential of ‘big data’ in revolutionizing clinical engagement with quality improvement activites is able to be realized.
Not applicable.
Ana Renker-Darby (Data curation, analysis (lead), original draft preparation, reviewing and editing), Shanthi Ameratunga (Analysis, reviewing & editing), Peter Jones (Analysis, reviewing & editing), Corina Grey (Analysis, reviewing & editing), Matire Harwood (Analysis, reviewing & editing), Roshini Peiris-John (Analysis, reviewing & editing), Timothy Tenbensel (Analysis, reviewing & editing), Sue Wells (Analysis, reviewing & editing), Vanessa Selak (Conceptuatlisation, Analysis, reviewing & editing).
Supplementary data is available at IJQHC online.
During the conduct of this research, S.A., C.G., M.H., P.J., R.P.J., V.S., and S.W. received funding for other research projects from the Health Research Council of New Zealand; S.A., C.G., M.H., V.S., and S.W. received funding from the National Heart Foundation of New Zealand and the National Science Challenge (Healthier Lives); V.S. and S.W. received funding from the Auckland Medical Research Foundation, and P.J. received funding from the A+ Trust.
A.R.’s work on this research was funded by a grant from the University of Auckland’s Faculty of Medical and Health Sciences Research Development Fund.
No new data were generated or analysed in support of this research.
Raleigh VS , Foot C . Getting the Measure of Quality. Opportunities and Challenges . London : The King’s Fund , 2010 .
Google Scholar
Google Preview
Solberg LI , Asche SE , Margolis KL et al. Measuring an organization’s ability to manage change: the change process capability questionnaire and its use for improving depression care . Am J Med Qual 2008 ; 23 : 193 – 200 . https://doi.org/10.1177/1062860608314942
Hemingway H , Asselbergs FW , Danesh J et al. Big data from electronic health records for early and late translational cardiovascular research: challenges and potential . Eur Heart J 2018 ; 39 : 1481 – 95 . https://doi.org/10.1093/eurheartj/ehx487
Roski J , Bo-Linn GW , Andrews TA . Creating value in health care through big data: opportunities and policy implications . Health Affairs 2014 ; 33 : 1115 – 22 .
Patel S , Rajkomar A , Harrison JD et al. Next-generation audit and feedback for inpatient quality improvement using electronic health record data: a cluster randomised controlled trial . BMJ Qual Saf 2018 ; 27 : 691 – 9 .
Hurrell M , Stein A , MacDonald S . Use of natural language processing to identify significant abnormalities for follow-up in a large accumulation of non-delivered radiology reports . J Health Med Inform 2017 ; 8 :2.
Liao KP , Cai T , Savova GK et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing . BMJ 2015 ; 350 :h1885.
Jones P , Shepherd M , Wells S et al. Review article: what makes a good healthcare quality indicator? A systematic review and validation study . Emergency Med Australasia 2014 ; 26 : 113 – 24 . https://doi.org/10.1111/1742-6723.12195
Critical Appraisal Skills Programme . CASP Qualitative Checklist 2018 . https://casp-uk.net/casp-tools-checklists/ ( 1 November 2019 , date last accessed).
Thomas J , Harden A . Methods for the thematic synthesis of qualitative research in systematic reviews . BMC Med Res Method 2008 ; 8 :10. https://doi.org/10.1186/1471-2288-8-45
Braun V , Clarke V . Using thematic analysis in psychology . Qual Res Psychol 2006 ; 3 : 77 – 101 . https://doi.org/10.1191/1478088706qp063oa
Jones P . Defining and Validating a Metric for Emergency Department Crowding . Auckland, New Zealand : University of Auckland , 2018 .
Bourgeault IL , Mulvale G . Collaborative health care teams in Canada and the USA: confronting the structural embeddedness of medical dominance . Health Sociol Rev 2014 ; 15 : 481 – 95 . https://doi.org/10.5172/hesr.2006.15.5.481
Month: | Total Views: |
---|---|
August 2024 | 107 |
Citing articles via.
Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide
Sign In or Create an Account
This PDF is available to Subscribers Only
For full access to this pdf, sign in to an existing account, or purchase an annual subscription.
BMC Medical Research Methodology volume 8 , Article number: 45 ( 2008 ) Cite this article
395k Accesses
4516 Citations
104 Altmetric
Metrics details
There is a growing recognition of the value of synthesising qualitative research in the evidence base in order to facilitate effective and appropriate health care. In response to this, methods for undertaking these syntheses are currently being developed. Thematic analysis is a method that is often used to analyse data in primary qualitative research. This paper reports on the use of this type of analysis in systematic reviews to bring together and integrate the findings of multiple qualitative studies.
We describe thematic synthesis, outline several steps for its conduct and illustrate the process and outcome of this approach using a completed review of health promotion research. Thematic synthesis has three stages: the coding of text 'line-by-line'; the development of 'descriptive themes'; and the generation of 'analytical themes'. While the development of descriptive themes remains 'close' to the primary studies, the analytical themes represent a stage of interpretation whereby the reviewers 'go beyond' the primary studies and generate new interpretive constructs, explanations or hypotheses. The use of computer software can facilitate this method of synthesis; detailed guidance is given on how this can be achieved.
We used thematic synthesis to combine the studies of children's views and identified key themes to explore in the intervention studies. Most interventions were based in school and often combined learning about health benefits with 'hands-on' experience. The studies of children's views suggested that fruit and vegetables should be treated in different ways, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective. Thematic synthesis enabled us to stay 'close' to the results of the primary studies, synthesising them in a transparent way, and facilitating the explicit production of new concepts and hypotheses.
We compare thematic synthesis to other methods for the synthesis of qualitative research, discussing issues of context and rigour. Thematic synthesis is presented as a tried and tested method that preserves an explicit and transparent link between conclusions and the text of primary studies; as such it preserves principles that have traditionally been important to systematic reviewing.
Peer Review reports
The systematic review is an important technology for the evidence-informed policy and practice movement, which aims to bring research closer to decision-making [ 1 , 2 ]. This type of review uses rigorous and explicit methods to bring together the results of primary research in order to provide reliable answers to particular questions [ 3 – 6 ]. The picture that is presented aims to be distorted neither by biases in the review process nor by biases in the primary research which the review contains [ 7 – 10 ]. Systematic review methods are well-developed for certain types of research, such as randomised controlled trials (RCTs). Methods for reviewing qualitative research in a systematic way are still emerging, and there is much ongoing development and debate [ 11 – 14 ].
In this paper we present one approach to the synthesis of findings of qualitative research, which we have called 'thematic synthesis'. We have developed and applied these methods within several systematic reviews that address questions about people's perspectives and experiences [ 15 – 18 ]. The context for this methodological development is a programme of work in health promotion and public health (HP & PH), mostly funded by the English Department of Health, at the EPPI-Centre, in the Social Science Research Unit at the Institute of Education, University of London in the UK. Early systematic reviews at the EPPI-Centre addressed the question 'what works?' and contained research testing the effects of interventions. However, policy makers and other review users also posed questions about intervention need, appropriateness and acceptability, and factors influencing intervention implementation. To address these questions, our reviews began to include a wider range of research, including research often described as 'qualitative'. We began to focus, in particular, on research that aimed to understand the health issue in question from the experiences and point of view of the groups of people targeted by HP&PH interventions (We use the term 'qualitative' research cautiously because it encompasses a multitude of research methods at the same time as an assumed range of epistemological positions. In practice it is often difficult to classify research as being either 'qualitative' or 'quantitative' as much research contains aspects of both [ 19 – 22 ]. Because the term is in common use, however, we will employ it in this paper).
When we started the work for our first series of reviews which included qualitative research in 1999 [ 23 – 26 ], there was very little published material that described methods for synthesising this type of research. We therefore experimented with a variety of techniques borrowed from standard systematic review methods and methods for analysing primary qualitative research [ 15 ]. In later reviews, we were able to refine these methods and began to apply thematic analysis in a more explicit way. The methods for thematic synthesis described in this paper have so far been used explicitly in three systematic reviews [ 16 – 18 ].
To illustrate the steps involved in a thematic synthesis we draw on a review of the barriers to, and facilitators of, healthy eating amongst children aged four to 10 years old [ 17 ]. The review was commissioned by the Department of Health, England to inform policy about how to encourage children to eat healthily in the light of recent surveys highlighting that British children are eating less than half the recommended five portions of fruit and vegetables per day. While we focus on the aspects of the review that relate to qualitative studies, the review was broader than this and combined answering traditional questions of effectiveness, through reviewing controlled trials, with questions relating to children's views of healthy eating, which were answered using qualitative studies. The qualitative studies were synthesised using 'thematic synthesis' – the subject of this paper. We compared the effectiveness of interventions which appeared to be in line with recommendations from the thematic synthesis with those that did not. This enabled us to see whether the understandings we had gained from the children's views helped us to explain differences in the effectiveness of different interventions: the thematic synthesis had enabled us to generate hypotheses which could be tested against the findings of the quantitative studies – hypotheses that we could not have generated without the thematic synthesis. The methods of this part of the review are published in Thomas et al . [ 27 ] and are discussed further in Harden and Thomas [ 21 ].
The act of seeking to synthesise qualitative research means stepping into more complex and contested territory than is the case when only RCTs are included in a review. First, methods are much less developed in this area, with fewer completed reviews available from which to learn, and second, the whole enterprise of synthesising qualitative research is itself hotly debated. Qualitative research, it is often proposed, is not generalisable and is specific to a particular context, time and group of participants. Thus, in bringing such research together, reviewers are open to the charge that they de-contextualise findings and wrongly assume that these are commensurable [ 11 , 13 ]. These are serious concerns which it is not the purpose of this paper to contest. We note, however, that a strong case has been made for qualitative research to be valued for the potential it has to inform policy and practice [ 11 , 28 – 30 ]. In our experience, users of reviews are interested in the answers that only qualitative research can provide, but are not able to handle the deluge of data that would result if they tried to locate, read and interpret all the relevant research themselves. Thus, if we acknowledge the unique importance of qualitative research, we need also to recognise that methods are required to bring its findings together for a wide audience – at the same time as preserving and respecting its essential context and complexity.
The earliest published work that we know of that deals with methods for synthesising qualitative research was written in 1988 by Noblit and Hare [ 31 ]. This book describes the way that ethnographic research might be synthesised, but the method has been shown to be applicable to qualitative research beyond ethnography [ 32 , 11 ]. As well as meta-ethnography, other methods have been developed more recently, including 'meta-study' [ 33 ], 'critical interpretive synthesis' [ 34 ] and 'metasynthesis' [ 13 ].
Many of the newer methods being developed have much in common with meta-ethnography, as originally described by Noblit and Hare, and often state explicitly that they are drawing on this work. In essence, this method involves identifying key concepts from studies and translating them into one another. The term 'translating' in this context refers to the process of taking concepts from one study and recognising the same concepts in another study, though they may not be expressed using identical words. Explanations or theories associated with these concepts are also extracted and a 'line of argument' may be developed, pulling corroborating concepts together and, crucially, going beyond the content of the original studies (though 'refutational' concepts might not be amenable to this process). Some have claimed that this notion of 'going beyond' the primary studies is a critical component of synthesis, and is what distinguishes it from the types of summaries of findings that typify traditional literature reviews [e.g. [ 32 ], p209]. In the words of Margarete Sandelowski, "metasyntheses are integrations that are more than the sum of parts, in that they offer novel interpretations of findings. These interpretations will not be found in any one research report but, rather, are inferences derived from taking all of the reports in a sample as a whole" [[ 14 ], p1358].
Thematic analysis has been identified as one of a range of potential methods for research synthesis alongside meta-ethnography and 'metasynthesis', though precisely what the method involves is unclear, and there are few examples of it being used for synthesising research [ 35 ]. We have adopted the term 'thematic synthesis', as we translated methods for the analysis of primary research – often termed 'thematic' – for use in systematic reviews [ 36 – 38 ]. As Boyatzis [[ 36 ], p4] has observed, thematic analysis is "not another qualitative method but a process that can be used with most, if not all, qualitative methods..." . Our approach concurs with this conceptualisation of thematic analysis, since the method we employed draws on other established methods but uses techniques commonly described as 'thematic analysis' in order to formalise the identification and development of themes.
We now move to a description of the methods we used in our example systematic review. While this paper has the traditional structure for reporting the results of a research project, the detailed methods (e.g. precise terms we used for searching) and results are available online. This paper identifies the particular issues that relate especially to reviewing qualitative research systematically and then to describing the activity of thematic synthesis in detail.
When searching for studies for inclusion in a 'traditional' statistical meta-analysis, the aim of searching is to locate all relevant studies. Failing to do this can undermine the statistical models that underpin the analysis and bias the results. However, Doyle [[ 39 ], p326] states that, "like meta-analysis, meta-ethnography utilizes multiple empirical studies but, unlike meta-analysis, the sample is purposive rather than exhaustive because the purpose is interpretive explanation and not prediction" . This suggests that it may not be necessary to locate every available study because, for example, the results of a conceptual synthesis will not change if ten rather than five studies contain the same concept, but will depend on the range of concepts found in the studies, their context, and whether they are in agreement or not. Thus, principles such as aiming for 'conceptual saturation' might be more appropriate when planning a search strategy for qualitative research, although it is not yet clear how these principles can be applied in practice. Similarly, other principles from primary qualitative research methods may also be 'borrowed' such as deliberately seeking studies which might act as negative cases, aiming for maximum variability and, in essence, designing the resulting set of studies to be heterogeneous, in some ways, instead of achieving the homogeneity that is often the aim in statistical meta-analyses.
However you look, qualitative research is difficult to find [ 40 – 42 ]. In our review, it was not possible to rely on simple electronic searches of databases. We needed to search extensively in 'grey' literature, ask authors of relevant papers if they knew of more studies, and look especially for book chapters, and we spent a lot of effort screening titles and abstracts by hand and looking through journals manually. In this sense, while we were not driven by the statistical imperative of locating every relevant study, when it actually came down to searching, we found that there was very little difference in the methods we had to use to find qualitative studies compared to the methods we use when searching for studies for inclusion in a meta-analysis.
Assessing the quality of qualitative research has attracted much debate and there is little consensus regarding how quality should be assessed, who should assess quality, and, indeed, whether quality can or should be assessed in relation to 'qualitative' research at all [ 43 , 22 , 44 , 45 ]. We take the view that the quality of qualitative research should be assessed to avoid drawing unreliable conclusions. However, since there is little empirical evidence on which to base decisions for excluding studies based on quality assessment, we took the approach in this review to use 'sensitivity analyses' (described below) to assess the possible impact of study quality on the review's findings.
In our example review we assessed our studies according to 12 criteria, which were derived from existing sets of criteria proposed for assessing the quality of qualitative research [ 46 – 49 ], principles of good practice for conducting social research with children [ 50 ], and whether studies employed appropriate methods for addressing our review questions. The 12 criteria covered three main quality issues. Five related to the quality of the reporting of a study's aims, context, rationale, methods and findings (e.g. was there an adequate description of the sample used and the methods for how the sample was selected and recruited?). A further four criteria related to the sufficiency of the strategies employed to establish the reliability and validity of data collection tools and methods of analysis, and hence the validity of the findings. The final three criteria related to the assessment of the appropriateness of the study methods for ensuring that findings about the barriers to, and facilitators of, healthy eating were rooted in children's own perspectives (e.g. were data collection methods appropriate for helping children to express their views?).
One issue which is difficult to deal with when synthesising 'qualitative' studies is 'what counts as data' or 'findings'? This problem is easily addressed when a statistical meta-analysis is being conducted: the numeric results of RCTs – for example, the mean difference in outcome between the intervention and control – are taken from published reports and are entered into the software package being used to calculate the pooled effect size [ 3 , 51 ].
Deciding what to abstract from the published report of a 'qualitative' study is much more difficult. Campbell et al . [ 11 ] extracted what they called the 'key concepts' from the qualitative studies they found about patients' experiences of diabetes and diabetes care. However, finding the key concepts in 'qualitative' research is not always straightforward either. As Sandelowski and Barroso [ 52 ] discovered, identifying the findings in qualitative research can be complicated by varied reporting styles or the misrepresentation of data as findings (as for example when data are used to 'let participants speak for themselves'). Sandelowski and Barroso [ 53 ] have argued that the findings of qualitative (and, indeed, all empirical) research are distinct from the data upon which they are based, the methods used to derive them, externally sourced data, and researchers' conclusions and implications.
In our example review, while it was relatively easy to identify 'data' in the studies – usually in the form of quotations from the children themselves – it was often difficult to identify key concepts or succinct summaries of findings, especially for studies that had undertaken relatively simple analyses and had not gone much further than describing and summarising what the children had said. To resolve this problem we took study findings to be all of the text labelled as 'results' or 'findings' in study reports – though we also found 'findings' in the abstracts which were not always reported in the same way in the text. Study reports ranged in size from a few pages to full final project reports. We entered all the results of the studies verbatim into QSR's NVivo software for qualitative data analysis. Where we had the documents in electronic form this process was straightforward even for large amounts of text. When electronic versions were not available, the results sections were either re-typed or scanned in using a flat-bed or pen scanner. (We have since adapted our own reviewing system, 'EPPI-Reviewer' [ 54 ], to handle this type of synthesis and the screenshots below show this software.)
The synthesis took the form of three stages which overlapped to some degree: the free line-by-line coding of the findings of primary studies; the organisation of these 'free codes' into related areas to construct 'descriptive' themes; and the development of 'analytical' themes.
In our children and healthy eating review, we originally planned to extract and synthesise study findings according to our review questions regarding the barriers to, and facilitators of, healthy eating amongst children. It soon became apparent, however, that few study findings addressed these questions directly and it appeared that we were in danger of ending up with an empty synthesis. We were also concerned about imposing the a priori framework implied by our review questions onto study findings without allowing for the possibility that a different or modified framework may be a better fit. We therefore temporarily put our review questions to one side and started from the study findings themselves to conduct an thematic analysis.
There were eight relevant qualitative studies examining children's views of healthy eating. We entered the verbatim findings of these studies into our database. Three reviewers then independently coded each line of text according to its meaning and content. Figure 1 illustrates this line-by-line coding using our specialist reviewing software, EPPI-Reviewer, which includes a component designed to support thematic synthesis. The text which was taken from the report of the primary study is on the left and codes were created inductively to capture the meaning and content of each sentence. Codes could be structured, either in a tree form (as shown in the figure) or as 'free' codes – without a hierarchical structure.
line-by-line coding in EPPI-Reviewer.
The use of line-by-line coding enabled us to undertake what has been described as one of the key tasks in the synthesis of qualitative research: the translation of concepts from one study to another [ 32 , 55 ]. However, this process may not be regarded as a simple one of translation. As we coded each new study we added to our 'bank' of codes and developed new ones when necessary. As well as translating concepts between studies, we had already begun the process of synthesis (For another account of this process, see Doyle [[ 39 ], p331]). Every sentence had at least one code applied, and most were categorised using several codes (e.g. 'children prefer fruit to vegetables' or 'why eat healthily?'). Before completing this stage of the synthesis, we also examined all the text which had a given code applied to check consistency of interpretation and to see whether additional levels of coding were needed. (In grounded theory this is termed 'axial' coding; see Fisher [ 55 ] for further discussion of the application of axial coding in research synthesis.) This process created a total of 36 initial codes. For example, some of the text we coded as "bad food = nice, good food = awful" from one study [ 56 ] were:
'All the things that are bad for you are nice and all the things that are good for you are awful.' (Boys, year 6) [[ 56 ], p74]
'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]
Some children reported throwing away foods they knew had been put in because they were 'good for you' and only ate the crisps and chocolate . [[ 56 ], p75]
Reviewers looked for similarities and differences between the codes in order to start grouping them into a hierarchical tree structure. New codes were created to capture the meaning of groups of initial codes. This process resulted in a tree structure with several layers to organize a total of 12 descriptive themes (Figure 2 ). For example, the first layer divided the 12 themes into whether they were concerned with children's understandings of healthy eating or influences on children's food choice. The above example, about children's preferences for food, was placed in both areas, since the findings related both to children's reactions to the foods they were given, and to how they behaved when given the choice over what foods they might eat. A draft summary of the findings across the studies organized by the 12 descriptive themes was then written by one of the review authors. Two other review authors commented on this draft and a final version was agreed.
relationships between descriptive themes.
Up until this point, we had produced a synthesis which kept very close to the original findings of the included studies. The findings of each study had been combined into a whole via a listing of themes which described children's perspectives on healthy eating. However, we did not yet have a synthesis product that addressed directly the concerns of our review – regarding how to promote healthy eating, in particular fruit and vegetable intake, amongst children. Neither had we 'gone beyond' the findings of the primary studies and generated additional concepts, understandings or hypotheses. As noted earlier, the idea or step of 'going beyond' the content of the original studies has been identified by some as the defining characteristic of synthesis [ 32 , 14 ].
This stage of a qualitative synthesis is the most difficult to describe and is, potentially, the most controversial, since it is dependent on the judgement and insights of the reviewers. The equivalent stage in meta-ethnography is the development of 'third order interpretations' which go beyond the content of original studies [ 32 , 11 ]. In our example, the step of 'going beyond' the content of the original studies was achieved by using the descriptive themes that emerged from our inductive analysis of study findings to answer the review questions we had temporarily put to one side. Reviewers inferred barriers and facilitators from the views children were expressing about healthy eating or food in general, captured by the descriptive themes, and then considered the implications of children's views for intervention development. Each reviewer first did this independently and then as a group. Through this discussion more abstract or analytical themes began to emerge. The barriers and facilitators and implications for intervention development were examined again in light of these themes and changes made as necessary. This cyclical process was repeated until the new themes were sufficiently abstract to describe and/or explain all of our initial descriptive themes, our inferred barriers and facilitators and implications for intervention development.
For example, five of the 12 descriptive themes concerned the influences on children's choice of foods (food preferences, perceptions of health benefits, knowledge behaviour gap, roles and responsibilities, non-influencing factors). From these, reviewers inferred several barriers and implications for intervention development. Children identified readily that taste was the major concern for them when selecting food and that health was either a secondary factor or, in some cases, a reason for rejecting food. Children also felt that buying healthy food was not a legitimate use of their pocket money, which they would use to buy sweets that could be enjoyed with friends. These perspectives indicated to us that branding fruit and vegetables as a 'tasty' rather than 'healthy' might be more effective in increasing consumption. As one child noted astutely, 'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]. We captured this line of argument in the analytical theme entitled 'Children do not see it as their role to be interested in health'. Altogether, this process resulted in the generation of six analytical themes which were associated with ten recommendations for interventions.
Six main issues emerged from the studies of children's views: (1) children do not see it as their role to be interested in health; (2) children do not see messages about future health as personally relevant or credible; (3) fruit, vegetables and confectionery have very different meanings for children; (4) children actively seek ways to exercise their own choices with regard to food; (5) children value eating as a social occasion; and (6) children see the contradiction between what is promoted in theory and what adults provide in practice. The review found that most interventions were based in school (though frequently with parental involvement) and often combined learning about the health benefits of fruit and vegetables with 'hands-on' experience in the form of food preparation and taste-testing. Interventions targeted at people with particular risk factors worked better than others, and multi-component interventions that combined the promotion of physical activity with healthy eating did not work as well as those that only concentrated on healthy eating. The studies of children's views suggested that fruit and vegetables should be treated in different ways in interventions, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective than those which were not.
The process of translation, through the development of descriptive and analytical themes, can be carried out in a rigorous way that facilitates transparency of reporting. Since we aim to produce a synthesis that both generates 'abstract and formal theories' that are nevertheless 'empirically faithful to the cases from which they were developed' [[ 53 ], p1371], we see the explicit recording of the development of themes as being central to the method. The use of software as described can facilitate this by allowing reviewers to examine the contribution made to their findings by individual studies, groups of studies, or sub-populations within studies.
Some may argue against the synthesis of qualitative research on the grounds that the findings of individual studies are de-contextualised and that concepts identified in one setting are not applicable to others [ 32 ]. However, the act of synthesis could be viewed as similar to the role of a research user when reading a piece of qualitative research and deciding how useful it is to their own situation. In the case of synthesis, reviewers translate themes and concepts from one situation to another and can always be checking that each transfer is valid and whether there are any reasons that understandings gained in one context might not be transferred to another. We attempted to preserve context by providing structured summaries of each study detailing aims, methods and methodological quality, and setting and sample. This meant that readers of our review were able to judge for themselves whether or not the contexts of the studies the review contained were similar to their own. In the synthesis we also checked whether the emerging findings really were transferable across different study contexts. For example, we tried throughout the synthesis to distinguish between participants (e.g. boys and girls) where the primary research had made an appropriate distinction. We then looked to see whether some of our synthesis findings could be attributed to a particular group of children or setting. In the event, we did not find any themes that belonged to a specific group, but another outcome of this process was a realisation that the contextual information given in the reports of studies was very restricted indeed. It was therefore difficult to make the best use of context in our synthesis.
In checking that we were not translating concepts into situations where they did not belong, we were following a principle that others have followed when using synthesis methods to build grounded formal theory: that of grounding a text in the context in which it was constructed. As Margaret Kearney has noted "the conditions under which data were collected, analysis was done, findings were found, and products were written for each contributing report should be taken into consideration in developing a more generalized and abstract model" [[ 14 ], p1353]. Britten et al . [ 32 ] suggest that it may be important to make a deliberate attempt to include studies conducted across diverse settings to achieve the higher level of abstraction that is aimed for in a meta-ethnography.
We assessed the 'quality' of our studies with regard to the degree to which they represented the views of their participants. In doing this, we were locating the concept of 'quality' within the context of the purpose of our review – children's views – and not necessarily the context of the primary studies themselves. Our 'hierarchy of evidence', therefore, did not prioritise the research design of studies but emphasised the ability of the studies to answer our review question. A traditional systematic review of controlled trials would contain a quality assessment stage, the purpose of which is to exclude studies that do not provide a reliable answer to the review question. However, given that there were no accepted – or empirically tested – methods for excluding qualitative studies from syntheses on the basis of their quality [ 57 , 12 , 58 ], we included all studies regardless of their quality.
Nevertheless, our studies did differ according to the quality criteria they were assessed against and it was important that we considered this in some way. In systematic reviews of trials, 'sensitivity analyses' – analyses which test the effect on the synthesis of including and excluding findings from studies of differing quality – are often carried out. Dixon-Woods et al . [ 12 ] suggest that assessing the feasibility and worth of conducting sensitivity analyses within syntheses of qualitative research should be an important focus of synthesis methods work. After our thematic synthesis was complete, we examined the relative contributions of studies to our final analytic themes and recommendations for interventions. We found that the poorer quality studies contributed comparatively little to the synthesis and did not contain many unique themes; the better studies, on the other hand, appeared to have more developed analyses and contributed most to the synthesis.
This paper has discussed the rationale for reviewing and synthesising qualitative research in a systematic way and has outlined one specific approach for doing this: thematic synthesis. While it is not the only method which might be used – and we have discussed some of the other options available – we present it here as a tested technique that has worked in the systematic reviews in which it has been employed.
We have observed that one of the key tasks in the synthesis of qualitative research is the translation of concepts between studies. While the activity of translating concepts is usually undertaken in the few syntheses of qualitative research that exist, there are few examples that specify the detail of how this translation is actually carried out. The example above shows how we achieved the translation of concepts across studies through the use of line-by-line coding, the organisation of these codes into descriptive themes, and the generation of analytical themes through the application of a higher level theoretical framework. This paper therefore also demonstrates how the methods and process of a thematic synthesis can be written up in a transparent way.
This paper goes some way to addressing concerns regarding the use of thematic analysis in research synthesis raised by Dixon-Woods and colleagues who argue that the approach can lack transparency due to a failure to distinguish between 'data-driven' or 'theory-driven' approaches. Moreover they suggest that, "if thematic analysis is limited to summarising themes reported in primary studies, it offers little by way of theoretical structure within which to develop higher order thematic categories..." [[ 35 ], p47]. Part of the problem, they observe, is that the precise methods of thematic synthesis are unclear. Our approach contains a clear separation between the 'data-driven' descriptive themes and the 'theory-driven' analytical themes and demonstrates how the review questions provided a theoretical structure within which it became possible to develop higher order thematic categories.
The theme of 'going beyond' the content of the primary studies was discussed earlier. Citing Strike and Posner [ 59 ], Campbell et al . [[ 11 ], p672] also suggest that synthesis "involves some degree of conceptual innovation, or employment of concepts not found in the characterisation of the parts and a means of creating the whole" . This was certainly true of the example given in this paper. We used a series of questions, derived from the main topic of our review, to focus an examination of our descriptive themes and we do not find our recommendations for interventions contained in the findings of the primary studies: these were new propositions generated by the reviewers in the light of the synthesis. The method also demonstrates that it is possible to synthesise without conceptual innovation. The initial synthesis, involving the translation of concepts between studies, was necessary in order for conceptual innovation to begin. One could argue that the conceptual innovation, in this case, was only necessary because the primary studies did not address our review question directly. In situations in which the primary studies are concerned directly with the review question, it may not be necessary to go beyond the contents of the original studies in order to produce a satisfactory synthesis (see, for example, Marston and King, [ 60 ]). Conceptually, our analytical themes are similar to the ultimate product of meta-ethnographies: third order interpretations [ 11 ], since both are explicit mechanisms for going beyond the content of the primary studies and presenting this in a transparent way. The main difference between them lies in their purposes. Third order interpretations bring together the implications of translating studies into one another in their own terms, whereas analytical themes are the result of interrogating a descriptive synthesis by placing it within an external theoretical framework (our review question and sub-questions). It may be, therefore, that analytical themes are more appropriate when a specific review question is being addressed (as often occurs when informing policy and practice), and third order interpretations should be used when a body of literature is being explored in and of itself, with broader, or emergent, review questions.
This paper is a contribution to the current developmental work taking place in understanding how best to bring together the findings of qualitative research to inform policy and practice. It is by no means the only method on offer but, by drawing on methods and principles from qualitative primary research, it benefits from the years of methodological development that underpins the research it seeks to synthesise.
Chalmers I: Trying to do more good than harm in policy and practice: the role of rigorous, transparent and up-to-date evaluations. Ann Am Acad Pol Soc Sci. 2003, 589: 22-40. 10.1177/0002716203254762.
Article Google Scholar
Oakley A: Social science and evidence-based everything: the case of education. Educ Rev. 2002, 54: 277-286. 10.1080/0013191022000016329.
Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, New York: Russell Sage Foundation
Google Scholar
EPPI-Centre: EPPI-Centre Methods for Conducting Systematic Reviews. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89 ]
Higgins J, Green S, (Eds): Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. 2006, Updated September 2006. Accessed 24th January 2007, [ http://www.cochrane.org/resources/handbook/ ]
Petticrew M, Roberts H: Systematic Reviews in the Social Sciences: A practical guide. 2006, Oxford: Blackwell Publishing
Book Google Scholar
Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Prof. 2002, 25: 12-37. 10.1177/0163278702025001003.
Article PubMed Google Scholar
Juni P, Altman D, Egger M: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.
Article CAS PubMed PubMed Central Google Scholar
Mulrow C: Systematic reviews: rationale for systematic reviews. BMJ. 1994, 309: 597-599.
White H: Scientific communication and literature retrieval. The Handbook of Research Synthesis. Edited by: Cooper H, Hedges L. 1994, New York: Russell Sage Foundation
Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J: Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003, 56: 671-684. 10.1016/S0277-9536(02)00064-3.
Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, Shaw RL, Smith JA, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006, 6: 27-44. 10.1177/1468794106058867.
Sandelowski M, Barroso J: Handbook for Synthesising Qualitative Research. 2007, New York: Springer
Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M: Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004, 14: 1342-1365. 10.1177/1049732304269888.
Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health. J Epidemiol Community Health. 2004, 58: 794-800. 10.1136/jech.2003.014829.
Article PubMed PubMed Central Google Scholar
Harden A, Brunton G, Fletcher A, Oakley A: Young People, Pregnancy and Social Exclusion: A systematic synthesis of research evidence to identify effective, appropriate and promising approaches for prevention and support. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=674 ]
Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J: Children and Healthy Eating: A systematic review of barriers and facilitators. 2003, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, accessed 4 th July 2008, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=246 ]
Thomas J, Kavanagh J, Tucker H, Burchett H, Tripney J, Oakley A: Accidental Injury, Risk-Taking Behaviour and the Social Circumstances in which Young People Live: A systematic review. 2007, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1910 ]
Bryman A: Quantity and Quality in Social Research. 1998, London: Unwin
Hammersley M: What's Wrong with Ethnography?. 1992, London: Routledge
Harden A, Thomas J: Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Meth. 2005, 8: 257-271. 10.1080/13645570500155078.
Oakley A: Experiments in Knowing: Gender and methods in the social sciences. 2000, Cambridge: Polity Press
Harden A, Oakley A, Oliver S: Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J. 2001, 60: 339-353. 10.1177/001789690106000406.
Harden A, Rees R, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Mental Health: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=256 ]
Rees R, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Physical Activity: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=260 ]
Shepherd J, Harden A, Rees R, Brunton G, Oliver S, Oakley A: Young People and Healthy Eating: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=258 ]
Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. BMJ. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.
Davies P: What is evidence-based education?. Br J Educ Stud. 1999, 47: 108-121. 10.1111/1467-8527.00106.
Newman M, Thompson C, Roberts AP: Helping practitioners understand the contribution of qualitative research to evidence-based practice. Evid Based Nurs. 2006, 9: 4-7. 10.1136/ebn.9.1.4.
Popay J: Moving Beyond Effectiveness in Evidence Synthesis. 2006, London: National Institute for Health and Clinical Excellence
Noblit GW, Hare RD: Meta-Ethnography: Synthesizing qualitative studies. 1988, Newbury Park: Sage
Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R: Using meta-ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy. 2002, 7: 209-215. 10.1258/135581902320432732.
Paterson B, Thorne S, Canam C, Jillings C: Meta-Study of Qualitative Health Research. 2001, Thousand Oaks, California: Sage
Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretative synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005, 10: 45-53. 10.1258/1355819052801804.
Boyatzis RE: Transforming Qualitative Information. 1998, Sage: Cleveland
Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006, 3: 77-101. 10.1191/1478088706qp063oa. [ http://science.uwe.ac.uk/psychology/drvictoriaclarke_files/thematicanalysis%20.pdf ]
Silverman D, Ed: Qualitative Research: Theory, method and practice. 1997, London: Sage
Doyle LH: Synthesis through meta-ethnography: paradoxes, enhancements, and possibilities. Qual Res. 2003, 3: 321-344. 10.1177/1468794103033003.
Barroso J, Gollop C, Sandelowski M, Meynell J, Pearce PF, Collins LJ: The challenges of searching for and retrieving qualitative studies. Western J Nurs Res. 2003, 25: 153-178. 10.1177/0193945902250034.
Walters LA, Wilczynski NL, Haynes RB, Hedges Team: Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res. 2006, 16: 162-8. 10.1177/1049732305284027.
Wong SSL, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in Medline. Medinfo. 2004, 11: 311-314.
Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P: Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998, 2 (16):
Seale C: Quality in qualitative research. Qual Inq. 1999, 5: 465-478.
Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: A framework for assessing research evidence. 2003, London: Cabinet Office
Boulton M, Fitzpatrick R, Swinburn C: Qualitative research in healthcare II: a structured review and evaluation of studies. J Eval Clin Pract. 1996, 2: 171-179. 10.1111/j.1365-2753.1996.tb00041.x.
Article CAS PubMed Google Scholar
Cobb A, Hagemaster J: Ten criteria for evaluating qualitative research proposals. J Nurs Educ. 1987, 26: 138-143.
CAS PubMed Google Scholar
Mays N, Pope C: Rigour and qualitative research. BMJ. 1995, 311: 109-12.
Medical Sociology Group: Criteria for the evaluation of qualitative research papers. Med Sociol News. 1996, 22: 68-71.
Alderson P: Listening to Children. 1995, London: Barnardo's
Egger M, Davey-Smith G, Altman D: Systematic Reviews in Health Care: Meta-analysis in context. 2001, London: BMJ Publishing
Sandelowski M, Barroso J: Finding the findings in qualitative studies. J Nurs Scholarsh. 2002, 34: 213-219. 10.1111/j.1547-5069.2002.00213.x.
Sandelowski M: Using qualitative research. Qual Health Res. 2004, 14: 1366-1386. 10.1177/1049732304269672.
Thomas J, Brunton J: EPPI-Reviewer 3.0: Analysis and management of data for research synthesis. EPPI-Centre software. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education
Fisher M, Qureshi H, Hardyman W, Homewood J: Using Qualitative Research in Systematic Reviews: Older people's views of hospital discharge. 2006, London: Social Care Institute for Excellence
Dixey R, Sahota P, Atwal S, Turner A: Children talking about healthy eating: data from focus groups with 300 9–11-year-olds. Nutr Bull. 2001, 26: 71-79. 10.1046/j.1467-3010.2001.00078.x.
Daly A, Willis K, Small R, Green J, Welch N, Kealy M, Hughes E: Hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007, 60: 43-49. 10.1016/j.jclinepi.2006.03.014.
Popay J: Moving beyond floccinaucinihilipilification: enhancing the utility of systematic reviews. J Clin Epidemiol. 2005, 58: 1079-80. 10.1016/j.jclinepi.2005.08.004.
Strike K, Posner G: Types of synthesis and their criteria. Knowledge Structure and Use: Implications for synthesis and interpretation. Edited by: Ward S, Reed L. 1983, Philadelphia: Temple University Press
Marston C, King E: Factors that shape young people's sexual behaviour: a systematic review. The Lancet. 2006, 368: 1581-86. 10.1016/S0140-6736(06)69662-1.
The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/8/45/prepub
Download references
The authors would like to thank Elaine Barnett-Page for her assistance in producing the draft paper, and David Gough, Ann Oakley and Sandy Oliver for their helpful comments. The review used an example in this paper was funded by the Department of Health (England). The methodological development was supported by Department of Health (England) and the ESRC through the Methods for Research Synthesis Node of the National Centre for Research Methods. In addition, Angela Harden held a senior research fellowship funded by the Department of Health (England) December 2003 – November 2007. The views expressed in this paper are those of the authors and are not necessarily those of the funding bodies.
Authors and affiliations.
EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, UK
James Thomas & Angela Harden
You can also search for this author in PubMed Google Scholar
Correspondence to James Thomas .
Competing interests.
The authors declare that they have no competing interests.
Both authors contributed equally to the paper and read and approved the final manuscript.
James Thomas and Angela Harden contributed equally to this work.
Below are the links to the authors’ original submitted files for images.
Authors’ original file for figure 2, rights and permissions.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Reprints and permissions
Cite this article.
Thomas, J., Harden, A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 8 , 45 (2008). https://doi.org/10.1186/1471-2288-8-45
Download citation
Received : 17 April 2008
Accepted : 10 July 2008
Published : 10 July 2008
DOI : https://doi.org/10.1186/1471-2288-8-45
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1471-2288
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
Ali hasanpour dehkordi.
Social Determinants of Health Research Center, Shahrekord University of Medical Sciences, Shahrekord, Iran
1 Health Information Technology Research Center, Student Research Committee, Department of Medical Library and Information Sciences, School of Management and Medical Information Sciences, Isfahan University of Medical Sciences, Isfahan, Iran
2 Department of International Relations, College of Law, Bayan University, Erbil, Kurdistan, Iraq
3 MSc in Biostatistics, Health Promotion Research Center, Iran University of Medical Sciences, Tehran, Iran
4 Spiritual Health Research Center, Research Institute for Health Development, Kurdistan University of Medical Sciences, Sanandaj, Iran
In recent years, published systematic reviews in the world and in Iran have been increasing. These studies are an important resource to answer evidence-based clinical questions and assist health policy-makers and students who want to identify evidence gaps in published research. Systematic review studies, with or without meta-analysis, synthesize all available evidence from studies focused on the same research question. In this study, the steps for a systematic review such as research question design and identification, the search for qualified published studies, the extraction and synthesis of information that pertain to the research question, and interpretation of the results are presented in details. This will be helpful to all interested researchers.
A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic.[ 1 ] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all relevant studies.[ 2 , 3 ] To identify assess and interpret available research, identify effective and ineffective health-care interventions, provide integrated documentation to help decision-making, and identify the gap between studies is one of the most important reasons for conducting systematic review studies.[ 4 ]
In the review studies, the latest scientific information about a particular topic is criticized. In these studies, the terms of review, systematic review, and meta-analysis are used instead. A systematic review is done in one of two methods, quantitative (meta-analysis) and qualitative. In a meta-analysis, the results of two or more studies for the evaluation of say health interventions are combined to measure the effect of treatment, while in the qualitative method, the findings of other studies are combined without using statistical methods.[ 5 ]
Since 1999, various guidelines, including the QUORUM, the MOOSE, the STROBE, the CONSORT, and the QUADAS, have been introduced for reporting meta-analyses. But recently the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement has gained widespread popularity.[ 6 , 7 , 8 , 9 ] The systematic review process based on the PRISMA statement includes four steps of how to formulate research questions, define the eligibility criteria, identify all relevant studies, extract and synthesize data, and deduce and present results (answers to research questions).[ 2 ]
Systematic reviews start with a protocol. The protocol is a researcher road map that outlines the goals, methodology, and outcomes of the research. Many journals advise writers to use the PRISMA statement to write the protocol.[ 10 ] The PRISMA checklist includes 27 items related to the content of a systematic review and meta-analysis and includes abstracts, methods, results, discussions, and financial resources.[ 11 ] PRISMA helps writers improve their systematic review and meta-analysis report. Reviewers and editors of medical journals acknowledge that while PRISMA may not be used as a tool to assess the methodological quality, it does help them to publish a better study article [ Figure 1 ].[ 12 ]
Screening process and articles selection according to the PRISMA guidelines
The main step in designing the protocol is to define the main objectives of the study and provide some background information. Before starting a systematic review, it is important to assess that your study is not a duplicate; therefore, in search of published research, it is necessary to review PREOSPERO and the Cochrane Database of Systematic. Sometimes it is better to search, in four databases, related systematic reviews that have already been published (PubMed, Web of Sciences, Scopus, Cochrane), published systematic review protocols (PubMed, Web of Sciences, Scopus, Cochrane), systematic review protocols that have already been registered but have not been published (PROSPERO, Cochrane), and finally related published articles (PubMed, Web of Sciences, Scopus, Cochrane). The goal is to reduce duplicate research and keep up-to-date systematic reviews.[ 13 ]
Writing a research question is the first step in systematic review that summarizes the main goal of the study.[ 14 ] The research question determines which types of studies should be included in the analysis (quantitative, qualitative, methodic mix, review overviews, or other studies). Sometimes a research question may be broken down into several more detailed questions.[ 15 ] The vague questions (such as: is walking helpful?) makes the researcher fail to be well focused on the collected studies or analyze them appropriately.[ 16 ] On the other hand, if the research question is rigid and restrictive (e.g., walking for 43 min and 3 times a week is better than walking for 38 min and 4 times a week?), there may not be enough studies in this area to answer this question and hence the generalizability of the findings to other populations will be reduced.[ 16 , 17 ] A good question in systematic review should include components that are PICOS style which include population (P), intervention (I), comparison (C), outcome (O), and setting (S).[ 18 ] Regarding the purpose of the study, control in clinical trials or pre-poststudies can replace C.[ 19 ]
After clarifying the research question and before searching the databases, it is necessary to specify searching methods, articles screening, studies eligibility check, check of the references in eligible studies, data extraction, and data analysis. This helps researchers ensure that potential biases in the selection of potential studies are minimized.[ 14 , 17 ] It should also look at details such as which published and unpublished literature have been searched, how they were searched, by which mechanism they were searched, and what are the inclusion and exclusion criteria.[ 4 ] First, all studies are searched and collected according to predefined keywords; then the title, abstract, and the entire text are screened for relevance by the authors.[ 13 ] By screening articles based on their titles, researchers can quickly decide on whether to retain or remove an article. If more information is needed, the abstracts of the articles will also be reviewed. In the next step, the full text of the articles will be reviewed to identify the relevant articles, and the reason for the removal of excluded articles is reported.[ 20 ] Finally, it is recommended that the process of searching, selecting, and screening articles be reported as a flowchart.[ 21 ] By increasing research, finding up-to-date and relevant information has become more difficult.[ 22 ]
Currently, there is no specific guideline as to which databases should be searched, which database is the best, and how many should be searched; but overall, it is advisable to search broadly. Because no database covers all health topics, it is recommended to use several databases to search.[ 23 ] According to the A MeaSurement Tool to Assess Systematic Reviews scale (AMSTAR) at least two databases should be searched in systematic and meta-analysis, although more comprehensive and accurate results can be obtained by increasing the number of searched databases.[ 24 ] The type of database to be searched depends on the systematic review question. For example, in a clinical trial study, it is recommended that Cochrane, multi-regional clinical trial (mRCTs), and International Clinical Trials Registry Platform be searched.[ 25 ]
For example, MEDLINE, a product of the National Library of Medicine in the United States of America, focuses on peer-reviewed articles in biomedical and health issues, while Embase covers the broad field of pharmacology and summaries of conferences. CINAHL is a great resource for nursing and health research and PsycINFO is a great database for psychology, psychiatry, counseling, addiction, and behavioral problems. Also, national and regional databases can be used to search related articles.[ 26 , 27 ] In addition, the search for conferences and gray literature helps to resolve the file-drawn problem (negative studies that may not be published yet).[ 26 ] If a systematic review is carried out on articles in a particular country or region, the databases in that region or country should also be investigated. For example, Iranian researchers can use national databases such as Scientific Information Database and MagIran. Comprehensive search to identify the maximum number of existing studies leads to a minimization of the selection bias. In the search process, the available databases should be used as much as possible, since many databases are overlapping.[ 17 ] Searching 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov, mRCTs, POPLINE, and SIGLE) covers all articles published in the field of medicine and health.[ 25 ] Some have suggested that references management software be used to search for more easy identification and removal of duplicate articles from several different databases.[ 20 ] At least one search strategy is presented in the article.[ 21 ]
The methodological quality assessment of articles is a key step in systematic review that helps identify systemic errors (bias) in results and interpretations. In systematic review studies, unlike other review studies, qualitative assessment or risk of bias is required. There are currently several tools available to review the quality of the articles. The overall score of these tools may not provide sufficient information on the strengths and weaknesses of the studies.[ 28 ] At least two reviewers should independently evaluate the quality of the articles, and if there is any objection, the third author should be asked to examine the article or the two researchers agree on the discussion. Some believe that the study of the quality of studies should be done by removing the name of the journal, title, authors, and institutions in a Blinded fashion.[ 29 ]
There are several ways for quality assessment, such as Sack's quality assessment (1988),[ 30 ] overview quality assessment questionnaire (1991),[ 31 ] CASP (Critical Appraisal Skills Program),[ 32 ] and AMSTAR (2007),[ 33 ] Besides, CASP,[ 34 ] the National Institute for Health and Care Excellence,[ 35 ] and the Joanna Briggs Institute System for the Unified Management, Assessment and Review of Information checklists.[ 30 , 36 ] However, it is worth mentioning that there is no single tool for assessing the quality of all types of reviews, but each is more applicable to some types of reviews. Often, the STROBE tool is used to check the quality of articles. It reviews the title and abstract (item 1), introduction (items 2 and 3), implementation method (items 4–12), findings (items 13–17), discussion (Items 18–21), and funding (item 22). Eighteen items are used to review all articles, but four items (6, 12, 14, and 15) apply in certain situations.[ 9 ] The quality of interventional articles is often evaluated by the JADAD tool, which consists of three sections of randomization (2 scores), blinding (2 scores), and patient count (1 scores).[ 29 ]
At this stage, the researchers extract the necessary information in the selected articles. Elamin believes that reviewing the titles and abstracts and data extraction is a key step in the review process, which is often carried out by two of the research team independently, and ultimately, the results are compared.[ 37 ] This step aimed to prevent selection bias and it is recommended that the chance of agreement between the two researchers (Kappa coefficient) be reported at the end.[ 26 ] Although data collection forms may differ in systematic reviews, they all have information such as first author, year of publication, sample size, target community, region, and outcome. The purpose of data synthesis is to collect the findings of eligible studies, evaluate the strengths of the findings of the studies, and summarize the results. In data synthesis, we can use different analysis frameworks such as meta-ethnography, meta-analysis, or thematic synthesis.[ 38 ] Finally, after quality assessment, data analysis is conducted. The first step in this section is to provide a descriptive evaluation of each study and present the findings in a tabular form. Reviewing this table can determine how to combine and analyze various studies.[ 28 ] The data synthesis approach depends on the nature of the research question and the nature of the initial research studies.[ 39 ] After reviewing the bias and the abstract of the data, it is decided that the synthesis is carried out quantitatively or qualitatively. In case of conceptual heterogeneity (systematic differences in the study design, population, and interventions), the generalizability of the findings will be reduced and the study will not be meta-analysis. The meta-analysis study allows the estimation of the effect size, which is reported as the odds ratio, relative risk, hazard ratio, prevalence, correlation, sensitivity, specificity, and incidence with a confidence interval.[ 26 ]
Estimation of the effect size in systematic review and meta-analysis studies varies according to the type of studies entered into the analysis. Unlike the mean, prevalence, or incidence index, in odds ratio, relative risk, and hazard ratio, it is necessary to combine logarithm and logarithmic standard error of these statistics [ Table 1 ].
Effect size in systematic review and meta-analysis
Systematic review type | Primary studies | Measures of interest |
---|---|---|
Prevalence systematic review | Cross-sectional studies Descriptive studies | Prevalence Mean,correlation |
Observational systematic review | Cohort studies Case-control studies Analytical descriptive studies | OR RR mean difference Standard mean difference |
Clinical trials systematic review | RCT Non-RCT | RR Risk difference NNT, NNH Mean difference |
Diagnostic systematic review | Diagnostic accuracy studies | Sensitivity Specificity PPV, NPV PLR, NLR DOR |
OR=Odds ratio; RR=Relative risk; RCT= Randomized controlled trial; PPV: positive predictive value; NPV: negative predictive value; PLR: positive likelihood ratio; NLR: negative likelihood ratio; DOR: diagnostic odds ratio
A systematic review ends with the interpretation of results. At this stage, the results of the study are summarized and the conclusions are presented to improve clinical and therapeutic decision-making. A systematic review with or without meta-analysis provides the best evidence available in the hierarchy of evidence-based practice.[ 14 ] Using meta-analysis can provide explicit conclusions. Conceptually, meta-analysis is used to combine the results of two or more studies that are similar to the specific intervention and the similar outcomes. In meta-analysis, instead of the simple average of the results of various studies, the weighted average of studies is reported, meaning studies with larger sample sizes account for more weight. To combine the results of various studies, we can use two models of fixed and random effects. In the fixed-effect model, it is assumed that the parameters studied are constant in all studies, and in the random-effect model, the measured parameter is assumed to be distributed between the studies and each study has measured some of it. This model offers a more conservative estimate.[ 40 ]
Three types of homogeneity tests can be used: (1) forest plot, (2) Cochrane's Q test (Chi-squared), and (3) Higgins I 2 statistics. In the forest plot, more overlap between confidence intervals indicates more homogeneity. In the Q statistic, when the P value is less than 0.1, it indicates heterogeneity exists and a random-effect model should be used.[ 41 ] Various tests such as the I 2 index are used to determine heterogeneity, values between 0 and 100; the values below 25%, between 25% and 50%, and above 75% indicate low, moderate, and high levels of heterogeneity, respectively.[ 26 , 42 ] The results of the meta-analyzing study are presented graphically using the forest plot, which shows the statistical weight of each study with a 95% confidence interval and a standard error of the mean.[ 40 ]
The importance of meta-analyses and systematic reviews in providing evidence useful in making clinical and policy decisions is ever-increasing. Nevertheless, they are prone to publication bias that occurs when positive or significant results are preferred for publication.[ 43 ] Song maintains that studies reporting a certain direction of results or powerful correlations may be more likely to be published than the studies which do not.[ 44 ] In addition, when searching for meta-analyses, gray literature (e.g., dissertations, conference abstracts, or book chapters) and unpublished studies may be missed. Moreover, meta-analyses only based on published studies may exaggerate the estimates of effect sizes; as a result, patients may be exposed to harmful or ineffective treatment methods.[ 44 , 45 ] However, there are some tests that can help in detecting negative expected results that are not included in a review due to publication bias.[ 46 ] In addition, publication bias can be reduced through searching for data that are not published.
Systematic reviews and meta-analyses have certain advantages; some of the most important ones are as follows: examining differences in the findings of different studies, summarizing results from various studies, increased accuracy of estimating effects, increased statistical power, overcoming problems related to small sample sizes, resolving controversies from disagreeing studies, increased generalizability of results, determining the possible need for new studies, overcoming the limitations of narrative reviews, and making new hypotheses for further research.[ 47 , 48 ]
Despite the importance of systematic reviews, the author may face numerous problems in searching, screening, and synthesizing data during this process. A systematic review requires extensive access to databases and journals that can be costly for nonacademic researchers.[ 13 ] Also, in reviewing the inclusion and exclusion criteria, the inevitable mindsets of browsers may be involved and the criteria are interpreted differently from each other.[ 49 ] Lee refers to some disadvantages of these studies, the most significant ones are as follows: a research field cannot be summarized by one number, publication bias, heterogeneity, combining unrelated things, being vulnerable to subjectivity, failing to account for all confounders, comparing variables that are not comparable, just focusing on main effects, and possible inconsistency with results of randomized trials.[ 47 ] Different types of programs are available to perform meta-analysis. Some of the most commonly used statistical programs are general statistical packages, including SAS, SPSS, R, and Stata. Using flexible commands in these programs, meta-analyses can be easily run and the results can be readily plotted out. However, these statistical programs are often expensive. An alternative to using statistical packages is to use programs designed for meta-analysis, including Metawin, RevMan, and Comprehensive Meta-analysis. However, these programs may have limitations, including that they can accept few data formats and do not provide much opportunity to set the graphical display of findings. Another alternative is to use Microsoft Excel. Although it is not a free software, it is usually found in many computers.[ 20 , 50 ]
A systematic review study is a powerful and valuable tool for answering research questions, generating new hypotheses, and identifying areas where there is a lack of tangible knowledge. A systematic review study provides an excellent opportunity for researchers to improve critical assessment and evidence synthesis skills.
All authors contributed equally to this work.
Conflicts of interest.
There are no conflicts of interest.
Objective The objective is to evaluate the diagnostic effectiveness of contrast-enhanced spectral mammography (CESM) in the diagnosis of breast cancer.
Data sources PubMed, Embase and Cochrane libraries up to 18 June 2022.
Eligibility criteria for selecting studies We included trials studies, compared the results of different researchers on CESM in the diagnosis of breast cancer, and calculated the diagnostic value of CESM for breast cancer.
Data extraction and synthesis Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) evaluated the methodological quality of all the included studies. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses specification. In addition to sensitivity and specificity, other important parameters were explored in an analysis of CESM accuracy for breast cancer diagnosis. For overall accuracy estimation, summary receiver operating characteristic curves were calculated. STATA V.14.0 was used for all analyses.
Results This meta-analysis included a total of 12 studies. According to the summary estimates for CESM in the diagnosis of breast cancer, the pooled sensitivity and specificity were 0.97 (95% CI 0.92 to 0.98) and 0.76 (95% CI 0.64 to 0.85), respectively. Positive likelihood ratio was 4.03 (95% CI 2.65 to 6.11), negative likelihood ratio was 0.05 (95% CI 0.02 to 0.09) and the diagnostic odds ratio was 89.49 (95% CI 45.78 to 174.92). Moreover, there was a 0.95 area under the curve.
Conclusions The CESM has high sensitivity and good specificity when it comes to evaluating breast cancer, particularly in women with dense breasts. Thus, provide more information for clinical diagnosis and treatment.
Data sharing not applicable as no datasets generated and/or analysed for this study.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .
https://doi.org/10.1136/bmjopen-2022-069788
Request permissions.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
This systematic review was a comprehensive search of experimental and observational studies on contrast-enhanced spectral mammography (CESM) in the diagnosis of breast cancer.
We included only prospective studies. Prospective studies were of higher quality with less bias, and our study screening criteria were developed prior to the meta-analysis.
The study was conducted by two people and was strictly based on inclusion criteria.
The data in this study were summarised using sound statistical methods.
A recent literature was added, and a literature from the same institution included only the most recent or the largest sample size.
We summarised the sensitivity and specificity of CESM in the diagnosis of breast cancer.
Globally, female breast cancer has overtaken lung cancer as the leading cause of cancer death, making it the fifth most common cause of death. 1 From the mid-20th century, the incidence of breast cancer in women has been increasing slowly by about 0.5% per year. 2 At present, the diagnostic methods of breast cancer include MRI, full field digital mammography (FFDM) and ultrasound (US). MRI is the most sensitive examination in the diagnosis of breast cancer at present. 3 However, it has some disadvantages such as no claustrophobic and high price. In addition, although FFDM is an effective diagnostic method for breast cancer, it also has the hazard of recall and needs further testing. 4 Ultrasonography has good diagnostic efficacy for breast cancer, especially in women with dense breasts; however, it has a relatively low positive predictive value. 5 Contrast-enhanced spectral mammography (CESM), which visualises breast neovascularisation in a manner similar to MRI, is an emerging technology that uses iodine contrast agent. 6 CESM has the advantages of patient friendliness and low cost. Previous studies have shown that CESM has obvious advantages in displaying lesions compared with US. The advantage of CESM is that it can show changes in anatomy and local blood perfusion, which may be caused by tumour angiogenesis. 7 Moreover, CESM is useful in detecting the suspicious findings in routine breast imaging 7 and the sensitivity and specificity of CESM are different in different studies.
It has been reported that several meta-analyses have been conducted regarding the diagnostic performance of CESM for breast cancer; however, their pooled results were different and had several limitations. 8–11 On the one hand, the sensitivity and specificity differed across the above-mentioned meta-analyses. 8 10 11 On the other hand, the numbers of included studies were limited. In addition, partial meta-analyses included none-English studies and overlapped studies, which might affect their pooled results. In the past few years, several studies evaluating the diagnostic value of CESM in breast cancer have been published. Therefore, we conducted this meta-analysis using available evidence to comprehensively determine whether CESM is effective in detecting breast cancer in women.
As recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), we conducted our study followed the PRISMA specification, 12 which met the requirements of diagnostic systematic review.
To evaluate the accuracy of CESM in diagnosing breast cancer, we retrieved the following databases: PubMed, Embase and Cochrane library. Two reviewers, JL and RX, independently searched the above databases up to the date of 18 June 2022. Our searching terms included ‘contrast-enhanced spectral mammography’, ‘Dual-Energy Contrast-Enhanced Spectral Mammography’, ‘CESM’, ‘contrast-enhanced digital mammography’, ‘CEDM’, ‘Breast Neoplasms’, ‘Breast Neoplasm’, ‘Breast Tumor’, ‘Breast Tumors’, ‘Breast Cancer’, ‘Malignant Neoplasm of Breast’, ‘Breast Malignant Neoplasm’, ‘Breast Carcinomas’, ‘Breast Carcinoma’, ‘breast mass’, ‘breast lesion’, ‘breast lesions’, ‘breast diseases’. In addition, the references of all the included studies were also reviewed.
Following is the list of inclusion criteria: (1) studies diagnosing breast cancer, (2) studies provided data on the sensitivity and specificity, (3) studies involving ≥10 patients or case, (4) English language and(5) prospective studies. Following is the list of exclusion criteria: (1) overlapped research, (2) commentaries, letters, editorials or abstracts or (3) studies referencing artificial intelligence and radiomics.
The titles and abstracts of the literature in the electronic databases were initially screened by two authors, following the above criteria for inclusion and exclusion. Each of the two researchers screened two times to avoid omission. If there is any disagreement, the third author was consulted to decide. Eligibly downloaded full texts and further screened. First, if the authors and institutions of the study are the same, we will include the most recently published studies with the largest sample size. If the research institutions are the same, but the authors are different, we will send an email to the corresponding authors to ask. If we do not receive a reply, we will include the most recently published studies having the largest sample size.
Two reviewers extracted data. If necessary, the difference shall be solved by the third reviewer. Each study was analysed for the following information: first author name, publication year, country, the numbers of patients and lesions, median age, the results of true positive (TP), false positive (FP), false negative (FN) and true negative (TN).
The quality of the methodology included in the publication was assessed by the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2). 13 QUADAS-2 were mainly focused on the following four domains: patient selection, index test, reference standard and flow and timing, with minimal overlapping, which present the main quality of the diagnostic study. Each domain is assessed according to risk of bias, with the three domains assessed according to applicability. The risk of bias was considered low if the study met the above criteria and high otherwise. Disagreements between the two reviewers on quality assessment were resolved by consensus.
STATA V.14.0 was used for all analyses. I 2 measure was used to quantify the heterogeneity between studies. If there is no statistical heterogeneity, the fixed effect model is used to consolidate the data. On the contrary, the random effect model is used to summarise the data. The sensitivity was shown in the form TP/(TP+FN), where TP represents the number of true-positive results and FN represent the number of FN results. The specificity was shown in the form TN/(TN+FP), where TN represent the number of TN results and FP represent the number of FN results. 14 We also computed other significant measures on the evaluation of diagnostic experiments such as positive likelihood ratio (PLR) and negative likelihood ratio (NLR) and diagnostic OR (DOR). The summary receiver operating characteristic curve ROC (SROC) curve and the area under the curve (AUC) of the SROC curve were also computed.
After a systematic search, we included 12 studies. 15–26 The complete selection process is in detail in PRISMA flowchart ( figure 1 ). From 544 screened studies, 85 studies were subjected to full text reading. The characteristics of all the 12 included studies are shown in table 1 . These 12 studies are all prospective studies published between 2014 and 2022. Most patients had US, mammography and related examinations before CESM examination. The dense breast we collected account for approximately two-thirds. In addition, the methodological quality assessment of all included studies was shown in online supplemental table 1 .
Study characteristics of each included study
The figure shows the workflow for study screening and selection. CESM, contrast-enhanced spectral mammography.
The sensitivity and specificity values were shown in Forest plots ( figure 2 ). A very high pooled test sensitivity of 0.97 (95% CI 0.92 to 0.98) was estimated. The pooled specificity was 0.76 (95% CI 0.64 to 0.85). The PLR was 4.03 (95% CI 2.65 to 6.11), NLR was 0.05 (95% CI 0.02 to 0.09) ( figure 3 ) and DOR was 89.49 (95% CI 45.78 to 174.92) ( online supplemental figure 1 ). I 2 values of sensitivity, specificity, PLR, NLR and DOR were 76.60%, 87.95%, 86.25%, 65.73% and 99.78%, respectively.
Forest plot of estimates of sensitivity and specificity for contrast-enhanced spectral mammography in the diagnosis of breast cancer.
Forest plot of estimates of positive likelihood ratio and negative likelihood ratio for contrast-enhanced spectral mammography in the diagnosis of breast cancer.
As shown in figure 4 , the SROC curve shows an AUC of 0.95 (0.93 to 0.97). CI is an interval estimation based on the average point estimation. The prediction interval is the interval estimation based on the individual value point estimation.
The plot shows the summary bivariate ROC curve for CESM diagnostic accuracy. AUC, area under the curve; CESM, contrast-enhanced spectral mammography; ROC, receiver operating characteristic curve; SENS, sensitivity; SPEC, specificity; SROC, summary receiver operating characteristic curve.
A confidence contour and a prediction contour were shown in the figure.
Fagan plots were drawn to understand the prior probability (current incidence) and the posterior probability (incidence estimated from this diagnostic experiment). In our sample, the pretest probability of malignancy was 50%, with a positive finding at CESM a post-test probability of 80% while a negative finding a post-test probability of 4% ( online supplemental figure 2 ).
We analysed some covariates (number of lesions, number of patients, being dense breast or not, year of publication) possible influence on the diagnostic accuracy of CESM. The regression analysis showed that the sensitivity of the studies that only included dense breast was different from that of other studies, but both were high ( online supplemental figure 3 ). In addition, a limited number of studies were included, which reduced the reliability of the regression analysis.
A funnel plot drawn with Stata V.14.0 software was used to analyse the publication bias of the included studies ( online supplemental figure 4 ). The included studies were evenly distributed on both sides of the regression line, showing that the included literatures had no obvious publication bias (p=0.78).
CESM is emerging as a valuable tool for the diagnosis and staging of breast cancer. CESM combines the contrast enhancement effect caused by tumour neovascularisation with the information of anatomical changes. The lesions were highlighted by reciprocal subtraction of the images, which further increased the sensitivity of CESM for the diagnosis of breast cancer. It improves the accuracy in diagnosing breast cancer, providing more accurate tumour size and identification of multifocal disease, especially in patients with the dense type of breast. 27
Results showed that the pooled sensitivity (0.97, 95% CI 0.92 to 0.98) was higher and the pooled specificity (0.76, 95% CI 0.64 to 0.85) was slightly lower than a previous meta-analysis 9 which indicated a pooled sensitivity of 0.89 (95% CI 0.88 to 0.91) and a pooled specificity of 0.84 (95% CI 0.82 to 0.85). The reason for the high sensitivity may be that our study went through more rigorous study screening, included the latest literature, and CESM has been increasingly used in clinical practice in recent years. Another point is that all the studies we included are prospective studies, which are less susceptible to bias than retrospective studies. Another previous meta-analysis 8 has obtained that CESM has high sensitivity for the diagnosis of breast cancer, but it has low specificity. This may be due to the following reasons: three studies included by the meta-analysis were similar and written by the same first author; the meta-analysis only included eight studies and the pooled specificity were obtained by six literatures. All the reasons may result in some bias. However, during our screening, there are five studies from the same authors 15 28–31 and with similar results, we only included one in which the study type was prospective and with large sample size and longest time span.
In addition, compared with other studies, this study included the latest studies in recent years, and conducted a more rigorous article screening, with each of the two researchers screening two times.
The DOR is a common statistic in epidemiology that expresses the strength of the association between exposure and disease. 32 The diagnostic DOR for a test is the ratio of the odds of being positive in the disease to the odds of being positive in the non-disease. In our meta-analysis, the DOR was 89.49 (95% CI45.78 to 174.92), which was high. It indicated that if CESM showed a positive result, the probability of a true breast cancer being correctly diagnosed was 89.49 to 1. DOR offers considerable advantages in a meta-analysis of diagnostic studies by combining results from different studies into a more precise pooled estimate. The I 2 statistic, also known as the inconsistency index, is a measure of heterogeneity or variability across studies in a meta-analysis. It quantifies the proportion of total variation in effect estimates that is due to heterogeneity rather than chance. Differences in study populations: the studies included in the meta-analysis may have varied in terms of patient characteristics, such as age, mammary gland type, disease severity or comorbidities. These differences can contribute to heterogeneity in the estimated DOR. Clinical and contextual factors: heterogeneity in DOR can also arise from differences in the clinical context, such as variations in disease prevalence, healthcare settings or geographic locations.
The SROC curve method takes into account the possible heterogeneity of thresholds. 33 The SROC indicates the relationship between the TP rate and FP rate at different diagnostic thresholds. 34 In general, the AUC of a diagnostic method between 0.5 and 0.7 means low accuracy, 0.7 and 0.9 means good accuracy, above 0.9 high accuracy. The SROC curve shows an AUC of 0.95, indicating high accuracy.
The study of Hobbs et al 35 reminds of that patients’ preferences for CESM will provide further evidence supporting the adoption of CESM as an alternative to ce-MRI in selected clinical indications, if diagnostic non-inferiority of CESM is confirmed. Ferranti et al 25 suggested that CESM may provide compensation for MRI through a slight FN tendency. Furthermore, Clauser et al 36 thought the specificity of CESM is higher than that of MRI. CEM determines breast cancer based on tumour angiogenesis assessment. 24 Growth factors secreted by cancer cells promote the formation of new blood vessels during division and proliferate to tumour cells. It is because of the increased vascular endothelial cell gap and permeability that the contrast in the tumour area is enhanced. CESM may combine the high sensitivity of MRI with the low cost and availability of FFDM. 37
However, there are some limitations in the study. First, primary source participants were all patients with lesions diagnosed by breast US or mammography. This may induce a selection bias. Second, the majority of the main participants were with dense breast. This point, while highlighting the superiority of CESM over dense breast examination, may still be subject to some bias. Third, due to the excessive number of retrieved literatures, we only included prospective studies and studies writing in English. In this way, some reliable studies and results may be missed.
The CESM has high sensitivity and good specificity when it comes to evaluating breast cancer, particularly in women with dense breasts. Thus, provide more information for clinical diagnosis and treatment.
Patient consent for publication.
Not applicable.
Contributors JL and RX designed the study. SZou and YH gathered data. JL and SZhen performed the analysis. HY and DH revised it critically for important intellectual content. DH acted as guarantor. All authors contributed to the article and approved the submitted version.
Funding This work has received funding by the Henan Medical Science and Technology Research Program (LHGJ20210498,LHGJ20230528).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Traumatic brain injury (TBI) presents complex management scenarios, particularly in patients requiring anticoagulation for concurrent conditions such as venous thromboembolism (VTE) or atrial fibrillation (AF). A systematic search of PubMed/MEDLINE, Embase, and the Cochrane Library databases was conducted to identify relevant studies. Inclusion criteria encompassed studies assessing the effects of anticoagulation therapy on outcomes such as re-hemorrhage, hematoma expansion, thrombotic events, and hemorrhagic events in TBI patients with subdural hematomas (SDH). This systematic review critically addresses two key questions: the optimal timing for initiating anticoagulation therapy and the differential impact of this timing based on the type of intracranial bleed, with a specific focus on subdural hematomas (SDH) compared to other types. Initially screening 508 articles, 7 studies met inclusion criteria, which varied in design and quality, precluding meta-analysis. The review highlights a significant knowledge gap, underscoring the lack of consensus on when to initiate anticoagulation therapy in TBI patients, exacerbated by the need for anticoagulation in the presence of VTE or AF. Early anticoagulation, particularly in patients with SDH, may elevate the risk of re-hemorrhage, posing a clinical dilemma. Evidence on whether the type of intracranial hemorrhage influences outcomes with early anticoagulation remains inconclusive, indicating a need for further research to tailor management strategies effectively. This review underscores the scarcity of high-quality evidence regarding anticoagulation therapy in TBI patients with concurrent conditions, emphasizing the necessity for well-designed prospective studies to elucidate optimal management strategies for this complex patient population.
This is a preview of subscription content, log in via an institution to check access.
Subscribe and save.
Price includes VAT (Russian Federation)
Instant access to the full article PDF.
Rent this article via DeepDyve
Institutional subscriptions
No datasets were generated or analysed during the current study.
Global, regional, and national burden of traumatic brain injury and spinal cord injury, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. (2018) [cited 2023 Aug 12]; https://doi.org/10.1016/S1474-4422(18)30415-0
Carney N, Totten AM, O’Reilly C, Ullman JS, Hawryluk GWJ, Bell MJ, et al. (2017) Guidelines for the Management of Severe Traumatic Brain Injury, Fourth Edition. Neurosurgery. [cited 2023 Jun 15];80(1):6–15. https://journals.lww.com/neurosurgery/Fulltext/2017/01000/Guidelines_for_the_Management_of_Severe_Traumatic.3.aspx
Yuan F, Ding J, Chen H, Guo Y, Wang G, Gao WW, et al. (2012) Predicting progressive hemorrhagic injury after traumatic brain injury: derivation and validation of a risk score based on admission characteristics. J Neurotrauma. [cited 2024 Mar 17];29(12):2137–42. https://pubmed.ncbi.nlm.nih.gov/22568757/
Delayed and Progressive Brain Injury in Closed-Head Trauma: Neurosurgery. [cited 2024 Mar 17]. https://journals.lww.com/neurosurgery/abstract/1993/01000/delayed_and_progressive_brain_injury_in.4.aspx
Chang EF, Meeker M, Holland MC (2006) Acute traumatic intraparenchymal hemorrhage: risk factors for progression in the early post-injury period. Neurosurgery. [cited 2024 Mar 16];58(4):647–55. https://pubmed.ncbi.nlm.nih.gov/16575328/
Chieregato A, Fainardi E, Maria Morselli-Labate A, Antonelli V, Compagnone C, Targa L, et al. (2005) Factors associated with neurological outcome and lesion progression in traumatic subarachnoid hemorrhage patients. Neurosurgery. [cited 2024 Mar 16];56(4):671–9. https://pubmed.ncbi.nlm.nih.gov/15792505/
Engström M, Romner B, Schalén W, Reinstrup P (2005) Thrombocytopenia predicts progressive hemorrhage after head trauma. J Neurotrauma. [cited 2024 Mar 16];22(2):291–6. https://pubmed.ncbi.nlm.nih.gov/15716634/
Narayan RK, Maas AIR, Servadei F, Skolnick BE, Tillinger MN, Marshall LF, et al. (2008) Progression of traumatic intracerebral hemorrhage: a prospective observational study. J Neurotrauma. [cited 2024 Mar 16];25(6):629–39. https://pubmed.ncbi.nlm.nih.gov/18491950/
Oertel M, Kelly DF, McArthur D, John Boscardin W, Glenn TC, Jae HL, et al. (2002) Progressive hemorrhage after head trauma: predictors and consequences of the evolving injury. J Neurosurg. [cited 2024 Mar 16];96(1):109–16. https://pubmed.ncbi.nlm.nih.gov/11794591/
Schnüriger B, Inaba K, Abdelsayed GA, Lustenberger T, Eberle BM, Barmparas G et al (2010) The impact of platelets on the progression of traumatic intracranial hemorrhage. J Trauma - Injury Infect Critic Care 68(4):881–885
Article Google Scholar
Servadei F, Murray GD, Penny K, Teasdale GM, Dearden M, Iannotti F et al. (2000) The Value of the ???Worst??? Computed Tomographic Scan in Clinical Studies of Moderate and Severe Head Injury. Neurosurgery. 70–7
Gardner RC, Dams-O’Connor K, Morrissey MR, Manley GT (2018) Geriatric traumatic brain injury: Epidemiology, outcomes, knowledge gaps, and future directions. Vol. 35, Journal of Neurotrauma. Mary Ann Liebert Inc. p. 889–906
Ramanathan DM, McWilliams N, Schatz P, Hillary FG (2012) Epidemiological shifts in elderly traumatic brain injury: 18-year trends in Pennsylvania. J Neurotrauma 29(7):1371–1378
Article PubMed Google Scholar
Coronado VG, Thomas KE, Sattin RW, Johnson RL (2005) The CDC traumatic brain injury surveillance system: characteristics of persons aged 65 years and older hospitalized with a TBI. J Head Trauma Rehabil. [cited 2023 Jun 13];20(3):215–28. Available from: https://pubmed.ncbi.nlm.nih.gov/15908822/
Chan V, Zagorski B, Parsons D, Colantonio A (2013) Older adults with acquired brain injury: a population based study. BMC Geriatr. [cited 2023 Jun 14];13(1). https://pubmed.ncbi.nlm.nih.gov/24060144/
De Guise E, Leblanc J, Dagher J, Tinawi S, Lamoureux J, Marcoux J, et al. (2014) Trends in Hospitalization Associated with TBI in an Urban Level 1 Trauma Centre. Can J Neurol Sci. [cited 2023 Jun 14];41(4):466–75. Available from: https://pubmed.ncbi.nlm.nih.gov/24878471/
Korhonen N, Niemi S, Parkkari J, Sievan̈en H, Kannus P (2013) Incidence of fall-related traumatic brain injuries among older Finnish adults between 1970 and 2011. JAMA. [cited 2024 Mar 16];309(18):1891–2. https://pubmed.ncbi.nlm.nih.gov/23652518/
Brazinova A, Mauritz W, Majdan M, Rehorcikova V, Leitgeb J (2015) Fatal traumatic brain injury in older adults in Austria 1980–2012: an analysis of 33 years. Age Ageing. [cited 2024 Mar 16];44(3):502–6. https://pubmed.ncbi.nlm.nih.gov/25520311/
Hamill V, Barry SJE, McConnachie A, McMillan TM, Teasdale GM (2015) Mortality from Head Injury over Four Decades in Scotland. J Neurotrauma. [cited 2024 Mar 16];32(10):689–703. https://pubmed.ncbi.nlm.nih.gov/25335097/
Haring RS, Narang K, Canner JK, Asemota AO, George BP, Selvarajah S et al. (2015) Traumatic brain injury in the elderly: morbidity and mortality trends and risk factors. J Surg Res. [cited 2024 Mar 16];195(1):1–9. https://pubmed.ncbi.nlm.nih.gov/25724764/
ACS TQIP Geriatric Trauma Management Guidelines. 2013
Kumar RG, Juengst SB, Wang Z, Dams-O’Connor K, Dikmen SS, O’Neil-Pirozzi TM et al. (2018) Epidemiology of Comorbid Conditions Among Adults 50 Years and Older With Traumatic Brain Injury. J Head Trauma Rehabil. [cited 2023 Jun 12];33(1):15–24. https://pubmed.ncbi.nlm.nih.gov/28060201/
Hawley C, Sakr M, Scapinello S, Salvo J, Wrenn P (2017) Traumatic brain injuries in older adults - 6 years of data for one UK trauma centre: Retrospective analysis of prospectively collected data. Emerg Med J 34(8):509–516
Chauhan AV, Guralnik J, Dosreis S, Sorkin JD, Badjatia N, Albrecht JS (2022) Repetitive Traumatic Brain Injury Among Older Adults. J Head Trauma Rehabil. [cited 2023 Jun 12];37(4):E242–8. https://pubmed.ncbi.nlm.nih.gov/34320558/
Lavoie A, Ratte S, Clas D, Demers J, Moore L, Martin M et al. (2004) Preinjury warfarin use among elderly patients with closed head injuries in a trauma center. J Trauma. [cited 2023 Jun 14];56(4):802–7. https://pubmed.ncbi.nlm.nih.gov/15187746/
Samuel S, Menchaca C, Gusdon AM (2022) Timing of anticoagulation for venous thromboembolism after recent traumatic and vascular brain Injury. J Thromb Thrombolysis
Iaccarino C, Carretta A, Demetriades AK, Di Minno G, Giussani C, Marcucci R et al. (2023) Management of Antithrombotic Drugs in Patients with Isolated Traumatic Brain Injury: An Intersociety Consensus Document. Neurocrit Care
Hawryluk GWJ, Austin JW, Furlan JC, Lee JB, O’Kelly C, Fehlings MG (2010) Management of anticoagulation following central nervous system hemorrhage in patients with high thromboembolic risk. J Thromb Haemost 8(7):1500–1508
Article CAS PubMed Google Scholar
Geerts WH, Code KI, Jay RM, Chen E, Szalai JP (1994) A prospective study of venous thromboembolism after major trauma. N Engl J Med. [cited 2023 May 28];331(24):1601–6. https://pubmed.ncbi.nlm.nih.gov/7969340/
Divito A, Kerr K, Wilkerson C, Shepard S, Choi A, Kitagawa RS (2019) Use of Anticoagulation Agents After Traumatic Intracranial Hemorrhage. World Neurosurg 1(123):e25-30
Connolly SJ, Crowther M, Eikelboom JW, Gibson CM, Curnutte JT, Lawrence JH et al (2019) Full Study Report of Andexanet Alfa for Bleeding Associated with Factor Xa Inhibitors. N Engl J Med 380(14):1326–1335
Article CAS PubMed PubMed Central Google Scholar
Staerk L, Fosbøl EL, Lamberts M, Bonde AN, Gadsbøll K, Sindet-Pedersen C, et al. (2018) Resumption of oral anticoagulation following traumatic injury and risk of stroke and bleeding in patients with atrial fibrillation: a nationwide cohort study. Eur Heart J. [cited 2023 May 28];39(19):1698–705. https://pubmed.ncbi.nlm.nih.gov/29165556/
Xu Y, Shoamanesh A, Schulman S, Dowlatshahi D, Al- R, Salman S et al. (2018) Oral anticoagulant re-initiation following intracerebral hemorrhage in non-valvular atrial fibrillation: Global survey of the practices of neurologists, neurosurgeons and thrombosis experts. [cited 2023 Jun 16]; https://doi.org/10.1371/journal.pone.0191137
Hawryluk GWJ, Furlan JC, Austin JW, Fehlings MG (2011) Survey of neurosurgical management of central nervous system hemorrhage in patients receiving anticoagulation therapy: current practice is highly variable and may be suboptimal. World Neurosurg. [cited 2023 May 28];76(3–4):299–303. https://pubmed.ncbi.nlm.nih.gov/21986428/
Gregson BA, Mendelow AD (2003) International variations in surgical practice for spontaneous intracerebral hemorrhage. Stroke (1970). (11):2593–7
Siddique MS, Gregson BA, Fernandes HM, Barnes J, Treadwell L, Wooldridge TD et al (2002) Comparative study of traumatic and spontaneous intracerebral hemorrhage. J Neurosurg 1:86–89
Park YA, Uhm JS, Pak HN, Lee MH, Joung B (2016) Anticoagulation therapy in atrial fibrillation after intracranial hemorrhage. Heart Rhythm. [cited 2023 Jun 15];13(9):1794–802. https://pubmed.ncbi.nlm.nih.gov/27554947/
Byrnes MC, Irwin E, Roach R, James M, Horst PK, Reicks P (2012) Therapeutic anticoagulation can be safely accomplished in selected patients with traumatic intracranial hemorrhage. World J Emerg Surg. [cited 2023 Jun 20];7(1). https://pubmed.ncbi.nlm.nih.gov/22824193/
Milling TJ, Refaai MA, Goldstein JN, Schneider A, Omert L, Harman A et al. (2016) Thromboembolic Events After Vitamin K Antagonist Reversal With 4-Factor Prothrombin Complex Concentrate: Exploratory Analyses of Two Randomized, Plasma-Controlled Studies. Ann Emerg Med. [cited 2023 May 28];67(1):96–105.e5. https://pubmed.ncbi.nlm.nih.gov/26094105/
Nielsen PB, Larsen TB, Skjøth F, Lip GYH (2017) Outcomes associated with resuming warfarin treatment after hemorrhagic stroke or traumatic intracranial hemorrhage in patients with atrial fibrillation. JAMA Intern Med 177(4):563–570
Article PubMed PubMed Central Google Scholar
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C (2016) PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. [cited 2024 Mar 16];75:40–6. https://pubmed.ncbi.nlm.nih.gov/27005575/
CASP Checklists - Critical Appraisal Skills Programme. [cited 2024 Mar 16]. https://casp-uk.net/casp-tools-checklists/
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. (2011) GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 64(4):383–94
Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, et al. (2011) GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. [cited 2024 Mar 16];64(4):401–6. https://pubmed.ncbi.nlm.nih.gov/21208779/
Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P et al. (2011) GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). J Clin Epidemiol. [cited 2024 Mar 16];64(4):407–15. https://pubmed.ncbi.nlm.nih.gov/21247734/
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M et al. (2011) GRADE guidelines: 7. Rating the quality of evidence--inconsistency. J Clin Epidemiol. [cited 2024 Mar 16];64(12):1294–302. https://pubmed.ncbi.nlm.nih.gov/21803546/
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M et al. (2011) GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol. [cited 2024 Mar 16];64(12):1303–10. https://pubmed.ncbi.nlm.nih.gov/21802903/
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D et al. (2011) GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. [cited 2024 Mar 16];64(12):1283–93. https://pubmed.ncbi.nlm.nih.gov/21839614/
Kia M, Saluja RS, Marcoux J (2023) Acute Traumatic Subdural Hematoma and Anticoagulation Risk. Can J Neurol Sci 50(2):188–193
Pandya U, Pattison J, Karas C, O’mara M (2018) Does the Presence of Subdural Hemorrhage Increase the Risk of Intracranial Hemorrhage Expansion after the Initiation of Antithrombotic Medication? Am Surg. 84(3):416–421
Matsushima K, Leichtle SW, Wild J, Young K, Chang G, Demetriades D et al. (2021) Anticoagulation therapy in patients with traumatic brain injury: An Eastern Association for the Surgery of Trauma multicenter prospective study. In: Surgery (United States). Mosby Inc. 470–6
Chipman AM, Radowsky J, Vesselinov R, Chow D, Schwartzbauer G, Tesoriero R et al (2020) Therapeutic anticoagulation in patients with traumatic brain injuries and pulmonary emboli. J Trauma Acute Care Surg 89(3):529–535
Ghenbot Y, Arena JD, Howard S, Wathen C, Kumar MA, Schuster JM (2023) Anticoagulation Holiday: Resumption of Direct Oral Anticoagulants for Atrial Fibrillation in Patients with Index Traumatic Intracranial Hemorrhage. World Neurosurg X 1:17
Google Scholar
Albrecht JS, Liu X, Baumgarten M, Langenberg P, Rattinger GB, Smith GS et al (2014) Benefits and risks of anticoagulation resumption following traumatic brain injury. JAMA Intern Med 174(8):1244–1251
Powers WJ (2010) Intracerebral hemorrhage and head trauma: common effects and common mechanisms of injury. Stroke 41(10 Suppl):S107–S110. https://doi.org/10.1161/STROKEAHA.110.595058
Cavallari LH, Shin J, Perera MA (2011) Role of pharmacogenomics in the management of traditional and novel oral anticoagulants. Pharmacotherapy 31(12):1192–1207. https://doi.org/10.1592/phco.31.12.1192
Download references
We would like to thank librarians Sonya Fogg, MLS and Celeste Perez, MLS, from the TMC Library, 1133 John Freeman Blvd., Houston, TX 77030, for their invaluable assistance with the literature search and data acquisition for this systematic review.
Not applicable.
Authors and affiliations.
Memorial Hermann-Texas Medical Center, 6411 Fannin Street, Houston, TX, 77030, USA
Sophie Samuel & Jennifer Cortes
McGovern Medical School at UT Health, University of Texas, 6431 Fannin Street, Houston, TX, 77030, USA
Eugene Uh & Huimahn Alex Choi
You can also search for this author in PubMed Google Scholar
S.S. conceptualized the study and formulated the research questions. S.S. conducted the initial review of all 508 articles, while J.C. and E.U. each reviewed 254 articles. Discrepancies were resolved by HAC. S.S., J.C., and E.U. used the JBI Critical Appraisal Checklist for Cohort Studies for data collection, screening, and appraisal. All authors contributed to writing and reviewing the manuscript.
Correspondence to Sophie Samuel .
Ethical approval, conflict of interest.
The authors declare no competing interests.
Publisher's note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Reprints and permissions
Samuel, S., Cortes, J., Uh, E. et al. A systematic review of the timing of therapeutic anticoagulation in adult patients with acute traumatic brain injury: narrative synthesis of observational studies. Neurosurg Rev 47 , 538 (2024). https://doi.org/10.1007/s10143-024-02717-1
Download citation
Received : 22 June 2024
Revised : 09 August 2024
Accepted : 18 August 2024
Published : 05 September 2024
DOI : https://doi.org/10.1007/s10143-024-02717-1
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
IMAGES
VIDEO
COMMENTS
Evidence syntheses are much more time-intensive than traditional literature reviews and require a multi-person research team. See this PredicTER tool to get a sense of a systematic review timeline (one type of evidence synthesis). Before embarking on an evidence synthesis, it's important to clearly identify your reasons for conducting one.
Qualitative systematic review: Qualitative synthesis: Synthesis of qualitative data a: Qualitative synthesis: Synthesis without meta-analysis ... Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016; 13 (5):1-31. doi: 10.1371 ...
1. INTRODUCTION. Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the "gold standard" of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search ...
As well as synthesis of these studies' findings, there should be an element of evaluation and quality assessment. ... 2015) - although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions. Systematic reviews can be inadvisable for a variety of reasons. It may be that the topic ...
The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information.
A Systematic Review (SR) is a synthesis of evidence that is identified and critically appraised to understand a specific topic. SRs are more comprehensive than a Literature Review, which most academics will be familiar with, as they follow a methodical process to identify and analyse existing literature (Cochrane, 2022).
A systematic review identifies and synthesizes all relevant studies that fit prespecified criteria to answer a research question (Lasserson et al. 2019; IOM 2011).What sets a systematic review apart from a narrative review is that it follows consistent, rigorous, and transparent methods established in a protocol in order to minimize bias and errors.
Research Synthesis Methods journal enables cross-fertilization across all scientific disciplines, publishing advances in the practices and methodologies for conducting research syntheses and systematic reviews. Spanning numerous disciplines, including health and social sciences, the journal allows researchers to learn, interact with and use one ...
Knowledge synthesis provides comprehensive overviews and interpretations of research data. Systematic reviews are highly methodological, whereas narrative reviews allow for a broader interpretation but lack methodological rigor. ... The methods section of a knowledge synthesis or review serves a critical function in ensuring the transparency ...
JBI Systematic Reviews The core of evidence synthesis is the systematic review of literature of a particular intervention, condition or issue. The systematic review is essentially an analysis of the available literature (that is, evidence) and a judgment of the effectiveness or otherwise of a practice, involving a series of complex steps.
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to ...
In a systematic review, researchers do more than summarize findings from identified articles. You will synthesize the information you want to include. While a summary is a way of concisely relating important themes and elements from a larger work or works in a condensed form, a synthesis takes the information from a variety of works and ...
Synthesis of existing research: Conclusions are more qualitative and may not be based on study quality. ... Types of evidence synthesis include: Systematic Review. Systematically and transparently collect and categorize existing evidence on a broad question of scientific, policy or management importance. ...
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
Synthesis is the process of combining the findings of research studies. A synthesis is also the product and output of the combined studies. This output may be a written narrative, a table, or graphical plots, including statistical meta-analysis. ... If a systematic review question is about the effectiveness of an intervention, then the included ...
Systematic Review: Comprehensive literature synthesis on a specific research question, typically requires a team: Systematic; exhaustive and comprehensive; search of all available evidence: Yes: Yes: Narrative and tables, describes what is known and unknown, recommendations for future research, limitations of findings:
In a qualitative systematic review, data can be presented in a number of different ways. A typical procedure in the health sciences is thematic analysis. As explained by James Thomas and Angela Harden (2008) in an article for BMC Medical Research Methodology: "Thematic synthesis has three stages: the coding of text 'line-by-line'
Formulating a research question is key to a systematic review. It will be the foundation upon which the rest of the research is built. At this stage in the process, you will have identified a knowledge gap in your field, and you are aiming to answer a specific question. ... Beyond PICO: The SPIDER tool for qualitative evidence synthesis ...
The first is a well-developed research question that gives direction to the synthesis (e.g., meta-analysis, systematic review, meta-study, concept analysis, rapid review, realist synthesis). The second begins as a broad general question that evolves and becomes more refined over the course of the synthesis (e.g., meta-ethnography, scoping ...
The term QES is used, and is the preferred term of the Cochrane Qualitative and Implementation Methods Group, as it acknowledges that qualitative research requires its own methods for synthesis which reflects the nature of the qualitative paradigm, rather than simply using the same methods devised for systematic reviews of quantitative research (Booth et al., 2016).
Create your own concept table for your research question and complete the keyword brainstorming section . Move on to the next page, where you'll learn about the specific controlled vocabulary PubMed uses and how to locate relevant terms within it ... In an evidence synthesis project like a systematic review, you must document: Where you ...
Rayyan is a user-friendly tool which enables a single person or a team to perform masked screening of references for evidence synthesis projects. It has some excellent features, especially if you're working with a large set of results. Rayyan is designed for screening, not for citation management or citing while writing!
This course brings the core components of the Evidence Synthesis Institute to a self-paced open online course. It contains 15 modules guiding learners through the ES process from an introduction to review types through writing a methods section for publication, with an emphasis on developing and using systematic search strategies. Development of this
This thematic synthesis of data identified from a systematic review of the literature was focused on physicians' views regarding the utility of clinical indicators in practice. This is important to understand given the increasing use of clinical indicators and expectations that physicians will use and act on clinical indicator data.
The systematic review is an important technology for the evidence-informed policy and practice movement, which aims to bring research closer to decision-making [1, 2].This type of review uses rigorous and explicit methods to bring together the results of primary research in order to provide reliable answers to particular questions [3-6].The picture that is presented aims to be distorted ...
Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic reviews. 2016; 5(1):74. ... J., & Copley, J. (2015). The meaning of leisure for children and young people with physical disabilities: A systematic evidence synthesis. Developmental Medicine & Child Neurology ...
Scoping reviews are a knowledge synthesis approach that aims to uncover the volume, range, reach and coverage of a body of literature on a specific topic.49 They differ from systematic reviews, another type of knowledge synthesis, in their objectives. Systematic reviews seek to answer clinical or epidemiological questions and are conducted to ...
In this study, the steps for a systematic review such as research question design and identification, the search for qualified published studies, the extraction and synthesis of information that pertain to the research question, and interpretation of the results are presented in details. This will be helpful to all interested researchers.
Data extraction and synthesis Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) evaluated the methodological quality of all the included studies. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses specification. In addition to sensitivity and specificity, other important parameters were explored in an analysis of CESM accuracy for breast ...
Eligibility criteria. The eligibility criteria were defined to ensure relevance and robustness in addressing the research questions. For a comprehensive overview of these criteria, refer to Table 1.Criteria encompassed various study designs, including meta-analyses, systematic reviews, randomized controlled trials, prospective and retrospective cohort studies, case-control studies, and ...