Evidence Synthesis

Type of Review

Typical Timeframe & Resources

Purpose

Databases/ Searches

Inclusion/ Exclusion & Appraisal

Synthesis

Limitations

Critical Review

~1-6 months, bulk of time is spent reading resources & critically analyzing.

Single person

Demonstrates author has extensively researched and critically evaluated the quality of literature in the area. Includes significant degree of analysis and conceptual innovation. Covers limited number of articles/ works.

Seeks to identify most significant works in the field, does not necessarily follow a standard protocol/ structured search, or include grey literature.

No formal quality assessment criteria, works are evaluated based on contribution to the field or other qualitative criteria

Narrative, highlighting the analysis component. May be thematic or chronological. Highlights future research areas or outlines a new theoretical framework.

Not a true “synthesis”, may be a starting point
No formal structure/ reporting requirement on methods
Interpretative, subjective,  high risk of bias
Search is not comprehensive

Comprehensive/ Systematic Literature Review

~1-6 months, bulk of time is spent reading resources & critically analyzing.

Single person

Provides summarization of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness.

May include exploratory search strategies rather than strict protocols; should take advantage of Boolean and controlled vocabularies. May include articles, books, theses, and other sources.

May or may not include quality assessment. Typically less analysis of quality in favour of objective summarization. Can be transparent and intentional in inclusion/ exclusion.

Descriptive summary of the findings, may be organized thematically or chronologically.

No formal structure/ reporting requirement on methods
Interpretative, subjective,  high risk of bias
No necessary quality assessment/ tests for validity of included studies

Mapping Review

~6-12+ months. Time is divided between search & visual depiction of the field.

May require more time screening articles due to the larger volume of studies from covering a wider scope.

Single person or team, experience with visualization tools

Provides a visual synthesis of data that responds to a particular question. Mapping reviews are best for:
- When there is an abundance and a diversity of research.
- As a first step to a systematic review.
- To identify gaps in a topic area.
- To assess whether a population of studies is suitable for a synthesis/ resources required for larger review.

Rapid/as time allows searching aimed to give a broad overview, still aims to be thorough and replicable.  May be limited to a certain type of article—eg. review articles, peer reviewed journals, or grey literature/research in progress.

Based on inclusion/exclusion criteria, should include a PRISMA flow diagram.  Appraisal/quality assessment may be limited in favour of classification & grouping.

Visual synthesis and classification of the available studies (graphical and/ or tabular). A high level map visualizing the status of the field related to the research question, identifying gaps.

Often do not include quality assessment, group studies only based on design not validity
Usually time constrained so may not be fully comprehensive
May have only one reviewer, increasing risk of bias

Rapid Review

~1-8 months. Time is divided between designing question/ search strategy, reviewing and evaluating sources, and summarizing.

Usually multiple people to reduce bias, may include non-blinded appraisal and selection

Applies parts of Systematic Review methodology within a time-constrained setting; faster, but less rigorous as a result. Usually for a tightly defined question. Employs "shortcuts" (eg: limiting search terms) at the risk of introducing bias/ reducing comprehensivity. Useful for addressing issues needing quick decisions, such as developing policy recommendations.

Sources are limited due to time constraints of searching, however still uses transparent and reproducible search methods. May exclude hand searching and grey lit, limited databases.
May restrict by years, language, and other parameters to reduce results.

Should document steps/ decisions according to PRISMA, even if they are not followed (eg, risk of bias assessment). Still likely to have inclusion/ exclusion, critical appraisal, but may be constrained/ streamlined due to time constraints.

Descriptive summary of the findings. Can be qualitative, quantitative, or mixed. Tailored to the specific research question.

Time and resource constraints limit generalizability and validity of findings
Limiting search duration and limits introduces publication bias and other risks
Methodology is not as agreed upon as other Reviews
May only be one reviewer, increasing bias
Search may not be comprehensive
Possible non-blinded appraisal and selection
Provide a limited interpretation of results

Critically Appraised Topic

From https://www.tandfonline.com/doi/full/10.1080/17489539.2017.1333683

~1-3 months. Time is dived between designing question, effective search strategy, screening, critical appraisal of selected articles, discussion of findings.

Single person

Subtype of rapid review. Provides a snapshot of the best available evidence on a focused question; typically identifies, critically appraises, and synthesizes evidence from 3 to 5 pieces of the best available evidence.

Search may be comprehensive, but sources selected will be narrow. Decision making on search & selection needs to be documented and transparent.

Transparent inclusion/ exclusion criteria, ranking of different types of studies into different levels of evidence based on methodological rigor and appropriateness to the question. Studies deemed to represent the highest level of evidence (based on study design) and have most relevance to the question are retrieved for further analysis.

For each study: Is the study valid? What are the results? Will the results help locally?
Authors should decide whether the evidence is weak or strong, if the study design and quality are good, and if the study findings are consistent. Recommendations may include implications for local context, need for updated studies, etc.

Source selection can introduce high risk of bias
Not comprehensive/ representative of full body of knowledge on a subject

Scoping Review

~8 weeks-2 years. Time is spent formulating question, developing a protocol, refining search, screening studies, summarizing and synthesizing relevant studies, reporting results and identifying gaps and/or trends

Conducted by a team including content and methodology specialists

Preliminary assessment of potential size and scope of available research literature on a given topic. Aims to identify nature and extent of research evidence (usually including ongoing research) – may not focus specifically on particular interventions and their analysis.

Comprehensiveness is the goal. Aim is for transparent, reproducible, comprehensive searches. Comprehensive searches using replicable syntax across databases. May involve multiple structured searches -hand searching is important, grey lit, theses, and other data types may be included, includes research in progress. Returns more results than systematic reviews as question/ terms are broader, requires more screening time.

Occasionally broader search terms than systematic reviews (not limited to particular intervention outcomes) and therefore more results, may not have formalized quality assessment. Based on inclusion/exclusion criteria, due to the iterative nature of a scoping review some changes may be necessary.  May require more time spent screening articles due to the larger volume of results from broader questions. Critical appraisal and risk of bias are optional/ where applicable.

Characterizes quantity and quality of literature, perhaps by study design and other key features. Attempts to specify a viable review.

Time and resource constraints limit generalizability and validity of findings
May not include quality assessment, cannot address validity of included studies or make best practice recommendations without this
Intended as a preliminary step to a further review that makes actual recommendations

“Systematized” Reviews

Fluid. Often a single person, reflecting the nature of these being typically “assignments”

Usually performed as an assignment, so students learn some elements of systematized evidence synthesis without having to follow all of the systems

Typically search strategies are the most similar to other evidence synthesis methods.

May or may not include quality assessment

Typically narrative, may include tables, or could have more systematized analysis/ synthesis from other methods

Not generalizable or necessarily accurate
Not fully transparent
High possibility of bias due to less rigorous quality assessment/ synthesis

Systematic Review

~1-2 years. Time is spent defining research question, developing a protocol (inclusion/ exclusion criteria), testing, refining, and translating search strategies, title and abstract screening, full text screening, risk of bias assessment, data extraction & analysis.

Teams of 2+ to reduce bias

Seeks to systematically search for, appraise and synthesize research evidence on a given topic, following strict guidelines as set out by PRISMA or AMSTAR

Aim is for transparent, reproducible, comprehensive searches following a described and registered protocol. Comprehensive searches using replicable syntax translated across databases. Includes searching multiple databases, grey literature/clinical trial registries, and handsearching of the literature (performed by the subject matter expert). It is best practice to involve a librarian or an information specialist in creating, translating, and documenting the search, as well as deduplicating the repeating references in a citation manager and writing the search methods for the review.

Strict screening and critical appraisal steps. Studies may be excluded based on quality assessment. Two phase screening using multiple reviewers first on title and abstract, then on full text. Inclusion & exclusion criteria, quality assessment means studies may be excluded based on methodology, quality, etc. Appraisal of quality typically requires data analysis and synthesis.

Descriptive summaries of the full methodology and results. Discussion usually covers what the strength of evidence is in the area, recommendations, and gaps in research.

Limiting only to particular study types (such as RCT) can reduce insights

Meta-analysis

~18 -24 months. Time is spent on all aspects of systematic review, as well as typically two stages of data analysis.

Technique that statistically combines the results of disparate quantitative studies to provide a more precise measure of the results. Not all systematic reviews include meta-analysis, but all meta-analyses will include a systematic review of the evidence.

Aim is for reproducible comprehensiveness, frequently 5 or more databases, see Systematic Review.

See Systematic Review. Inclusion & exclusion criteria, studies may be excluded based on methodology, quality, etc. Focus on identifying the data and translating the data across studies.

Numerical analysis of measures of effect.

Required advanced statistical knowledge
Dependent on the quality of the included studies
May combine studies that are not sufficiently similar (should be carefully controlled for)

Qualitative Meta Synthesis

~3-12 months. Time is divided between search/ discovery and analysis - repeated reading of articles to connect and record concepts or themes.

2+ people, usually multidisciplinary. Should have teams familiar with content and methodology.

An intentional approach to analyzing data and conclusions across qualitative studies. Researchers identify a specific research question and then search for, select, appraise, summarize, and combine qualitative evidence to address the research question. May use PICO (Population, Interest, Context), SPICE, SPIDER or PerSPECTiF question formats.

Search focuses specifically on qualitative studies, may include more grey literature, book chapters or conference proceedings or theses. Refer to the 7 S Framework of Qualitative Searching.

Appraisal: Focus is on the translation of elements of qualitative studies. Three common primary levels of analysis:
Reciprocal Translation: Concepts in one study can incorporate those of another.
Refutational Translation: Concepts in different studies contradict one another.
Line of Argument Synthesis: Studies identify different aspects of the topic that can be drawn together in a new interpretation.

An interpretive and descriptive summary showing the relationships between qualitative studies on the question.

Methodology is not agreed upon
Analysis stage is likely to be subjective/ at risk of bias (which is moderated by having multiple team members)
May end up aggregating research, rather than synthesizing/ analyzing