Evidence Synthesis

Type of Review

Timeframe

Purpose

Resources

Databases/ Searches

Inclusion/Exclusion & Appraisal

Synthesis

Critical Review

~1-6 months, bulk of time is spent reading resources & critically analyzing.

Demonstrates author has extensively researched and critically evaluated the quality of literature in the area. Includes significant degree of analysis and conceptual innovation. Covers limited number of works.

Single person

Seeks to identify most significant works in the field, does not necessarily follow a standard protocol/ structured search, or include grey literature.

No formal quality assessment criteria, works are evaluated based on contribution to the field or other qualitative criteria

Narrative, highlighting the analysis component. May be thematic or chronological. Highlights future research areas or outlines a new theoretical framework.

Comprehensive/ Systematic Literature Review

~1-6 months, bulk of time is spent reading resources & critically analyzing.

Provides summarization of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness.

Single person

May include exploratory search strategies rather than strict protocols; should take advantage of Boolean and controlled vocabularies. May include articles, books, theses, and other sources.

May or may not include quality assessment. Typically less analysis of quality in favour of objective summarization. Can be transparent and intentional in inclusion/ exclusion.

Descriptive summary of the findings, may be organized thematically or chronologically.

Mapping Review

~6-12+ months. Time is divided between search & visual depiction of the field.

May require more time screening articles due to the larger volume of studies from covering a wider scope. 

Provides a visual synthesis of data that responds to a particular question. Mapping reviews are best for:
- When there is an abundance and a diversity of research.

- As a first step to a systematic review.

- To identify gaps in a topic area.

Single person, needs expertise in visualization tools

Rapid/as time allows searching aimed to give a broad overview, still aims to be thorough and replicable.  May be limited to a certain type of article—eg. review articles, peer reviewed journals, or grey literature/research in progress.

Based on inclusion/exclusion criteria, should include a PRISMA flow diagram.  Appraisal/quality assessment may be limited in favour of classification & grouping.

Visual synthesis and classification of the available studies (graphical and/ or tabular). A high level map visualizing the status of the field related to the research question, identifying gaps.

Rapid Review

~1-8 months. Time is divided between designing question/ search strategy, reviewing and evaluating sources, and summarizing.

Applies parts of Systematic Review methodology within a time-constrained setting; faster, but less rigorous as a result. Usually for a tightly defined question. Employs "shortcuts" (eg: limiting search terms) at the risk of introducing bias/ reducing comprehensivity. Useful for addressing issues needing quick decisions, such as developing policy recommendations.

Usually requires multiple people to reduce bias, may include non-blinded appraisal and selection

Sources are limited due to time constraints of searching, however still uses transparent and reproducible search methods. May exclude hand searching and grey lit, limited databases.

May restrict by years, language, and other parameters to reduce results.

Should document steps/ decisions according to PRISMA, even if they are not followed (eg, risk of bias assessment). Still likely to have inclusion/ exclusion, critical appraisal, but may be constrained/ streamlined due to time constraints.

Descriptive summary of the findings. Can be qualitative, quantitative, or mixed. Tailored to the specific research question.

Critically Appraised Topic

~1-3 months. Time is dived between designing question, effective search strategy, screening, critical appraisal of selected articles, discussion of findings.

Subtype of rapid review. Provides a snapshot of the best available evidence on a focused question; typically identifies, critically appraises, and synthesizes evidence from 3 to 5 pieces of the best available evidence.

https://www.tandfonline.com/doi/full/10.1080/17489539.2017.1333683

Single person

Search may be comprehensive, but sources selected will be narrow. Decision making on search & selection needs to be documented and transparent.

Transparent inclusion/ exclusion criteria, ranking of different types of studies into different levels of evidence based on methodological rigor and appropriateness to the question. Studies deemed to represent the highest level of evidence (based on study design) and have most relevance to the question are retrieved for further analysis.

For each study: Is the study valid? What are the results? Will the results help locally?

Authors should decide whether the evidence is weak or strong, if the study design and quality are good, and if the study findings are consistent. Recommendations may include implications for local context, need for updated studies, etc.

Systematic Review

~1-2 years. Time is spent defining research question, developing a protocol (inclusion/ exclusion criteria), testing, refining, and translating search strategies, title and abstract screening, full text screening, risk of bias assessment, data extraction & analysis.

Seeks to systematically search for, appraise and synthesize research evidence on a given topic, following strict guidelines as set out by PRISMA or AMSTAR

Teams of 2+ to reduce bias

Aim is for transparent, reproducible, comprehensive searches following a described and registered protocol. Comprehensive searches using replicable syntax translated across databases. Includes searching multiple databases, grey literature/clinical trial registries, and handsearching of the literature (performed by the subject matter expert). It is best practice to involve a librarian or an information specialist in creating, translating, and documenting the search, as well as deduplicating the repeating references in a citation manager and writing the search methods for the review.

Two phase screening using multiple reviewers first on title and abstract, then on full text. Inclusion & exclusion criteria, quality assessment means studies may be excluded based on methodology, quality, etc. Appraisal of quality typically requires data analysis and synthesis.

Descriptive summaries of the full methodology and results. Discussion usually covers what the strength of evidence is in the area, recommendations, and gaps in research.

Scoping Review

~8 weeks-2 years. Time is spent formulating question, developing a protocol, refining search, screening studies, summarizing and synthesizing relevant studies, reporting results and identifying gaps and/or trends

Preliminary assessment of potential size and scope of available research literature on a given topic. Aims to identify nature and extent of research evidence (usually including ongoing research) – may not focus specifically on particular interventions and their analysis.

May require larger teams because of more search results to screen, may take longer than systematic review

Comprehensiveness is the goal. Aim is for transparent, reproducible, comprehensive searches. Comprehensive searches using replicable syntax across databases. May involve multiple structured searches -hand searching is important, grey lit, theses, and other data types may be included, includes research in progress. Returns more results than systematic reviews as question/ terms are broader, requires more screening time.

Occasionally broader search terms than systematic reviews (not limited to particular intervention outcomes) and therefore more results, may not have formalized quality assessment. Based on inclusion/exclusion criteria, due to the iterative nature of a scoping review some changes may be necessary.  May require more time spent screening articles due to the larger volume of results from broader questions. Critical appraisal and risk of bias are optional/ where applicable.

Characterizes quantity and quality of literature, perhaps by study design and other key features. Attempts to specify a viable review.

Meta-analysis

~18 -24 months. Time is spent on all aspects of systematic review, as well as typically two stages of data analysis.

Technique that statistically combines the results of disparate quantitative studies to provide a more precise measure of the results. Not all systematic reviews include meta-analysis, but all meta-analyses will include a systematic review of the evidence.

Experience in advanced statistical methods

Aim is for reproducible comprehensiveness, frequently 5 or more databases, see Systematic Review.

See Systematic Review. Inclusion & exclusion criteria, studies may be excluded based on methodology, quality, etc. Focus on identifying the data and translating the data across studies.

Numerical analysis of measures of effect.

Qualitative Meta Synthesis

~3-12 months. Time is divided between search/ discovery and analysis - repeated reading of articles to connect and record concepts or themes.

An intentional approach to analyzing data and conclusions across qualitative studies. Researchers identify a specific research question and then search for, select, appraise, summarize, and combine qualitative evidence to address the research question. May use PICO (Population, Interest, Context), SPICE, SPIDER or PerSPEcTiF question formats.

Single person. Requires significant methodological skill and experience with qualitative methods

Search focuses specifically on qualitative studies, may include more grey literature, book chapters or conference proceedings or theses. Refer to the 7 S Framework of Qualitative Searching.

Appraisal: Focus is on the translation of elements of qualitative studies. Three common primary levels of analysis:

Reciprocal Translation: Concepts in one study can incorporate those of another.
Refutational Translation: Concepts in different studies contradict one another.
Line of Argument Synthesis: Studies identify different aspects of the topic that can be drawn together in a new interpretation.

An interpretive and descriptive summary showing the relationships between qualitative studies on the question.