Systematic reviews

Aim

To describe the documentation that is necessary when undertaking a systematic review.

Requirements

Clear documentation of the various steps undertaken during the review process.

Documentation

Search and selection criteria;

  • The full search that has been performed, mentioning the database with suppliers, the search dates and all search strategies with all search terms (in a appendix);
  • Selection of the articles presented in a flow diagram;
  • Evaluation of the methodological quality (i.e. risk of bias assessment) of the articles;
  • Data analysis and evaluation of the quality of the evidence.
Responsibilities

Executing researcher:

  • Ensures the necessary background knowledge is present, gained through sources and experts, prior to writing the protocol.
  • Write the protocol. This should include the following broad topics: Criteria for including studies (i.e. inclusion/exclusion criteria); Search methods for identification of studies; Data collection and analysis (e.g. selection of studies, data extraction and management, assessment of risk of bias, measures of treatment effect).
  • Consult a search specialist in the process.
  • Include two independent reviewers for the selection of the abstracts and full-text articles, and risk of bias assessment.
  • Document the selection and evaluation of the articles in the report.
  • Select a report format and follow the guidelines described for a proper report. See the ‘Checklists for writing’ in the table below.

Project leader:

  • Consult sources and content experts before writing the protocol, if necessary.
  • Inspect the protocol by reading and adding comments.
  • Consult a search specialist in the process, if necessary.
  • Ensure that the selection of articles and risk of bias assessment has been carried out by two independent reviewers.
  • Check the documentation of the executing researcher as proposed in the section ‘documentation’.
  • Review the data extraction and evaluation for the most important outcome measures in the review.
  • If the executing researcher is also the project leader then ensure that your documentation is reviewed by an independent researcher. This may include a co-author.
How To

A systematic review is a comprehensive overview of the available evidence based upon the literature within a particular topic. In this guideline, we focus on the quality aspects when conducting a systematic review. It is imperative that you consult other sources when preparing your review, such as the Cochrane handbook. It is important to note that a systematic review is written together with at least two persons, preferably with the help of a search specialist. At least one reviewer should be a content expert. A well-conducted review includes the following steps:

Preparation

Books:

It is advisable to check the Prospero database for ongoing reviews in order to see if your review is currently being conducted, and to register your topic.

Library websites:

 

Type of systematic review Guidelines for executing Checklists for writing
Interventions; (R)CT’s

SysRev3 (york.ac.uk)

Cochrane Handbook for Systematic Reviews of Interventions | Cochrane Training

 PRISMA (prisma-statement.org)
Diagnostic test accuracy Handbook for DTA Reviews | Cochrane Screening and Diagnostic Tests Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement | The EQUATOR Network (equator-network.org)
Measurement instruments SysRev3 (york.ac.uk) COSMIN - Improving the selection of outcome measurement instruments
Meta-analyses Cochrane Handbook for Systematic Reviews of Interventions | Cochrane Training PRISMA (prisma-statement.org)
Observational studies MOOSE Statement 2000.pdf (consort-statement.org) Microsoft Word - 14MOOSE_SS.docx (ijo.in)
Course:

It is advisable to follow a course in conducting systematic reviews, for example, the course offered by EpidM. The VU library offers an online search course.

Advice:

Should you be interested in working on a systematic review which examines outcomes instruments, then we recommend contacting Dr. Wieneke Mokkink, Dr. Caroline Terwee or Prof. Riekie de Vet (from the clinimetric group). Should you be interested in working on a (Cochrane) diagnostic test accuracy systematic review, then you could contact Dr. Mariska Leeflang (m.m.leeflang@amsterdamumc.nl)

Protocol:

Write a protocol for your systematic review before you start your search. Examples of protocols for a Cochrane review of intervention studies or for diagnostic reviews can be found in the Cochrane Library or CRD Handbook. The CRD Handbook also describes examples of other types of review, such as clinical tests, public health interventions, adverse events, or economic evaluations. For other types of systematic reviews, examples include:

  • Albers G, Echteld MA, de Vet HC, Onwuteaka-Philipsen BD, van der linden MH, Deliens L. Evaluation of quality-of-life measures for use in palliative care: A systematic review. Palliat Med. 2010;24(1):17-37.
  • van den Dungen P, van Kuijk L, van Marwijk H, van der Wouden J, Moll van Charante E, van der Horst H, van Hout H. Preferences regarding disclosure of a diagnosis of dementia: a systematic review. Int Psychogeriatr 2014;26:1603-18
  • Rietmeijer-Mentink M, Paulis WD, van Middelkoop M, Bindels PJ, van der Wouden JC. Difference between parental perception and actual weight status of children: a systematic review. Matern Child Nutr 2013;9:3-22.

A systematic review may age rapidly; therefore, you need to plan properly when you perform the literature search and when you evaluate the articles. It is, therefore, best if you have your search strategy, selection criteria, data extraction formula, and evaluation list ready prior to actually searching the databases.

Searching and Search specialist

It is advisable to consult a search specialist when undertaking a systematic review. The university library employs search specialists, who will be able to advise you.

Selection and evaluation of articles

The selection, assessment of risk of bias and data extraction should be undertaken by two independent reviewers. The initial selection should be based upon the title, abstract, and keywords, and the final selection should be based upon the full tekst and combined with the methodological evaluation of the article, although this is not advised. Make sure to document the selection, separately for both independent reviewers. Various standardized checklists are available for assessing the methodological quality of (randomized) controlled trials, diagnostic research (e.g. QUADAS) and observational studies. Consult the Cochrane website for the most appropriate tool.

Disagreements between the reviewers regarding the selection and methodological assessment can be discussed during consensus meeting. If necessary, a third independent reviewer should be consulted in order to reach consensus.

Documentation

Proper documentation of all decisions and choices made is essential for a systematic review. Maintain a separate logbook (see the ‘Logbook’ guideline) for systematic reviews. The documentation which is required is noted under ‘Documentation’ above.

In addition:
  1. Store all references retrieved in a database (e.g. Reference Manager, EndNote, Mendeley). Make a selection of the references included at a later stage and store this in a second database;
  2. Make a note of the reason for exclusion when excluding any references. Consult the PRISMA flow diagram and be familiar with what is required in the final report. Note down all the comments from consensus meetings. This prevents you from forgetting why a given article was or wasn’t included;
  3. Note down how many references in total have been retrieved, how many references you have excluded on the basis of the title, how many on the basis of the abstract and how many on the basis of full-text articles;
  4. Evaluation of the methodological quality of the articles: include, for example in a table, the reasons for allocating a particular score. This may appear to be quite a bit of work, but it will make your report more transparent for the reader.
Reporting:

Use reporting guidelines specifically developed for different systematic reviews.

A number of tools are available for summarizing results. A quantitative analysis is commonly used to statistically pool the results of the included studies. The decision to conduct a quantitative analysis should be based upon the relative homogeneity of the data, which will determine whether statistical pooling of the results is possible. When heterogeneity exists (i.e. too much variation) between studies then a qualitative analysis will be necessary. This might be also necessary if too few data are available for pooling. Note: it is advisable to discuss the data analysis an presentation with an expert.

In a qualitative analysis the results are summarized in a systematic way, taking into consideration various aspects such as methodological quality, consistency of results, statistical significance and clinical relevance of the effects. The conclusion can be formulated by evaluating the strength of the available evidence. In order to summarize these effects and for formulating levels of evidence, it is advisable to use GRADE.

 

Questions?

Contact Dr. Sidney Rubinstein s.m.rubinstein@vu.nl or Dr. Miranda Langendam m.w.langendam@amsterdamumc.nl 

Â