Aim

To specify and correctly implement post-hoc and sensitivity analyses.

Requirements

Post-hoc and sensitivity analyses are performed when needed.

Documentation
  • Data analysis plan: Describe which post-hoc tests and sensitivity analyses will be performed.
  • Syntax: Document the post-hoc tests and sensitivity analyses.
  • Logbook: Document the decisions you made based on the results of post-hoc tests and sensitivity analyses.
Responsibilities
  • Executing researcher:
    • To make sure you do post-hoc tests and sensitivity analyses when needed;
    • To discuss the results with your supervisors;
    • To document the points mentioned under 3 (documentation).
    • To make sure post-hoc tests and sensitivity analyses are reported in the manuscript accordingly.
  • Project leaders:
    • To inspect the data analysis plan on post-hoc tests and sensitivity analyses when needed;
    • To discuss the results of sensitivity analyses with the executing researcher
  • Research assistant: N.a.
How To
Post-hoc analyses

Post-hoc analyses are required when a significant relationship has been found between the dependent variable and a categorical, independent variable with more than two categories.  It is needed to confirm where the differences occur between the groups. An example of this is variance analysis, in which so-called post-hoc tests can be used (examples include Tukey, Duncan, Scheffé of Bonferroni tests). The tests differ in the way they correct for multiple testing.

Moreover, another form of post-hoc analysis is subgroup analysis. A subgroup analysis can be performed to test whether the association of two variables differs depending on a third variable. For example: the association between alcohol consumption and heart attacks might differ for smokers and non-smokers. Sometimes these are not thought of when writing the analysis plan, but they arise in discussions with your supervisors, colleagues, or are asked by reviewers. Make sure you update your analysis plan with these analyses and correctly write it down in your manuscript.

Sensitivity analyses

There is always more than one way to carry out an analysis. In order to be more certain about the results it is advisable to redo the analyses in a slightly different way, often by changing one or more (external) parameters. There are a number of cases where a sensitivity analyses is almost always desirable. These will be discussed here.

Firstly, when a cut-off has been selected for the dependent or independent variable for which there is, as yet, no consensus. Even if there is a consensus, there is the question of whether this cut-off is applicable to the study population. It is advisable to repeat the analyses with different cut-off values.

Secondly, there may be variables with “unused” categories. This may be either missing values or variables that are composed of data from various sources where there may occasionally be conflicts between both sources. An example of the latter is a disease diagnosis based on data provided by the general practitioner and the respondent. Missing values can be substituted, meaning the respondent can be retained for the analysis. Advanced statistical imputation methods can be used for this. Substitution can also be based on the basis of a “best guess”. It is good practice to carry out the analyses both with and without respondents with missing values, and to compare the results.

An example of this is an uncertain diagnosis where all uncertain cases are set at “no disease” in one analysis, and as “diseased” in another. All uncertain cases can be omitted in a third analysis.

Finally, sensitivity analyses are a standard component of economic evaluations. The opportunities for multivariable analysis in economic evaluations are very limited, owing to the fact that the distribution of cost data is skewed. Sensitivity analyses are used to study the effect of, for instance, the value of cost points on outcomes. Often subgroup analyses and analyses with imputed missing values are carried out as sensitivity analyses (see Drummond et al., 1997).

Post-hoc and sensitivity analyses should be reported as such and cannot replace the primary analysis.