Assessing the Evidence Base

Close up of opened hard cover books on top of each other

Assessing the evidence base is an important part of Health Impact Assessment (HIA) process and ensures that findings and recommendations are as scientifically sound and robust as possible. This can be done by completing a systematic literature review (SLR) on the potential health impacts being assessed.

An SLR helps to provide evidence to support or disprove inferences about cause and effect relationships in HIAs.1 According to The Campbell Collaboration, systematic reviews use “transparent procedures to find, evaluate and synthesize the results of relevant research. Procedures are explicitly defined in advance, in order to ensure that the exercise is transparent and can be replicated. This practice is also designed to minimize bias”.As a result, SLRs must have inclusion and exclusion criteria, clear search strategies, and coding and analysis of included studies.3

HIA practitioners can use previously published systematic reviews or conduct their own.4 The Guide to Community Preventive Services and the Cochrane Collection are two sources that provide existing systematic reviews, while databases such as PubMed and ScienceDirect allow the user to search for scientific evidence to conduct their own review. 

One limitation to note when working on relationships between health and the built environment is that built environment studies are typically more observational in nature so causal effects for the relationships of interest are difficult to demonstrate.

Evaluating the Strength of Evidence
Harris County Public Health (HCPH) developed a hybrid approach to evaluate the literature gathered for a systematic review by combining and slightly modifying the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) approach 5 and the Guide to Community Preventive Services’ approach to evaluating the strength of a body of evidence.The strength of evidence for the hypothesized associations is rated as “strong,” “sufficient,” or “insufficient” based on:
  1. The GRADE quality level given to each study by two different graders 
  2.  The percentage of studies with results in the hypothesized direction (Table 1)
Table 1. Evaluating the Strength of Evidence For or Against the Associations of Interest*
Strength of Evidence For the Hypothesized Association
Level of Quality Based on GRADE Approach
Percentage of Studies With Results in the Hypothesized Direction
Strong
High and/or Moderate
≥ 60%
Sufficient
High and/or Moderate
> 50% - < 60%
Insufficient
High and/or Moderate
≤ 50%
Insufficient
Low and/or Very Low
≥ 60%
*Adapted from Briss, P. A., Zaza, S., Pappaioanou, M. et al. (2000). Developing an evidence-based Guide to Community Preventive Services – Methods. Am J Prev Med, 18(1S), 35-43.

 

In the GRADE approach, sound observational studies are usually given a quality rating of “low,” while a rating of “high” is given to randomized trials.7 However, due to the nature of built environment studies, randomized trials are rarely, if ever, used since it would be unethical and highly unfeasible to randomize community-level exposures in humans. So, sound observational studies are given a rating of “moderate” or are upgraded or downgraded, based on quality.

Two graders systematically evaluate each study using the GRADE approach. If the quality level does not match between the two graders for any study, a third grader can read the article and resolve the discrepancy. For each association, graders then determine the percentage of studies with results in the hypothesized direction that have “high,” “moderate,” “low,” and “very low” quality. Graders use Table 1 to rate the strength of evidence for each hypothesized association as “strong,” “sufficient,” or “insufficient.”

Impact Assessment Table
The impact assessment table visually displays the direction, strength, likelihood, severity, magnitude, and distribution of these potential health associations. These methods were based on Health Impact Assessment: A Guide for Practice8 and A Health Impact Assessment Toolkit: A Handbook to Conducting HIA, 3rd Edition.9

Impact Table Key

Direction:
Shows the impact direction of the development features and their outcomes:

Positive: The changes may improve health
Negative: The changes may impair health
Uncertain: It is unknown how health may be affected
No effect: There will be no health effect

Strength of Evidence:
Evaluated strength of evidence for each association using Table 1:

Strong: There is strong evidence for the association
Sufficient: There is sufficient evidence for the association
Insufficient: There is insufficient evidence for the association

Likelihood:
Answers the question, “How certain is it that the development features and outcomes will affect health outcomes, irrespective of the frequency, magnitude, or severity?”

Very likely: There is enough evidence for a causal, generalizable effect
Likely: The effect is logically plausible with substantial, consistent supporting evidence
Possible: The effect is logically plausible with limited supporting evidence
Unlikely: The effect is logically implausible, with substantial evidence against the mechanism of effect

Severity:
Answers the question, “How severe is the negative health consequence that results or is avoided after the proposed changes are implemented with regards to human well-being, function, or longevity, considering the community’s ability to manage the health effects?”

High: Health effects that are chronic, irreversible, or fatal
Medium: Health effects that necessitate treatment or medical management; effects are reversible
Low: Health effects that can be quickly and easily managed or that do not need treatment

Magnitude of Association: 
Answers the question, “How large is the measure of association  between the proposed change and the outcome of interest?” The levels of magnitude are considered to be “substantial,” “moderate,” or “limited” based on Table 2 below. If all of the results for an association do not agree on a specific level of magnitude, an average is taken. For example: If three articles find a “substantial” level of magnitude, while three articles find a “limited” level of magnitude, a level of “moderate” would be reported for that association. If there is an equal number of articles reporting a “limited” and “moderate” or a “moderate” and “substantial” magnitude, then the lower level is reported.

Distribution: 
Answers the question, “Who will be affected by the proposed changes?


Table 2. Evaluating the Magnitude of Association*
Index
Effect Size
Index
Limited = >0 to <0.2

Moderate = 0.2 to <0.5
Substantial = ≥0.5

Odds ratio (OR) (if <1, take inverse)
Limited = >1 to <1.5

Moderate = 1.5 to <2
Substantial = ≥2

Relative risk or risk ratio (RR) (if <1, take inverse)
Limited = >1 to <2

Moderate = 2 to <3
Substantial = ≥3

Pearson’s r correlation
Limited = >0 to <0.2 OR >-0.2 to <0

Moderate = 0.2 to <0.5 OR >-0.5 to -0.2
Substantial = ≥0.5 OR ≤-0.5

r2 coefficient of determination
Limited = >0 to <0.04

Moderate = 0.04 to <0.25
Substantial = ≥0.25

*Based on Sullivan, G. M. & Feinn, R. (2012). Using effect size – or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279-282.

 

NEXT: Community Engagement


References

1. Bhatia, R. (2011). Health Impact Assessment: A Guide for Practice. Oakland, CA: Human Impact Partners.
2. The Campbell Collaboration. What is a Systematic Review? Accessed on September 6, 2016. Retrieved from http://www.campbellcollaboration.org/what_is_a_systematic_review/
3. The Campbell Collaboration.
4. Bhatia, 2011.
5. Higgins, J. P. T. & Green, S. (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. 12.2 Assessing the Quality of a Body of Evidence. Accessed on June 13, 2016. Retrieved from http://handbook.cochrane.org/chapter_12/12_2_assessing_the_quality_of_a_body_of_evidence.htm
6. Briss, P. A., Zaza, S., Pappaioanou, M. et al. (2000). Developing an evidence-based Guide to Community Preventive Services – Methods. Am J Prev Med, 18(1S), 35-43.
7. Higgins & Green, 2011.
8. Bhatia, 2011.
9. [HIP] Human Impact Partners. (2011). A Health Impact Assessment Toolkit: A Handbook to Conducting HIA, 3rd Edition. Oakland, CA: Human Impact Partners.