Grading Guide

Abstract

While grading strength of recommendations and quality of underlying evidence enhances the usefulness of clinical guidelines, the profusion of guideline grading systems undermines the value of the grading exercise. The international GRADE group has suggested an approach that may be useful for many groups developing guidelines; UpToDate has chosen a modification of the GRADE approach. The grading scheme classifies recommendations as strong (Grade 1) or weak (Grade 2), according to the balance between benefits, risks, burden, and cost, and the degree of confidence in estimates of benefits, risks, and burden. The system classifies quality of evidence as high (Grade A), moderate (Grade B), or low (Grade C) according to factors that include the study design, the consistency of the results, and the directness of the evidence.

Introduction

Treatment decisions involve a trade off between benefits on the one hand, and risks, burden, and costs on the other. UpToDate provides recommendations for management of typical patients. To integrate these recommendations with their own clinical judgment, and with individual patient's values and preferences, clinicians need to understand the basis for the recommendations that expert guidelines offer. A systematic approach to grading the strength of management recommendations can minimize bias and aid interpretation [2]. Indeed, most guideline groups have accepted the necessity for some sort of grading scheme.

While grading of recommendations represents a positive development for guideline development and interpretation, the proliferation of grading systems has proved an unfortunate consequence [4]. Methodologists and guideline developers have given much thought and effort to considering criteria and approaches to an optimal grading system. An international group of guideline developers and methodologists (the GRADE group) has developed a system that remedies some of the disadvantages of prior systems, and that a number of organizations and groups are finding useful [5].

Strength of recommendation

UpToDate should make recommendations to administer, or not administer an intervention on the basis of tradeoffs between benefits on the one hand, and risks, burden, and costs on the other. If benefits outweigh risks and burden, experts will recommend that clinicians offer a treatment to appropriately-chosen patients. The uncertainty associated with the trade-off between the benefits and risks and burdens will determine the strength of recommendations.

The UpToDate approach classifies recommendations into two levels, strong and weak (see table 1). A two-level grading system has the merit of simplicity. Two levels also facilitate a clear interpretation of the implications of strong and weak recommendations by clinicians. We offer three ways editors and clinicians can interpret strong and weak recommendations.

If editors are confident that, on the basis of the existing evidence, most or all patients will be best served by a particular management strategy, they will make a strong recommendation: Grade 1. This confidence can arise in three ways. First, high quality evidence may provide precise estimates of both benefits and risk, and the balance may be clear (recommendation for statins in patients with known atherosclerotic disease). Second, high quality evidence may suggest that two therapies share equivalent benefits, and low quality evidence points to appreciably more harm in one than the other (recommendation for acetaminophen over aspirin in children with chickenpox). Third, high quality evidence may suggest that two therapies share equivalent harms, and low quality evidence points to appreciably more benefit for one (recommendation for mammary artery implants over venous grafting in coronary artery bypass grafting). If they believe that benefits and risks and burdens are finely balanced, or appreciable uncertainty exists about the magnitude of both benefits and risks, they must offer a weak (Grade 2) recommendation.

Clinicians are becoming increasingly aware of the importance of patient values and preferences in clinical decision making [6]. A second way to interpret strong and weak recommendations is in relation to patient values and preferences. For decisions in which it is clear that benefits far outweigh risks, or risks far outweigh benefits, virtually all patients will make the same choice (see Box 1). In such instances, editors can offer a strong (Grade 1) recommendation. In contrast, there are other choices in which patient values and preferences will play a crucial role and in which patients will, as a result, make different choices (see Box 2 and 3). When, across the range of patient values, fully informed patients are liable to make different choices, editors should offer weak (Grade 2) recommendations.

Following closely from this reasoning, a third way for clinicians to interpret strong recommendations is, for typical patients, just do it. On the other hand, when clinicians face weak recommendations, or when they face patients with very atypical circumstances or values, they should carefully consider the benefits, risks, and burden in the context of the individual patient before them.

How to individualize decision-making in weak recommendations remains a challenge. For weak recommendations, clinicians should have a detailed conversation with the patient to ensure that the ultimate decision is consistent with the patient's values or even use a decision aid that presents patients with both benefits and down sides of therapy [7]. Because of time constraints, clinicians cannot use decision aids in all patients. For strong recommendations, using a decision aid is likely, for most patients, to constitute a poor use of time and energy.

Factors that influence the strength of a recommendation

Editors must consider a number of factors in grading recommendations (see table 2). One issue is their confidence in the best estimates of benefit and harm. The rating of methodological quality, which we discuss below, captures that degree of confidence.

Prevention of outcomes with high patient-importance should, in general, lead to stronger recommendations than prevention of outcomes of lesser patient importance [8]. For instance, one needs to expose 4 patients to a respiratory rehabilitation program for 1 patient to gain a small but important improvement in dyspnea in daily life [9]. In low risk patients who have suffered a myocardial infarction one might need to treat 100 patients with agents such as aspirin, beta blockers, ACE inhibitors, or statins to extend one life. Despite the much higher number needed to treat (NNT), since we value prolongation of life more highly than relieving dyspnea, the latter intervention may warrant a stronger recommendation.

The choice of adjusted dose warfarin versus aspirin for prevention of stroke in patients with atrial fibrillation illustrates a number of the factors that will influence the strength of a recommendation. A systematic review and meta-analysis found a relative risk reduction (RRR) of 46 percent in all strokes with warfarin versus aspirin. This large effect supports a strong recommendation for warfarin. Furthermore, the relatively narrow 95 percent confidence interval (RRR 29 to 57 percent) suggests that warfarin provides a RRR of at least 29 percent, and further supports a strong recommendation. At the same time, warfarin is associated with an inevitable burden of keeping dietary intake of vitamin K constant, monitoring the intensity of anticoagulation with blood tests, and living with the increased risk of both minor and major bleeding. Most patients, however, are much more stroke averse than they are bleeding averse [10]. As a result, almost all patients with high risk of stroke would choose warfarin, suggesting the appropriateness of a strong recommendation.

This last point emphasizes the importance of the patient's baseline risk (sometimes called control event rate) of the adverse outcome that treatment is designed to avoid. Consider a 65 year-old patient with atrial fibrillation and no other risk factors for stroke. This individual's risk for stroke in the next year is approximately 2 percent. Considering the relative risk reduction and this baseline risk, one can derive the absolute magnitude of an effect (see table 2). Dose-adjusted warfarin can, relative to aspirin, reduce the risk to approximately 1 percent for an absolute risk reduction of 1 percent (= 2 percent - 1 percent). Some patients who are very stroke averse may consider the down sides of taking warfarin well worth it. Given the relative narrow confidence interval that follows from the confidence interval around the relative risk reduction one could make a strong recommendation to use warfarin if all patients were equally stroke adverse. Other patients, however, are likely to consider the benefit not worth the risks and inconvenience. When, across the range of patient values, fully informed patients are liable to make different choices, editors should offer weak (Grade 2) recommendations.

When UpToDate editors make recommendations, they assume a particular set of values as they weigh the possible beneficial and detrimental outcomes. When value or preference judgments are particularly salient, editors should describe the key values attached to these outcomes and that influenced the direction of a recommendation or its grade. The limited literature regarding what average patient values and preferences actually are, and the range of preferences, emphasizes the importance of making explicit the key values and preference judgments that drive their recommendations.

Wording of recommendations

It is desirable to provide clinicians with as many indicators as possible in interpreting strength of recommendations. UpToDate editors, when they are making a strong recommendation, should use the terminology "We recommend...". When they make a weak recommendation, they should use less definitive wording, such as, "We suggest...".

Confidence in estimates of magnitude of benefits, risks, burden, and costs

Early systems of grading methodologic quality relied primarily on the basic study design (ie, randomized control trials (RCTs), or observational studies). The fundamental study design remains critically important in determining our confidence in estimates of beneficial and detrimental treatment effects. Because of prognostic differences between groups, and lack of safeguards such as blinding that can avoid biased ascertainment of outcomes, evidence based on observational studies will, in general, be appreciably weaker than evidence from RCTs. Recent years have seen, however, an increased awareness of a number of other factors that influence our confidence in our estimates of risk and benefit (see table 3).

UpToDate has chosen a three-category system of quality of evidence: high (Grade A), moderate (Grade B), and low quality (Grade C) (see table 1). The strongest evidence comes from systematic reviews that summarize one or more well-designed and well-executed randomized control trials (RCTs) yielding consistent directly applicable results. Strong evidence can also come, under unusual circumstances, from observational studies yielding very large effects.

The moderate strength category is populated by randomized trials with important limitations and by exceptionally strong observational studies. Observational studies, and on occasion RCTs with multiple serious limitations, will fill the low quality evidence category. This categorization follows the principle that all relevant clinical studies provide evidence, the strength of which varies.

Factors that modify the quality of evidence

• Limitations in RCTs — When RCTs have addressed the impact of alternative management strategies (both benefits and harms) on all relevant outcomes they will yield high quality evidence unless they suffer from one of a number of limitations. The following limitations may decrease the quality of evidence supporting a recommendation (see table 3).

1) Our confidence in recommendations decreases if the available RCTs suffer from major deficiencies that are likely to result in a biased assessment of the treatment effect. These methodological limitations include a very large loss to follow-up, or an unblinded study with subjective outcomes highly susceptible to bias. How lack of blinding can influence the grading is exemplified by a recommendation to treat heparin-induced thrombocytopenia (HIT) complicated by thrombosis with danaparoid sodium. The randomized trial evidence for danaproid use in HIT comes from an unblinded trial in which the outcome was the clinicians' assessment of when the thromboembolism had resolved, a subjective judgment. As a result, an ACCP guideline panel, using a system very close to UpToDate's, rated the quality of the evidence as moderate rather than strong [11].

2) When several RCTs yield widely differing estimates of treatment effect (heterogeneity or variability in results) investigators look for explanations for that heterogeneity. For instance, drugs may have larger relative effects in sicker, or in less sick, populations. When heterogeneity exists but investigators fail to identify a plausible explanation, the strength of recommendations from even rigorous RCTs is weaker. For example, RCTs of pentoxifylline in patients with intermittent claudication have shown conflicting results that so far defy explanation. Acknowledging the unexplained heterogeneity, a guideline panel rated the quality of the evidence for pentoxifylline as moderate, rather than high [12].

3) Investigators may have undertaken RCTs in populations similar, but not identical, to those under consideration. Editors should consider this indirect evidence and, to the extent they are uncertain about the applicability to their relevant population, downgrade the strength of evidence. For instance, while graduated compression stockings have proved of benefit in a variety of populations at risk of venous thrombosis, they have never been tested directly in trauma patients. An ACCP guideline panel judged the available RCTs relevant to trauma patients in whom administration of low molecular weight heparin is contraindicated, but because of concern about generalizing from other populations (that is, concern about the indirectness of the evidence), rated the quality of the evidence as moderate. Had they had no concerns about directness, they would have considered the evidence high quality, whereas if there were no relevant RCTs available, and the best evidence came from observational studies, they would have rated the evidence low quality [13].

Indirectness may also apply to the intervention (RCTs of similar but not identical interventions, different doses and formulations, for instance) and outcomes (RCTs measuring laboratory exercise capacity, for instance, when a panel is really interested in quality of life improvement). Consider sigmoidoscopic screening for colon cancer. The relevant evidence includes not only direct but weak evidence from observational studies, but also stronger but indirect evidence from randomized trials of fecal occult blood screening.

4) Investigators may have conducted RCTs but included very few patients, and observed very few events. For instance, a well-designed and rigorously conducted RCT addressed the use of nadroparin, a low molecular weight heparin, in patients with cerebral venous sinus thrombosis. Of 30 treated patients, 3 had a poor outcome, as did 6 of 29 patients in the control group. The investigators' analysis suggests a 38 percent reduction in relative risk of a poor outcome, but the result was not statistically significant [14]. Because of the small number of patients, and small number of events, a guideline panel judged the quality of the evidence for anticoagulation in cerebral sinus thrombosis as moderate rather than high [13].

• Observational studies can provide moderate or high quality evidence — While observational studies will generally yield only low quality evidence, there may be unusual circumstances in which editors will classify such evidence as of moderate, or even high quality.

1) On the rare occasions when they yield extremely large and consistent estimates of the magnitude of a treatment effect, we may be confident about the results of observational studies. For example, oral anticoagulation in mechanical heart valves has not been compared to placebo in an RCT. However, evidence from observational studies suggests that the probability of suffering thromboembolic events without anticoagulation is 12.3 percent annually in bileaflet prosthetic aortic valves and higher for other valve types, and estimates of the relative risk reduction with oral anticoagulation are in the range of 80 percent. While the observational studies are likely to overestimate the true effect, the weak study design is very unlikely to explain the entire benefit. Thus, an ACCP guideline panel concluded that these data, despite the absence of randomized trials, constituted high quality evidence of the effectiveness of anticoagulation in bileaflet aortic prosthetic valves [15].

Anticoagulation among patients with AF for more than 48 hours undergoing cardioversion provides another example. In a large observational study, patients who had presented with AF and were already receiving anticoagulation were continued on warfarin; in contrast, those not already on anticoagulation underwent cardioversion without warfarin [16]. The incidence of embolization following DC electroversion was much lower in the warfarin group (0.8 versus 5.3 percent). A review of observational studies among patients undergoing cardioversion found rates of thromboembolism of 2 percent in patients who were not anticoagulated and 0.33 percent among those who received anticoagulation [17]. The magnitude of the association in these trials constitutes high quality evidence.

2) On other occasions, all plausible biases from observational studies may be working to underestimate an apparent treatment effect. In other words, the actual treatment effect is very likely to be larger than what the data suggests. For instance, a rigorous systematic review of observational studies including a total of 38 million patients compared private for-profit versus private not-for-profit hospital care. The meta-analysis demonstrated higher death rates in the private for-profit hospitals [18].

The investigators postulated two likely sources of bias. The first was residual confounding with disease severity. It is likely that, if anything, patients in the not-for-profit hospitals were sicker than those in the for-profit hospitals. Thus, to the extent that residual confounding existed, it would bias results against the not-for-profit hospitals.

The second likely bias was the possibility that higher numbers of patients with excellent private insurance coverage could lead to a hospital having more resources and a "spill-over" effect that would benefit those without such coverage. Since for-profit hospitals are likely to admit a larger proportion of such well-insured patients than not-for-profit hospitals, the bias is once again against the not-for-profit hospitals. Because the plausible biases would all diminish the demonstrated treatment effect, one might consider the evidence from these observational studies as moderate rather than low quality.

What to do when strength of evidence differs across outcomes?

UpToDate will provide a single rating of quality of evidence for every recommendation. Recommendations, however, depend on evidence regarding a variety of outcomes. Thus, it may occasionally be necessary to report a single evidence grade when the quality of evidence differs across important outcomes. Consider, for instance, administration of clopidogrel versus aspirin for threatened stroke. A very large well-conducted RCT has shown a small incremental benefit of clopidogrel over aspirin in reducing vascular events, clearly high quality (Grade A) evidence [19]. In deciding on whether to recommend clopidogrel over aspirin, however, one must also consider toxicity. Case reports have suggested the clopidogrel may, on rare occasions, cause thrombotic thrombocytopenic purpura [20]. The quality of this evidence is low. Should the overall quality of evidence for clopidogrel versus aspirin, therefore, be considered high, moderate, or low?

In such instances, we suggest that editors should consider whether toxicity endpoints are crucial to the decision regarding the optimal management strategy. If they are, one must rate the overall quality of the evidence according to the studies that address toxicity. If not, the overall rating of the evidence is based on the evidence regarding benefit. For example, if one considers thrombotic thrombocytopenic purpura to be a crucial outcome, then one would rate the overall quality of the evidence regarding clopidogrel versus aspirin as low. If, on the other hand, one considered that the outcome was so rare that it should not be considered crucial, the overall evidence rating would remain high.

The process of grading: a checklist and an example

UpToDate editors may benefit, in developing recommendations and grading them, from reference to a checklist (see table 4). The following examples (see box 4 and 5) from the management of Wegener's granulomatosis show how editors might work through the issues.

Summary

In the UpToDate grading system, the strength of any recommendation depends on two factors: the tradeoff between benefits and risks and burden, and the quality of the evidence regarding treatment effect. We grade the tradeoff between benefits and risks and burden in two categories; 1, in which the tradeoff is clear enough that most patients, despite differences in values, would make the same choice, leading to a strong recommendation; and 2, in which the tradeoff is less clear, and individual patients values will likely lead to different choices, leading to a weak recommendation. We grade methodological quality in three categories: randomized trials that show consistent results, or observational studies with very strong treatment effects; randomized trials with limitations, or observational studies with exceptional strengths; and observational studies without exceptional strengths or randomized trials with major weaknesses. The framework summarized in Table 1 generates recommendations from the very strong (benefit/risk tradeoff unequivocal, high quality evidence, 1A) to the very weak (benefit/risk questionable, low quality evidence, 2C).

Table 1: Grading Recommendations

Table 2. Factors panels should consider in deciding on a strong or weak recommendation

Table 3. Factors panels should consider in deciding on their confidence in estimates of benefits, risks, burden, and costs

Table 4. A checklist for developing and grading recommendations

References

1. Buller HA, G. Hull, RD. Hyers, TM. Prins, MH. Raskob GE. Antithrombotic Therapy for Venous Thromboembolic Disease. CHEST 2004; 126:401S— 428S.

2. Guyatt GS, J. Cook, D. Jaeschke, R. Schünemann, H. Pauker S. Grading recommendations: A qualitative approach. In: Guyatt GR, D., ed. Users' Guides to the Medical Literature: A manual for evidence-based practice. Chicago, Ilinois: AMA Press, 2002.

3. Collins R, MacMahon S, Flather M, et al. Clinical effects of anticoagulant therapy in suspected acute myocardial infarction: systematic overview of randomised trials. Bmj 1996; 313:652-659.

4. Schunemann HJ, Best D, Vist G, et al. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. Cmaj 2003; 169:677-680.

5. GRADE wg.

6. Guyatt GS, S. McAlister, F. Haynes, RB. Sinclair, J. Devereaux, PJ. Lacchetti, C. Incorporating Patient Values. In: Guyatt GR, D., ed. Users' Guide to the Medical Literature. Chicago, Illinois: AMA Press, 2002.

7. O'Connor AM, Stacey D, Entwistle V, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2003:CD001431.

8. Guyatt GM, V. Devereaux, PJ. Schunemann, H. Bhandari, M. Patients at the centre: In our practice, and in our use of language. ACP Journal Club 2004; 140:A-11.

9. Goldstein RG, EH. Gort, Guyatt, GH. Feeny, D. Economic Analysis of Respiratory Rehabilitation. Chest 1997; 112:370-379.

10. Devereaux PA, DR. Gardner, MJ. Putnam, W. Flowerdew, GJ. Brownell, BF. Nagpal, S. Cox, JL. Differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation: observational study. Bmj 2001; 323:1218-1222.

11. Warkentin TE, Greinacher A. Heparin-induced thrombocytopenia: recognition, treatment, and prevention: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126:311S-337S.

12. Clagett GP, Sobel M, Jackson MR, et al. Antithrombotic therapy in peripheral arterial occlusive disease: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126:609S-626S.

13. Geerts WH, Pineo GF, Heit JA, et al. Prevention of venous thromboembolism: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126:338S-400S.

14. de Bruijn SF, Stam J. Randomized, placebo-controlled trial of anticoagulant treatment with low-molecular-weight heparin for cerebral sinus thrombosis. Stroke 1999; 30:484-488.

5. Salem DN, Stein PD, Al-Ahmad A, et al. Antithrombotic therapy in valvular heart disease--native and prosthetic: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126:457S-482S.

16. Bjerkelund CJ, Orning OM. The efficacy of anticoagulant therapy in preventing embolism related to D.C. electrical conversion of atrial fibrillation. Am J Cardiol 1969; 23:208-216.

17. Moreyra E, Finkelhor RS, Cebul RD. Limitations of transesophageal echocardiography in the risk assessment of patients before nonanticoagulated cardioversion from atrial fibrillation and flutter: an analysis of pooled trials. Am Heart J 1995; 129:71-75.

18. Devereaux PJ, Choi PT, Lacchetti C, et al. A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. Cmaj 2002; 166:1399-1406.

19. A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE). CAPRIE Steering Committee. Lancet 1996; 348:1329-1339.

20. Bennett CC, JM. Carwile, JM. Moake, JL. Bell, WR. Tarantolo, SR. McCarthy, LJ. Sarode, R. Hatfield, AJ. Feldman, MD. Davidson, CJ. Tsai, HM. Thrombotic thrombocytopenic purpura associated with clopidogrel. New England Journal of Medicine. 2000; 342:1773-1777.

21. CAPRIE-Steering-Committee. A randomized, blinded trial of clopidogrel versus aspirin in patients at risk of ischemic events. Lancet 1996; 348:1329-1339.