Comments on FDA Draft Guidance on Conduct of Clinical Trials in Community-Acquired Bacterial Pneumonia
Re: Docket FDA-2009-D-0136, Draft Guidance for Industry on Community-Acquired Bacterial Pneumonia
Division of Dockets Management (HFA-305)
Food and Drug Administration
5630 Fishers Lane, Room 1061
Rockville, MD 20852
Dear Food and Drug Administration,
The Food and Drug Administration’s recent draft guidance on community-acquired bacterial pneumonia (CABP) is significant step forward from the previous draft guidance, issued in 1998. However, it includes recommendations that contradict established principles of rigorous clinical trial conduct, as defined in the International Conference on Harmonization documents E-9 and E-10, and a number of issues are left unaddressed. The recommendations for the conduct of noninferiority trials are most problematic and of greatest relevance, as nearly all recent clinical trials for CABP have been noninferiority trials.
1) Endpoints
“Resolution of signs and symptoms” (lines 433-434) is an inappropriate primary endpoint, as it represents a composite of biomarkers rather than something of clear relevance to patients. A well-validated patient-reported outcome (PRO) tool may be used as an endpoint in a superiority trial for CABP, or mortality.
On the other hand, a noninferiority comparison can be made only when substantial and reproducible evidence for an effect size of an active comparator has been demonstrated. The draft guidance clearly states that for bacterial pneumonia due to atypical pathogens (M. pneumoniae, C. pneumonia, or L. pneumophilia) and for viral pneumonia, this evidence does not exist. In pneumococcal pneumonia, both penicillin and sulfa result in a reproducible and compelling reduction in mortality, especially in patients over the age of 50 or with bacteremia. For a noninferiority comparison to be valid, the outcome on which a margin is based must be the same outcome for which an effect size for the active comparator has been demonstrated, and in a study population with similar characteristics as those from the historical data. For CABP, this outcome is mortality.
Therefore, it is troubling that the draft guidance (lines 773-784) argues that a treatment effect for a reduction in mortality with antibiotics can be extrapolated to a “clinical outcome.” This is neither logical, nor based on sound scientific reasoning. A noninferiority margin based on a clinical outcome other than mortality can be justified only if compelling evidence of an effect size for this outcome were available, but such data do not exist. Of note, a reduction in fever, which is a biomarker and not a relevant clinical outcome, does not comprise a valid clinical outcome.
2) Size of Margin in Noninferiority Trials
The draft guidance (lines 796-944) provides an excellent review of the uncontrolled observational and historical data establishing an effect size for antibiotics in reducing morality in patients with CABP. From this analysis, the lower bound of the 95% confidence interval for the effect size of antibiotics (M1) is 15-22%, in patients with pneumococcal pneumonia and a high incidence of bacteremia. From this M1, it is unclear how the FDA justifies an M2 of 15%, which essentially preserves none of the treatment effect of antibiotics. In other words, an experimental antibiotic could be declared “noninferior” when, in fact, it is only marginally better than placebo, and substantially worse than an active comparator. While M1 is calculated from historical data, M2 is determined by clinical reasoning, and it is impossible to justify approval of an antibiotic that is as much as 15% worse in preventing death than an existing therapy. For very ill patients – those over the age of 50 with bacteremia – a margin of at most 10% can be tolerated. In studies with less-ill patients, only smaller margins can be tolerated, or superiority trials must be conducted.
3) Study Population
A common feature of most contemporary trials in CABP is the low rate of microbiologic confirmation of the presence of typical pathogens, suggesting a high rate of atypical and viral pneumonias, and the inclusion of patients who are not very ill. This is not particularly problematic in a superiority trial, because including patients without the disease in question or who are not ill generates bias towards the null finding, making Type 1 error unlikely. Likewise, the “assay sensitivity” of the trial, or the ability of the trial to detect a difference between the experimental drug and an active comparator or placebo is assured by a positive finding. However, including such patients in a noninferiority trial reduces the ability of the trial to identify a treatment difference between the experimental drug and the active comparator, reducing the assay sensitivity of the trail. Unlike in superiority trials, assay sensitivity in noninferiority trials is not assured by a positive finding of noninferiority. Rather, it is an un-testable assumption and it is preserved only through rigorous trial design and conduct. In other words, enrolling patients without pneumonia and who are not ill in a noninferiority trial may lead to scenario in which no treatment difference is observed in the study population, while in patients at higher risk of death – the elderly and those with bacteremia – the experimental treatment may be substantially worse.
Thus, the draft guidance’s recommendation that at least 75% percent of patients be over the age of 50 (lines 1020-1021) and that no more than 25% of patients have a PORT score of II or lower (lines 307-310) in a noninferiority trial are appropriate. In contrast, the recommendation that only at least 30-40% of patients have bacteriologic confirmation of typical pathogens (lines 115-118) is insufficient. Rather, all patients in a noninferiority trial must have microbiologic confirmation of a typical pathogen through culture, serology or antigen testing. Otherwise, the margin established from the historical data, which almost without exception included patients with clear radiographic findings of lobar pneumonia or microbiological evidence of streptococcal pneumonia, cannot be used and a noninferiority comparison cannot be justified.
4) Prior and Concomitant Antibiotic Therapy
Similarly, the inclusion of patients who have received prior and concomitant antibiotic therapy diminishes the ability of a noninferiority trial to detect a differential treatment effect between an experimental drug and an active comparator, reducing the assay sensitivity of the trial and biasing the results toward a positive finding of noninferiority. Not only must this be avoided (lines 410-425), as the draft guidance recommends, but patients with prior antibiotic therapy should not be randomized and concomitant therapy must be absolutely prohibited. On the other hand, prior or concomitant therapy in a superiority trial creates bias toward the null finding of no difference, and can be allowed.
5) Exclusions Based on Post-Randomization Events
In most contemporary trials of CABP, a large number of patients are excluded from analysis because of premature discontinuation of therapy, “intercurrent illnesss,” use of non-study antibiotics, and other protocol violations. This must be absolutely prohibited, as a high rate of exclusions based on post-randomization events results in an effective loss of randomization, turning a randomized clinical trial into a mere observational study. Often, these exclusions are imbalanced between the two treatment groups, creating the potential for systematic bias.
6) A Recent Example: Cethromycin
The antibiotic cethromycin (New Drug Application 22-398), which was discussed by the Anti-Infective Drugs Advisory Committee on June 2, 2009, highlights all of the problems that I have identified above. Although the two Phase 3 noninferiority trials comparing cethromycin to clarithromycin in CABP followed most of the principles detailed in the updated draft guidance, they were wholly incapable of demonstrating noninferiority and, even worse, may have obscured that the drug is actually worse than clarithromycin, based on a sensitivity analysis conducted by the FDA. Specifically, these trials enrolled few patients with microbiologic confirmation of typical pathogens (one-quarter), most patients were at low risk of complications and death (1% with bacteremia and half with PORT score of 1), prior and concomitant antibiotics were allowed, many patients were excluded based on post-randomization events, and mortality was not evaluated as the primary endpoint. My testimony before the advisory committee detailing these problems is attached as an appendix.
7) Conclusion
Most of the concerns detailed above are specific to noninferiority trials, which comprise the vast majority of contemporary trials in CABP. The many problems with the design and conduct of noninferiority trials may lead to an incorrect conclusion of noninferiority, a Type 1 error, when an experimental treatment is actually substantially worse than a comparator drug. A positive finding from a noninferiority trial only provides reassuring evidence that an experimental drug is better than placebo and not substantially worse than an active comparator when the trial has been conducted rigorously.
Protests from sponsors that the rigorous conduct of noninferiority trials is not feasible are wholly inappropriate, and the scientific principles that provide the foundation for efficacy comparisons should not be bent to accommodate them. Otherwise, there is a real possibility that a new antibiotic will be approved when it is, in fact, worse than existing treatments. This does not serve patients well and it contradicts the public health mission of the Food and Drug Administration. Furthermore, allowing sponsors to conduct noninferiority trials in this fashion does not serve sponsors well, as time and money are wasted on trials that have little capacity to demonstrate efficacy.
To summarize, the only acceptable endpoint on which a noninferiority margin can be established is mortality, and a margin (M2) of greater than 10% is clinically unacceptable. Validated PROs may be used in superiority comparisons. In addition, noninferiority trials must not allow prior or concomitant antibiotic therapy, exclusions based on post-randomization events, or the enrollment of patients without a confirmed diagnosis of bacterial pneumonia with typical pathogens.
Sincerely,
James Floyd, M.D.
Researcher
Public Citizen