Bias in measuring blood oxygen in patients with dark skin: comment on the FDA’s latest draft guidance for pulse oximeter makers
Comment to Docket #
FDA-2023-N-4976
Pulse Oximeters for Medical Purposes – Non-Clinical and Clinical Performance Testing, Labeling, and Premarket Submission Recommendations; Draft Guidance for Industry and Food and Drug Administration Staff, January 7, 2025
Michael T. Abrams, M.P.H., Ph.D.
Public Citizen’s Health Research Group
March 10, 2025
A 2024 systematic review and meta-analysis by Parr et al.[1] offered one of the most recent and complete characterizations of racial bias associated with pulse oximeter use. That review identified fifteen primary research studies pertaining to the incidence of “occult hypoxemia,” i.e., blood oxygenation deficiency that a through-the-skin pulse oximeter failed to detect.
The studies that met the inclusion criteria of Parr et al were published on or before January 2024 and indexed in a major biomedical database, e.g., MEDLINE. Additionally, studies reviewed were those limited to adults, and pulse oximeter use in formal inpatient or outpatient clinical settings.
These selection criteria allowed the researchers to identify a total of 207,464 patients attached to 732,505 discrete moments in the clinic when the same patient’s blood oxygen (02) was measured with both pulse oximetry and arterial blood gas (the gold standard) assays. The analyses found that Black patients were 67% (95% C.I. 47% to 90%) more likely to experience occult hypoxemia than White patients, while the pooled prevalence for Asian, Latinx, Indigenous, multi-racial, or other race or ethnicity was 39% (95% C.I. 19% to 64%) more likely than Whites to experience such apparent pulse oximeter failures.
Parr et al.’s study was motivated by several clinical observational studies from the COVID-19 epidemic in the United States that showed pulse oximeter errors are associated with racial disparities in delayed access to clinically needed supplemental oxygen, dexamethasone or remdesivir in response to serious infection; and increased organ failure or death related to occult hypoxemia. Moreover, recent empirical studies add to a growing body of research, dating back to at least 1991 and the work of Zeballos and Weisman,[2] that continues to show that pulse oximeters overestimate true blood 02 readings in persons with darker skin.
In anticipation of the February 2022 advisory committee meeting on this topic, the FDA compiled fifteen studies revealing racial bias in pulse oximeters.[3] By the next meeting, held in February 2024, the FDA added thirteen studies, including seven that confirmed the racial bias. For example, the Fawzy et al. 2023 study of over 24,000 COVID-19 hospitalizations found that occult hypoxemia (pulse oximeter minus blood gas discrepancies of at least 4%) were evident in nearly 20% of Black or Hispanic patients, compared to 13% of White patients.[4]
Parr et al.’s November 2024 meta-analysis confirmed the FDA’s observations in the draft guidance and used the existing publication record to clarify the magnitude and certainty of the effects observed. Basic elements of that work’s pooled results are summarized below.
Table 1 shows that a consistent and statically significant racial/ethnic bias is evident in the functioning of pulse oximeters that indicates an average 3.1% to 3.5% absolute increase in occult hypoxemia in Black compared to White patients. The studies in this analysis were rated as having moderate bias, mostly because of uncertainties regarding the device used, limits of the administrative records relied upon, and the lack of standardized collection methods for the oximeter and blood gas data.
Overall Parr et al. concluded that occult hypoxemia is “likely” to be more common in Black than in White patients and it is conceivable that such disparities are “commonplace” in modern medicine. Still, these researchers note the following limitations of their findings: 1) effects were small (absolute racial/ethnic bias estimates averaging 1.16% in patients classified as Black or African American), 2) race/ethnicity characterization was typically not or incompletely standardized and 3) other factors besides skin tone may influence these evident disparities, e.g., perfusion differences.
In 2022, to address this persistent yet nuanced problem of skin pigmentation bias with pulse oximetry, the FDA partnered with the University of California, San Francisco (UCSF) pulse oximetry laboratory to conduct prospective clinical studies.[5] This effort has to date tested at least 33 unique products to see which pass standards set in 2013. Those results are summarized in Table 2.[6]
The following observations can be made from Table 2. Across the 33 devices thus far reviewed by USCF, the accuracy (ARMS, a measure of both bias [error compared to the gold standard of a blood gas reading] and precision [variation]) of these devices ranges from 1.4% to 7.75% and the retail cost ranges from $10 to $5999 (median= $250). Additionally, the third column of Tabl3 2 shows that 13 of 33 (39%) devices failed to achieve the FDA’s 2013 accuracy standards. Although these findings cannot distinguish between racial bias and other forms of device error or unpredictability, they do show that many contemporary pulse oximeters do not have blood 02 measurements that clinicians and patients can rely upon.
Since the COVID-19 pandemic highlighted this persistent problem with pulse oximetry, the FDA has had the following responses:[7]
- February 19, 2021: Issued a safety communication to patients and health care providers stating that “multiple factors may affect the performance of a pulse oximeter’s readings such as poor circulation, skin pigmentation, skin thickness, skin temperature, current tobacco use, and use of fingernail polish.”
- 2022: Partnered with UCSF and Stanford University researchers to conduct various clinical studies on pulse oximeters, including those examining skin pigmentation effects.
- November 1, 2022: Convened the Anesthesiology and Respiratory Therapy Devices Panel (ARTDP) to review evidence about prescription pulse oximeter use. The panel concluded that people with dark skin were at increased risk for negative disease outcomes related to pulse oximetry errors compared to those with lighter skin, and that this risk should be addressed by encouraging the production of more accurate devices.
- November 16, 2023: Released a discussion paper for public comment entitled: “Approach for Improving the Performance Evaluation of Pulse Oximeter Devices Taking into Consideration Skin Pigmentation, Race and Ethnicity.” The focus of the paper was the quality of the pre-market studies the FDA uses when considering the marketing of new devices.
- February 2, 2024: Convened the ARTDP for a second time to discuss an approach “to improve the quality of premarket studies and associated methods used to evaluate the performance of pulse oximeters submitted for pre-market review, taking into consideration a participant’s skin pigmentation and participant-reported race and ethnicity.”
- January 7, 2025: Issued a guidance document for industry entitled: “Pulse Oximeters for Medical Purposes – Non-Clinical and Clinical Performance Testing, Labeling, and Premarket Submission Recommendations.”
The January 7, 2025, guidance document was based on the two ARTDP meetings and the discussion paper issued in November 2023.
On behalf of Public Citizen’s Health Research Group, I reviewed the guidance document. In general, the document enhances FDA standards for new pulse oximeters by requiring more test subjects overall, more subjects across a spectrum of human skin tones using two standardized scales, and more stringent performance results for a device to achieve before it enters the market. Furthermore, I mostly agree with the assessment of Lipinski et al.,[8] scientists from the USCF pulse oximeter laboratory who recently reviewed the guidance.
Under each bullet point, the first part is the assessment of Lipinski et al. My additional comments appear in parentheses:
- This is a long-anticipated update of FDA guidance that has not changed since 2013. Because the 2013 guidance was inadequate, there is concern about the potential liability of device manufacturers for apparent inaccuracies, especially in persons with darker skin. (In November 2023, 25 state Attorneys General sent a letter to the FDA calling for immediate action to mitigate the adverse effects of racially biased pulse oximeters.[9] I am unaware of a public FDA response.)
- Increasing the number of participants in a pre-market trial from 10 to 150; increasing the number of required persons with dark skin from 15% to 25% of each sample; use of two standardized skin tone scales; and decreases in the acceptable ARMS thresholds, i.e., more quantitative accuracy required. (These changes encourage the development of more accurate devices that are more likely to be safe and effective in racially diverse populations).
- Creation of a new metric, “non-disparate performance,” which includes the comparison of bias evident in those at the extremes of skin pigmentation, i.e., lightest versus darkest. Demonstrating non-disparate performance across subgroups of people with different skin tones is important. (Rather than using this statement on product labels: “This pulse oximeter has been evaluated to perform comparably across groups of individuals with a wide variety of skin tones based on [details provided consistent with the study conducted],”[10] the FDA should require a statement based on a single point-estimate that likely represents the extreme of the potential error of concern: “This pulse oximeter was tested in groups of individuals with very light or very dark skin and the differences in the accuracy at these extremes averaged X% based on data from Y persons tested using Z pairs of pulse oximeter/blood gas measurements over a range of blood 02 saturation from 70-95%.” )
- Proposed product labeling changes to address racial bias and other concerns about pulse oximeter accuracy represent “positive steps to address ambiguity in performance reporting but are likely insufficient.” It is unlikely that most clinicians will notice, read and fully understand the product label for the pulse oximeters they use. (Public Citizen especially agrees with the concerns that many clinicians will neither read nor fully understand the product label. The FDA should be tracking, with provider surveys and other monitoring of patient outcomes, whether the product label and other messaging about the safe use of pulse oximeters are being appropriately disseminated and interpreted.)
In addition to the concerns raised by Lipinski et al, the proposed labeling changes inadequately highlight skin pigmentation bias. For example, in warning and precaution lists proposed in the guidance, the first mention of skin pigmentation bias follows routine information, such as: “Only a health care provider can diagnose medical conditions.” The proposed warnings and precautions should, therefore, be revised and re-ordered to clearly state that darker skin tone can decrease the accuracy of pulse oximeter readings and may lead to the delay or denial of urgently needed medical care. The guidance should be revised to better frame this critical issue. The guidance should emphasize that racial bias is the main reason for the current concern about pulse oximeters.
Lipinski et al. and others have warned that stronger pre-market standards for pulse oximeters could have the unintended consequence of making these devices more expensive and thus less available.[11],[12],[13] Moreover, Lipinski et al. note that that the physics and physiology of the reasons for skin pigmentation bias in pulse oximetry are not fully understood. Nonetheless, the data in Table 2 and the Figure below (created from that data) show that the retail price of pulse oximeters is not strongly correlated with their accuracy; several less expensive oximeters perform as well as more expensive devices.
Table 2 and the Figure further show that six of the tested pulse oximeters performed quite poorly (ARMS >4). These devices, in particular, represent reasonable targets for post-marketing enforcement action including recalls— an action that Public Citizen strongly encouraged at both FDA advisory committee meetings.[14],[15] Regrettably, the FDA has yet to recall any pulse oximeters because of their deficient performance characteristics. The reasonable standards proposed in the draft guidance may identify additional devices that fail similar ARMS testing. FDA must act when pulse oximeters fail ARMS testing, especially when the pulse oximeters are medical grade, i.e., for regular use in health care settings.
Thank you for the opportunity to comment on the important topic of improving the accuracy of pulse oximeters and mitigating their risks of racial bias.
[1] Parr NJ, Beech EH, Young S, Valley TS. Racial and ethnic disparities in occult hypoxemia prevalence and clinical outcomes among hospitalized patients: a systematic review and meta-analysis. J Gen Intern Med. 2024;39(13):2543-2553.
[2] Zeballos RJ, Weisman IM. Reliability of noninvasive oximetry in Black subjects during exercise and hypoxia. Am Rev Respir Dis. 1991;144(6):1240-4.
[3] Abrams MT. Improving the Performance of Pulse Oximeters Taking into Consideration Race, Ethnicity, and Differences in Skin Pigmentation. Testimony Before the Food and Drug Administration’s Anesthesiology and Respiratory Therapy Devices Panel of the Medical Devices 2, 2024. https://www.citizen.org/article/improving-the-performance-of-pulse-oximeters-taking-into-consideration-race-ethnicity-and-differences-in-skin-pigmentation/. Accessed March 6, 2025.
[4] U.S. Food and Drug Administration. FDA Executive Summary: Performance Evaluation of Pulse Oximeters Taking into Consideration Skin Pigmentation, Race and Ethnicity. Prepared for the February 2, 2023 meeting of the Anesthesiology Devices Advisory Committee Center for Devices and Radiological Health. https://www.fda.gov/media/175828/download. Accessed January 31, 2024.
[5] Food and Drug Administration. Pulse oximeters for medical purposes—non-clinical and clinical performance testing, labeling, and premarket submission recommendations. Published January 2025. Accessed January 7, 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/pulse-oximeters-medical-purposes-non-clinical-and-clinical-performance-testing-labeling-and. Accessed March 8, 2025.
[6] Food and Drug Administration. Prospective Clinical Study of Pulse Oximeter Errors in Adult Hospitalized Patients with Varying Skin Pigmentation. October 13, 2022. https://www.fda.gov/science-research/advancing-regulatory-science/prospective-clinical-study-pulse-oximeter-errors-adult-hospitalized-patients-varying-skin. Access March 9, 2025.
[7] Food and Drug Administration. Pulse oximeters for medical purposes—non-clinical and clinical performance testing, labeling, and premarket submission recommendations. Published January 2025. Accessed January 7, 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/pulse-oximeters-medical-purposes-non-clinical-and-clinical-performance-testing-labeling-and. Accessed March 8, 2025.
[8] Lipnick MS, Ehie O, Igaga EN, Bicker P. Pulse Oximetry and Skin Pigmentation-New Guidance From the FDA. JAMA. 2025 Mar 5. doi: 10.1001/jama.2025.1959. Epub ahead of print. PMID: 40042945.
[9] McFarling UL. Pulse oximeters’ inaccuracies in darker-skinned people require urgent action, AGs tell FDA. STAT. November 7,2023.
[10] Food and Drug Administration. Pulse oximeters for medical purposes—non-clinical and clinical performance testing, labeling, and premarket submission recommendations. Published January 2025. Accessed January 7, 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/pulse-oximeters-medical-purposes-non-clinical-and-clinical-performance-testing-labeling-and. Accessed March 8, 2025.
[11] Lipnick MS, Ehie O, Igaga EN, Bicker P. Pulse Oximetry and Skin Pigmentation-New Guidance From the FDA. JAMA. 2025 Mar 5. doi: 10.1001/jama.2025.
[12] Shachar C, Drabo EF, Iwashyna TJ, Ferryman K. Addressing Racial and Ethnic Bias in Pulse Oximeters-A Wicked Problem. JAMA. 2025;333(7):563-564.
[13] American Society of Anesthesiologists. Letter regarding: [FDA-2023-N-4976] Approach for Improving the Performance Evaluation of Pulse Oximeter Devices Taking into Consideration Skin Pigmentation, Race and Ethnicity. January 16, 2024. Regulations.gov. Accessed March 9, 2025.
[14] Abrams MT. Improving the Performance of Pulse Oximeters Taking into Consideration Race, Ethnicity, and Differences in Skin Pigmentation. Testimony Before the Food and Drug Administration’s Anesthesiology and Respiratory Therapy Devices Panel of the Medical Devices 2, 2024. https://www.citizen.org/article/improving-the-performance-of-pulse-oximeters-taking-into-consideration-race-ethnicity-and-differences-in-skin-pigmentation/. Accessed March 6, 2025.
[15] Abrams MT. Review of Pulse Oximeters and Factors that can Impact their Accuracy, Especially Including Skin Pigmentation Levels Testimony Before the Food and Drug Administration’s Anesthesiology and Respiratory Therapy Devices Panel of the Medical Devices Advisory Committee. November 1, 2022. https://www.citizen.org/wp-content/uploads/2644.pdf. Access March 9, 2025.