HEALTH AND SAFETY

» Drug, Devices, and Supplements

» Physician Accountability

» Consumer Product Safety

» Worker Safety

» Health Care Delivery

» Auto and Truck Safety

» Global Access to Medicines

» Infant Formula Marketing

 

More Information on Ethics and Clinical Trials

Letter Urging an Investigation of Medical Schools Conducting Unethical Research

July 22, 2003 

Appendix: Published articles using data from the Graduation Questionnaire (pdf)

Bernard Schwetz, D.V.M., Ph.D
Acting Director
Office for Human Research Protections
Department of Health and Human Services
The Tower Building
1101 Wooton Parkway, Suite 200
Rockville, MD 20852

Dear Dr. Schwetz:

We are writing to urge you to conduct immediately an investigation into unethical and possibly illegal research currently being conducted annually on the nation’s over 15,000 graduating medical students.  This research is, in many instances, coercive, is conducted without independent ethical review and does not inform students that they are in a research project, thereby violating the Declaration of Helsinki, the Nuremberg Code and the Federal Policy for the Protection of Human Subjects. 

Since 1978, the Graduation Questionnaire (GQ) has been administered to graduating medical students by each of the nation’s medical schools, with study design, coordination and analysis coming from the Association of American Medical Colleges (AAMC).  Despite the general increased focus on safeguards in human subjects research over that quarter century, the AAMC and U.S. medical schools have continued to conduct research in a manner that, far from providing an example of how research should be conducted ethically, instead instills in medical students the notion that coercion of research subjects is somehow acceptable and that ethical oversight of such studies is not necessary.  It is time for this research to be brought in line with contemporary ethical thinking.

A number of medical schools use extremely coercive techniques to compel their students to fill out the survey.  Based on our contacts with 23 of the U.S.’s 126 medical schools, we have determined that at least seven, including the University of California at Los Angeles (UCLA), the University of Chicago and UMDNJ – New Jersey Medical School make completing the GQ a graduation requirement and six, including the University of California, San Francisco and Harvard University, assess other penalties on students if they do not comply (such as withholding tickets for friends and family to attend graduation or not refunding financial deposits that students have made).  An administrator at UCLA explained its method of assuring compliance by saying, “I'll simply say — that we get every student to comply — because if they don't, then they don't get their diploma.  We start telling them — in February of their 4th year.  It’s just that simple.  We get 100% cooperation…”[1] 

A number of sensitive topics are covered in the questionnaire, including sexual and other harassment during medical school (and the adequacy of the school’s response to such allegations) and both private and school loan debt.  Because the data are collected in a confidential, rather than an anonymous fashion,[2] there is the potential for very personal information to be disseminated if adequate measures for protecting confidentiality are not in place.  Despite this, the AAMC does not run its procedures through an ethical review committee known as an Institutional Review Boards (IRB).  In addition, of nine schools that indicated whether or not they sought IRB approval, only Emory University runs the questionnaire through its own IRB. 

We agree that the GQ is an important tool.  It gives policy-makers information on the current state of medical education.  It tracks rates of medical student indebtedness and sexual harassment.  It enables researchers to answer valuable questions about medical school graduates’ career choices.  For these reasons, we do not wish to see the GQ ended; we simply seek to ensure that the data are collected ethically.  The remainder of this letter outlines our concerns in more detail and indicates the modest and reasonable changes that the AAMC and its members could make to bring this research into compliance with current ethical and legal standards. 

Background

The AAMC is a non-profit organization that represents the 126 accredited allopathic medical schools in the U.S.[3] When it was first administered by the AAMC in 1978, the GQ was conducted with pencil and paper, but since 1999 it has been conducted on a password-protected site on the Internet.[4] Evidently, the AAMC is responsible for the design and administration of the GQ, while the individual schools are responsible for assuring that the students complete the survey.  The AAMC claims that there was a 91% response rate for the 2002 GQ,[5] unusually high for a research survey of students or physicians.

The GQ asks questions about a wide range of the experiences of medical students.  Sections on the 2003 GQ included:

  • Pre-medical and pre-clinical experiences;
  • Clinical experiences;
  • Specialty and career plans;
  • Medical school experiences (includes questions on the student’s personal experiences with harassment and mistreatment);
  • General opinion questions on the current state of medicine; and
  • Financial Aid information (includes questions on both private and medical school-related debt accumulated by the student).[6]

Upon completing the survey, medical students are directed to a page entitled “Release Permission.”  There they can choose to have their responses remain anonymous, so that their responses are kept in a file separate from any personal information.  Alternatively, students can choose to have their answers remain confidential.  If this option is selected, responses of students can be linked later to other personal information kept by the AAMC or other organizations.  According to the AAMC, over half of the responding medical students choose to have their responses be confidential rather than anonymous.[7] 

While the AAMC administers the GQ and hosts the survey on its website, medical schools are responsible for encouraging their students to complete the survey.  As an Academic Officer at Baylor University explained, “The AAMC requires us to have a role.  We’re required to try and get the forms filled out.”[8] For this function, they require the names of non-responders.  The GQ is thus decidedly not anonymous while it is being administered, regardless of the anonymous/confidential option selected by the student.  Each medical school designates a GQ coordinator, with whom the AAMC communicates.  This coordinator publicizes the survey to medical students and monitors student compliance by logging onto the AAMC GQ website with a secure password.  On this site, the coordinator is able to see by name whether or not the student has completed the survey.  The AAMC states that the coordinator cannot see the responses themselves.  Each school then develops its own strategy for maximizing response rates.

Information from the GQ appears to be used in four ways.  First, the AAMC publicly releases the aggregate results (no schools or individuals named) of the GQ on its website.[9] Second, school administrators are provided with a summary report showing how their school compares to the national averages on every question.  Third, for more elaborate analyses, schools can provide the AAMC with additional sets of information that the AAMC can link to students’ responses on the GQ using the students’ personal identifiers.  For example, schools could ask to see how in-state students compare to out-of-state students by providing the AAMC with a list of students who are state residents and those who are not.  The AAMC would then run the analysis and provide the school with the results.  According to the AAMC,

If a school asks, we will prepare a data set for them.  It is up to medical schools to decide how to use their own data.  Around 30-40 medical schools request this every year.  We don’t track what they use it for… In the past, we sometimes would merge two data sets – the AAMC GQ with another one (like the AMA Masterfile) using anonymous identifiers.  We aren’t doing that at the present time.[10] 

Fourth, a number of journal articles have been published using data from the GQ.  This is relevant to determining whether the GQ is “research” or “evaluation” (see discussion below).  MEDLINE searches with the Medical Subject Heading terms “Students, Medical” and “Questionnaires” and additional keyword searches for “graduation questionnaire” yielded 21 articles that have been published using data from the GQ alone or using the GQ and other sources.  The federal government funded one of these studies[11] and another had an author who was a federal employee.[12] (See Appendix for details)  Sixteen articles had at least one AAMC author and in 12 of these all authors were from the AAMC.  No article mentioned that an IRB had reviewed the study design.  This may underreport the true extent of federal funding and IRB approval, as journal articles often fail to report these items.[13],[14] Of these 21 studies, 17 were published in the AAMC’s own journal, Academic Medicine, or its predecessor, The Journal of Medical Education.  In all but one study, the published article was based on the full national dataset, not on data about specific medical schools.  Some of the studies include the students’ responses on the questions about harassment and debt.

In eight studies, responses from the GQ were linked to other data.  The most common linked datasets were the Matriculating Student Questionnaire (MSQ; five studies), the Premedical Student Questionnaire (PSQ; three studies), and the Medical College Admission Test scores (MCAT; two studies).[15] 

Ethical Standards

Three documents – the Nuremberg Code, the World Medical Association’s Declaration of Helsinki, and the Council for International Organizations of Medical Sciences’ (CIOMS) International Ethical Guidelines for Biomedical Research Involving Human Subjects – provide standards for assessing whether researchers’ behavior is ethical.

The first principle of the Nuremberg Code (1949) states that:

The voluntary consent of the human subject is absolutely essential.  This means that the person involved should have legal capacity to give consent; should be so situated as to be able to exercise free power of choice, without the intervention of any element of force, fraud, deceit, duress, over-reaching, or other ulterior form of constraint or coercion…[16] 

Three principles from the Declaration of Helsinki (first developed in 1975) are relevant to the case at hand.  Like the Nuremberg Code, the Declaration of Helsinki heavily emphasizes the voluntary participation of subjects:

The subjects must be volunteers and informed participants in the research project… The subject should be informed of the right to abstain from participation in the study or to withdraw consent to participate at any time without reprisal.[17] 

The Declaration was also one of the first documents to emphasize the need for review by an ethics committee:

The design and performance of each experimental procedure involving human subjects should be clearly formulated in an experimental protocol.  This protocol should be submitted for consideration, comment, guidance, and where appropriate, approval to a specially appointed ethical review committee, which must be independent of the investigator, the sponsor or any other kind of undue influence.

Finally, the Declaration of Helsinki discusses the need to minimize risks and provide benefits.

Medical research is only justified if there is a reasonable likelihood that the populations in which the research is carried out stand to benefit from the results of the research.

CIOMS is also clear on these points in its International Ethical Guidelines for Biomedical Research Involving Human Subjects (first published in 1993).  The following guidelines are applicable to the present discussion.

Guideline 2: Ethical review committees

All proposals to conduct research involving human subjects must be submitted for review of their scientific merit and ethical acceptability to one or more scientific review and ethical review committees.  The review committees must be independent of the research team… The investigator must obtain their approval or clearance before undertaking the research…

Guideline 5: Obtaining informed consent: Essential information for prospective research subjects

Before requesting an individual's consent to participate in research, the investigator must provide the following information, in language or another form of communication that the individual can understand:

1. that the individual is invited to participate in research, the reasons for considering the individual suitable for the research, and that participation is voluntary;

2. that the individual is free to refuse to participate and will be free to withdraw from the research at any time without penalty or loss of benefits to which he or she would otherwise be entitled;

15. the limits, legal or other, to the investigators' ability to safeguard confidentiality, and the possible consequences of breaches of confidentiality;

26. that an ethical review committee has approved or cleared the research protocol.[18] 


Ethical Problems with the GQ

The GQ is coercive

In order to examine the methods used by the medical schools to increase their response rates, we contacted individuals at 27 medical schools.  The schools were selected in non-random fashion to focus disproportionately on the most influential schools, to achieve some geographic diversity, and to use contacts we had at specific medical schools. At 13 of these schools, we received data directly from administrators, while at 10 we obtained information from graduating medical students.  All 23 respondents were from different medical schools.  At the remaining four medical schools, administrators did not respond to our questions.

Several of the schools use the names of students to determine who has completed the survey and then contact the students individually (in-person or through email), asking them to complete the survey.  As an administrator at the University of Connecticut stated, “We track completion and send reminders to those who have not completed it.”[19] Similarly, an official at Emory University explained that they use “verbal requests” to follow up with students who have not started or finished the GQ.[20] 

Thirteen of 23 schools contacted punish students in some way for not filling out the survey (see Table).  At seven of these 13 schools, filling out the GQ is a graduation requirement.  An administrator at the University of Miami said, “Completion of the AAMC GQ is a requirement for graduation… Students need to complete all requirements for graduation to receive their diploma.”[21] The other six of the 13 assess other penalties if students do not fill out the GQ.  For example, a student at the University of California at San Diego (UCSD) explained that, “I had to do the very, very long survey in order to get the $90 deposit for my ‘learning resources center’ key refunded. I wasn’t so happy about that but I need the cash!  I guess that's the UCSD way.”[22] An administrator at TulaneUniversitysent out an email telling students that, “If you do not at least start the questionnaire by March 17, you will not be able to participate in the Match Day ceremonies.”[23] (Match Day is when students find out what residency they will start a few months later.)  An administrator at Harvard said that, “In order to assure compliance, we don’t distribute graduation lunch until a student has completed his or her questionnaire.”[24] Another technique is to withhold tickets for friends and family to attend graduation until the survey is complete.

Table: Incentives and IRB approval for the GQ at selected U.S. medical schools

School

Information Source[25]

Incentive Technique

Use of Local IRB

Albany Medical College

Student

Graduation Requirement

 

Baylor

Administrator

None

 

Boston University

Student

Tickets for family and friends to attend graduation withheld

No

Chicago Medical School

Student

Graduation Requirement

 

Columbia

Student

Extra funds for graduation celebration with 100% compliance

 

Dartmouth

Administrator

Graduation activity subsidized with 90% participation rate

 

Emory

Administrator

None

Yes

George Washington

Student

Graduation Requirement

 

Harvard

Administrator

Graduation lunch not distributed unless student has completed questionnaire

 

Johns Hopkins

Administrator

None

No

University of Miami

Administrator

Graduation Requirement

No

Stanford University

Student

Prizes with early completion

 

Tulane

Administrator (email forwarded by student)

Need to complete questionnaire to participate in Match Day ceremonies

 

University of California, Los Angeles

Administrator

Graduation Requirement

 

University of California, San Diego

Student

$90 key deposit not refunded until survey completed

 

University of California, San Francisco

Administrator

Invitations for family and friends to attend graduation withheld until survey completed

No

University of Connecticut

Administrator

None

No

University of Massachusetts

Administrator

Graduation Requirement

No

UMDNJ – New Jersey Medical School

Student

Graduation Requirement

 

University of Pennsylvania

Student

Tickets/Invitations for family and friends to attend graduation withheld

 

Vanderbilt

Administrator

$25 gift certificate raffled off if you complete the survey by March 31

No

Yale

Administrator

None

No

University of Vermont

Student

None

 


Four other schools use incentives (rather than punitive measures) to get students to fill out the GQ.  One student from Columbia University told us that, “Columbia promises extra funds for our graduation celebration if 100% of the class fills out the survey, then follows up with a barrage of e-mail ‘reminders’ to members of the class who have not yet filled it out.”[26] An administrator at DartmouthUniversitystated that, “The only incentive we (the offices of student affairs and medical education) provide is to subsidize a major graduation student social activity (i.e., Mount Sunapee Cruise) providing we obtain 90% or more class participation in the survey.”[27] These techniques rely upon peer pressure from other students to ensure a high response rate for the GQ. 

Six of the 23 schools responding use neither incentives nor disincentives.  An example is the University of Connecticut, where an administrator told us, “No incentives other than stressing the value to upcoming classes and to the administration in order to address issues of concern to the students.”[28] 

We acknowledge that minor, non-coercive forms of encouragement to complete the questionnaire are acceptable.  Notwithstanding the confidentiality and the IRB review issues (see below), simple reminders to non-responding students are also acceptable.  But threatening the students with preventing them from graduating or attending valued graduation activities after they have labored long and hard for their medical degree is wholly inappropriate.  Using peer pressure — by not funding a class activity until a certain percentage of the class has completed the survey — is also unacceptable.

The Nuremberg Code, the Declaration of Helsinki and the CIOMS guidelines are all clear on this point: coercion to participate has no place in research.  Despite this, the AAMC and many medical schools continue to conduct research in which medical students have no real choice but to participate.

The GQ has not undergone independent ethical review

It is at least possible that oversight from an IRB would have eliminated or at least greatly diminished the coercive elements of this research.  But no IRB has central oversight over this research project, and, of nine medical schools from whom we were able to obtain such information, we found only one (Emory[29]) that involves a local IRB.  The failure to secure IRB approval violates both the Declaration of Helsinki and the CIOMS guidelines.

The AAMC does not use an IRB, even though it engages in a series of activities that are clearly part of the research process: it develops the questions, posts the questionnaire on its website and maintains it there, provides each of the medical schools with the names of non-responders, sends separate reports to the individual medical schools, provides independent researchers with data, and utilizes in-house researchers to analyze data and publish in medical journals.  In an interview with us, the AAMC stated that it does not use an IRB, but:

We have a “data-clearance process” that reviews all surveys that we conduct, to make sure that the instrument is appropriate, that it needs to be done, etc.  They’ve done this — I assume — every year that the GQ has been used… It’s up to each school [whether or not to use an IRB] — we do not encourage or discourage it.[30] 

Ethical review should also occur at the school level, because each school provides the AAMC with the names of graduating students, uses the AAMC website to identify the non-responders and then, in many cases, contacts the students individually to improve response rates.  In some cases, school faculty publish medical journal articles.  These are all research functions.  

It is the nature of IRBs that no one can predict what aspects of a protocol it may find troubling.  But it is very likely that any IRB that reviewed this research would concentrate on the coercive elements, when present, and the efficacy of the procedures used to maintain student confidentiality (see below). 

We recognize that there is currently much debate over the form of ethical review for multi-center studies.[31],[32],[33] Some authors have argued for a single, centralized review, but this study illustrates a circumstance in which additional review at the “local” (school) level would add an additional measure of protection for participants.  Precisely because the schools are left by the AAMC to develop their own techniques for maximizing participation, resulting in an array of techniques ranging from the acceptable to the blatantly coercive, some degree of local review, in addition to central review, is necessary.  Neither is properly in place here. 

The confusion over the responsibility for review of multi-center studies is reflected in bureaucratic finger-pointing.  The AAMC states that it is up to medical schools to decide whether or not to utilize the IRB.[34] However, the registrar at Johns Hopkins University, when asked if an IRB had been used, stated that, “No.  That would be something the AAMC would do.”[35] In response to the same question, the Associate Dean of Student Affairs at the University of Miami stated that, “No. The AAMC is the sponsoring agency of the GQ, not the University of Miami.”[36] Until the debate over the review of multi-center studies is resolved, full ethical review should occur at both a central and local level.

With the GQ, the AAMC is assuming functions similar to those of Contract Research Organizations (CROs), organizations that commonly conduct research at multiple sites, usually on behalf of pharmaceutical companies.  In the case of CROs, each site (typically an academic center) is required to have its own IRB review the study protocol.  In addition, the CRO typically uses a central IRB to review the study.  There are now a number of for-profit IRBs that can review such research.  Because these IRBs have an obvious incentive to approve research (and increase the number of protocols they subsequently receive for review), the GQ should be reviewed by an IRB in the government instead.  (The IRBs at the individual schools should not be acting as both central and local reviewers.)

Sensitive topics are included on the GQ; AAMC procedures may not be sufficient to maintain confidentiality

While most of the questions on the GQ have to do with student satisfaction with their medical school curriculum and career plans, other questions stray into personal areas, such as harassment, debt level and opinions about the healthcare system.  Questions from the GQ include:

[With what frequency have you] been asked to exchange sexual favors for grades or other awards?

[How satisfied are you that your school’s mistreatment procedures] equitably investigated and adjudicated complaint(s)?

Do you have any outstanding loans for your medical school education, which you are legally required to repay?  [If yes, also indicate the amount.]

Do you have any debts other than educational (e.g., credit cards, car loans, etc.)?  [If yes, also indicate the amount.]

These differ from the rest of the questions due to the personally sensitive subject areas that they explore.  The AAMC states that over half of the responses to the GQ are confidential, not anonymous.  Thus, if the AAMC’s procedures are somehow inadequate, it is possible to track potentially damaging responses to these questions back to the respondent. 

It is quite possible that the AAMC’s safeguards to protect confidentiality are entirely appropriate and would be considered so by an IRB.  But the safeguards are complicated and there is at least the potential of their being breached because the study is not anonymous.  It is not the place of the AAMC to declare its own safeguards sufficient or for the medical schools to passively assume that this is so; independent bodies must confirm that this is the case.  This is the very rationale for the IRB system.

Students are not informed that the study results may be published

A fundamental right of research participants is to know that they are included in a study.  The GQ fails even to meet this minimum standard.  Students may assume that the results are intended only for evaluation purposes and not that their responses may be included in published research articles.

The GQ is not the only example of how data on health professionals in an educational setting have later been published as research, without the approval of the research participants.  At one pediatric residency program at Harvard, researchers collected data on alcohol use among residents as part of an educational intervention.  Without seeking the consent of the residents, they then published an article outlining alcohol problems among these residents.[37] The authors were forced to acknowledge that they had made errors in conducting the research.[38] 

Ensuring that medical students will benefit

As mentioned earlier, the Declaration of Helsinki recommends that the research populations must have a reasonable likelihood of benefiting from the research.  However, the AAMC and medical schools are preventing the maximum benefit of the GQ from being realized.  In particular, the schools do not in general release their data publicly.  If the schools were to share their results publicly, at least two benefits would occur.  First, students would be able to advocate more effectively for curricular and other improvements in their medical schools and to monitor their schools’ progress.  Second, prospective medical students could use these data to make more informed choices about the medical schools to which they wished to apply.  Graduating medical students could also use them in selecting their residency programs.  (We realize that some of these benefits would not accrue to the graduating medical students themselves.  However, the ethical requirement to ensure that the participants benefit from the research can be fulfilled if the members of the group from which the subjects are drawn would benefit.) 

Federal Law

The Code of Federal Regulations 45 CFR § 46 (Federal Policy for the Protection of Human Subjects) explains what the federal government considers research, the role of IRBs in overseeing research and the role of informed consent in the research process.

Research means a systematic investigation, including research development, testing and evaluation [emphasis added] designed to develop or contribute to generalizable knowledge.  Activities which meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program which is considered research for other purposes.[39] 

Thus, evaluations that are systematic and/or designed to produce “generalizable knowledge” are considered research.  Research that is exempt from 45 CFR § 46 is discussed below under “The GQ is much more than just curriculum evaluation.”

Later sections discuss the role of IRBs.

An IRB shall review and have authority to approve, require modifications in (to secure approval), or disapprove all research activities covered by this policy.[40] 

Such review would include

When appropriate, [that] there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data.[41] 

Finally, 45 CFR § 46 discusses what must be included in any informed consent form.

A statement that participation is voluntary, refusal to participate will involve no penalty or loss of benefits to which the subject is otherwise entitled and the subject may discontinue participation at any time without penalty or loss of benefits to which the subject is otherwise entitled.[42] 

For the reasons discussed in the next section, we believe that the medical schools have administered the GQ in violation of these regulations.

Legal Issues Related to the GQ

The research takes place at federally funded institutions and is thus covered by federal law

In order to receive federal funds, universities must demonstrate to the federal government that they have a functioning IRB and appropriate ethical safeguards in place.  Research submitted to the Food and Drug Administration (FDA) as part of a New Drug Application is also expected to have met similar standards.  However, 45 CFR § 46 provides that, for all federally supported projects, an institution must provide an “assurance” that it will comply with the requirements stated in 45 CFR § 46.[43] The assurance must include a statement of the principles for protecting human subjects, “regardless of whether the research is subject to federal regulation.”[44] 

It is likely that every medical school receives federal monies for research,[45] even if those funds are not related to the GQ,[46] and thus must provide the government with an assurance that all research at the institution “protect[s] the rights and welfare of human subjects of research.”[47] This would include conducting only research that is free from coercion and is reviewed by an IRB. 

Because, to our knowledge, the AAMC is not a recipient of federal research funds, it would not be subject to 45 CFR § 46.  However, this does not release the AAMC from its ethical responsibilities.  It is hardly admirable for the organization representing the nation’s medical schools to conduct research on medical students that would be impermissible under the guidelines covering all federally funded research.  On the other hand, the participating universities that receive federal funding do appear to be in direct violation of federal law.

The fact that the AAMC is not subject to federal regulations highlights an underlying problem in the structure of federal research ethics oversight: many categories of research are simply not covered.  For example, much research conducted in private settings — such as a physician’s office, a weight loss clinic, an in-vitro fertilization clinic, or corporate health programs — are not covered unless they are federally funded or are part of a New Drug Application.  A number of groups have issued recommendations addressing this gap in federal law.  One of the most prominent of these, the presidentially established National Bioethics Advisory Commission (NBAC), expressed frustration with the gaps in the current legal and regulatory system.

Recommendation 2.3:  A unified, comprehensive federal policy embodied in a single set of regulations and guidance should be created that would apply to all types of research involving human participants.[48] 

Unfortunately, this NBAC recommendation has not been implemented.  The circumstances described in this letter adds further urgency to the need to implement those recommendations.

The GQ creates generalizable knowledge and is thus “research”

The AAMC and the medical schools may counter these criticisms by claiming that the GQ is not research at all and thus is not subject to 45 CFR § 46.  As noted above, federal law requires that an activity develop or contribute to “generalizable knowledge” for it to be considered research.  We believe that this standard has been met in this case.

To begin with, the complete results of the GQ, not broken down by medical school, are posted on the AAMC’s website.  In addition, the GQ tells medical students at its beginning:

Thank you for participating in this year's AAMC Medical School Graduation Questionnaire (GQ). The Association is very pleased to be able to offer you this survey to assure your input into important medical education issues both at your school and nationally [emphasis added]. The results of the GQ have been used since 1978 to assist the Association and medical schools in priority setting and program and policy development.[49] 

This confirms that producing generalizable (in this case national) data is a central purpose of the GQ.  The fact that a minimum of 21 studies have appeared in the peer-reviewed medical literature, that only one of these does not use the full national data set and that none involve the evaluation of educational techniques provides further confirmation. 

The GQ is much more than just curriculum evaluation

It is true that there are exemptions to 45 CFR § 46 for what might be termed evaluation.  Research that is exempt includes:

(1) Research conducted in established or commonly accepted educational settings, involving normal educational practices, such as (i) research on regular and special education instructional strategies, or (ii) research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods.

(2) Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless: (i) information obtained is recorded in such a manner that human subjects can be identified, directly or through identifiers linked to the subjects; and (ii) any disclosure of the human subjects' responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, or reputation.[50] 

However, the GQ does not fall within either of these exemptions.  Turning to Exemption 1, as noted earlier, the GQ asks questions in a number of topic areas that have more relation to personal feelings, experience and plans than to educational practices – debt levels, experiences with harassment, and general opinion questions.  We estimate that 12% of the questions are on such topics.  Furthermore, the results are used in such a way that it is impossible to compare various instructional techniques reliably.  Schools are provided only with their own data compared to the national average, and the national results that are released do not enable anyone to compare the efficacy of different didactic methods.  Indeed, it is noteworthy that not a single one of the published papers focuses on comparisons of educational techniques.  Instead, the papers focus on describing the medical student population in some respect or in following some characteristic of medical students over time.  It is therefore clear that the criteria for exemptions 1(i) and 1(ii) are not met.

While we acknowledge that schools may indeed use their own schools’ GQ data in part for evaluation purposes, the question is not whether any evaluation is being conducted, but rather whether any research is being conducted, because that is the threshold for federal oversight. 

With respect to Exemption 2(i), as described above, the data are collected in such a manner that requires both names and personal identifiers. The AAMC has the names of students on file, in order to aid the schools in maximizing their response rates.  They may have other personal identifiers on file, as these can also be used to link individual responses from the GQ to other data sets.  With respect to Exemption 2(ii), for a school to know who has and who has not completed the questionnaire is itself a modest threat to students.  If the AAMC’s confidentiality procedure is somehow breached (and it is an IRB’s responsibility to ensure that this is not likely), students could theoretically be placed at risk of reprisal if in the GQ they reported sexual harassment at the school or the school’s inadequate response to a complaint they had filed.  The IRB would also be charged with ensuring that personal political views or data on personal debt remained private.  Thus, neither exemption 2(i) nor 2(ii) is met.

The NBAC agreed that data collection efforts such as the GQ should be considered research.

Recommendation 2.4:  Federal policy should cover research involving human participants that entails systematic collection or analysis of data with the intent to generate new knowledge.  Research should be considered to involve human participants when individuals 1) are exposed to manipulations, interventions, observations, or other types of interactions with investigators or 2) are identifiable through research using biological materials, medical or other records, or databases.[51] 

Both of these criteria have been met.  The first criterion is met because there is interaction between researcher and student (generally electronically, although some of the interactions between students and representatives of their medical schools exhorting them to complete the survey occur in person).  With regard to the second point, we have already established that respondents to the GQ are identifiable in an AAMC database.

The NBAC report did briefly discuss health services research and program evaluation in its report.

If the purpose [of the evaluation] is to assess the success of an established program, and the information gained from the evaluation will be used to improve that program, the activity should not be considered research involving human participants. Evaluation is a program monitoring tool, and the information gained will immediately benefit the program and/or the individuals involved.[52] 

If the results from GQ were only used locally at each medical school, the GQ would not be considered research according to the NBAC.  However, given that the results from the GQ are used in many other capacities and contribute to generalizable knowledge, the GQ falls under the NBAC’s definition of research.  As mentioned above, the sine qua non of generalizable knowledge is published studies, of which there have been almost two dozen drawn from the GQ.

Other authors state that studies that have dual research and evaluation functions, similar to the GQ, do constitute research and thus need to be overseen by an IRB.  In the AAMC’s own journal, Academic Medicine, two researchers argued that evaluation projects that gather data for the purpose of producing “generalizable knowledge” do constitute research and that students have a right to know when published articles are likely to ensue. The authors go on to compare the similar conflicts of interest faced by both clinician-researchers and educator-evaluators and state that is possible to abuse one’s position as a clinician or as an educator in order to conduct research.  The solution, they state, is to utilize IRBs and have proper informed consent not just for clinical trials but also for evaluation projects that contribute to generalizable knowledge.  They continue:

We recommend several steps to strengthen institutional policies to protect the rights of learners when scholarship rests upon student data…

1. Medical schools should implement requirements that all institutionally sponsored evaluations that constitute research and that use student data must include documentation of IRB approval [53]

Conclusion

We reiterate our belief that the GQ serves a useful function.  But we believe just as strongly that it can only properly serve that function if it is conducted ethically and legally.  Fortunately, the changes necessary to meet those goals are modest and achievable.  We thus call for the following measures to address the problems described in this letter:

1. IRBs at each medical school need to review the administration of the GQ at their school.  Furthermore, the AAMC should contract with an independent IRB, ideally the IRB at a federal agency, to review the administration of the survey.  It is possible that this central IRB would permit the IRBs at each medical school to perform only a limited review focusing particularly on the issue of coercion. 

2. Coercion of medical students to complete the GQ should be eliminated.  Very modest, non-coercive inducements to participate are acceptable, but punitive measures or the removal of significant benefits that students would normally enjoy are inappropriate.

3. Students taking the GQ must be clearly informed that they are participating in research and that their responses may form part of published research articles.

4. For each of the years of the student’s enrollment in medical school, he or she should receive a copy of the school’s performance on the previous year’s GQ, comparing the school to the national average.  In addition, school-specific reports should be posted on the AAMC’s website along with the aggregate report.

5. The AAMC and the medical schools should issue a formal apology on its website and in a press release.

It is extremely disappointing that in this day and age some medical schools and the AAMC continue to conduct unethical, and in some respects illegal, research on medical students.  Rather than taking the opportunity to educate medical students on the importance of research ethics, the AAMC and the individual medical schools have taken the easy but unethical road.  The AAMC’s own website states that,

Medical schools and teaching hospitals value integrity and accountability in the conduct of all research and vigilant protection of the rights and welfare of patients and other human subjects participating in research.[54] 

It is time to live up to the lofty ideals ostensibly promoted by the organization.  By their evasion of the accepted legal oversight mechanisms, the AAMC and the individual medical schools have demonstrated their inability to police themselves.  It is time for the OHRP to step in.

Yours sincerely,

Joshua Rising, M.D., M.P.H.
PGY-1
Department of Pediatrics
University of California at San Francisco*

Peter Lurie, M.D., M.P.H.
Deputy Director
Public Citizen’s Health Research Group

Sidney M. Wolfe, M.D.
Director
Public Citizen’s Health Research Group

*Affiliation for identification purposes only

cc: Jordan Cohen, AAMC 
Deans of all U.S. medical schools


[1] Email conversation with Charlotte Myers, Director of the Office of Student Affairs at the University of California at Los Angeles School of Medicine on April 3, 2003.

[2] In anonymous data collection, the name of the respondent is not even collected.  In confidential data collection, the name of the student is collected, but kept in a separate electronic data file.  This file allows certain individuals within the universities to contact non-responders, while keeping their names separate from the database that is used for analysis or shared with the schools.

[7] Phone conversation with Deborah Danoff, Marquita Williams and Raj Sabharwal of the AAMC on April 8, 2003.

[8] Phone conversation with Pamela Ashley at Baylor University School of Medicine on March 27, 2003.

[10] Phone conversation with Deborah Danoff, Marquita Williams and Raj Sabharwal of the AAMC on April 8, 2003, Ms. Danoff speaking.

[11] Dial TH, Lindley DW.  Predictive Validity of Specialty Choice Data From AAMC GQ.  Journal of Medical Education 1987;62:955-958.

[12] Bowman MA, Haynes RA, Rivo ML, Killian CD, Davis PH.  Characteristics of Medical Students by Level of Interest in Family Practice.  Family Medicine 1996;28:713-9.

[13] Olson CM, Jobe KA.  Reporting Approval by Research Ethics Committees and Subjects’ Consent in Human Resuscitation Research. Resuscitation 1996;31:255-263.

[14] Rennie D, Yank V.  Disclosure to the Reader of Institutional Review Board Approval and Informed Consent. Journal of the American Medical Association 1997;277:922-923.

[15] The MSQ is a survey administered by the AAMC and medical schools when students begin their medical education.  The PSQ is administered by the AAMC to premedical students when they take the MCAT, the required exam for entering medical school.  Some studies linked to more than one database.  We did not collect information about the administration of the MSQ and PSQ as part of this project.

[16] Trials of War Criminals before the Nuremberg Military Tribunals under Control Council Law No. 10, Vol. 2, pp. 181-182. Washington, D.C.: U.S. Government Printing Office, 1949.  Available online at http://ohsr.od.nih.gov/nuremberg.php3.

[17] World Medical Association Declaration of Helsinki: Recommendations Guiding Physicians in Biomedical Research Involving Human Subjects.  Adopted by the 18th World Medical Assembly, Helsinki, 1964 and revised by the 29th World Medical Assembly, Tokyo, 1975, the 35th World Medical Assembly, Venice, 1983, the 41st World Medical Assembly, Hong Kong, 1989, the 48th World Medical Assembly, Somerset West, Republic of South Africa, 1996, and the 52nd World Medical Assembly, Edinburgh, Scotland, 2000. Note of Clarification on Paragraph 29 added by the WMA General Assembly, Washington 2002. Available online at http://www.wma.net/e/policy/b3.htm.

[18] Council for International Organizations of Medical Sciences, World Health Organization.  International Ethical Guidelines for Biomedical Research Involving Human Subjects, 2002.  Available online at http://www.cioms.ch/frame_guidelines_nov_2002.htm.

[19] Email conversation with Keat Sanford, Assistant Dean and Director of Student Affairs at the University of Connecticut School of Medicine, on March 29, 2003.

[20] Email conversation with Fran Wiggins at Emory University School of Medicine April 18, 2003.

[21] Email conversation with Robert Hernandez, Associate Dean for Student Affairs at the University of Miami, on April 3, 2002.

[22] Email conversation with a student from the University of California at San Diego on March 25, 2003.

[23] Email sent from Charlotte Steger at the Tulane University School of Medicine to Tulane medical students on March 10, 2003.

[24] Email conversation with Carla Fujimoto, Special Assistant to the Dean at Harvard University School of Medicine, on March 17, 2003.

[25] Administrative sources as following:  Baylor = Pamela Ashley, Academic Officer, Dean of Students Office.  Dartmouth = Sue Ann Hennessy, Assistant Dean for Student Affairs.  Emory = Jonas Shulman, Executive Associate Dean for medical education and student affairs, via Fran Wiggins, Executive Administrative Assistant.  Harvard = Carla Fujimoto, Office of Student Affairs.  Johns Hopkins = Pat Elardo, Administrative Secretary, Office of Student Affairs, Mary Foy, Registrar, and Frank Herlong, Associate Dean for Student Affairs.  University of Miami = Robert L. Hernandez, Jr., Associate Dean for Student Affairs.  Tulane = Charlotte Steger, Assistant Director of Student Affairs.  UCLA = Charlotte Myers, Director of Student Affairs Office.  UCSF = Francis Harvey, Administrator, Office of Student Affairs.  UConn = Keat Sanford, Assistant Dean and Director of Student Affairs.  UMass = Mai-Lan Rogoff, Associate Dean for Student Affairs.  Vanderbilt = Tamie Swah, Administrative Assistant for Associate Dean for Student Affairs.  Yale = Nancy Angoff, Associate Dean of Student Affairs.

[26] Email conversation with a student from Columbia University on March 29, 2003.

[27] Email conversation with Sue Ann Hennessy, Assistant Dean for Student Affairs at Dartmouth University, on March 18, 2003.

[28] Email conversation with Keat Sanford at the University of Connecticut School of Medicine on March 29, 2003.

[29] Email conversation with Fran Wiggins at Emory University School of Medicine April 18, 2003.

[30] Phone conversation with Deborah Danoff, Marquita Williams and Raj Sabharwal of the AAMC on April 8, 2003, Ms. Danoff speaking.

[31] Christian MC, Goldberg JL, Killen J, et al.  A Central Institutional Review Board for Multi-Institutional Trials.  New England Journal of Medicine 2002;346:1405-1408.

[32] Slater EE.  IRB Reform.  New England Journal of Medicine 2002;346:1404. 

[33] Steinbrook R.  Improving Protection for Research Subjects.  New England Journal of Medicine 2002;346:1425-1430. 

[34] Phone conversation with Deborah Danoff, Marquita Williams and Raj Sabharwal of the AAMC on April 8, 2003.

[35] Phone interview with Mary Foy at the Johns Hopkins University School of Medicine on March 27, 2003.

[36] Email interview with Robert Hernandez, Jr. at University of Miami School of Medicine on April 3, 2003.

[37] Knight JR, Palacios J, Shannon M.  Prevalence of Alcohol Problems Among Pediatric Residents Archives of Pediatric and Adolescent Medicine 1999;153:1181-1183.

[38] Knight JR, Palacios J, Shannon M.  Unanticipated Reactions to a Recent Report on Alcohol Problems Among Pediatric Residents. Archives of Pediatric and Adolescent Medicine 2000;154:635.

[39] 45 CFR § 46.102(d)

[40] 45 CFR § 46.109(a)

[41] 45 CFR § 46.111(a)(7)

[42] 45 CFR § 46.116(a)(8)

[43] The assurance may be either project-specific or a multiple project, or “federalwide,” assurance approved by and on file with and the HHS Office of Protection from Research Risks.  45 CFR § 46.103 (a)

[44] 45 CFR § 46.103 (b)(1)

[45] Using data from the AAMC’s website, we have determined that all but three of the 126 U.S. medical schools received awards from just the National Institutes of Health in Fiscal Year 2002.  Of course, schools may have received funding from other federal sources and in other years.

[46] To the best of our knowledge, the AAMC and the medical schools do not receive federal monies to conduct the GQ.  (The one study that has federal funding may be funded to link databases to the GQ and then conduct analyses, or the data collection for the other databases may have been federally funded.)

[47] 45 CR § 46.103 (b)(1)

[48] National Bioethics Advisory Committee, Ethical and Policy Issues in Research Involving Human Participants, Bethesda, Maryland, August 2001, p. 9.  Available online at http://www.georgetown.edu/research/nrcbl/nbac/human/overvol1.pdf.

[50] 45 CFR § 46.101(b)(1). Available online at http://ohrp.osophs.dhhs.gov/humansubjects/guidance/45cfr46.htm.

[51] National Bioethics Advisory Committee, Ethical and Policy Issues in Research Involving Human Participants, Bethesda, Maryland, August 2001, p. 9.  Available online at http://www.georgetown.edu/research/nrcbl/nbac/human/overvol1.pdf.

[52] National Bioethics Advisory Committee, Ethical and Policy Issues in Research Involving Human Participants, Bethesda, Maryland, August 2001, p. 37.  Available online at http://www.georgetown.edu/research/nrcbl/nbac/human/overvol1.pdf.

[53] Henry RC, Wright DE.  When Do Medical Students Become Human Subjects of Research?  The Case of Program Evaluation.  Academic Medicine 76:871-875.

Copyright © 2014 Public Citizen. Some rights reserved. Non-commercial use of text and images in which Public Citizen holds the copyright is permitted, with attribution, under the terms and conditions of a Creative Commons License. This Web site is shared by Public Citizen Inc. and Public Citizen Foundation. Learn More about the distinction between these two components of Public Citizen.


Public Citizen, Inc. and Public Citizen Foundation

 

Together, two separate corporate entities called Public Citizen, Inc. and Public Citizen Foundation, Inc., form Public Citizen. Both entities are part of the same overall organization, and this Web site refers to the two organizations collectively as Public Citizen.

Although the work of the two components overlaps, some activities are done by one component and not the other. The primary distinction is with respect to lobbying activity. Public Citizen, Inc., an IRS § 501(c)(4) entity, lobbies Congress to advance Public Citizen’s mission of protecting public health and safety, advancing government transparency, and urging corporate accountability. Public Citizen Foundation, however, is an IRS § 501(c)(3) organization. Accordingly, its ability to engage in lobbying is limited by federal law, but it may receive donations that are tax-deductible by the contributor. Public Citizen Inc. does most of the lobbying activity discussed on the Public Citizen Web site. Public Citizen Foundation performs most of the litigation and education activities discussed on the Web site.

You may make a contribution to Public Citizen, Inc., Public Citizen Foundation, or both. Contributions to both organizations are used to support our public interest work. However, each Public Citizen component will use only the funds contributed directly to it to carry out the activities it conducts as part of Public Citizen’s mission. Only gifts to the Foundation are tax-deductible. Individuals who want to join Public Citizen should make a contribution to Public Citizen, Inc., which will not be tax deductible.

 

To become a member of Public Citizen, click here.
To become a member and make an additional tax-deductible donation to Public Citizen Foundation, click here.