Simulated Patient Studies: An Ethical Analysis (2024)

Abstract

Keywords: research ethics, simulated patient studies, audit methodology, deceptive design, exceptions to consent, access to care, public health, health services

Simulated customers—commonly called secret or mystery shoppers—have long been used for quality control purposes by retail establishments and other companies that serve the public. These mystery shoppers seek or use a company's own services and then report back to the quality control staff. Research using a similar technique was proposed in 2011 in anticipation of the large expansions in coverage by Medicaid and private insurance as a result of the Patient Protection and Affordable Care Act. The U.S. Department of Health and Human Services (DHHS) commissioned a study in which researchers posing as prospective patients would use a script to call doctors' offices and attempt to obtain appointments for primary or specialty medical care. The purpose was to better understand the extent to which new patients with Medicaid or private insurance could expect to gain access to medical care. Announcement of the proposed study in the Federal Register (Federal Register, April 28, 2011), followed by an article describing the study in the New York Times, triggered strong public criticism by some doctors and members of Congress (Pear 2011). One doctor remarked, “I don't like the idea of the government snooping. It's a pernicious practice—Big Brother tactics, which should be opposed.” Another stated, “If federal officials are worried about access to care, they could help us. They don't have to spy on us.” Echoing these sentiments, Senator Orrin Hatch (R-UT) declared that the Obama administration was “wasting taxpayer dollars to snoop into the care physicians are providing patients.” In response to the public criticism, the plans for the research were dropped.

Although this controversy certainly had a political element, the question we address in this article is how such methods should be viewed from the perspective of research ethics, particularly with regard to the use of deception with human subjects without soliciting their informed consent. Mystery shopper methods are not new in research, having been used to generate powerful evidence of negligent business practices that impact public health (Sorenson and Vittes 2003) and to document racial discrimination in fields such as real estate and mortgage lending (Fix and Struyk 1993; Ross et al. 2002; Turner et al. 2002; Turner and Ross 2005). Such methods have been used, as well, in health services research to measure access to and the timeliness of needed outpatient medical appointments (Asplin, Rhodes, and Levy 2005; Bisgaier and Rhodes 2011; Medicaid Access Study Group 1994; Rhodes et al. 2009), to study the consultation practices of pharmacists (Weiss et al. 2010), and to examine the prescribing practices of primary care physicians (Kravitz et al. 2005). But previous use does not answer the question of how such methods comport with ethical principles and the regulatory framework that applies to research involving human subjects.

Addressing this issue is now of particular concern, as we have relatively few reliable methods for measuring access to care. Gathering knowledge regarding access to care is relevant to current public policy, as both Medicaid and private insurance coverage will undergo major expansions under the Patient Protection and Affordable Care Act (2010, Pub. L. No. 111–148) (Rosenbaum 2011). While mystery shopper research using simulated patients adopts a naturalistic and scientifically sound study design, this mode of research raises ethical concerns relating primarily to the use of deception with human subjects without soliciting their informed consent. This article provides an ethical analysis of simulated patient techniques used for health services research, with a particular focus on research measuring access to care. To describe this ethical analysis, we present next a brief case study of one completed simulated patient study of access to health care.

A Case Study of a Simulated Patient Study of Access to Urgent Outpatient Medical Care

In 2002/2003, a group of emergency care health services researchers used simulated patients to measure access to outpatient primary care following a supposed emergency room visit for an urgent medical condition (Asplin, Rhodes, and Levy 2005). The aim was to evaluate the association between insurance status and access to needed outpatient medical care. Research assistants posing as patients telephoned randomly selected clinics and medical practice groups in nine U.S. cities. The sampling frame from which they drew their random sample was the lists of primary care providers who were on the hospital emergency departments’ “on call” rosters and were thus identified as having some obligation to provide follow-up care for emergency department patients who did not have a primary care physician. Using a standardized script, a trained research assistant presented herself to the receptionist answering the telephone as a patient who had been seen the previous night at the local hospital emergency room. She said she had been given this number and had been told to seek urgent follow-up medical attention. The researcher described herself according to one of three common clinical vignettes involving pneumonia, hypertension, or a possible ectopic pregnancy. The same researcher called the same practice twice with calls separated by several weeks; the only significant difference between the calls was the reported insurance status. Callers were randomly assigned to report either private insurance (Blue Cross/Blue Shield) or Medicaid or to explain that they were uninsured. If she was not given an appointment, the uninsured simulated patient asked how much it would cost to pay in full at the time of the visit. After receiving this information, she inquired if it would be possible to pay $20 initially and make arrangements for paying the balance over time. If the receptionist indicated a willingness to schedule an appointment, the simulated patient used a scripted response to cancel the appointment at the end of the call.

This simulated patient study was conducted without prior notice to the clinics and physicians’ practices and without informed consent. Two institutional review boards (IRBs) approved the research with a “waiver” of consent, as permitted in the U.S. federal regulations governing human subjects research, which we discuss later in this article. To avoid inconveniencing the study subjects and health care providers, the research was designed to minimize the time spent on the telephone calls. At the conclusion of the study, a debriefing letter was sent to all the clinics and office practices in the larger sampling frame, explaining that they may or may not have been contacted as part of a research study. The debriefing letter also provided the study results.

We briefly summarize the study results to indicate the potential social value of the knowledge that can be gained from this type of research—an essential element of ethical assessment. Simulated patients posing with private insurance were more likely to be offered appointments than those insured by Medicaid (64% vs. 34%) or than the uninsured who offered to pay a small amount initially and the balance over time (65% vs. 25%). There was no significant difference in the likelihood of receiving an appointment between those privately insured and those willing to pay in full at the visit (66% vs. 63%) (Asplin, Rhodes, and Levy 2005). The results revealed important primary care capacity issues, even among the fully insured patients, as well as disparities in access in accordance with insurance status and ability to pay. In sum, this study uncovered important knowledge about access to medical care following emergency room visits that was helpful to both the practice of emergency medicine and public policy.

Do Simulated Patient Studies Pose Legitimate Ethical Concerns?

From the perspective of traditional research ethics, it would seem obvious that simulated patient studies, which involve deceptive research interventions without consent, are ethically problematic. They contravene the basic norm that interventional research should proceed only on the basis of informed consent by research subjects or by surrogate decision makers for those subjects not capable of giving informed consent. However, research ethics is often discussed in a parochial context that raises ethical concerns that are disproportionate to the way in which people assess related activities outside the research setting (Wertheimer 2011). To evaluate the ethical significance of this departure from informed consent in simulated patient research, we compared this type of research with other, related activities outside the context of human subjects research. Our example is the use of mystery shopper techniques for quality improvement in service industries and the practice of restaurant reviews.

Mystery shoppers have been widely used in various industries, such as retail and hotel businesses, for evaluating the quality of customer service (Lazarus 2009). Marketing researchers disguised as customers visit such businesses to gather data to assess performance and to guide quality improvement. In the business world, employees are often informed in advance that they may be visited by mystery shoppers, although they are not told when the visit will take place. Typically, employees are given evaluative feedback based on the results of the mystery shoppers’ assessment.

Businesses and nonprofit institutions providing services to the public have a legitimate interest in—or even a responsibility for—evaluating the quality of services provided by employees, as do hospitals with regard to physicians with practice privileges (Baily et al. 2006). Employees expect to have their performance evaluated; having consented to employment, they arguably have no grounds for objecting to reasonable methods of employer-sponsored quality assessment. Mystery shopper techniques have obvious value in quality assessment by simulating naturalistic interactions between customers and service providers. Any concerns with the use of deception are obviated, or at least mitigated, by prior notice that mystery shopper techniques will be used for evaluating performance.

This legitimating context of employer authority and employee consent is not operative in simulated patient studies conducted by researchers who have no institutional connections with the research subjects and the health care practices evaluated by this type of research. While staff responsible for answering telephones in medical clinics and physicians’ offices have consented (as part of their job) to take calls from members of the public, it does not seem reasonable to construe their consent to employment as authorizing researchers not affiliated with their employer to gather data from them while posing as patients seeking access to medical care. In summary, simulated patient studies of access to medical care, such as the study just described, are ethically distinct from employer-sponsored quality improvement projects.

Restaurant evaluations might also seem comparable to simulated patient research techniques from an ethical perspective. Restaurant reviewers, writers of restaurant guides, and travel writers all routinely evaluate food quality and service based on their experience of dining at restaurants. Many restaurant reviewers commonly use disguises and typically make reservations using a phony name. For their part, restaurant owners expect to receive such reviews and benefit from them financially when they are positive. By consenting to serve meals to diners, restaurant owners and staff cannot have any legitimate objections to the practice of restaurant reviewing by some of the diners who visit the restaurant. Indeed, some restaurants provide opportunities for all diners to evaluate the quality of their meals and the service through satisfaction forms provided at the time of payment. Likewise, many hospitals regularly pay independent survey firms to collect anonymous patient satisfaction surveys (AMGA 2012; Press Ganey 2010). The key aspect of mystery shopper quality improvement projects that is lacking in simulated patient research is that there is a context of expectation and consent that legitimates the former but not the latter. Therefore, ethical comfort with using mystery shopper techniques in quality improvement projects and in the practice of restaurant reviews by unannounced reviewers is not sufficient to ground the ethical legitimacy of simulated patient studies.

Ethical Justification of Simulated Patient Research

Simulated patient studies of access to medical care pose two distinctive ethical problems: the use of deception and the absence of consent. Requirements for informed consent serve the well-being and protect the autonomy of research subjects. With respect to protecting well-being, making research participation conditional on informed consent protects people from unreasonable exposure to research risks. However, the simulated patient studies comparable to our case study pose, at most, minor risks to participants. More strongly in favor of consent to this type of research is a general principle of respecting autonomy, according to which people are entitled to a protected zone of control over their lives (and their property), such that others are prohibited from intervening or interacting within this zone without valid consent (Feinberg 1986). Thus, as a rule, medical interventions constitute a violation of bodily integrity and count legally as battery if they are performed without consent; likewise, interventional research with capacitated adult human subjects is (generally) impermissible without their informed consent. In other words, consent is morally transformative, making permissible those interventions that would be prohibited in the absence of consent (Kleinig 2010). A case can be made that in simulated patient studies without consent, there is no authorization by human subjects (e.g., receptionists answering telephone calls at clinics) of their research participation and this is therefore an infringement of their autonomy. The contrary argument would be that there is no infringement of the autonomy of the receptionists because they are conducting public, as opposed to private, business. They are not exercising discretion but are instead being paid to answer the phone and schedule appointments for whoever calls in accordance with organizational policy. But being a research subject lies outside the receptionists’ job description, thus raising the question of what legitimates researchers interacting with them for the purpose of research without their consent. Moreover, simulated patient studies also involve deception, which violates a moral rule against lying—for example, researchers posing as patients are lying about their medical situation and their need for a doctor's appointment. As such, simulated patient studies can be construed as violating the expectations of honesty that characterize normal human interactions in the medical workplace.

We argue that simulated patient studies require ethical justification in view of concerns regarding the absence of consent and the use of deception. Justification can be grounded in considerations of the social value of policy-relevant knowledge, scientific validity, and risk-benefit assessment.

Regulatory Considerations

There has long been controversy over whether low-risk social science research should be overseen by IRBs governed by federal regulations to protect human research subjects (De Vries, DeBruin, and Goodgame 2004; Patullo 1985; Pettit 1992). We do not take a stand on this complex issue, as our focus is primarily on ethical considerations pertaining to simulated patient studies without consent. As we have pointed out, this type of health services research deserves ethical attention, regardless of policy questions relating to the proper regulatory approach to social science research. To be sure, with IRB oversight, there is the potential for arbitrarily disapproving or restricting valuable research. Conversely, significant ethical issues relevant to justifying social research, especially when it involves deception and the absence of consent, may be ignored in the absence of IRB review or some comparable mechanism of independent oversight.

It is worth considering whether government-sponsored simulated patient research of access to medical care (or relating to other services) should also undergo ethical review. U.S. federal regulations governing human subjects research, which we examine in the following section, exempt from review by institutional review boards those “research and demonstration projects which are conducted by or subject to the approval of Department or Agency heads, and which are designed to study, evaluate, or otherwise examine: (i) Public benefit or service programs; (ii) procedures for obtaining benefits or services under those programs; (iii) possible changes in or alternatives to those programs or procedures; or (iv) possible changes in methods or levels of payment for benefits or services under those programs” (Code of Federal Regulations 45CFR46.101(b)(5), 2009). While IRB oversight would appear not to be required for the proposed DHHS-sponsored simulated patient study of access to primary care, we suggest that some form of independent review would be desirable to address the need to waive informed consent requirements to produce scientifically valid data and to ensure that risks to subjects are minimal and that adverse consequences to clinical and medical practices are prevented.

A threshold issue of regulatory jurisdiction is whether simulated patient studies even constitute human subjects research under the federal regulations, given that “observation of public behavior” does not meet the definition of research with human subjects (Code of Federal Regulations 45CFR46.102f, 2009). Indeed, a recently published report of a simulated patient study concerning the practices of pharmacies in providing access to emergency contraceptive medication stated that “the Boston University Medical Center Institutional Review Board deemed this study to be non-human subject research” (Wilkinson et al. 2012, 625). The current regulatory definition of “human subject” is the following: “Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains (1) Data through intervention or interaction with the individual, or (2) Identifiable private information. Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the subject or the subject's environment that are performed for research purposes. Interaction includes communication or interpersonal contact between investigator and subject” (Code of Federal Regulations 45CFR46.102f, 2009). Simulated patient studies of access to pharmaceutical products or medical care involve interaction with human subjects—the pharmacy, clinic, or medical office staff who respond to telephone calls from researchers posing as patients seeking information or appointments. Yet it might be argued that simulated patient research on access to medical care via telephone calls with receptionists at medical offices is not collecting data about the receptionists. Rather, it is collecting data about the appointment practices of these organizations. Conversely, this type of research is collecting data about the behavior of receptionists in scheduling appointments, thus arguably making them human subjects under the regulatory definition. We believe that there is room for debate as to whether an individual answering the phone on behalf of a business is the “research subject” when the goal of simulated patient studies is to study “business as usual” across a number of medical care organizations. Nonetheless, most IRBs that have reviewed studies employing the simulated patient method do classify it as collecting data on the behavior of human subjects. The rationale is that it constitutes a manipulation of the subjects’ environment and involves communicative interaction between the researchers and the staff who speak on behalf of the business. They approve such studies when they believe that the design meets the regulatory conditions necessary for waiving consent. From an ethical perspective, this is desirable; otherwise, research involving deception and without consent could be conducted without the IRB's review and approval of the study protocol. But in view of the “public versus private” nature of studying the health system and the divergent IRB practices in applying the definition of “human subject” to simulated patient studies and other forms of mystery shopper research, it would be desirable for the U.S. Office for Human Research Protections to provide additional guidance on this issue.

The current regulations are as follows: The U.S. federal regulations permit IRBs to approve research without informed consent in some circumstances. Four conditions must be satisfied: “(1) the research involves no more than minimal risk to the subjects; (2) the waiver or alteration will not adversely affect the rights and welfare of the subjects; (3) the research could not practicably be carried out without the waiver or alteration; and (4) whenever appropriate, the subjects will be provided with additional pertinent information after participation” (Code of Federal Regulations 45CFR46.116d, 2009).

The case study of simulated patient research on access to medical care qualifies for each of these requirements. The minimal intervention with subjects and the data confidentiality make the study no greater than minimal risk. It might seem impossible for simulated patient studies to satisfy the second condition, as they do infringe on rights not to be deceived and to be free of research interventions without consent. Strict interpretation of this regulatory provision, however, would rule out all studies employing deception or conducted without consent, which would defeat the purpose of permitting IRBs to approve studies that deviate from a requirement of informed consent. We suggest that the second condition be interpreted as holding that the waiver of consent does not unreasonably infringe on the subjects’ rights. The simulated patient study would easily qualify under this interpretation. Also, posing no (or, at most, minimal) risk to the subjects, it does not adversely affect their welfare. We argued earlier that scientifically valid data for this type of study, aimed at investigating the health care system, cannot be obtained with the standard practices of soliciting consent. Finally, our case study used a debriefing mechanism that satisfies the fourth condition. Arguably, the publication of results could also be construed as meeting this obligation.

Conclusion

Simulated patient studies conducted as health services research pose ethical concerns related to the deception and absence of consent that deserve careful attention. We contend that such studies are ethically justified and can be approved under U.S. federal regulations without consent when (1) they expose research participants to no more than minimal risks or burdens; (2) the waiver of consent is necessary to produce scientifically valid data; and (3) the potential social value of the knowledge to be gained from the research is substantial.

Acknowledgments

The opinions expressed are those of the authors and do not necessarily reflect the position or policy of the National Institutes of Health, the Public Health Service, or the U.S. Department of Health and Human Services.

References