Incentivizing Information Exchange Within Groups: The Role of Voting Protocols in U.S. Food and Drug Administration Advisory Committees

1. Introduction

Organizations involved in the production of innovation must evaluate their projects and decide whether to continue or terminate development. Because the success of an innovation project is often sensitive to many factors, decision makers—privy to only a limited amount of information about the problem—face large ex ante uncertainty on the project outcomes (Pich et al. 2002, Sommer and Loch 2004, Chao and Kavadias 2008, Sommer et al. 2020). Such project selection and resource allocation decisions are therefore often delegated to committees of experts with diverse areas of expertise (Van de Ven and Delbecq 1971, 1974).¹ The primary advantage of committees is that they enable a more comprehensive evaluation of the problem at hand by assembling a set of diverse sources of knowledge and perspectives (Hong and Page 2004, Oraiopoulos and Kavadias 2020).

Yet, simply assembling a committee of diverse experts is insufficient for effective decision making. Despite their potential, committees often struggle to surface and assimilate the knowledge dispersed throughout the group (Stasser and Titus 1985). The reason is that myriad frictions exist that hinder the exchange of information (see, e.g., Janis 1971, Cottrell 1972, Sah and Stiglitz 1988, Von Hippel 1998). These serve to conceal the diversity of knowledge and perspectives distributed among the individual members, undermining the advantages of group decision making.

How can committees be designed to facilitate information exchange? In this paper, we focus on voting protocols—specifically, sequential versus simultaneous voting—and how they influence the discussion, voting outcomes, and quality of a committee’s recommendation. The empirical basis of our investigation is U.S. Food and Drug Administration (FDA) advisory committee (AdCom) meetings that involve the evaluation of new drug or medical device applications. These committees are pivotal in ensuring the safety and efficacy of medical products before they reach the market, making them an ideal setting to study the impact of voting protocols on decision making. The FDA leverages such committees of external experts when evaluating novel drugs and medical devices prior to market approval. The AdComs meet and deliberate on the set of available data and evidence before forming a collective recommendation by voting on the perceived safety, efficacy, and benefit-to-risk profile of the product.

FDA AdComs provide an ideal context to study committee rules for project evaluation, for three reasons. First, the FDA assembles an AdCom to deliberate on only the most challenging evaluation decisions; hence, the problems that AdComs discuss are multifaceted and beyond the ability of any single individual to evaluate.² Second, because such decisions carry substantial societal consequences, AdComs’ decision-making processes are transparent, enabling us to investigate both discussions and outcomes. Third, the FDA in 2007 amended the rules concerning how AdComs vote on product approval recommendations. Whereas AdCom members had previously voted and explained their reasoning in sequential order (members revealed their votes and expressed justifications orally in a sequential manner, one after the other), after mid-2007, the AdComs adopted simultaneous voting (members cast their votes at the same time, and votes and justifications are revealed only after voting had closed). This exogenous change in the voting protocol—how committee members execute a round of voting—offers a real-world opportunity to examine the effects of voting protocols on information exchange in committees and on decisions involving the evaluation of new products.

Our methodological approach starts with collecting the verbatim transcripts and meeting minutes of 563 FDA AdCom meetings, held from 1999 through 2016, that covered specific drug or medical device evaluations and approval decisions. We manually read through these minutes and transcripts to identify and extract the specific questions that AdComs voted on, as well as the voting outcomes. We also use the full transcripts to investigate how the linguistic and behavioral characteristics of the discussions changed. We then treat the 2007 change in voting protocol as an exogeneous shock to the AdCom “rules of engagement” and to the incentives for committees to engage in information exchange.

We first reaffirm prior evidence that documents a reduction in the likelihood of unanimous votes under simultaneous voting (Urfalino and Costa 2015, Newham and Midjord 2019). Whereas around 45% of all questions that were voted on sequentially ended in unanimity prior to the FDA guidance, this proportion dropped to 26% after that voting guidance and the change to simultaneous voting.

We then establish three new results. First, we show new evidence that the nature of discussions in which committee members engaged changed in several ways after the introduction of simultaneous voting. We find that discussions broadened in scope: AdCom meetings covered more disparate topics, thus increasing the diversity of information that surfaced during discussions. We attribute this difference to the following dynamic: simultaneous voting, by depriving later voters of the ability to observe earlier voters’ votes and justifications, better incentivizes committee members to elicit each others’ understandings of the problem during prevote discussions. Our theory is further corroborated by evidence on behavioral and linguistic changes among the committee members. After the switch to simultaneous voting, members directed a greater proportion of questions to each other, showed greater equality in amount of speaking time, and used language that was more positive in tone, more authentic, and exhibited greater equality in social status. These all reflect behaviors that are known to encourage other members of a group to share their views (Edmondson 2003, Siemsen et al. 2009, Woolley et al. 2010, Keck and Tang 2018).

Second, we show that the diversity of information surfaced during discussions explains the lowered likelihood of a unanimous vote. Thus, unlike prior explanations that attribute the decreased likelihood of unanimous votes under simultaneous voting to “vote herding” and do not consider the role of discussions (U.S. Food and Drug Administration 2008), our analysis shows that improvements in information exchange were a fundamental driver of the reduction in unanimous voting outcomes observed after the change in voting protocol. In fact, the diversity of information exchanged during discussions predicts the likelihood of unanimous votes more strongly than voting protocols: after controlling for the diversity of discussion content, the direct effect of voting protocol on the likelihood of unanimous voting outcomes is more than halved and becomes statistically indistinguishable from zero. Thus, we uncover novel evidence showing that the large reduction in unanimous votes after 2007 can be attributed to discussions that broached broader topics and involved the exchange of more diverse, individually held information.

Third, we provide evidence that the change in voting protocol is associated with better decision quality. By supplementing our study with data on the outcomes of all drugs in our sample, we capture the prevalence of type I “errors of commission” before and after the voting change.³ We define an error as an event in which a drug was recommended for approval by an AdCom but was subsequently either (1) withdrawn from the market for reasons of safety or efficacy or (2) issued a postmarket boxed warning.⁴ Both are errors in that the drug should not have been recommended for approval in the first place (in the case of a market withdrawal), or should not have been recommended without severe usage restrictions (in the case of a postmarket boxed warning). We find that, compared with under sequential voting, a smaller proportion of drugs recommended under simultaneous voting were either withdrawn or issued boxed warnings, even though the overall proportion of drugs recommended for approval by AdComs (and eventually approved by the FDA) slightly increased after the 2007 voting rule change.

Our work contributes to the discussion on the rules of engagement for group decision making and new product evaluation. Whereas the innovation management literature has focused on such rules in the production of innovations (see, e.g., Van de Ven and Delbecq 1971, Criscuolo et al. 2017), we see two distinct areas of contribution and address the following pair of related questions. First, do committee voting protocols affect decision quality? This is a first-order question if one is to offer any prescriptions based on the superiority of one protocol over another. Although prior work has extensively examined decision making by committee from analytical and normative angles (see, e.g., Sah and Stiglitz 1988, Hao and Suen 2009, Csaszar and Eggers 2013, Oraiopoulos and Kavadias 2020), there is scant real-world empirical evidence that links protocols to decision outcomes, especially when dealing with the evaluation of complex new products. Our work provides evidence for that link.

Second, how do voting protocols improve decision quality? The arguments and evidence presented in the literature have tended to focus on the possibility, under sequential voting, of vote herding (Urfalino and Costa 2015, Newham and Midjord 2019), whereby later voters observe the votes of earlier voters and base their own votes on the decisions of others. Thus, simultaneous voting can improve decision-making processes by rendering votes more independent (Banerjee 1992, Callander 2007, U.S. Food and Drug Administration 2008). Our results show that this vote herding argument ignores the importance of the discussions that transpire during committee meetings. Voting protocols have a significant effect on the diversity of information revealed during the discussions, and it is this diversity of information that more strongly predicts (the lack of) unanimity in voting outcomes. As a result, the benefits of simultaneous voting go beyond promoting independence in judgment: the most meaningful benefit may be in how simultaneous voting incentivizes committee members to engage in the robust discussions needed to uncover information relevant to evaluating the focal problem.

Because many government agencies, private firms, and other organizations rely on expert committees to evaluate and deliver recommendations on complex problems, our work has broader and actionable implications for how organizations should structure their committee meetings. In the ensuing pages, we first hypothesize in Section 2 as to why and how voting protocols affect committee meeting discussions and voting outcomes. We then introduce the empirical context in Section 3. In Section 4, we describe our data and present our main findings. We explore the data in greater depth and provide supporting evidence as to how the changes in discussions may have come about and whether these changes resulted in better-quality recommendations in Section 5. Section 6 demonstrates the robustness of our results. Finally, Section 7 discusses contexts to which insights are likely to generalize and clarifies limitations to our theory and empirical setting.

2. Theory Development

2.1. The Difficulty with New Product Evaluations

The development of new products is often characterized as a stage-gate process involving a series of go/no-go decision points (Cooper 1990, Ding and Eliashberg 2002). At these gates, decision makers evaluate incomplete information about a new product’s potential to determine whether development should continue or be terminated. Such decisions are challenging because they usually must be made before the often considerable uncertainties surrounding a new product’s viability can be resolved. These uncertainties may stem from unproven new technologies (Krishnan and Bhattacharya 2002, Bhaskaran and Ramachandran 2011), unknown market interest (Terwiesch and Loch 2004), or a complex combination of these and other factors (Pich et al. 2002, Sommer and Loch 2004).

Given these challenges, new product evaluations and decisions are therefore often delegated to groups of decision makers rather than individuals (Csaszar and Eggers 2013, Criscuolo et al. 2017, Oraiopoulos and Kavadias 2020). The rationale for leveraging committees in these and many other decision-making contexts reflects the recognition that the scope of knowledge relevant to solving any complex problem is so wide that no single person could possibly cover its entirety (Jones 2009, Staats et al. 2012). Thus, a committee’s assembling of individuals with diverse perspectives and areas of expertise helps to fill in knowledge gaps and also helps individuals reduce their own uncertainties (Hong and Page 2001).

Because such evaluations tend to be delegated to committees, scholars in innovation management and economics have examined many questions on optimal “group design.” Broadly, research on this topic can be classified into three interrelated streams based on research focus: (i) group composition, or how the type and amount of group diversity affects decision making (see Kavadias and Sommer 2009, Keck and Tang 2018, Oraiopoulos and Kavadias 2020, Chan et al. 2023); (ii) organizational decision-making structure, or how decision-making power is distributed across group members (e.g., delegating the decision to an individual, requiring a majority or consensus vote, or other decision-making mechanisms; see Sah and Stiglitz 1988, Csaszar and Eggers 2013, Chao et al. 2014, Kurtuluş et al. 2022, Böttcher and Klingebiel 2025); and (iii) information exchange, or what kind and amount of information should be exchanged among members. Work in this last stream has explored whether members should work independently or together to exchange ideas on how to solve a problem (see Van de Ven and Delbecq 1971, Hegedus and Rasmussen 1986, Girotra et al. 2010, Crama et al. 2019, Cornelius and Gokpinar 2020, Sommer et al. 2020, Becker et al. 2022) and the merits of sharing rich information about the strengths and weaknesses of the project (see Van de Ven and Delbecq 1971) or only binary approval/disapproval decisions (see Sah and Stiglitz 1988, Banerjee 1992, Callander 2007, Criscuolo et al. 2021).

Our study contributes to the third research stream, as we investigate the impact of voting protocols (in particular, the mechanism by which a committee of individuals casts votes) on project evaluation and selection decisions. As we argue below, the design of effective voting protocols is crucial in any innovation context, because voting protocols can affect information exchange among individuals, and how they evaluate and select projects.

2.2. Voting Protocols, Outcomes, and the Vote Herding Argument

Across many contexts, group decisions are often reached through voting. There are numerous means by which a committee may execute a round of voting, but two overarching protocols exist: sequential and simultaneous. Under sequential voting, committee members cast their individual votes one by one, in succession; thus, earlier votes are visible to later voters. Under simultaneous voting, everyone casts their votes at the same time. This protocol eliminates the possibility of members observing how others vote before doing so themselves.

Because a vote reflects the information possessed by an individual, the economics literature has long highlighted vote herding as a key concern with sequential decision making (Banerjee 1992, Bikhchandani et al. 1992): given their own limited information, individuals tend to seek the information embedded in others’ decisions and actions before making their own decisions. As Banerjee (1992, p. 798) notes, the very act of “trying to use the information contained in the decisions made by others makes each person’s decision less responsive to her own information and hence less informative to others.” So as individuals obtain information by observing the preceding votes of others before having to cast their own vote, they may discount their own information and simply “follow the leader.” This notion is treated explicitly in DeGroot’s (1974) model of information exchange. In that model, a voter places fixed positive weights on the votes revealed by others, which in turn reduces the weight placed on the voter’s own evaluation of the situation. Game-theoretic models of vote herding do not presuppose fixed weights placed on the revealed votes, but they similarly establish conditions under which vote herding could occur as a consequence of individuals’ rational calculations (see Banerjee 1992, Bikhchandani et al. 1992, Callander 2007).

The key concern inherited from prior literature, then, is that sequential voting results in the “broad tendency to grow the initial majority” (Becker et al. 2022, p. 3960). Stated differently, under sequential voting, earlier voters have an outsize impact on vote outcomes. This is because, given enough signals in one direction, people who come later in the order may infer that previous voters have some superior information. This raises the prospect of individuals voting in the same direction. Simultaneous voting obviates that possibility, precisely because there is no voting order and thus no earlier or later voters. Because votes must be cast at the same time, it precludes the ability of any individual to see others’ votes and attempt to infer information prior to casting their own votes.

The FDA specifically emphasized the problem of vote herding under sequential voting when it issued the guidance on adopting a simultaneous voting protocol (U.S. Food and Drug Administration 2008), citing the research of Banerjee (1992) and Callander (2007) as a major reason for the move. Empirical evidence appears to suggest that simultaneous voting reduces the likelihood of members voting in the same direction, which scholars have argued is evidence of vote herding (see Urfalino and Costa 2015, Newham and Midjord 2019). We establish the baseline Hypothesis 0 about how voting protocol affects the likelihood of vote unanimity that is in line with the prior empirical results.

The above hypothesis does not consider the existence of information exchange prior to voting, as information is merely inferred from others’ votes. Yet, in many project evaluation settings (and other committee contexts), votes are cast and selection decisions are made only after a prolonged period of discussion during which committee members interact and jointly evaluate the project at hand (Criscuolo et al. 2017, Oraiopoulos and Kavadias 2020). For example, FDA AdCom meetings typically last an entire day, and the bulk of the meeting is devoted to discussing the various dimensions of the product and evaluating data from clinical trials. It is not clear, then, whether vote herding is the only (or even the main) mechanism driving Hypothesis 0. In what follows, we will argue that the reduction in the likelihood of vote unanimity under simultaneous voting is not so much evidence of vote herding, but rather is due to how voting protocols shape the preceding discussions.

2.3. Prevote Discussions and the Challenge of Revealing Diverse Information

Discussions provide an explicit phase for obtaining and exchanging information that is distinct from voting. They offer an opportunity for committee members to actively share their own knowledge, information, and evaluations of the new product, and they further allow members to glean nuanced information about the opinions of other experts and thereby fill gaps in their individual knowledge (Urfalino and Costa 2015). For these reasons, innovation scholars have stressed that groups operate most effectively when individuals are able to openly and freely express their ideas, knowledge, and evaluations (Van de Ven and Delbecq 1971, Sutton and Hargadon 1996), allowing the group to benefit from the diversity of information distributed across its members. A robust discussion allows for the committee to pool information, more comprehensively attending to the problem at hand so as to arrive at a better decision.

However, groups often fail in effectively pooling the unique, unshared knowledge distributed across their members (Stasser and Stewart 1992, Gigone and Hastie 1993). This failure stems from a variety of “costs”—both informational and social—that hinder the free exchange of knowledge in group settings (Robertson 1980, Sunstein and Hastie 2015). These costs serve to constrain the reduction of uncertainty, resulting in discussions that are biased toward common, shared information (Kim 1997, Wittenbaum et al. 1999) and away from the unique, unshared information that the committee is assembled to expose.

Among these, communication costs are particularly significant (Arrow 1969). The process of exchanging private information often requires substantial effort, as private knowledge is typically embedded within its context. To make this information accessible and meaningful to others, the individual who holds it must carefully explain and elaborate on that context (Dougherty 1992). At the same time, other group members must expend time and attention engaging in debates and discussions, clarifying opinions, and asking questions to attempt to reveal and integrate the focal member’s private information (Von Hippel 1998, Casciaro et al. 2019).

For example, individuals may refrain from expressing novel ideas because they are averse to being evaluated negatively by peers or an audience (Cottrell 1972, Amabile et al. 1990, Kavadias and Sommer 2009), especially in situations where individuals view other group members as experts (Collaros and Anderson 1969) or where reputations and careers are at stake (Zwiebel 1995). Similarly, the desire for group cohesiveness may prevent an individual from sharing privately held information (Janis 1971). In comparison, communicating commonly held knowledge leads to mutual enhancement (Wittenbaum et al. 1999), as discussing shared information helps validate the knowledge and competence of the group, leading to positive evaluations of one another.

In sum, exchanging privately held information within a group is costly both to the individual who shares and to the rest of the group that must elicit and process this information. Therefore, groups of individuals exhibit a tendency to discuss information that is already common to the group instead of uncovering and exchanging the diverse information that is privately held by their constituents (Stasser and Titus 1985). Hence, the exchange of diverse knowledge and expertise requires more than simply assembling a group of individuals together in the same room for a joint discussion period.

2.4. Voting Protocols and Incentives to Exchange Information

Fortunately, individuals can adopt behaviors to improve knowledge sharing during group discussions (Edmondson 1999). Behaviors such as regulating one’s speaking frequency (Duhigg 2016), minimizing concerns about power and status differences among a group’s members (Edmondson 2003), adopting positive tones in speech (Keck and Tang 2018), and directing questions to each other (Casciaro et al. 2019) can all encourage others to share knowledge and unique expertise about less common aspects of the problem. Empirical evidence across a range of work contexts has shown that such behaviors improve not only knowledge sharing (Siemsen et al. 2009) but also the odds of discussions yielding new insights (e.g., through the recombination of shared knowledge; Choo et al. 2007) and of overall project success (Lee et al. 2011, Chandrasekaran and Mishra 2012).

Importantly, although groups can actively seek out ways to elicit the diversity of information hidden among the other members, doing so requires effort. There must be incentives to overcome the costs—with respect both to the processing of complex information (Kool et al. 2010) and to the related social risks (Visser and Swank 2007, Siemsen et al. 2007)—required for eliciting and sharing information across the group.

Voting protocols constitute one such mechanism that meaningfully alters individuals’ incentives to expend effort on eliciting information from others (Persico 2004). We argue that, in comparison with sequential voting, a simultaneous voting protocol more strongly motivates a committee to engage in productive knowledge exchange during the preceding discussions, thereby facilitating the revelation of the diversity of group members’ knowledge. To the extent that individuals are all aware of the meeting process at the beginning of the meeting, and can anticipate the voting protocol adopted at the end of the meeting, they can adjust their behavior and actions accordingly during discussions.

It is the anticipation of extra information during a sequential vote that disincentivizes information exchange during discussions. Under sequential voting, members can expect to be able to see at least some votes before having to cast their own. Earlier votes communicate information about how others think about the focal decision (see, e.g., Banerjee 1992, Bikhchandani et al. 1992). Therefore any member has a chance to receive, “for free,” information about others’ evaluations while the vote is ongoing. That opportunity reduces the need for active engagement during the discussion phase (Mukhopadhaya 2003), forgoing the costs of information exchange previously noted. Thus, a sequential voting protocol disincentivizes the expenditure of committee members’ efforts (and the incurrence of communication costs and social risks) to extract each other’s individually held information in the preceding discussions.

In contrast, simultaneous voting removes each individual’s ability to observe others’ votes before casting their own votes, thereby eliminating the possibility of obtaining informative signals during the voting phase. In this way, simultaneous voting incentivizes committee members to engage in effortful behaviors (such as those described in previous paragraphs) to elicit views from other members during the discussion phase. Under this protocol, remaining passive during discussions is penalized more highly because there is a nonnegligible risk of voting with neither a good appreciation of the problem (through the discussion) nor a good understanding of how the others would vote (through the vote). We therefore expect the discussions that precede simultaneous voting to be more effective at uncovering diverse, individually held knowledge relevant to assessing the problem at hand. These considerations lead to our first main hypothesis.

We further argue that the greater information diversity revealed during the discussion phase of meetings has a meaningful impact on the likelihood of vote unanimity. Specifically, a greater diversity of information revealed during discussions heightens all members’ awareness of the full extent of the facets and complexity of the evaluation problem at hand. In other words, all members establish a better appreciation that the same problem can be scrutinized from many different angles (Oraiopoulos and Kavadias 2020). Such appreciation should reduce the likelihood of all group members voting in the same direction. The opposite case is illustrative: when information diversity is low (i.e., the discussion focuses only on specific aspects of the problem), convergence is easier because the original problem has been (artificially) reduced to a simpler one with a straightforward solution (Simon 1962, Janis 1971).

Another benefit of discussions that reveal greater information diversity is that they bring to the fore the wide diversity of experience and expertise across the committee. Observing this diversity reduces the fear of being caught out voting differently from others (Hong et al. 2000). Again, it is useful to consider the opposite case: when information diversity is low, a discussion focused on narrow aspects of the problem can lull members into thinking that those narrow aspects are pivotal to the decision and that most others in the group share this view (Janis 1971).

To summarize, whereas prior research on voting protocols tends to focus on vote herding, we argue that such an argument is oversimplified when voting is executed after a lengthy discussion phase during which members discuss different aspects of the problem and their evaluations of the new product. Our arguments therefore examine a longer causal path and theorize on how voting protocols affect the diversity of information revealed during discussions (Hypothesis 1a) and on how revealed information diversity subsequently affects vote unanimity (Hypothesis 1b). We believe that our model—as encapsulated by Hypothesis 1a and Hypothesis 1b—provides a more complete explanation than the standard vote herding argument, when members are able to discuss the problem before voting. In particular, the model posited by Hypotheses 1a and 1b should partially or fully explain (i.e., mediate) the relationship observed in Hypothesis 0—we formulate this mediation effect explicitly as Hypothesis 2. Figure 1 illustrates our full conceptual model.

3. FDA Advisory Committees

In this section, we provide an overview of new product evaluations at the FDA and the agency’s use of advisory committees. We also refer the reader to Online Appendix A, where we detail pertinent information about the conduct of these meetings and the implications of the 2007 change in voting protocols.

3.1. New Product Evaluations at the FDA

Following clinical trials, pharmaceutical companies and medical device manufacturers must submit an application to the FDA before they are able to market their products to U.S. consumers and patients. The FDA must then evaluate the application. This evaluation task is difficult: clinical trials are often rendered incomplete because of the substantial financial costs associated with conducting them (DiMasi et al. 2016), imperfect or unavailable measures of clinically meaningful outcomes (Fleming 2005, Markou et al. 2023), and inconsistencies between trial populations and the general population (Corrigan-Curay et al. 2018). Predicting a product’s safety and efficacy on the broader public thus remains an “inexact science” (Lewis 1993, p. 1038), as evidenced by how frequently unanticipated side effects and harmful interactions are discovered in late-stage clinical trials (Girotra et al. 2007, Cook et al. 2014) or even after the product is launched in the market (Lasser et al. 2002).⁵

In many cases, the FDA reaches a decision on an application without convening an AdCom. However, in some situations, reaching a decision can be exceedingly difficult. This could be due to the complexity of the new drug or device, complicated clinical trial designs, and/or conflicting, indirect, or inconclusive results from clinical trials, counterbalanced by a lack of alternative solutions or an urgency to decide before sufficient evidence can be gathered (e.g., the emergency use authorization of the Pfizer-BioNTech COVID-19 vaccine). In such situations, the FDA can convene an AdCom to provide recommendations (Smith et al. 2012).

The FDA’s AdComs therefore embody a context in which the evaluation of complex new products requires a committee of diverse experts to facilitate decision making. Accordingly, not only do AdCom members scrutinize the available quantitative evidence (e.g., statistical tables, clinical trial outcomes), but they also assess the many different possible consequences of a new product’s approval. For example, AdComs often discuss the suitability of the target population, trial design, adequacy of safety and well-being measures, acceptability of surrogate endpoints, logistical or behavioral issues that may lead to lower levels of efficacy observed outside of trials, and/or possible spillover effects to other people associated with the individual.⁶

An AdCom (e.g., the Dental Products Committee, or the Antiviral Drugs Committee) comprises individuals with a diverse set of expertise and professional experience in areas such as clinical medicine, engineering, biostatistics, patient advocacy, and others. Those nominated to be part of an AdCom may be academic researchers, practicing doctors, directors of centers, patient representatives, or others. The vast majority hold an MD and/or PhD degree, although they may hold other (or no) higher-education degrees. Many committee members are standing members who serve over multiple years, but individuals with specific experience or expertise are frequently tapped to serve as temporary members.

The AdCom meetings that address new product approvals follow a prescribed meeting structure. Prior to the meeting, the FDA sets a specific date on which the meeting will take place, and most meetings last an entire day. In preparation for a meeting, both the FDA and the product sponsor create briefing materials that are made available to all AdCom members in the form of a briefing book or panel pack (McIntyre et al. 2013). These materials include background information, statistical data, and evidence that has been collected through the course of clinical trials; also included are labeling information and the proposed postmarketing monitoring plans. In addition, the FDA prepares a set of questions—typically concerning the efficacy, safety, and benefit-to-risk profile of the product being reviewed—that the AdCom is expected to discuss and hold a formal vote on.⁷

Each meeting convenes with some preliminaries, followed by sponsor and FDA presentations, and then one or two question-and-answer (Q&A) periods with the AdCom members. After the Q&A period, committee members discuss, ask questions, and deliberate extensively among themselves before voting on each question posed by the FDA. After all voting questions have been discussed and voted on, the meeting concludes. After the meeting, the FDA takes into consideration all the discussions and votes when it ultimately decides whether to approve the product.⁸

3.2. Changes to Committee Voting Protocols

In August 2007, the FDA published guidance stipulating that its AdComs switch from sequential to simultaneous voting (U.S. Food and Drug Administration 2008, Urfalino and Costa 2015). Note that the change occurred in 2007, even though the draft guidance (cited above) was not finalized until one year later in 2008: by checking the meeting transcripts, we confirm that the change to simultaneous voting occurred on August 1, 2007. Prior to this guidance, votes were conducted in a protocol under which each member revealed their vote (and any explanation for that vote) as follows: At the start of the voting period, the chairperson would choose one committee member to announce their name, orally cast their vote, and explain their decision. A second member would then announce their vote and comment, followed by a third, and so on. This sequential format allowed individuals later in the order to observe the preceding individuals’ votes as well as their reasoning.

The 2007 guidance stipulated that all votes henceforth be conducted in a simultaneous format: all members cast their votes at the same time using electronic buttons, then voting is closed and the votes are tallied, and only after the tally is announced do members reveal their individual votes and provide justifications. The FDA’s stated aim in switching to simultaneous voting was to eliminate the possibility of later voters being influenced by earlier voters (U.S. Food and Drug Administration 2008; see also Banerjee 1992, Callander 2007). Not only does this procedure eliminate the possibility of anyone observing how others vote before casting their own vote,⁹ but it also retains the FDA’s commitment to transparency by ensuring that each vote is documented in the official record.

Importantly, the highly standardized and transparent AdCom meeting process (which we detail in Online Appendix A) ensures that voting participants are aware of the overall voting protocol (i.e., whether they would be voting simultaneously or sequentially) even before the meeting has started. As mentioned above, such awareness of the meeting rules of engagement enables members to adjust their behaviors during the discussion phase, which occurs prior to voting.

4. Analysis of FDA Advisory Committee Meetings

4.1. Data Sample

We first identified all meetings held by FDA advisory committees from 1999 through 2016, focusing on meetings that were held to discuss and decide on the approval of specific drug or medical device applications.¹⁰ This period covers about nine years before and after the change, which gives us a time window of suitable duration for investigating the FDA change’s effects. The starting and ending years for our sample were chosen for reasons both analytical (ensuring an 18-year time span bifurcated by the switch from sequential to simultaneous voting) and practical (despite the FDA’s folder name, file name, and web location changing over time, for our chosen study period, we have strong confidence that our script can reliably obtain the transcript and meeting data we seek).

Although all the data we need are publicly available, curating the data for analysis was challenging. Beyond file formats and naming conventions that change over time, there is no public database that catalogs the purpose of each meeting: some meetings involved no new product approvals, yet others involved approvals of multiple new products in a single day. To ensure our sample’s integrity, we manually examined the meeting minutes and verbatim transcripts from the FDA’s public online archives. We then used these documents to identify whether each meeting involved an AdCom voting on a specific product or on multiple products (splitting the transcripts into two half days where appropriate). Our final sample consists of 563 meetings whose focus was to evaluate and vote on the approval of (i) a new drug or medical device or (ii) a supplement to an existing drug or device.¹¹

From this sample, we examine the full transcripts to obtain information about all committee members and other speakers who participated in the meeting. Each transcript begins with a roster of participants, from which we hand-collected information on every individual’s full name, degrees (e.g., MD, PhD), and role (designated voting member, FDA participant, etc.). From the meeting minutes and full transcripts, we also extracted the specific question voted on as well as the final totals of “yes,” “no,” and “abstain” votes cast by the voting members on each question. Our sample comprises 1,091 voting questions discussed over the 563 meetings.

4.2. Preliminary Evidence

Prior to formally testing our hypotheses, we explore whether the raw data reveal any effects of a voting protocol change on discussion and outcomes. Figure 2 presents model-free evidence of how the change in voting protocol affects how committees vote on each individual voting question; the graph contrasts the distribution of majority vote share before and after the FDA guidance.¹² A clear postguidance effect is visible: whereas nearly 45% of all questions ended with a unanimous vote prior to the guidance (majority vote share = 1; i.e., the green bar near the horizontal axis value of one), the proportion of unanimous votes dropped to about 26% after the guidance (the lavender bar near the horizontal axis value of one). Overall, the proportion of questions that ended in unanimity is greater in the preguidance period than in the postguidance period. Thus, the raw data strongly suggest that the shift from sequential to simultaneous voting reduced the number of unanimous voting decisions, a finding that is consistent with past findings (Hypothesis 0).

We next analyze the meeting discussion and product evaluation process by exploiting textual information in the AdCom meeting transcripts. Our core argument centers on the diversity of information that is revealed during a committee’s meeting. To measure the extent of that diversity, we leverage natural language processing. We first combine all verbatim transcripts, then feed them into Word2Vec (a standard word embedding technique; see Mikolov et al. 2013) to transform words into an embedding. The output is a representation of words as points in a high-dimensional vector space (100 in our case), with words more similar in meaning placed closer together in this space. Each transcript is then represented by a collection of words scattered over the same real vector semantic space. We measure the diversity of information revealed during a discussion as the “volume” of the resulting word cloud, where a larger volume corresponds to discussions in which more information diversity is revealed. The volume of the word cloud is calculated by the (log of) generalized variance (InfoDiversity). Intuitively, generalized variance, or the determinant of the variance-covariance matrix, generalizes the one-dimensional notion of variance, which captures spread over a set of words in one dimension (Wilks 1932). Thus, if the discussion ranges across different issues, our measure would capture that through a larger spread of meanings over the words used. We share technical details in Online Appendix B. There, we also visualize two-dimensional representations of word clouds representing the distribution of words uttered in meetings, and show further analysis that establishes the validity and robustness of the InfoDiversity measure.

Figure 3 illustrates how meeting information diversity progressed over time. The data suggest a discontinuity in the information diversity revealed in meetings between the 1999–2006 period and the 2008–2016 period, coinciding with introduction of the FDA’s 2007 guidance. This provides initial support in favor of Hypothesis 1a: the advent of simultaneous voting seems to have induced a marked increase in the information diversity revealed during AdCom meeting discussions.

4.3. Formal Models and Estimation Results

4.3.1. Identification Approach.

We leverage the exogenous change in voting protocol to deliver identification of its effects on information diversity and voting outcomes revealed during the discussions. In August 2007, the FDA published guidance recommending that AdComs switch from sequential to simultaneous voting. This change affected all AdComs at the same time, so we focus on the contrast in outcomes before and after the switch. A key independent variable is therefore ${\mathit{\text{Post}}}_t$, which takes a value of one if the day t of an AdCom meeting is on or after August 1, 2007 (and otherwise takes a value of zero).

We study two dependent variables. The first one is binary, and it indicates whether a vote by committee i on question q ends up being unanimous (${\mathit{\text{Unanimous}}}_{\mathit{\text{itq}}}$).¹³ Our second dependent variable is continuous; it represents the level of information diversity elicited by committee i’s discussion on day t (${\mathit{\text{InfoDiversity}}}_{it}$).

We estimate the following models:

The terms ${\beta }_0$ in Equation (0), ${\beta }_{1a}$ in Equation (1a), and ${\beta }_{1b}$ in Equation (1b) are our coefficients of interest: ${\beta }_0$ and ${\beta }_{1a}$ capture the extent (if any) of the voting protocol change’s effect on vote unanimity (Hypothesis 0) and information diversity (Hypothesis 1a), respectively, and ${\beta }_{1b}$ captures the effect of changes in discussion information diversity on vote unanimity (Hypothesis 1b). The term ${\mathbf{X}}_{it}$ is a meeting-level vector of control variables whose values depend both on the committee i and time t; ${\mathbf{W}}_{\mathit{\text{itq}}}$ is a question-level vector of control variables that varies by question; and $c_i$ denotes committee-level fixed effects. We estimate conditional logit models to study binary dependent variables (e.g., Unanimous) and linear fixed-effects models for continuous variables (e.g., InfoDiversity).

The empirical identification approach that we adopt is the one-group pretest–posttest design (Bonate 2000). It is called “one-group” because every unit in the sample experienced the change at exactly the same time; hence, the identification strategy is distinguished by the absence of a control group that did not experience the change. In terms of ability to deliver causal identification with few assumptions, this design lies between empirical identification where we observe variations in voting protocol but such variations are not exogenous (i.e., where committees themselves choose how to vote), and a difference-in-differences approach with variation in exogenous assignment (where a group of committees not assigned to change their voting protocol could serve as a control group).

Finally, we posit that information diversity might mediate the relationship between voting protocol change and the likelihood of vote unanimity (Hypothesis 2). We test Hypothesis 2 by estimating ${\beta }_{1a}\times {\beta }_{1b}$ from Equations (1a) and (1b) (i.e., a product of coefficients approach; MacKinnon et al. 2007). Our testing of mediation effects is complicated slightly by (1) the presence of a nonlinear dependent variable (i.e., Unanimous is binary) and (2) multilevel data (i.e., we have data at the committee level, each committee may have multiple meetings, and at each meeting there may be multiple votes taken). We discuss technical details of estimating such mediation models in Online Appendix C.

4.3.2. Controls.

We introduce committee-level fixed effects $c_i$ in all models, as each product being evaluated falls under the purview of a specific committee (e.g., Oncologic Drugs Advisory Committee, Circulatory System Devices Panel, etc.). This allows us to control for many inherent, unobservable features of the committee or product being evaluated (e.g., features of the therapeutic area) that are arguably time invariant. However, this approach is susceptible to estimation bias if there are unobserved but time-varying features that correlate with either of our outcome variables and Post. The following set of controls aims to account for such features. (See Online Appendix D for details on variable construction and summary statistics.)

Meeting/Product Controls.

We control for a host of variables related to the meeting and the product being evaluated. Because new products may be more difficult to evaluate than new extensions to existing products, we control for whether a meeting concerns a new application or a supplement to an existing product (a binary variable Supplement; see also Endnote 11), a count of the number of special designations (e.g., for Accelerated Approval) that the application has received from the FDA (Status), and whether the product under evaluation involves a biologic (i.e., large-molecule) or small-molecule drug (a binary variable BLA that is one if the product is filed with a Biologics License Application). We assess heterogeneity in meeting length by distinguishing among meetings that are scheduled for a half day, a full day, or two days (MeetDuration) and by measuring the total transcript length in words (Words). In addition, we control for the number of voting questions posed by the FDA (VotingQs).

Composition Controls.

We include several committee-level variables to control for temporal changes in committee composition. We control for the proportion of members who have standing versus temporary assignments (PctTemp), are patient representatives (PctPatientRep), are consumer representatives (PctConsumerRep), and are female (PctFemale). We further control for background experience by controlling for the proportion of members with MD (PctMD) and PhD (PctPhD) degrees. We account for diversity in professional expertise (ExpertiseDiversity) with the log count of the number of unique professions of the members. We further control for committee familiarity by measuring the log sum of the total number of prior meetings that each pair of individuals has attended together (Familiarity). Finally, the FDA has gradually limited the amount of conflict of interest waivers that can be granted (see Wood and Perosino 2008), and so we control for the total dollar value of all potential conflicts of interest across the committee (Conflicts).

Question Controls.

Analyzing outcomes at the vote level requires that we also control for the topic of the vote—as a categorical variable capturing whether a voting question concerned the product’s approvability, efficacy, safety, or other (the variables $\mathit{\text{Motion}}\_A,E,S,O$). We control also for the wording of the voting question by using the indicator variable MotionInFavor, whose value is one if a question is worded “positively” (e.g., “Has the safety of the product been adequately demonstrated?”) or zero if it is worded “negatively” (e.g., “Does this application raise concerns about safety?”).

4.3.3. Main Results.

Table 1 reports results of our formal analyses of how the voting protocol affects vote unanimity and discussion information diversity. Consider first Models (1) and (2), which show how the switch from sequential to simultaneous voting affected the likelihood of vote unanimity. Model (1) does not account for the effects of any time-varying control variables, whereas Model (2) includes all controls. Both models show a statistically significant reduction in vote unanimity after the FDA’s guidance to switch to simultaneous voting. Focusing on the full Model (2), there is a negative and statistically significant coefficient for Post (${\beta }_0=-0.528$, $p<0.05$). This result translates into an estimated decline in the probability of a unanimous vote from 0.445 under sequential voting to 0.321 under simultaneous voting.¹⁴ We thus find support for Hypothesis 0, which posited that unanimous votes are less likely to occur under simultaneous voting.

Table 1. Voting Protocol, Information Diversity, and Vote Unanimity

Next we examine Models (3) and (4), which address the effect of voting protocol on the diversity of information discussed during a meeting. Focusing on the full Model (4), we find a positive and statistically significant coefficient for Post (${\beta }_{1a}=3.287$, $p<0.01$). Thus, there is a significant increase in information diversity: our results imply that the change from sequential to simultaneous voting is associated with committees sharing more diverse information, lending support to Hypothesis 1a.¹⁵

The statistically significant coefficients among the set of controls in Model (4) also offer some insights into the drivers of information diversity. At the committee level, we observe evidence consistent with the idea that greater committee diversity increases information diversity. Most directly, information diversity increases with the diversity in expertise among members (${\beta }_{\mathit{\text{ExpertiseDiversity}}}=0.663,p<0.01$). Two other pieces of evidence support the importance of committee diversity in eliciting diverse information. First, the presence of consumer representatives, who share information from the angle of consumer and community-based organizations, helps increase information diversity in a committee that would otherwise be dominated by doctors and technical experts (${\beta }_{\mathit{\text{PctConsumer}}}=3.061,p<0.10$; see, e.g., Von Hippel 1998, Chan and Lim 2023). Second, the presence of some conflicts of interest also increases information diversity. Consistent with the analysis of Cooper and Golec (2019), such individuals are selected to serve despite a significant declarable conflict of interest, so they tend to have deep relevant expertise, likely bringing unique knowledge to the discussions (${\beta }_{\mathit{\text{Conflicts}}}=0.101,p<0.01$). Importantly, we also observe a positive effect of member familiarity on information diversity (${\beta }_{\mathit{\text{Familiarity}}}=0.249,p<0.05$). This result is consistent with the idea that evaluation apprehension exists among committee members—individuals become more willing to share private information when members are more familiar with each other (Huckman and Staats 2011). At the product level, the presence of special FDA designations (which represent more difficult, uncertain, and/or clinically meaningful evaluation problems, like whether the product is under priority review or has a Breakthrough Therapy designation) is associated with an increase in the diversity of information discussed in the meeting (${\beta }_{\mathit{\text{Status}}}=0.379,p<0.05$).

Finally, Models (5) and (6) assess the effect of voting protocol on vote unanimity while accounting for the possible mediating effect of information diversity. We first observe that information diversity has a negative effect on vote unanimity: in the full Model (6), the coefficient for InfoDiversity is both negative and statistically significant (${\beta }_{1b}=-0.083$, $p<0.05$). Thus, we find support also for Hypothesis 1b.

Importantly, we also observe a sharp drop in the coefficient for Post when moving from Model (2) (which does not account for the effects of information diversity) to Model (6) (which does). Conceptually, the coefficient on Post in Model (2) aggregates all effects of the protocol change on unanimous voting outcomes, including both vote-herding effects and information-diversity effects. Model (6) disentangles these effects, explicitly measuring an effect from informationally diverse discussions (InfoDiversity) and any other effects (e.g., vote herding; Post).

That the coefficient of Post becomes statistically insignificant in Model (6) is notable, as it suggests that a vote-herding explanation of the effect of voting protocols on vote unanimity is likely incomplete. In comparison, the mediation effect proposed in Hypothesis 2—that voting protocols affect the likelihood of vote unanimity through the amount of information diversity gained during the discussions—is corroborated statistically (${\beta }_{1a}\times {\beta }_{1b}=-0.273,p<0.05$; see Online Appendix C for details of the estimation of the mediation effect). Indeed, the magnitude of the mediation effect is sizable: compared with the direct effect of Post on vote unanimity (${\beta }_{\mathit{\text{Post}}}=-0.247$ from Model (6)), the mediation effect ${\beta }_{1a}\times {\beta }_{1b}=-0.273$ is slightly larger, and accounts for 52.5% of the total effect of Post (Breen et al. 2013).¹⁶ This implies that more than half of the effect in the reduction in unanimous votes is due to the discussions in which committee members engage.

To summarize, we first find strong evidence for a direct effect of voting protocol on voting outcomes: the change from sequential to simultaneous voting effected a substantial decrease in unanimous voting outcomes. However, we also show that this switch changed the nature of committee discussions, such that they broached on more diverse topics. When we account for this increase in discussion information diversity, we find that the direct effect of voting protocol on voting outcomes is more than halved and becomes statistically indistinguishable from zero. In other words, the documented decrease in unanimous voting can be explained, to a large extent, by the greater levels of information diversity revealed during discussions that occurred after the change to simultaneous voting.

5. Supplementary Evidence and Analyses

In this section, we present results of various analyses that corroborate our theory and supplement the main findings. Specifically, Section 5.1 analyzes various behavioral and linguistic characteristics of the discussions, with the objective of explaining how committee members changed their behaviors to uncover more diverse knowledge and information. Section 5.2 presents evidence that committees made better recommendations under simultaneous than under sequential voting.

5.1. Changes in Behaviors and Linguistic Characteristics

As we have argued in Section 2, eliciting private information is improved when individuals are incentivized to behave in ways that encourage and reduce the costs of sharing knowledge. In this section, we explore whether we can find any evidence of behavioral changes that facilitated members’ revealing their individual knowledge.

One straightforward and effective way to encourage other committee members to share knowledge is to direct questions to them (see, e.g., Keck and Tang 2018). We operationalize such a measure by first identifying the number of questions in each meeting transcript (indicated by the presence of a question mark that ends a sentence). We then find the subset of questions that also contain a reference to the name(s) of at least one committee member. Doing so allows us to capture the proportion of questions that were posed directly to other voting members (pDirectedQs). When we examine the raw numbers, the proportion of questions directed to other voting members increased from 0.080 in the preperiod to 0.099 in the postperiod, providing direct evidence for our theory that members were working harder to elicit information from each other under simultaneous voting. Model (1) in Table 2 reports regression results showing that this increase is statistically significant (coefficient of $\mathit{\text{Post}}=0.028,p<0.05$).

Table 2. Effect of Voting Protocol on Speech Patterns

Further, one of the main causes of a lack of knowledge sharing in a group is often the lack of sharing from quieter members. A large variation in speaking time among voting members is therefore a signal that particular members are dominating the conversation, which could prevent quieter members from sharing their knowledge (Woolley et al. 2010). We create a measure of the variation in speaking time by first counting the total number of words uttered by voting members. We then create a measure of the amount of variation in speaking time by calculating the standard deviation of the distribution of the (logged) number of words among voting members (sdTalkTime). Model (2) in Table 2 shows that the postperiod resulted in a significant decrease in the variation of speaking time (coefficient of $\mathit{\text{Post}}=-0.282,p<0.01$). Thus, quieter members are speaking up relatively more in the postperiod.

We also examine changes in higher-level linguistic characteristics. To do this, we leverage the linguistic analysis software Linguistic Inquiry and Word Count (LIWC-22; Pennebaker et al. 2014). This software generates multiple summary measures that capture a meeting’s linguistic characteristics: how the words and language used by committee experts reflect their behaviors and psychology, and how they changed after the FDA’s 2007 guidance. We focus on three measures relevant to assessing (i) the extent to which individuals feel safe about revealing individual knowledge and (ii) the extent to which individuals encourage others to do so. The first measure is Authentic, which reflects the level of either spontaneity or self-regulation that speakers exhibit (Pennebaker et al. 2014). The second measure, Tone, captures whether speakers make relatively positive remarks—that is, they speak in a tone that may encourage others to be more open about sharing their knowledge. The third measure is Clout, which captures the extent to which the language used by an expert projects higher relative social status, confidence, and/or leadership (Kacewicz et al. 2014, Jordan et al. 2019).¹⁷

Models (3)–(5) in Table 2 report how each meeting-level measure differs between pre- and post-FDA guidance. Specifically, we first used the standardized measure generated by LIWC for each speaker who attended the meeting, and we then took the average over all speakers. In this manner, we obtain an “average” level of, for example, authenticity and positive tone in the focal meeting. Focusing first on Model (4), which analyzes the tone of meetings, we find that the move to simultaneous voting resulted in committee members adopting more positive tones during the discussion (the coefficient for $\mathit{\text{Post}}=0.060$, $p<0.01$). In line with arguments that positivity encourages others to share their individual knowledge (or helps avoid deterring others from sharing), we also observe that committee members tended to speak more spontaneously (the coefficient for Post in Model (3), which analyzes speakers’ authenticity, is positive, $\beta =0.037$, $p<0.05$). In other words, after the switch to simultaneous voting, individuals were less likely to self-regulate or “filter” what they said.

We found no evidence of a significant reduction in the average level of social status/confidence projected by speakers (in Model (5) with Clout as dependent variable, $\beta =-0.018$, $p>0.1$). Yet in Model (6), we examine the distribution of clout across speakers in a meeting (measured as the standard deviation in clout across speakers, sdClout) and find that the advent of simultaneous voting resulted in a statistically significant reduction in the differences in clout across speakers (in Model (6), $\beta =-0.056$, $p<0.01$). Put differently, speakers are more equal—in terms of projecting confidence and social status—under simultaneous than under sequential voting. This result reinforces the earlier finding about the distribution in speaking time: quieter members found the confidence to share knowledge, especially under simultaneous voting. The evidence here suggests that the contributions to knowledge sharing under simultaneous voting may have resulted from increased sharing from the originally more reserved members.

5.2. Changes in Quality of Recommendations

For researchers, measuring the quality of go/no-go decisions is difficult. There is seldom a clear, objective measure of what constitutes a “correct” decision, as projects can fail for a multitude of reasons (e.g., poor marketing, portfolio considerations, regulatory changes) that are outside the decision-making committee’s purview. Also, researchers can be aware of (and therefore measure) only those decisions and outcomes that are public and visible. Yet in many settings, high-level project-selection decisions are made behind closed doors and success targets are confidential.

Our context helps to circumvent both issues. First, the role of FDA AdComs is relatively clearly circumscribed: they are tasked with making recommendations as to the safety and efficacy of new medical products. This generates clarity on the measures (safety and efficacy) on which their decisions must be judged. Second, FDA decisions are made in full view of the public, and product outcomes (reported drug withdrawals, or the imposition of boxed warnings) are also publicly available.

We assess decision quality by capturing the prevalence of type I decision errors in AdCom recommendations before and after the voting protocol change. More specifically, we first consider drugs recommended for approval by AdComs (and then approved by the FDA) that were subsequently found to have an unacceptable benefit-to-risk profile and so were withdrawn from the market. These events reflect the AdCom recommending approval of a new drug when it should not be approved, and therefore constitute clear errors.¹⁸

To investigate withdrawals and discontinuations of drugs, we first subsampled the 215 meetings in which a new pharmaceutical drug was evaluated, recommended for approval by the AdCom, and ultimately approved by the FDA—this is the set of products where we can observe postmarket events indicative of decision errors by the AdComs.¹⁹ We then manually searched through the FDA’s Drugs@FDA database, scientific journal articles, historical news publications and articles, and the Drugs.com website for information on the drug’s approval history. This process included collecting information on whether the focal drug was discontinued or withdrawn (for safety- or efficacy-related reasons only) from the market and, if so, the year of withdrawal.

Column (1) in Table 3, panel A, displays a raw tabulation of the number and proportion of drugs that were still on the market or eventually withdrawn/discontinued when the data were collected. Whereas 8.6% of drugs evaluated under a sequential voting protocol were withdrawn/discontinued, this decreased to 3.4% under simultaneous voting, providing evidence for the reduction in type I errors. Even though more than twice as many drugs were recommended for approval by the AdComs (and eventually approved by the FDA) after the 2007 voting protocol change, we document both a smaller number and a lower proportion of withdrawals or discontinuations in the postguidance period.

Table 3. How Voting Protocol Affects Recommendation Quality

Given that our analysis centers on type I errors, one might conjecture that the reduction in such errors is due less to better decision making than to more conservative decision making (e.g., a committee could reduce type I errors by rejecting all drugs with the slightest hint of risk, though doing so would increase the opposite type II errors, i.e., rejecting good drugs). Table 3, panel B, compares the approval rates of drugs before and after guidance. We find no differences in approval rates for the AdComs (difference = 0.049; $p=0.358$) or the FDA (difference = 0.038; $p=0.307$), a result that is consistent with reductions in type I errors being driven by better decision making rather than an increase in conservatism after guidance.²⁰

Because wholesale withdrawals from the market are the most severe actions the FDA can take, they are quite rare: only 11 drugs in our sample were removed from the market over our 17-year window (see Table 3, panel A, column (1)). We therefore also consider a less drastic action known as a boxed warning. The FDA requires producers to display these warnings prominently at the beginning of the package insert for drugs, so as to clearly inform prescribers and users that use of the specific product is associated with severe consequences and side effects.

Drugs in our sample may receive a boxed warning immediately upon approval. However, we are concerned only with postmarket performance, that is, after the drug has been in the market for some time. Whereas a boxed warning upon approval represents identification of severe risks by the AdCom during evaluation and steps taken to limit those risks, postmarket boxed warnings may constitute decision errors because the AdCom failed to identify such severe risks at the time of evaluation. For transparency, we segregate our analysis by separately considering the postmarket performance of drugs that were approved without a boxed warning ($n=111$; Table 3, panel A, column (2)) and those that received a boxed warning upon approval ($n=104$; column (3)). For drugs that were approved without such a warning, column (2) shows that postmarket boxed warnings decreased substantially for those drugs that were evaluated in the years after the voting protocol change. Around 32.6% of those drugs that were evaluated in the preguidance period and were approved without a boxed warning did eventually receive a postmarket boxed warning by the FDA, whereas postmarket boxed warnings dropped to 8.8% for drugs evaluated in the postguidance period. Column (3) in Table 3 similarly shows that, among the drugs that did receive a boxed warning immediately upon approval, 22.2% of drugs evaluated in the years prior to the voting protocol change experienced a further postmarket upgrade in their warning labels. Postmarket boxed warning upgrades dropped slightly to 18.2% in the years after the voting protocol change.

Overall, the results are consistent with the idea that, subsequent to the FDA’s guidance, AdComs more accurately characterized the risk profile of the evaluated drugs before their introduction to the market. The implication of these results underscores an improvement in drug safety oversight, as more accurate AdCom assessments likely resulted in the reduced type I error rates observed across all three measures of postmarket outcomes.

Finally, Table 4 displays results from Cox proportional hazards models that estimate the hazard rate of a drug experiencing an unfavorable postmarket outcome: being withdrawn from the market, receiving a postmarket boxed warning, or receiving an upgrade to its boxed warning. These models help us address the issue of censoring (because products that are approved very late in the study period have less time under observation to experience a negative event). We do see that for drugs evaluated under a simultaneous committee voting protocol, the hazard rate of receiving a postmarket boxed warning decreased significantly (coefficient of $\mathit{\text{Post}}=0.063,p<0.01$; note that hazard ratios of less than one imply a decrease in the hazard rate). Although we did not observe statistically significant effects on the other two models, we note that the rarity of such postmarket events makes statistical estimation difficult. Furthermore, although not statistically significant, the hazard rate of withdrawal (the most severe kind of decision error) is directionally consistent with our expectations and economically meaningful. Thus, the overall body of evidence suggests that the AdComs made more accurate recommendations under simultaneous than sequential voting.

Table 4. Cox Proportional Hazards Model

6. Robustness

We ensure that our analysis is robust to three categories of empirical issues. We briefly describe these checks below, and the detailed analyses and results are in Online Appendix E.

First, we want to ensure that any temporal changes, such as increasing public scrutiny or changes in committee composition (e.g., due to the gradual tightening in conflict of interest requirements; see Section 4.3.2), are not confounding our results. To rule these out, we looked at Google Trends data and Wall Street Journal articles focusing on the FDA AdComs. Across multiple measures, we did not find significant changes in public interest over time, which supports the validity of our pre–post comparison. We then took steps to check that our insights are valid to the presence of confounding time trends by limiting the time frame to ±3 years around the 2007 change and adding time fixed effects. Finally, to further control for committee composition, we implemented first a model with chairperson fixed effects (thus ensuring that we are comparing across meetings with the same chair), and second another model where a supermajority (i.e., 75%) of the committee members served as experts both before and after the 2007 change.

Second, we want to ensure that our analysis is robust to possible sources of “leakage” in the visibility of votes. For example, one source of leakage comes from the 23 meetings we have identified in the postperiod where voting was conducted via simultaneous, physical hand raising (as opposed to electronically). Unlike electronic voting, simultaneous hand raising affords some level of visibility into how others voted before voting. A second source of leakage comes from multiple FDA voting questions in some meetings: members voting on later questions would have seen how others voted in earlier rounds and could use that information to infer how later voting rounds might unfold. Our results are robust to either dropping all data where voting is done via simultaneous hand raising or dropping all data after the first round of voting.

Finally, we also tested different model specifications, such as probit and linear probability, to ensure that our findings hold up to various other form assumptions.

7. Conclusion

Important and challenging problems are often attended to by committees, and decisions are reached by voting. Despite their ubiquity, there are few real-world studies that examine the effect of voting protocols on how committees discuss and how individual members vote, and the resulting quality of their decisions. This paper provides a comprehensive analysis that compares the impact of voting protocols on committee discussion and voting dynamics.

We leverage a structural change in the way that FDA advisory committees conducted their voting: a switch from sequential to simultaneous voting. We find that simultaneous voting led not only to a reduction in the likelihood of unanimous committee votes but also to an increase in the diversity of information elicited and exchanged during discussion. Indeed, it is the amount of diverse information surfaced by discussions that better predicts (the decrease in) unanimous voting, rather than the voting protocol itself.

The key point is that the impact of voting protocols is not limited to voting outcomes per se; in fact, they can fundamentally alter the dynamics of the entire discussion and voting process. This is not a priori obvious: the FDA’s originally stated intention in moving from a sequential to a simultaneous voting protocol was to avoid vote herding—that is, to prevent later voters from being unduly influenced by the votes of earlier voters (U.S. Food and Drug Administration 2008). In citing research that examines decision situations with no discussions (Banerjee 1992, Callander 2007), the FDA ostensibly did not anticipate that voting protocols would affect the dynamics of the discussions. Nonetheless, our analysis shows that although the FDA did achieve its objective of improving the independence of the committee votes, the more important effect is that the change led to richer discussions. Indeed, the decrease in unanimous votes arose through an indirect route whereby the simultaneous voting protocol incentivized better discourse and greater levels of diverse information elicitation during discussions, and this increased information diversity in turn reduced vote unanimity.

7.1. Discussion and Implications

Our findings extend the literature focusing on how group structures and protocols improve the generation of ideas and solutions (Van de Ven and Delbecq 1974, Kavadias and Sommer 2009, Girotra et al. 2010, Sommer et al. 2020) by offering insights on how to strategically design the incentives for individuals to overcome the various costs and barriers to information exchange within a group. In particular, we show that voting protocols function as structural elements that modify individual incentives to share their own information and elicit information from others during discussions. When committee members know they will vote simultaneously, without access to others’ voting intentions, they face stronger incentives to draw out those diverse perspectives that can inform their own decisions. This incentive-based perspective extends beyond voting protocols to suggest a broader principle: procedural rules in committee settings should be evaluated not only for their direct effects on final decisions, but also for how they reshape incentives for information exchange during deliberations.

Further, our work also allows us to better characterize what it means to have a “good discussion.” By analyzing the behaviors of committee members and the linguistic characteristics of their meetings, we observe a meaningful change in how individuals discuss. In particular, information-rich discussions tend to coincide with members asking each other more questions and spending more equal time speaking during the meeting. Moreover, we document an aggregate increase in authenticity and positiveness, as well as a reduction in the spread of clout (i.e., the most assertive members tend to “hold back” more, whereas the less assertive members project more confidence). These observations correspond to dynamics that promote greater sharing of private information by members and also greater effort devoted to encouraging others to share their knowledge. In so doing, our work establishes rare real-world evidence complementing microbehavioral theories on how discussions shape group decision making (Woolley et al. 2010, Keck and Tang 2018).

For organizational leaders, two clear implications emerge. First, we show just how important it is for managers and other authorities to pay attention to the meeting process, focusing on the extent to which a meeting is structured in a way that incentivizes the free flow of discussion and information. For those committees where a final vote is executed in a round-robin fashion, we are able to provide a clear and actionable prescription: implement a voting protocol that better incentivizes information exchange and suppresses the revelation of votes until after they have been cast. We could summarize our message here as share information, but suppress sharing of judgments (Urfalino and Costa 2015).

Second, how does an organizational leader know if information is shared in a meeting? On top of the many linguistic and behavioral cues (e.g., distribution of speaking time or asking directed questions), we show that diversity of information can be measured easily, and with sensible outputs, via a standard word embedding technique. The increasing prevalence of virtual or hybrid meetings in fact makes the generation of such metrics straightforward. This means that any actions that are taken to improve committee functioning can be directly tracked and improved (though, to the extent that any performance metrics can shape or distort incentives, we also caution business leaders from adopting singular metrics).

7.2. Limitations

The FDA AdCom context represents a distinctive institutional arrangement characterized by its temporary membership structure and service orientation. Committee members operate as external experts providing independent assessments without direct organizational affiliation to the FDA. This arrangement likely attenuates certain strategic voting behaviors we do not consider, but that might emerge in permanent organizational committees where members maintain hierarchical relationships and repeated interactions. Organizations with embedded evaluation committees face additional complexities as members navigate both immediate decision requirements and long-term professional relationships. In these contexts, however, the costs associated with expressing divergent perspectives may actually be greater, potentially amplifying the value of voting protocols that encourage diverse information exchange. Our theoretical framework, however, does not explicitly account for strategic voting behaviors that might emerge in more permanent structures where evaluation decisions carry implications for organizational influence.

Likewise, our results are driven by the presence of (dis)incentives for information exchange. In the current context, these emerge from the transparent nature of deliberations, the technical complexity of evaluation tasks, and the significance of AdCom recommendations for the public good. When discussions and voting occur in closed environments with limited external scrutiny and minimal consequence, the social costs of information sharing may already be reduced, potentially diminishing the incremental impact of voting protocol adjustments. Nevertheless, it is important to recognize that all committee deliberations, regardless of privacy levels, remain social processes subject to interpersonal dynamics that create barriers to comprehensive information exchange. Status differentials, conformity pressures, and professional relationship concerns persist even in confidential settings, suggesting that procedural mechanisms to enhance information sharing are relevant across diverse organizational contexts.

Beyond the context, we must acknowledge limitations relating to the pre–post design of our empirical analysis, which may confound unobserved time-varying elements that we have attributed to the effects of a change in voting protocol. Under a difference-in-differences approach, these are addressed by using a control group; however, our context does not lend itself to identifying a comparable control group. For example, other FDA rules introduced contemporaneously (such as limitations to conflicts of interest) could confound our results on the voting protocol change. That said, our results are robust both to controlling for many identifiable time-varying confounding effects (e.g., changes in committee composition, including the proportion of members with conflicts of interest) and to restricting the analysis to a much narrower window (to minimize the impact of longer lagging effects). These outcomes give us assurance about the robustness of our insights.