A Decidedly Insufficient Statistical Analysis

Among the races contested during the 2022 U.S. general election was one for governor of the state of Arizona. That race pitted Katie Hobbs, a Democrat (then serving as secretary of state) against Kari Lake, a Republican. Voting was conducted by mail or in person, both early and on Election Day, Nov. 8, 2022.

On Nov. 15, the race was called for Hobbs, who received 1,287,891 votes, whereas Lake received 1,270,774, a margin of 17,117 votes [1]. This margin was 0.67% of votes cast for the two candidates, exceeding the margin of 0.5% that would have triggered an automatic recount [2]. Arizona certified the election results on Dec. 5, declaring Katie Hobbs to be the governor-elect and opening a five-day window for formal election challenges [3].

On Dec. 9, Lake’s attorneys filed a 70-page lawsuit against Hobbs (in her personal capacity as a candidate and in her official capacity as secretary of state) and against various officials of Maricopa County [4]. Maricopa County, which, ironically, is controlled by Republicans, was named in part because of its strategic importance given that more than 60% of statewide votes were cast in that county. The lawsuit included 10 election challenge counts and sought to have Lake declared the winner of the election or, alternatively, order a do-over in Maricopa County.

On Dec. 20, Maricopa Superior Court Judge Peter Thompson dismissed eight of the 10 counts, but ruled that two – pertaining to the impact of malfunctioning printers on in-person Election Day results and mail-in ballot chain of custody – could go to trial. At numerous voting centers in the county, ballot printers, for one reason or another, printed ballots that could not be read by tabulators, causing what Maricopa County Board of Supervisors Chairman Bill Gates referred to as Election Day “chaos,” including long lines. An evidentiary hearing was scheduled for Dec. 21 and 22.

At trial, Lake’s attorneys called several witnesses, including Richard Baris, director of Big Data Poll. Baris indicated that in his more than six years with Big Data Poll, the organization had never inaccurately predicted the winner outside of the sampling error. Big Data Poll signed up 813 respondents from Maricopa County for a 2022 general election exit poll. Although 93% of respondents who indicated they would vote in advance of Election Day completed the survey, only 72% who indicated they would vote on Election Day completed the survey. According to Baris, “that [the over 20% difference] has never happened to me before, ever.” He stated that in his professional opinion, “these people didn’t complete the questionnaire because they didn’t vote, they didn’t get to vote.”

Baris presented results of analysis based on the exit poll response shortfall that suggested that 25,000-40,000 voters who intended to vote on Election Day were unable to do so. Republicans, who represented 70% or more of Election Day voters, were disproportionately affected. Baris concluded, “in my professional opinion … I believe … that it was substantial enough to change the leaderboard.” Asked whether “Plaintiff Kari Lake would have won this race BUT FOR the Election Day chaos,” Baris replied, “I have no doubt, I believe it strongly, it’s my opinion, that strongly, yes.”

On cross-examination, Baris was asked the following questions (to name a few) regarding the credibility (or lack thereof) of Big Data Poll. Are you familiar with The New York Times? Are you familiar with FiveThirtyEight [a website that focuses on, among other things, opinion poll analysis and politics]? Are you aware that FiveThirtyEight aggregates more than 450 different polls for its analysis? Are you aware that Big Data Poll is excluded from FiveThirtyEight’s aggregated polls? Are you aware that Big Data Poll is one of only 11 out of 500 to receive a failing grade from FiveThirtyEight [as unreliable, not transparent or inaccurate]? Are you a member of the National Council on Public Polls, the Association of Public Opinion Research or the Roper Center? Have you ever published a paper in a peer-reviewed journal?

Baris’ report (which was not admitted into evidence but was shared with the defendants and discussed during cross-examination) included an analysis of what could have happened had Election Day voter turnout in Maricopa County been higher. For example, if 2.5% (or around 39,000) more overall voters voted on Election Day, then based on the Election Day margin (conservatively 70% for Lake and 30% for Hobbs), the election result would have been reversed. Baris argued that, based on his exit poll results, 2.5% or more was reasonable.

Governor-elect Hobb’s attorney reviewed the underlying calculations with Baris. This questioning included a bit of humor as the attorney noted that “lawyers are notoriously bad at math” and “I used a calculator.” The upshot of this exchange was that, whereas 39,000 is 2.5% of the overall number of Maricopa County voters who voted (approximately 1.5 million), 39,000 is more than 15% of the roughly 250,000 voters who voted in person in Maricopa County on Election Day.

Judge Thompson issued his ruling on Dec. 24, which included the following remarks regarding Baris’ testimony [5]:

“Indeed, to the extent that a range of outcomes was suggested by Mr. Baris, he suggested that – with his expected turnout increase on Election Day of 25,000-40,000 votes the outcome could be between a 2,000-vote margin for Hobbs to a 4,000-vote margin for Plaintiff. Taking Mr. Baris’s claims at face value, this does not nearly approach the degree of precision that would provide clear and convincing evidence that the result did change as a result of BOD [ballot-on-demand] printer failures. While this Court (in the absence of controlling authority) is reticent to state that statistical evidence is always insufficient as a matter of law to demonstrate a direct effect on the outcome of an election, a statistical analysis that shows that the current winner had a good chance of winning anyway is decidedly insufficient. Cf. Moore, 148 Ariz. at 159 (suggesting that population data might in some cases be admissible to prove voter disenfranchisement).”

The judge ruled, with respect to both counts, that the court DOES NOT find clear and convincing evidence that any misconduct was intended to or did in fact affect the result of the 2022 general election. “Therefore: IT IS ORDERED: confirming the election of Katie Hobbs as Arizona Governor-elect pursuant to A.R.S. § 16-676(B)” [5]. (Katie Hobbs was sworn in as governor on Jan. 2, 2023.)

In subsequently ruling on a motion for sanctions filed by the Defendant Secretary of State and Governor-elect Katie Hobbs, Judge Thompson further commented on Baris’ statistical analysis as follows [6]:

“The case law regarding Election Challenges dates back to Territorial days when ballots were counted by hand. There are no cases that rule in or out a statistical analysis as a method for proving elements of an Election Challenge under A.R.S. § 16-672. But, the law makes clear that Plaintiff is required to show a specific number of votes affected which would impact the Election outcome. The Court was not persuaded that, among other failures of proof, statistical analysis with projected ranges of votes based on assumptions as to people who did not vote, met the burden of clear and convincing evidence of a specific number or [sic] votes to be subtracted or added to either Plaintiff or Defendant under A.R.S. § 16-672.”

During Baris’ testimony, the term “reasonable degree of mathematical certainty” was used to describe his conclusions. A subsequent witness (Dr. Kenneth Meyer, a political scientist from the University of Wisconsin-Madison) indicated that the National Institute of Standards and Technology discouraged use of this term because it conveys an air of scientific precision but is in fact meaningless.

Lake’s attorneys needed someone to state that the results of the election were wrong or uncertain, and Baris was that witness. Baris’ conclusion that but for Election Day chaos, Plaintiff Kari Lake would have won indicates an extreme degree of certainty. And yet, can results based on statistical sampling with assumptions to boot justify such a degree of certainty? What ethical questions arise regarding how such analysis results should be presented in a court of law?

Judge Thompson concluded that Baris’ statistical analysis was decidedly insufficient but did not rule out the possibility that statistical analysis could play a role in contesting election results. What sort of analysis (if any) could meet the burden of providing specific, clear and convincing evidence? What are the limits of our craft?

References

1. https://azsos.gov/sites/default/files/2022Dec05_General_Election_Canvass_Web.pdf
2. https://www.azleg.gov/ars/16/00661.htm
3. https://www.pbs.org/newshour/politics/arizona-certifies-2022-election-results-despite-gop-complaints
4. https://www.clerkofcourt.maricopa.gov/home/showpublisheddocument/4151
5. https://www.clerkofcourt.maricopa.gov/home/showpublisheddocument/4531/638074764564181393
6. https://www.clerkofcourt.maricopa.gov/home/showpublisheddocument/4567