Voting Without (Statistical) Controls

Presentation at the 2004 Fall Technical Conference

By Fritz Scheuren

ASA President-elect



Does our election system have the accuracy to calculate a winner when vote totals are close? Could our election be too close to call? What can we do to help as professional statisticians? These are questions asked this August in an article I had published in Amstat News. An update on how we might answer them now will be given in this talk.[1]


The impact of errors in the voting system on our democracy will be discussed as a statistical quality control problem. Specifically, the discussion will center on how the steps for ensuring a Quality Process might be applied to the problem of increasing the accuracy of elections – both by reducing the bias in election outcomes due to errors in the voting system and increasing the precision in that system. Because the topic of voting is so much on everyone’s mind right now the hope is that the talk will be highly interactive.


Voting as a System


I am going to be treating voting as a “system.” So we need to be applying our “systems thinking” skills, as Deming advocates. We will also be looking in a few moments at the major portions of the voting system whose variability we need to understand. So Deming’s second major recommendation will be followed too of (trying) to understand (and reduce) system variability.


For the sake of completeness (but not to cover all his 14 points) let me speak about Deming’s notion of the need to have “profound knowledge” of human nature and finally on the essential ingredient to quality improvement -- “constancy of purpose.”


Now based on the side comments made yesterday at this conference very few people here want to talk about the election. Virtually all of us (me included) are uncomfortable with what seems to be an excessive degree of partisanship and the attendant ugliness and puffery that accompany it.


That observation may not pass for “profound knowledge” but we do, as statistical practitioners, have a problem of being objective when we see a problem that we are inside of ourselves. It can be dark inside. The search for rationality is difficult but we must try, if we are to overcome mere rationalizations.


Remember the Greek philosopher, Diagones? He was the one who went about with a lantern looking for an honest man. Well, given the heat of the moment we may be hard pressed to find in ourselves the nonpartisan statistician we need to be, with or without Diagones’ lantern


Our Professional Role


Many of us are angry about what is happening and that could cloud our critical thinking skills. Try anyway is my plea. We cannot afford to just leave this issue to an army of lawyers. We need to be players too. A squad of statisticians? Who knows it could make the right difference!


Data, if properly collected and competently interpreted, in my experience, reduces conflict and we owe it to our country to try to do this. The blue and red maps we see almost daily need not be turned into blue and gray maps. We need to find, and I expect Dean Neubauer (yesterday’s luncheon speaker on the civil war) would agree, a way to put patriotism above party. After all one of the elements in the mission statement of the American Statistical Association is to use our discipline to enhance human welfare. And here is a chance to do just that.


Getting Started


How would I recommend we go about using our discipline in the current context? Well, our profession and allied professions have not been asleep since the Florida 2000 debacle, although more attention would have been welcome from more of us.


To bring you up to date on what has been done and help you to get started, let me mention five sets of references. These are --


          The Kirk Wolter et al article in the February 2003 American Statistician where Kirk describes the recount NORC, where I work, did of Florida. 


A second important source is the special issue of Chance in 2002 which usefully analyzes the implications of the 2000 election more generally.


A third source is a series of articles, also in 2002, which appear in Statistical Science on voting.


I have just two more references to mention now, not to drag this part out.


A fourth source, for those interested in the role of polling and the election, are pieces by Warren Mitofsky and others that appear in the 2003 and 2004 Public Opinion Quarterly. After all polling (and the media) arguably is a part of the election system. The system is by no means only what happens after you get into the voting booth.


Finally, there is an article by Ted Selker that appears in the October 2004 Scientific American just out. Selker is part of the Caltech/MIT voting project.


The Caltech/MIT group were referenced by me in the short appeal I made to our profession on this topic which appeared in the August AMSTAT News.[2]


If you only have a little time to sharpen your systems thinking skills before November 2, then I would recommend the Scientific American piece. Selker is not a statistician, so you will have to add in your own measurement perspectives but this piece is the most comprehensive, short treatment out there.



Measurement Problem


The 2004 election presents us s a very hard measurement problem. The “margin of error” on our voting yardstick could well turn out to be higher, in all likelihood that the real difference in the “true” vote (if we could even measure that.) Such a situation cries out for major system changes


To get the details here it may be useful today to divide quality (like Caesar did Gaul) into three parts: prevention, detection, and repair. Naturally, we all want to spend as much of our resources on prevention as possible (allowing us to spend much less altogether). Now a prevention effort was made here by the politicians in passing the Help America Vote Act (HAVA). Good try too.


Some HAVA provisions will not kick in until 2006, however. Some are inadequate and much of the money set aside to do this job remains unspent. (Of course, some of the expenditures, no surprise, were misspent).


Given that the 2004 prevention efforts were at best incomplete, we are largely left with is the problems of detection and repair (and of course, replanning for the next time). Our data analysis skills and patience will be crucial to reducing the heat and increasing the light that gets shed on these issues.


Right now nearly every informed American is distrustful of the way we vote and whether the quality of the results can be used. Still we have no choice but to go forward anyway. What then should we be looking at as the election approaches?


Isikawa (Fishbone) Diagram of Voting


Allow me to drop deeper down now into the strengths but, sadly, mostly weaknesses of our existing highly decentralized voting system. To do this I will employ a very general Isikawa or Fishbone Diagram of the US voting system. See next page.





Now in the usual familiar fashion we employ the head of the fish to state the system goal -- here having a trustworthy voting system. There are seven major bones off the spine and I will take up each of these in turn. They are expressed in a positive way: certified equipment, secured tabulation, tested ballot, educated voters, poll workers, verified identity, and media support.


For each bone off the spine I look first at barriers to achieving the desired state; then some possible quality improvement approaches to overcoming those barriers. Finally, some “starter set” comments are given on how we might be able to measure success.  In a following section I will sketch in the brief time available what we, as statisticians, can do now and how what we do can help replan the US Voting System for 2008


General Closing Comments


I’ve laid out problems very roughly. Key issue is continuing to look for more and developing adequate well planned investments and real-time monitoring so system works as it should.


Still there needs to be some conformance to requirements checking (tests for poll workers?, review of equipment at end of day, on a sample basis, etc.?) Plus fitness for use confirmations (like exit poll monitoring if its many problems can be overcome)


Your statistical imaginations, considerable experience and willingness to help are needed.


This clearly is a time, when our discipline (as the ASA mission statement says) should be used to enhance human welfare.


After word on Constancy of Purpose


In Deming’s terms, after 2000 the country as a whole and statisticians as a profession could have demonstrated more constancy of purpose in dealing with this problem. Well we have another chance ib getting ready for 2006 nd 2008. Let’s be about it.










28% electronic for 2004 but more next time.


Software bugs are known to exist but not easily identified


In many case there are reports of unverified, inadequately tested code.


Need an audit trail that is in place fukky before the vote.




New contracts are needed with software and hardware vendors.


The standard setting process now underway needs to be finished and an acceptance process devised for its implementation.


From the voter perspective the process must be more transparent (e.g., Nevada)











Hacking in concerns exist in many systems?


Corrupting vote by adding viruses have also been a concern?


Chain of custody concerns (Two party observation) up to the final vote tally by precinct




Vendor contracts again must allow for third party checking before voting and be auditable afterwards.


Reduction of variability (but no enough to create a bigger) a (Microsoft) target for hackers.









Butterfly Ballot is the most famous of the problems here but there are others recorded too.


Cognitive problems are simply not addressed in ballot design and implementation.


The untested nature of ballots, might have special consequence for minorities and those with weak language skills (new immigrants, say).



Usual survey approaches should work. Getting them used is problem.


Research tools for questionnaire design, as in Census. Again science needs a role here in place of traditional (and sometimes) even bad practice


Don Dillman and his students need to be brought in, for sure but others too.








Many voters do not read or follow instructions (simplify of course but…)


New technology will be a problem for many voters not just first time voters


Preparation of absentee and overseas ballots a big problem, including determining fatal flaws



Voter needs to take more time to understand process before voting


Parties need to better prepare voters  (especially, first time voters)


Classes might be offered ahead of time by League of Women Voters, etc.











Part-time and at quite distant intervals


Workers are underpaid and under appreciated.


They typically are older and more technologically challenged.


Happens too quick to do on the job training



Stretch out election process, allow more early voting.


Vote by mail even (as in Oregon) but quality of this may be a problem for some groups


Stabilize technology at point of deliver, so changes in technology are as transparent as possible.









Registration systems weak on identity information, not kept up-to-date. Multiple registrations not checked or even checkable across jurisdictions. Citizenship not confirmed (motor voter)


Checking after the fact in Florida revealed many hundreds of double voters and many thousands of double registrations (FL/NY)



Citizenship check needed


Lists needs to be adequate to deduce reliably and to define uniquely or nearly so for all practical purposes.


Massive record linkage issues and confidentiality problems need to be faced here.


Recourse needed when adverse action taken (as in felon purge in Florida)









Pressure from media to rush results, leading to mistakes by media, others (Florida again)


Overuse of exit polls by untrained partisan “volunteers” looking for problems (Venezuela example)


Tainted evidence by voter (?) and observer



Slow down process, handle controversies better, individually


Solution may be trained exit poll watcher plus better partisan observers inside who have some authority














[1] Presented at the Fall Technical Conference in Roanoke VA on October 15, 2004.

[2]  A still incomplete but broader set of references are to be found elsewhere on this website with links or copied posted where possible. Also the results of the Albuquerque exit polling done during the election on November 2 can be found there as well.