Florida 2000 and Washington 2004

"If we make the reasonable approximating assumption that the percentage of votes given to Rossi in a count is a normal random variable, we can use statistics to calculate the odds that Rossi truly won more than 50%. His share in the first count was 50.004722%. His share in the second count was 50.000729%. Let the null hypothesis be that Rossi's true share was > 50%. Use the t-distribution (Excel TDIST() function). Calculate the sample mean and standard error and you get a t-statistic of about 1.36. The one-tailed t-distribution with 1 degree of freedom gives the answer that we can reject the null hypothesis at the 20% level. In other words, the probability is 80% to 20% that Rossi beat Gregoire."

(Sharkansky, 2004)

The analysis he describes uses a "student's" t-test which derives a value for T by comparing the difference between two sample means with the standard error of the mean differences in a much larger sample (the name is historical and has nothing to do with students or education per se). This is incorrect in itself. The student's t-test assumes random draws from a pool in which the variables involved (in this case votes for each candidate) are unrelated to each other. For runoff elections this is not the case. Libertarian Ruth Bennett's counts were for all intents and purposes negligible, and the contest was essentially a runoff between Rossi and Gregoire. Thus, their vote percentages will be related to each other. As we've already seen, this requires a proportional t-test where the relevant dispersion is given by the standard error of the proportion (equation 4).

Even if we grant Sharkansky his method, it's difficult to tell how he obtained this result. His discussion is vague to the point of being almost indecipherable, and at several points it's even self-contradictory. He states that he used MS Excel to calculate his T, but Excel's TTEST() function requires data to be entered as two arrays of values and does not allow for the means of each vote count to be input directly. The Excel TDIST() function he refers to derives probabilities associated with any given value of T but does not derive T itself (this is where his 20 percent figure comes from). For a proper student's t-test he needs a reliable estimate of the standard error associated with a large number of vote counting efforts--which he does not have. Without one, he will have to have made some assumptions. There are two basic philosophies he might have used. One the one hand he could have assumed that county level differences would reliably represent the standard error of the statewide recount process. In this case he could have used the MS Excel TTEST() function. This would be incorrect because county level samples differ widely in size, canvassing protocols, and (most importantly) voting technology implementations and will not reflect the state as a whole (WA Sec. of State, 2005b). I was unable to reproduce his results this way using any of his cited sources. It's more likely that he either assumed a standard error or derived one in some indeterminate manner, and then did his calculation by hand.

How this was done is a mystery. He says that he calculated "the sample mean and standard error" but nowhere are we told what "sample" he is referring to or how his calculations were done. The only clues we have as to what might have been going through his head are his comments regarding residual votes and tabulation error. Using data from the Secretary of State's office he obtains statewide residual vote rate of 0.85 percent not including write-ins (WA Sec. of State, 2004e; 2004g). Supplementing this with county Elections Division data he obtains a corresponding rate of 0.39 percent for King County after 1,194 write-ins are removed from the county total of 4,704 total residuals (King Cty. RELS, 2004). His discussion of write-ins was limited to King County by necessity as he did not have access to write-in totals for other counties (I was unable to obtain these figures as well). But even though he doesn't say so explicitly, it's clear that he considers these figures to be representative of the statewide gubernatorial election. From which he concludes that nearly all Washington's 2004 gubernatorial residual vote was intentional abstentions. This whole line of reasoning displays a profound lack of understanding of residual votes and how they originate.

First, the presidential residual vote will have limited relevance to the Gregoire/Rossi runoff. Senatorial and gubernatorial elections are typically very similar to each other and run anywhere from one to three percent higher than their presidential counterparts. They also tend to have differing rates of intentional abstention and machine (or "technology") driven vote spoilage, though differences in the latter are usually much smaller (Caltech/MIT, 2001; Brady, 2001; Ansolabehere & Stewart, 2005). This can be seen in Sharkansky own sources. A comparison of the gubernatorial base count and machine recount is shown in Figure 2. Here it can be seen that for all ballots cast (Ruth Bennett included) we have a statewide residual vote of 75,158, or 2.6 percent--more than three times the figure he quotes (WA Sec. of State, 2004b; 2004g). The corresponding senatorial rate is 2.2 percent (WA Sec. of State, 2004f; 2004g). Furthermore, these figures are actually on the low side of national trends, even where they're restricted to the more reliable optical scanning technology (Ansolabehere & Stewart, 2005).

Sharkansky ridicules David Goldstein for claiming that there may have been 14,000 unintended residuals in the election--"There is absolutely no basis for screaming that there were '14,000 erroneous votes!'" he tells us. Yet his own sources reveal more than five times this amount. It is true that this does not account for write-ins and intentional abstentions. The latter are also consistently higher for senatorial and gubernatorial elections than for presidential ones. But we're talking about a total that is close to one-half the population of Spokane. It's difficult to see how this many spoiled ballots can be explained away as voter choice. Sharkansky's attempts to do so are weak at best. He points to 717 spoiled ballots that the King County Canvassing Board managed to convert during the election (0.08 percent of all King County ballots) and claims that the true unintended residual vote rate could not be any higher than this. He fails to notice that this only includes ballots that needed to be redone to make them machine-readable. When ballots have been mutilated, mismarked, or otherwise damaged enough that they cannot be input to optical scan equipment, it is standard canvassing practice to transfer them to new forms to render them countable. This has nothing to do with the much larger question of random machine and/or voter/polling place related errors. It must be remembered that a machine recount is not a manual count and does not involve hand examination of every cast ballot. If anything, Sharkansky's 0.08 percent is an absolute lower bound on King County's unintended residual vote.

Which brings us to the next point--tabulation errors. Goldstein's figure of 0.56 percent, he tells us, is 125 times larger than reality. Linking to the Secretary of State's web site (WA Sec. of State, 2004b), he uses the base count and machine recount table (Figure 2) to derive a tabulation error of 0.004 percent. "Copy this table into Excel and do the math," he insists.

I did--and not only are all of his numbers incorrect, he doesn't even understand what he's calculating.

The tabulation invalidation rate is the percent change in vote count between a base election count and a recount per candidate and office. The value Goldstein cites was derived from a study of contested New Hampshire elections between 1946 and 2002 (Ansolabehere & Reeves, 2004). A total sample of 415 cases was evaluated. Each case compared the percent change between the base and recount tabulations in a selected town or district for a candidate in a contested election. The 0.56 percent figure represents the average invalidation rate among these cases that can be attributed to optical scan technologies, weighted by population and corrected for "office effects" (that is, the differences between senatorial, gubernatorial, and presidential elections in demographic and residual vote trends that are specific to the office rather than technology). To derive a directly comparable case from the last fall's gubernatorial runoff we would examine the change in tabulations between the base count and machine recount on a county level basis for Gregoire and Rossi separately (towns or precincts would be a more direct comparison, but the Secretary of State's data Sharkansky cites is by county). Using the same data Sharkansky did, the base count for Rossi in King County was 350,779 votes. The corresponding machine recount tabulation was 351,127 for a total tabulation discrepancy of 348. This gives a tabulation invalidation rate of 348/350,779, or 0.099 percent. The corresponding figure for Gregoire is 0.117 percent. A population weighted average for the county would be (0.099*350,779 + 0.117*505,243)/(350,779 + 505,243), or 0.110 percent (WA Sec. of State, 2004b). Statewide, the same dataset shows tabulation invalidations in optical scan counties that range from zero (Gregoire in San Juan County and both candidates in Skamania County) to 0.791 percent (Gregoire in Adams County). Goldstein's figure falls nicely within this observed spread. The statewide population weighted average is 0.115 percent for optical scan technologies, and 0.094 percent for the entire state. To be sure, this is noticeably lower than 0.56 percent but nowhere near Sharkansky's figure, which is smaller by a factor of over 28.

How can this be? Once again, Sharkansky is silent as to how his figures were obtained. But he does tell us that his numbers were based on the Secretary of State's machine recount data (WA Sec. of State, 2004b) which he supplements with the offhand comment that the "true number of erroneously counted votes" will be equal to "half the discrepancy." A careful examination of that data reveals that his figure is based not on the tabulation invalidation, but the change in Rossi's victory margin divided by the total ballot count. He then compounds the error even further by dividing this ratio by two. In other words, his "tabulation error" is 0.5*(261 - 42)/2,742,567, or 0.004 percent (Sharkansky, 2004; WA Sec. of State, 2004b). This is a serious misunderstanding. By definition, an error in tabulation is the actual discrepancy between two counts, not the change in a victory margin. It's defined on a per-candidate basis rather than by total ballot count so that it will be normalized to variations in the latter even where margins are large. According to Sharkansky's method, if Rossi and Gregoire had respectively suffered recount tabulation discrepancies of 50.002 percent and 49.998 percent of all ballots cast, the "true tabulation error" for the election would work out to 0.004 percent even though 1.4 million ballots would have been lost, spoiled, or otherwise not registered in the machine recount. This, of course, is patent nonsense.

By contrast, Goldstein's discussion of residual votes and tabulation invalidation is accurate at every point indicating that he did in fact read the papers he cited. He also addressed a crucial point that Sharkansky carefully avoided. Tabulation discrepancies are a poor metric for quantifying unintended ballot spoilage. They fail to account for a wide range of errors resulting from human interactions with vote-counting equipment, bureaucratic errors, and other factors. Most reliable studies rely on the residual vote rate for evaluating technology driven ballot spoilage--in particular, discrepancies in residual vote for differing vote counting technologies (Caltech/MIT, 2001; Ansolabehere & Reeves, 2004; Ansolabehere & Stewart, 2005). Ansolabehere and Reeves discuss this fact in their New Hampshire paper. Sharkansky appears to have missed that as well.

At times Sharkansky's comments are almost schizophrenic. He begins by incorrectly referring to tabulation error as "the error rate of machine counting" (his point 2). In fact, technology is only one of many sources of tabulation error. A few paragraphs later he refers to it as "the difference between the outcomes of the first count and the recount" which appears to conflate variations in the victory margin with tabulation discrepancies (consistent with his numbers). Then, he tells us that Goldstein's cited figure of 0.56 percent is 7 standard deviations off of Washington's actual tabulation error. Goldstein's source reports a 95 percent confidence interval of 0.42 to 0.70 percent for this figure (Ansolabehere & Stewart, 2005). For a normally distributed random variable this equates to a standard deviation of 0.07 percent (one-fourth of a "2-sigma" confidence interval), which is very close to the actual state average tabulation discrepancy (see Figure 4) indicating that Sharkansky may have understood the concept tabulation invalidation after all. But then, in the very next sentence he returns to square-one and compares the 0.56 percent figure to his own margin based one. Two sentences later he correctly refers to tabulation error as "the discrepancy between two counts," only to compare it yet again to his own margin based estimate in the very next sentence.

What the man was thinking is anyone's guess. I've been over his commentary countless times and I still can't decipher what he was actually trying to claim. Despite several hours of effort I was unable to reproduce his value for T using any figure he quoted or referenced, including the base and machine recount results (by county or state), residual vote, or tabulation invalidation. Even so, his methods can still be evaluated from scratch using the student's t-test method to back out an assumed standard error from his T of 1.36. In a student's t-test (the name is historical and has nothing to do with students or schools), T is given directly by equations 1 and 1A where we have either a statistically significant number of recounts from which S_G and S_R can be evaluated (which we do not) or reliable estimates of the standard errors of all known noise sources impacting the vote counting process. In this case both counts used the same machine technology and the difference in tabulation due to the addition of new, previously uncounted ballots was negligible compared to the overall sample size of roughly 2.8 million so the same raw standard error would have driven both. In this case equations 5 and 6 give,

For a T of 1.36 this equates to an assumed standard error S of 0.00207. Interestingly, this is very close to one-half of Sharkansky's margin invalidation rate implying that he may have erroneously divided by two a second time. Rerunning his numbers with the correct statewide weighted tabulation invalidation of 0.094 percent yields a T of 0.030 and a corresponding probability of 98.1 percent that his null hypothesis is valid. In other words, when used properly even Sharkansky's own flawed method concludes that there is less than one chance in 50 that Rossi's "victory" was statistically significant.

Point to Goldstein. Game to Goldstein.

All this is moot however because tabulation invalidation is not a proper measure of ballot spoilage. The relevant metric is the unintended residual vote, which will be given by the total residual vote (not including write-ins) minus intentional abstentions. Sharkansky's investigation was based on the most complete information available at the time. His correction for write-ins was limited to King County out of necessity because, as he rightly pointed out, the Secretary of State's counts do not include write-ins. These were only available for King County without resorting to unreasonable and time-consuming effort (I too was unable to obtain write-in stats for more than King). From these figures he obtained overall presidential residual vote rates for the state and King County that appear to be quite close to the mark. But his argument breaks down when he attempts to extrapolate this data to intentional abstentions and the unintended residual vote for the statewide gubernatorial race, which as we saw earlier ran considerably higher than the presidential rate even with fewer write-in candidates. In the wake of the fall 2004 election, other data has become available that allow us to test the validity of these assumptions.

Early this year Federal Election Survey data gathered after Election Day became available. That data, which was graciously provided to me by the Secretary of State's office (Wa. Sec. of State, 2005), gives far more detail regarding residual votes and provisional ballots than was available when Sharkansky wrote his commentary. Figure 5 shows the results of that survey that are most relevant to this study. Here we find data for presidential and senatorial vote counts, total ballots cast, and complete breakdowns of the residual vote into undervotes and overvotes--all given by county and statewide totals. An examination of the King County residuals for the presidential race reveals a total of 3,390 undervotes and 120 overvotes for a total of 3,510 residuals. Sharkansky cited a 4,704 residuals of which 1,194 were write-ins giving the exact same total (WA Sec. of State, 2004b; King Cty. RELS, 2004). This highlights a significant point--the Federal Election Survey residual tallies already take write-ins into account giving us a reliable snapshot of the raw residual vote. No FES data was available for the gubernatorial race, but senatorial data was. We saw earlier that like most similar elections on record nationwide, last fall's senatorial runoff was much closer to the gubernatorial runoff than the presidential one. In fact, the senatorial residual rate is 0.4 percent lower than the gubernatorial rate so that if anything, predictions based on it are likely to be conservative. Comparisons of Sharkansky's statements to this data are revealing.

The final FES presidential residual vote for Washington was 21,024, or 0.73 percent of all ballots cast (WA Sec. of State, 2005). Of these, 16,452 were undervotes and 4,572 were overvotes. The senatorial residual rate was 61,306 (2.1 percent) with 59,927 undervotes and 1,379 overvotes (WA Sec. of State, 2005)--again, not including write-ins. Sharkansky made much if his 0.85 percent presidential figure and implied that after write-ins were removed the actual rate might be less than half of that if King County could be taken as an example. In fact, write-ins accounted for barely 0.11 percent of all cast ballots--a mere 13 percent of the total presidential residual count. Examination of the county level figures reveals why (WA Sec. of State, 2005; 2005b). King County (which uses Global Accuvote optical scan systems) had a total residual rate of 0.52 percent making it the best performing county in the state by a noticeable margin. Only 4 other Washington counties use similar systems. All did worse. The rest of the state's optical scan systems are nearly ten years older. Coming in behind King for second and third among optical scan counties were San Juan (also a Global Accuvote county) which boasted a residual rate of 0.44 percent, and Spokane (an ES&S OPSCAN 650 county) which ended up at 0.49 percent. Pierce--Washington's largest ballot contributor behind King--came in at over 1.0 percent. The next largest contributor, Snohomish, came in at 0.52 percent. Snohomish used a mix of OPTECH optical scan and AVC Edge Direct Recording Equipment (DRE) systems. The latter do not allow for overvoting so this figure likely underestimates the Snohomish optical scan related residual rates. Overall, Washington's optical scan counties show a total residual vote rate of 0.61 percent. Not surprisingly, the state's punch-card counties did much worse. All but two had residual rates of more than 1.0 percent with most coming in at 1.2 to 1.5 percent. Franklin County suffered a whopping 2.2 percent residual rate, the state's worst. Overall, Washington's punch-card counties had a residual rate of 1.2 percent (WA Sec. of State, 2005; 2005b).

To obtain the true unintended residual vote rate these figures will have to be corrected for intentional abstentions. Given that all balloting is secret, this can be difficult to estimate. Historically, the best estimates have come from exit poll studies of presidential elections which typically have the lowest rates. Overall, these show a national average of around 0.5 percent (Ansolabehere & Stewart, 2005). Senatorial and gubernatorial elections usually run higher. Both optical scan and punch-card technologies show their best residual vote performance in presidential elections where intentional abstention data is best (Ansolabehere & Stewart, 2005). Therefore, last fall's presidential race can be used to set a lower bound on Washington's unintended residual vote. The last two presidential elections were among the most heated in the nation's history, as was the Gregoire/Rossi runoff in Washington--possibly the only election in the state's history where a death threat was made against the victor (Ammons, 2005). Therefore, it's likely that intentional abstentions were lower than normal making this a conservative estimate. With a 0.5 percent intentional abstention rate we get a statewide unintended residual vote of 0.11 percent of all ballots cast for Washington's optical scan counties, 0.70 for punch-card counties, and a statewide average of 0.23 percent. This yields a bare minimum of more than 6,600 spoiled ballots. The true rates are likely to be higher, especially for the senatorial and gubernatorial races.

With these results we can revisit Sharkansky's four original arguing points (Sharkansky, 2004), and his claim that the true margin in Washington's 2004 gubernatorial election went to Rossi.

1)   The "Residual Rate" (blank and otherwise disqualified ballots) in Washington was far less than 1 percent [in 2004].

In fact, it was far greater. The original gubernatorial residual, defines here as all "blank and otherwise disqualified ballots" was 2.6 percent. With write-ins accounted for, it the final total will be somewhat less, but very likely to be greater than the senatorial residual of 2.1 percent. This puts it over the high end Goldstein's stated range of 1 to 2 percent. Only the presidential rate was less, and that by only 0.27 percent with write-ins removed.

2)   The "Tabulation Error Rate" (the difference between the outcomes of the first count and the recount) in the governor's race was nowhere near 0.56 percent. It was 0.0040 percent when looking at the entire state.

As a matter of fact, they were close. The statewide optical scan tabulation error observed in the machine recount was 0.115 percent with the county level figures running anywhere from zero to as high as 0.79 percent (WA Sec. of State, 2004b). Ansolabehere and Reeves' figure falls nicely within this spread. Apart from the high end outliers (Adams and Walla Walla Counties) several counties had rates that approached the lower end of their 95 percent confidence interval. Thus, in terms of overall averages Washington's optical scan performance is noticeably better than that observed by Ansolabehere and Reeves in New Hampshire but not by anything like as much as Sharkansky claims, especially since the data these figures came from were based on a single runoff for one office. A larger dataset would likely reveal more noise and bring the two figures even closer. Sharkansky's figure is based on a confusion of tabulation invalidation with variance in margin.