CYLAB/ISR SEMINAR APRIL 16, 2007 COPYRIGHT © 2007 MICHAEL I. SHAMOS What Happened to 18,000 Votes? Results of the Sarasota Source Code Audit Michael I.

CYLAB/ISR SEMINAR

APRIL 16, 2007

COPYRIGHT © 2007 MICHAEL I. SHAMOS

What Happened to 18,000 Votes?

Results of the SarasotaSource Code Audit

Michael I. Shamos, Ph.D., J.D.Institute for Software ResearchSchool of Computer ScienceCarnegie Mellon University

CYLAB/ISR SEMINAR

APRIL 16, 2007


Outline

• What happened in Sarasota County?

– The problem

– Political events

• Source code review

– What was done

– What was found

– Vote flipping

– Touchscreen delay

• Where did the votes go?

Buchanan Jennings

CYLAB/ISR SEMINAR

APRIL 16, 2007


Florida U.S. House District 13

• Includes all of Sarasota, De Soto and Hardee Counties• Parts of Manatee and Charlotte Counties

CYLAB/ISR SEMINAR

APRIL 16, 2007


Voting Methods in District 13 • Manatee, De Soto and Hardee Counties use Diebold opscan• Sarasota and Charlotte Counties use ES&S iVotronic

touchscreen machines (no VVPAT), version 8.0.1.2

Touchscreen

Opscan

CYLAB/ISR SEMINAR

APRIL 16, 2007


Florida U.S. House District 13

• Vern Buchanan (R) beat Christine Jennings (D) by 369 votes out of 238,249 cast, a 0.15% margin

• In Sarasota County, 18,412 ballots showed no vote at all in that race, an undervote of 15%

• Jennings beat Buchanan, 65,487-58,632 in Sarasota• If the 18,412 undervotes followed that percentage

(52.76%-47.24%), Jennings would win by 648 votes• The other counties in District 13 had an average

undervote of 2.5% (range: 2.1-4.0%)

CYLAB/ISR SEMINAR

APRIL 16, 2007


What Happened?

• Jennings has filed suit in Florida. Can she find out?• The Florida Secretary of State ordered an audit. Can

he find out?• Congress is investigating. Can it find out?• What sort of forensic investigation is needed?

CYLAB/ISR SEMINAR

APRIL 16, 2007


A New Election?

• Legal scholars believe Jennings must show there was a machine malfunction to win a new election

• Voter “confusion” is not enough• Fla. Stat. §102.168(4) lists all grounds for a contest:

– (c) Receipt of a number of illegal votes or rejection of a number of legal votes sufficient to change or place in doubt the result of the election.

– (e) Any other cause or allegation which … would show that a person other than the successful candidate was … elected

CYLAB/ISR SEMINAR

APRIL 16, 2007


U.S. Const. Art I, Sec. 5

• “Each House shall be the judge of the elections, returns and qualifications of its own members”

• Election matters are referred to the Committee on House Administration (9 members: 6 Dem, 3 Rep.)

• Federal Contested Elections Act, 2 U.S.C. §318ff.• Chairwoman Millender-McDonald: “Florida law will facilitate

the evaluation of the election contest – to the extent that it provides access to relevant and critical evidence … the House may not have to get involved at all if the state court does a thorough job.”

• Jennings is trying to show that the court is not doing a thorough job. April 13 memorandum.

http://www4.law.cornell.edu/uscode/html/uscode02/usc_sec_02_00000381----000-.html



CYLAB/ISR SEMINAR

APRIL 16, 2007


ES&S iVotronic Voting System• Ballot (eligible candidates)

loaded from infrared device (“personal electronic ballot” – PEB)

• Choices (votes) recorded in 4 places: 3 on the machine, 1 on removable memory device

• Totals printed at polling location AND sent to county on media for tabulation AND retained in machines

• 1498 machines in Sarasota

• Allegheny County uses a later version of iVotronic: 9.

• Touchscreen DRE

CYLAB/ISR SEMINAR

APRIL 16, 2007


Some Possible Explanations

• Software error– Voters cast votes, but no votes were recorded– Unlikely, because 85% of votes were counted– Post-election testing, source code review

• Tampering (malicious software)– Post-election testing– Source code review

• Conscious voter protest– Unlikely, because of comparison demographics– Absentee (opscan) undervote in Sarasota was 2.6%

• Bad ballot layout – voters missed the race– Compare with Charlotte County

The Sarasota Ballot

UNDERVOTE 1.1%

The Sarasota Ballot

UNDERVOTE 15%

UNDERVOTE 1.3%

The Sarasota Ballot

UNDERVOTE 4.4%

UNDERVOTE 4.4%

UNDERVOTE 5.2%

Sarasota Summary Page

CYLAB/ISR SEMINAR

APRIL 16, 2007


Comparison with Charlotte County• Sarasota and Charlotte used the same touchscreen

system• In Sarasota, House and Governor were on the same

screen• In Charlotte, House had its own screen, but Attorney

General and Governor were on the same screen• Sarasota had a 13% undervote for House, but 1.3% for

Governor• Charlotte had a 2.4% undervote in the U.S. House race,

26% undervote for attorney general (would not have made a difference statewide). 41% undervote for Florida House District 71

CYLAB/ISR SEMINAR

APRIL 16, 2007


Ballot Comparison

SARASOTA CHARLOTTE

4.4% UNDERVOTE

26% UNDERVOTE

4.4% UNDERVOTE

5.2% UNDERVOTE

0.7% UNDERVOTE

CYLAB/ISR SEMINAR

APRIL 16, 2007


The Dent Memo (Nov. 3, 2006)

CYLAB/ISR SEMINAR

APRIL 16, 2007


Timeline

• Nov. 3Sarasota SOE letter to precincts warning of potential to overlook the race

• Nov. 7Election Day• Nov. 8Hell breaks loose with 15% undervote• Nov. 9FL Sec’y of State announces audit• Nov. 13 Canvassing commission orders recount• Nov. 20 Canvassing commission certifies election• Nov. 20 Jennings sues in FL to contest election• Nov. 21 Voters sue for new election• Dec. 5Florida forms source code task force

Timeline• Dec. 20 Jennings contests election in Congress• Dec. 26 Florida judge rules against source code access by

Jennings• Jan. 4 Buchanan seated by House of

Representatives• Jan. 4 Jennings appeals denial of source code access to

Fla. Court of Appeal• Jan. 4 Rep. Millender-McDonald urges Court of

Appeal to expedite the case• Jan. 10 Court tells Millender-McDonald to butt out• Feb. 14 House of Representatives forms Subcommittee on

Elections• Feb. 23 Source code task force report released

CYLAB/ISR SEMINAR

APRIL 16, 2007


Secretary of State Audit

1. Review of election, procedures, results, and certification examination

2. Testing machines actually used in election and machines held aside as spares

3. Independent Source Code Review

CYLAB/ISR SEMINAR

APRIL 16, 2007


Post-Election Testing Results

• The machines properly recorded votes• The software was certified and unaltered• The internal audit trail shows the undervotes• No evidence of tampering or vote-dropping• No evidence of vote-flipping

CYLAB/ISR SEMINAR

APRIL 16, 2007


Source Code Task Force

• Florida State University was prime contractor• Alec Yasinsac, FSU

– Director, Security and Assurance in Information Technology Lab

• Ted Baker, FSU– Device drivers, hardware/software interaction

• Matt Bishop, UC Davis– Author: Computer Security: Art & Science

• Mike Burmeister, FSU Co-Director, SAIT

• Breno de Medeiros, FSU Information security

• Michael Shamos, CMU Voting systems examiner

• Gary Tyson, FSU Architecture and compilers • David Wagner, UC Berkeley Secure software, e-voting

CYLAB/ISR SEMINAR

APRIL 16, 2007


Ground Rules

• Total independence from Secretary of State

• All source code provided

• Access to actual voting machines

• Vendor furnished documentation and briefings

• No confidentiality restriction for discoveries relevant to the District 13 race or any system flaws

CYLAB/ISR SEMINAR

APRIL 16, 2007


Evidence Considered

• Source code• Machine behavior• Election statistics• Ballot definition files• Ballot images, electronic files• Election event logs• Court filings, county documents• Poll-worker logs of voter complaints• News stories, blogs• Did not review: firmware of I/O devices, 3rd-party utility

libraries

PEB

CFCF

TF

CompactFlash

Processor

RAM

Video Card

RAM

Firmware

Display Data

DisplayData

Audio Ballots Ballot

Images

Summary Data

Ballot Images

Ballot Style

Removable components are pinkDashed lines are memory mappings

TF – Terminal Flash Memory, PIC = Programmable Interrupt Controller, PEB – Personal Electronic Ballot

Controller

Touch Screen

EPROM

Touches

PIC

InterruptInterrupt

Interrupt

iVotronic Hardware Architecture

SOURCE: TASK FORCE REPORT

Intel 386 EX

TRIPLY REDUNDANT

CYLAB/ISR SEMINAR

APRIL 16, 2007


iVotronic Software Architecture

• NO operating system• Low-level and machine interface code

– Mostly C, some assembly language – all was available• Application code

– All C• COTS

– Very little, e.g. C libraries, driver for compact flash card

CYLAB/ISR SEMINAR

APRIL 16, 2007


iVotronic Software Properties

• Good– No GOTOs

– No dynamic memory allocation

– No multithreading

– Single address space

– Not object-oriented, so no fragile base class problem

– After each voter, processor is reset, program reloaded from EPROM and variables re-initialized

• Bad– No high-level design

– Limited code readability

– Aging code base, numerous updates

– Global variables updated by main program and interrupt handlers

CYLAB/ISR SEMINAR

APRIL 16, 2007


Technical Approach

• Follow the evidence• Consider all proposed hypotheses• We traced program execution

1. Voting machine initialization

2. Voter selections & screen review

3. Ballot image creation

4. Ballot image storage

5. Asynchronous system faults not associated with a voting phase.

• Used Fortify Source Code Analysis (SCA) tool from Fortify Software

CYLAB/ISR SEMINAR

APRIL 16, 2007


Unanimous Findings

• Complete ballot was presented to each voter• All selections presented on review screens• All selections recorded to terminal flash memory• All flash memory selections recorded to external media• No queueing or stacking of interrupts• No malware• No time-sensitive code• No serial race effect

– Race A unaffected by race B for A≠B

• No serial voter effect– Voter n unaffected by voters 1, …, n-1

CYLAB/ISR SEMINAR

APRIL 16, 2007


Vote Flipping

• Some voters reported vote-flipping• Voter presses the square next to a Democrat, but the

square next to the Republican gets marked(Reported widely, especially Broward County, FL)

• This is not caused by malware, but by miscalibrated touchscreens

• How do we know? The problem goes away when the screens are recalibrated.

CYLAB/ISR SEMINAR

APRIL 16, 2007


Touchscreens

1. Sensor

2. Controller

3. Software driver

CYLAB/ISR SEMINAR

APRIL 16, 2007


Resistive Touchscreens1. Polyester Film

2. Upper Resistive Circuit Layer

3. Conductive ITO (Indium-Tin Oxide, transparent metal coating)

4. Lower Resistive Circuit Layer

5. Insulating Dots

6. Glass/Acrylic Substrate

7. Touching the overlay surface causes the (2) Upper Resistive Circuit Layer to contact the (4) Lower Resistive Circuit Layer, producing a circuit switch from the activated area.

8. Touchscreen controller measures alternating voltages between the circuit layers (7) and converts them into the digital X and Y coordinates of the activated area.

Resistive Touchscreens

• Screen is fed clock signals

• Touching the screen creates voltage dividers in two dimensions

• Transient signals from the wires must be interpreted to determine (x, y) coordinates

SOURCE: RICK DOWNS

• Smoothing of the signal is required

• This is done in software by a “smoothing filter”

CYLAB/ISR SEMINAR

APRIL 16, 2007


Calibrating Touchscreens

SOURCE:WWW.EMBEDDED.COM

A circle on the display and in touchscreen coordinates

CYLAB/ISR SEMINAR

APRIL 16, 2007


The Smoothing Filter

• The iVotronic smoothing filter was slow, sometimes 3 seconds until a touch was registered

• Florida’s primary election was on September 5, 2006• About August 21, 2006, the Sarasota Supervisor of Elections

received a letter from the vendor advising of the slow response and suggesting either:

– Installing a new version with a faster filter; or

– Alerting the voters to the slow response

• Sarasota did neither for the primary or the November election

The ES&S Letter (condensed)

CYLAB/ISR SEMINAR

APRIL 16, 2007


The Smoothing Filter Hypothesis

• It is now alleged that the smoothing filter was the cause of the undervote

• Theory: Voters pressed “Jennings.” This did not register immediately, so they pressed it again. This had the effect of selecting and then deselecting Jennings.

• Plausible but incorrect:– Interrupts (touches) are not queued. Only the last touch takes effect.

If a voter touches again before the first touch registers, the second one registers, does not cancel the first.

– If the effect existed, it would have affected other races in Sarasota and other jurisdictions.

– If the effect were widespread (15%), it would have been observed in testing, but was not.

CYLAB/ISR SEMINAR

APRIL 16, 2007


What Caused the Undervote?

• Bad ballot design COMBINED WITH ineffective undervote warning

• WHY DO WE BELIEVE THIS?

• No other hypothesis is confirmed by the facts

• WHAT IS THE FIX?

• Do not allow exit from an undervoted screen without warning and express confirmation

• EFFECT ON PENNSYLVANIA?

• Vendor will not receive any new certification until all vulnerabilities and the undervote warning are repaired

CYLAB/ISR SEMINAR

APRIL 16, 2007


Aftermath

Go hence, to have more talk of these sad things;

Some shall be pardon'd, and some punishèd;

For never was a story of more note

Than this of Jennings and her undervote.

CYLAB/ISR SEMINAR

APRIL 16, 2007


QA&

CYLAB/ISR SEMINAR APRIL 16, 2007 COPYRIGHT © 2007 MICHAEL I. SHAMOS What Happened to 18,000 Votes? Results of the Sarasota Source Code Audit Michael I.

Documents

cylabisr seminar

shamos florida

buchanan jennings slide

shamos outline

memorandum slide

charlotte counties slide

touchscreen opscan slide

house district