1 A Vision for the Testing A Vision for the Testing of Election Systems in a of Election Systems in a HAVA World HAVA World Eric Lazarus [email protected]
1 A Vision for the Testing of Election Systems in a HAVA World Eric Lazarus [email protected].
Jan 02, 2016
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
A Vision for the Testing of Election A Vision for the Testing of Election
Systems in a HAVA WorldSystems in a HAVA World
Eric Lazarus
2
How Rate a Testing Capability?How Rate a Testing Capability?
Transparent Identification Recommendation Cost effective Broad coverage
– Reliability– Accessibility– Usability – Security
Encourage high-value innovation
Pick correct structure given success
3
This is a tough problemThis is a tough problem
1983 Turing Award Lecture – Ken Thompson showed that conventional methods will fail
A Trojan Horse can live in a compiler, linker, loader, interpreter, micro code, BIOS, hardware…
Testing is hard and limited
4
Types of TestingTypes of Testing Acceptance/Qualification
Testing Code inspections/code walk
through Concurrency testing Data table testing Disability Access Testing
(Variation of usability testing) Installation testing Integration Testing Legal Validation/Verification
(Validate legal requirements then verify legal requirements met)
Load/Stress Testing Performance testing (test
response times) Recovery testing
Regression testing Reliability Testing Scalability testing (variation
of load/stress testing) Security Testing / Penetration
Testing Spike testing (Variation of
Load/Stress Testing) Uninstallation testing
(variation of installation testing)
Unit Testing Upgrade/Patch testing
(variation of installation testing)
Usability testing
5
Applied Common Sense Applied Common Sense
Vision is not hard to come byCreate a vision
– What are the questions?– What are common-sense answers?
Bring together smart people to think about the obvious vision
6
Q: States Testing Independently? Q: States Testing Independently?
go it alone, or…or Voluntary Consortium of States?
7
Q: States Testing Independently? Q: States Testing Independently?
Voluntary Consortium of States:– Hire more and/or better people– Save $ on duplicated effort – Better shared knowledge gained in
• Product evaluation • Use
8
Q: Who should pay for it?Q: Who should pay for it?
Not vendor funded as with ITA system– Interest clash – Barrier to new entries
Pooled state election money What about others including
– Political parties – Good government groups – Civil rights groups – Academic institutions
9
Q: Big-Bang or Continuous? Q: Big-Bang or Continuous?
10
Q: Big-Bang or Continuous? Q: Big-Bang or Continuous?
Like getting regular checkups Nevada gaming control board takes
machines out of service
11
Q: White Box or Black Box?Q: White Box or Black Box?
Why handicap our testers by not giving them source?
We want to find bugs – source code review is good for this
Every branch much be run – too many to realistically be done in voting system software
12
Q: Partisans Included?Q: Partisans Included?
Brennan Center for Justice projects worked both ways
Working with people on both sides of debates has brought out insights
Smart and knowledgably is important – such people often have opinions
13
Q: Team must have…Q: Team must have…
Understand election processes Understand computer security techniques Testing in other domains Background from other industries
including gaming International perspective Heterogeneous team how do find
problems
14
Q: Product RoadmapQ: Product Roadmap
Can election officials impact product direction via a consortium?
15
Q: Consortium Services?Q: Consortium Services?
What can they offer?
16
Q: Develop Risk ModelsQ: Develop Risk Models
Testing should be driven by clear view of the risks testing is attempting to address– “We might buy a machine that is not as
accessible as we are told.”– “…not as secure.”– “…not as reliable.” – “…not as easy to administer.”
Good to develop and maintain these jointly
17
Q: Shared Repository of Knowledge?Q: Shared Repository of Knowledge?
What was learned under testing? What was learned in use? What procedures work well with this
technology? Model: Information Sharing and Analysis
Centers (ISACs) e.g., Financial Services Information Sharing and Analysis Center www.fsisac.com/about.htm
18
Q: Evaluating Election Procedures? Q: Evaluating Election Procedures?
Could this same team evaluate procedure manuals?
Should be able to evaluate procedures against best practices
19
Q: Testing When?Q: Testing When?
Product Evaluation Certification AcceptanceLogic & AccuracyContinuous
20
Q: Other services? Q: Other services?
Negotiate joint purchasing agreements (like GSA Schedule)– Products– Services
Transparency: Arrange for purchasable by responsible organizations
Encourage innovation by– Adhering to open standards
21
Q: Make policy?Q: Make policy?
Should such consortia of states do testing and provide testing information or should they take on policy making role?
22
Q: Make policy?Q: Make policy?
Should such consortia of states do testing and provide testing information or should they take on policy making role? – I’ve been assuming that these staff would
make no policy but only provide the results of their tests. They would not, for example, certify or decertify machines but would report on results of testing.
23
So one vision emergesSo one vision emerges
Multiple states group into a consortium (or two)
Has own staff and/or consultants, small contractors, academics
Performs Testing for:– Usability – Security
Evaluates– Procedures– New technology – Cost
24
Does this make sense?Does this make sense?
Very interested in collaborating around a proposal to create a consortium
How can we improve this vision? Please contact me if you want to work on
this
25
Testing is Testing is notnot an end in itself an end in itself
Current State
GOAL= ImprovedElections
Testing
Skills
Authority
Resources
Commitment
26
Illustration: Gaming – What’s Different?Illustration: Gaming – What’s Different?
Ladder of trust with signed firmware at bottom Multiple people with different keys Field trails as part of certification Hash compare in the field randomly every two years Auditing the auditors Certification done by government employees willing to
share/discuss their methods Post-employment restrictions on working for vendors Penalties for messing up Assumption of cheating
Related Documents
