Preserving Peer Replicas By Rate-Limited Sampled Voting Petros Maniatis, Mema Roussopoulos, TJ Giuli, David Rosenthal, Mary Baker, Yanto Muliandi.

Preserving Peer Replicas Preserving Peer Replicas By Rate-Limited Sampled By Rate-Limited Sampled

VotingVoting

Petros Maniatis, Mema Roussopoulos, Petros Maniatis, Mema Roussopoulos, TJ Giuli, David Rosenthal, Mary Baker, TJ Giuli, David Rosenthal, Mary Baker,

Yanto MuliandiYanto Muliandi

ProblemProblem

Academic publishing is moving to the WebAcademic publishing is moving to the Web

Libraries rent accesses to publisher’s copyLibraries rent accesses to publisher’s copy

But…But…

What if publishers go out of business?What if publishers go out of business?

Solution: LOCKSSSolution: LOCKSS Digital preservation among librariesDigital preservation among libraries Need to address scalability and security Need to address scalability and security

issuesissues

Characteristics of LOCKSSCharacteristics of LOCKSS

Long-term large-scaleLong-term large-scale

Lack of central controlLack of central control

Avoid long-term secrets like encryption Avoid long-term secrets like encryption keyskeys

Resist random failures and deliberate Resist random failures and deliberate attack for a long timeattack for a long time

Design AssumptionsDesign Assumptions

Storage is unreliableStorage is unreliable

Third-party reputation is problematicThird-party reputation is problematic Vulnerable to slander and subversionVulnerable to slander and subversion Can cash in a history of good behaviorCan cash in a history of good behavior

Strong adversaryStrong adversary Need to prepare for unforeseen attacksNeed to prepare for unforeseen attacks

Design PrinciplesDesign Principles

No long-term secretsNo long-term secrets Secrets require storage that is effectively Secrets require storage that is effectively

impossible to replicate, audit, repair, or impossible to replicate, audit, repair, or regenerateregenerate

Use inertiaUse inertia Rate-limit changesRate-limit changes

Design PrinciplesDesign Principles

Reduce predictabilityReduce predictability

Intrinsic intrusion detectionIntrinsic intrusion detection Bimodal behaviorBimodal behavior

The Existing LOCKSS SystemThe Existing LOCKSS System

Use persistent Web cachesUse persistent Web caches Crawl the journal websites Crawl the journal websites Distribute to local readersDistribute to local readers Preserve by cooperating with other cachesPreserve by cooperating with other caches

Use “opinion polls” in a peer-to-peer Use “opinion polls” in a peer-to-peer networknetwork Compare the hash values of specified part of Compare the hash values of specified part of

the contentthe content

The Opinion PollsThe Opinion Polls

Provide content authenticity and integrityProvide content authenticity and integrity Based on independently obtained copiesBased on independently obtained copies

Peers vote on large archived units (AUs)Peers vote on large archived units (AUs) An AU is checked every three monthsAn AU is checked every three months With ~17 peersWith ~17 peers

Only repair a replica if it participated in the Only repair a replica if it participated in the pastpast Prevent free-loading and theftPrevent free-loading and theft

The New Opinion Poll ProtocolThe New Opinion Poll Protocol

AssumptionsAssumptions Each peer uses one of a number of Each peer uses one of a number of

independent implementations of the LOCKSS independent implementations of the LOCKSS protocol to limit common-mode failuresprotocol to limit common-mode failures

Each peer’s AU is subject a low rate of Each peer’s AU is subject a low rate of undetected random damageundetected random damage

Polling rate >> random damage ratePolling rate >> random damage rate

The New Opinion Poll ProtocolThe New Opinion Poll Protocol

DefinitionsDefinitions Malign peer: one tries to subvert the systemMalign peer: one tries to subvert the system Loyal peer: one that follows the LOCKSS Loyal peer: one that follows the LOCKSS

protocol at all timesprotocol at all times Damaged peer: a loyal peer with a damage Damaged peer: a loyal peer with a damage

AUAU Healthy peer: a loyal peer with the correct AUHealthy peer: a loyal peer with the correct AU

Goal: high probability of healthy peers Goal: high probability of healthy peers despite failures and attacksdespite failures and attacks

The Idea of PollingThe Idea of Polling

A peer invites a small subset of the peers A peer invites a small subset of the peers it has recently encounteredit has recently encountered

Each computes a fresh digest of its AUEach computes a fresh digest of its AU

If the caller of the pool receives votes that If the caller of the pool receives votes that overwhelmingly agree with its own versionoverwhelmingly agree with its own version Do nothingDo nothing

The Idea of PollingThe Idea of Polling

If the caller of the pool receives votes that If the caller of the pool receives votes that overwhelmingly disagree overwhelmingly disagree Ask for a copy to repair its ownAsk for a copy to repair its own Vote againVote again

If the result of the poll is neither a landslide If the result of the poll is neither a landslide win nor a landslide loss, then the caller win nor a landslide loss, then the caller raises an alarm to attract human attention raises an alarm to attract human attention to the situationto the situation

Voting MembershipVoting Membership

Inner circleInner circle Decide the poll outcomeDecide the poll outcome

Outer circleOuter circle Nominated by inner circleNominated by inner circle May become members of the inner circle in May become members of the inner circle in

the futurethe future

Sybil-Attack PreventionsSybil-Attack Preventions

Sybil attack: Use an unlimited number of forged Sybil attack: Use an unlimited number of forged identities to subvert a systemidentities to subvert a system Prevention schemes:Prevention schemes: Infrequent voting (Limits the rate of change in the Infrequent voting (Limits the rate of change in the

systemsystem Bimodal distribution of system states (increase the Bimodal distribution of system states (increase the

chance to trigger alarms)chance to trigger alarms) Require each peer to expend significant computing Require each peer to expend significant computing

power for each steppower for each stepComputing the hash for an AUComputing the hash for an AU

Churn (to be explained later)Churn (to be explained later)

DetailsDetails

Each peer maintains two listsEach peer maintains two lists Reference listReference list

Recently encountered peersRecently encountered peers Friends listFriends list

Peers with out-of-band relationshipPeers with out-of-band relationship

BootstrappingBootstrapping

Copy all entries from its current friends list Copy all entries from its current friends list into its reference listinto its reference list

Each reference has a random expiration Each reference has a random expiration timetime

Poll InitiationPoll Initiation

Choose N random peers from the Choose N random peers from the reference list (inner circle)reference list (inner circle)Send encrypted poll messagesSend encrypted poll messagesRemove peers that cannot answer the Remove peers that cannot answer the challenge-response questions within a challenge-response questions within a specified time frame from the inner circlespecified time frame from the inner circleIf too few inner circle members, invites If too few inner circle members, invites additional peers from the reference listadditional peers from the reference listAbort when the reference list is exhausted Abort when the reference list is exhausted

Poll EffortPoll Effort

Receiver must solve a puzzle to show Receiver must solve a puzzle to show effort effort Make it computationally difficult for attackers Make it computationally difficult for attackers

to forge multiple identitiesto forge multiple identities

Inner circle also nominates outer circle Inner circle also nominates outer circle members members Every inner circle nominator affects the outer Every inner circle nominator affects the outer

circle equallycircle equally Initiator also polls outer circle membersInitiator also polls outer circle members

Vote VerificationVote Verification

If the proof of effort is incorrect, the vote is If the proof of effort is incorrect, the vote is invalid, and the peer if black listedinvalid, and the peer if black listed

If the proof is correct, and the hash If the proof is correct, and the hash matches, it is valid and agreeingmatches, it is valid and agreeing

If the proof is correct, and the hash If the proof is correct, and the hash mismatches, it is valid and disagreeingmismatches, it is valid and disagreeing

Vote TabulationVote Tabulation

Agreeing votes are smaller than a Agreeing votes are smaller than a threshold (landslide loss), the initiator threshold (landslide loss), the initiator needs to repair its copyneeds to repair its copy

Agreeing votes are greater than a Agreeing votes are greater than a threshold (landslide win), the initiator threshold (landslide win), the initiator updates its reference list and schedules updates its reference list and schedules the next pollthe next poll

Otherwise, raise an alarmOtherwise, raise an alarm

Inter-poll AlarmInter-poll Alarm

Triggered if an initiator fails to collect Triggered if an initiator fails to collect enough votes for a long timeenough votes for a long time

RepairRepair

Need to detect inconsistencies between Need to detect inconsistencies between the voting information and the repaired AUthe voting information and the repaired AU

If initiator cannot complete the repair If initiator cannot complete the repair process, raise the corresponding alarmprocess, raise the corresponding alarm

Reference List UpdateReference List Update

Remove all disagreeing peers and some Remove all disagreeing peers and some randomly chosen agreeing peers from the randomly chosen agreeing peers from the inner circleinner circleResets the expiration time for the Resets the expiration time for the remaining peersremaining peersInsert all outer circle peers whose votes Insert all outer circle peers whose votes were valid and agreeingwere valid and agreeingInsert randomly chosen entries from Insert randomly chosen entries from friends list up to a churn factorfriends list up to a churn factor

Vote ConstructionVote Construction

Consists of a hash of AU and interleaved Consists of a hash of AU and interleaved with provable computational effortwith provable computational effort

Vote computation is divided in rounds, Vote computation is divided in rounds, each with computational effort and the each with computational effort and the hashed portion double in sizehashed portion double in size

A subsequent challenge is dependent on A subsequent challenge is dependent on the previous challengethe previous challenge

Protocol AnalysisProtocol Analysis

Need to achieve the followingNeed to achieve the following Prevent one from gaining a footholdPrevent one from gaining a foothold Make it expensive for the adversary to waste Make it expensive for the adversary to waste

another peer’s resourcesanother peer’s resources Make it likely for attacks to be detectedMake it likely for attacks to be detected

Effort SizingEffort Sizing

Use memory-bound computationsUse memory-bound computations

An initiator needs to expend more effort An initiator needs to expend more effort than the cumulative effort it imposes on than the cumulative effort it imposes on the voters the voters

Timeliness of EffortTimeliness of Effort

Only proofs of recent effort can affect the Only proofs of recent effort can affect the systemsystem

Need to expend resources to maintain Need to expend resources to maintain footholdfoothold

Rate LimitingRate Limiting

Loyal peers call polls autonomously and Loyal peers call polls autonomously and infrequentlyinfrequently

The rate of progress for an attack is limited The rate of progress for an attack is limited by victims, not by attackersby victims, not by attackers

Reference List ChurningReference List Churning

Avoid depending on a fixed set of peersAvoid depending on a fixed set of peers They become easy targetsThey become easy targets

Avoid depending on entirely on random Avoid depending on entirely on random peerspeers They can launch Sybil attacksThey can launch Sybil attacks

With friends listWith friends list Attackers can gain foothold on the outer circle Attackers can gain foothold on the outer circle

list but not the friends listlist but not the friends list

Obfuscation of Protocol StateObfuscation of Protocol State

Encrypt all but the first protocol message Encrypt all but the first protocol message exchanged by a poll initiator and each exchanged by a poll initiator and each potential voterpotential voter

Make all loyal peers invited into a poll, Make all loyal peers invited into a poll, even those who decline to voteeven those who decline to vote

Can’t deduce the number of loyal peers Can’t deduce the number of loyal peers who are involved in deciding the outcome who are involved in deciding the outcome of a pollof a poll

AlarmsAlarms

Raising an alarm is expensiveRaising an alarm is expensive Involve human examinationsInvolve human examinations

If an attacker’s goal is to raise alarms….If an attacker’s goal is to raise alarms….

Adversary AnalysisAdversary Analysis

Complete parameter knowledgeComplete parameter knowledgeExploitation of common peer vulnerabilityExploitation of common peer vulnerability Take over a fraction of populations running Take over a fraction of populations running

the same implementationthe same implementation

Unconstrained identitiesUnconstrained identities Infinite IP addressesInfinite IP addresses

StealthStealth One cannot discern loyal peers from One cannot discern loyal peers from

compromised onescompromised ones

Adversary AnalysisAdversary Analysis

Total information awarenessTotal information awareness Identities of all malign peersIdentities of all malign peers

Perfect work balancingPerfect work balancingPerfect digital preservationPerfect digital preservation Incorruptible copies of good and bad AusIncorruptible copies of good and bad Aus

Local eavesdroppingLocal eavesdroppingLocal spoofingLocal spoofing One end of the communication needs to be in One end of the communication needs to be in

the local networkthe local network

Adversary AttacksAdversary Attacks

Platform attacksPlatform attacks Can take over a fraction of peers Can take over a fraction of peers

instantaneouslyinstantaneously

Protocol attacksProtocol attacks Play against the LOCKSS protocolPlay against the LOCKSS protocol

Protocol AttacksProtocol Attacks

Stealth modificationStealth modification Replace good AUs with bad onesReplace good AUs with bad ones

NuisanceNuisance Raise many alarmsRaise many alarms

AttritionAttrition Prevent loyal peers from repairsPrevent loyal peers from repairs

TheftTheft Obtain published content without payingObtain published content without paying

Protocol AttacksProtocol Attacks

Free-loadingFree-loading Obtain services without supplying services in Obtain services without supplying services in

returnreturn

Counter-Attack TechniquesCounter-Attack Techniques

Adversary foothold in a reference listAdversary foothold in a reference list Need to wait for invitation to voteNeed to wait for invitation to vote Need to behave well for a long time before the Need to behave well for a long time before the

attack (without raising alarms)attack (without raising alarms)

Vote base on good AU, supply the bad AU Vote base on good AU, supply the bad AU for repairfor repair Ask random sample bits (verified) before each Ask random sample bits (verified) before each

pollpoll The repair AU must match the initial bitsThe repair AU must match the initial bits

Stealth Modification Attack StrategyStealth Modification Attack Strategy

Two phasesTwo phases Lurk to build a foothold in loyal peers’ Lurk to build a foothold in loyal peers’

reference listsreference lists AttackAttack

Need to have the majority of votesNeed to have the majority of votes

Need to have loyal peers < the alarm Need to have loyal peers < the alarm thresholdthreshold

An adversary…An adversary…

Needs to wait for an initiator to call for Needs to wait for an initiator to call for votesvotes

Needs to go through many rounds of Needs to go through many rounds of voting without triggering an alarmvoting without triggering an alarm

Needs to expend effort to maintain the Needs to expend effort to maintain the foothold in the reference listfoothold in the reference list

SimulationSimulation

Running LOCKSS for 30 yearsRunning LOCKSS for 30 years

1000 peers1000 peers Clusters of 30 peersClusters of 30 peers 29 peers in the initial friends list29 peers in the initial friends list

80% from the local cluster80% from the local cluster

20 years of lurking20 years of lurking

10 years of attacking10 years of attacking

ResultsResults

Low rates of false alarms in the absence Low rates of false alarms in the absence of attacksof attacks

Can sustain up to 1/3 of the peers Can sustain up to 1/3 of the peers subverted (with 10% churn)subverted (with 10% churn)

System degrades gracefullySystem degrades gracefully