Indexing Big Data in the Cloud
Indexing Big Data in the Cloud 2
Me
Scott StultsCo-Founder of OpenSource Connections
Solr / Lucene
Bash / Python / Java
Indexing Big Data in the Cloud 3
Eric
Indexing Big Data in the Cloud 4
Big Data
Indexing Big Data in the Cloud 5
Big Data Wrangler
Indexing Big Data in the Cloud 6
How?
Address a Real ProjectBe Agile
Make Small Mistaeks FastSucceed BIG
Indexing Big Data in the Cloud 7
USPTO Goals
Prototype Search UX
Prove Solr:Scales
IntegratesExcels
Indexing Big Data in the Cloud 8
Scale?
Indexing Big Data in the Cloud 9
Our Approach
KISSYAGNI
(This space intentionally left blank)
Indexing Big Data in the Cloud 10
Minimal Flair
Indexing Big Data in the Cloud 11
Record Everything!
Indexing Big Data in the Cloud 12
Some Numbers
Doc Count 1.1 MillionZip Files 313
Docs per Zip File 4,000
Zip File Size 75M
File Size 300M
Indexing Big Data in the Cloud 13
Testing
Start some serversProcess a batchCheck the clock
Indexing Big Data in the Cloud 14
start_nodes
start_nodes() { ec2-run-instances ami-1b814f72 \ --block-device-mapping '/dev/sdb=snap-48adde35::true' \ --block-device-mapping '/dev/sdi1=:10:false' \ --block-device-mapping '/dev/sdi2=:10:false' \ --block-device-mapping '/dev/sdi3=:20:false' \ --instance-type m1.large \ --key uspto-proto \ --instance-count $MAX_NODES \ --group default > ~/run-output}
Indexing Big Data in the Cloud 15
Gut Check
How fast can we do this?
What can we do in parallel?
Indexing Big Data in the Cloud 16
Scaling
Raise our instance limit
xargs -P GNU parallel
Indexing Big Data in the Cloud 17
Shortcomings
SSH?Error recovery
One Solr
Indexing Big Data in the Cloud 18
Alternatives
CloudFormationPuppet / Chef
Multiple Cores / ShardsHadoop
Indexing Big Data in the Cloud 19
Success
Indexing Big Data in the Cloud 20
Victory Lap
Indexing Big Data in the Cloud 21
Instances / Time
Indexing Big Data in the Cloud 22
Thank You
https://github.com/sstults/patent-indexing
@scottstults#o19s
