PATENTLY INNOVATIVEFinding where innovation lives!
!!
David E Drummond Insight Data Engineer
INTRO
Which state has more innovation? Use number of patents to see which regions are “patently innovative”.
DATA PIPELINE Ingestion Batch Processing Real-Time Queries
XML, TSVData Cleansing
JSON Hive SerDe
HappyBase
DATA PIPELINE Ingestion Batch Processing Real-Time Queries
XML, TSVData Cleansing
JSON Hive SerDe
HappyBase
DATA SET ~90 MB/WEEK ~10 MILLION PATENTS >45GB
TSV XML
CONVERT TO JSON AND PARSE
NODE PARALLELIZATION
.PY/.SH
Name Node
Worker Nodes
SINGLE LINE JSON RECORDS
DATA PIPELINE Ingestion Batch Processing Real-Time Queries
XML, TSVData Cleansing
JSON Hive SerDe
HappyBase
JSON HIVE SER-DE
CLEANED TABULAR DATA
DATA PIPELINE Ingestion Batch Processing Real-Time Queries
XML, TSVData Cleansing
JSON Hive SerDe
HappyBase
HBASE SCHEMA
State 2005 2006 2007 … 2011 2012 2013CA 8530 7411 7120 … 7849 7799 9185TX 2167 1961 1806 … 2050 2121 2500
State 200501 200502 … 201408 201409CA 512 538 … 1380 1194
Denormalized schema for faster queries
TX 102 217 … 350 263
Yearly
Monthly
DAVID DRUMMONDEarned Ph.D in Physics from UC Riverside,
simulating fault tolerant parallel Quantum Computing systems. !
Love to travel and learn about everything!
