Couchbase at Cox Automotive – Couchbase Live New York 2015
Post on 27-Jan-2017
673 Views
Preview:
Transcript
Couchbase at Cox Automotive
3
Who we are
4
Who we are
5
Who we are
6
What we do at
7
What we do at
8
What we do at
9
Our data changes quickly
Couchbase at Cox AutomotiveKeeping pace with our data
11
What sort of data do we store?
12
What sort of data do we store?
13
What sort of data do we store?
14
How do we keep pace with our data?
15
How do we keep pace with our data?
16
How do we keep pace with our data?
17
How do we keep pace with our data?
18
How do we keep pace with our data?
Two weeks to get to production!
Couchbase at Cox AutomotiveAccelerating our pace
20
We churn through truckloads of data
21
We churn through truckloads of data
22
An API with Couchbase storage
23
An API with Couchbase storage
24
An API with Couchbase storage
25
Implementation
API design features
• RESTful endpoints• Use of JSON format and immutable documents• Cached access to SQL Server look ups for configuration• Couchbase storage for speed and flexibility
26
It’s simple to move our data into documents
SQL Server crashes when we try to list all our “staging” tables!
27
It’s simple to move our data into documents{ "ImageURLs": [ "http://www.xhjsdfk.com/15234.jpg", "http://www.xhjsdfk.com/93784.jpg", "http://wwwxhjsdfk.com/13378.jpg" ], "Properties": { "BodyStyle": "2dr Car", "TransmissionSpeed": 6, "TransmissionTypeCode": "6-Speed Ma", "VehicleCondition": "New", "Year": 2021, }, "VIN": "1TAGP28135G148557" }, "Timestamp": "2015-10-05T13:38:46.2979624-04:00", "Version": 1}
JSON lends itself to flexible storage
There’s no schema
28
It’s simple to move our data into documents
We coded an additive storage process.
Say we get this input first….
29
It’s simple to move our data into documents
And then we get this data, for the same vehicle…
30
It’s simple to move our data into documents
We produce a resulting document…
31
It’s simple to move our data into documents
Now we’ve solved our
archiving problem andour rollback problem!
Version 1 Version 2
32
How we got operational in two data centers
Now we use Couchbase as our database of record for this data,
we want resiliency with XDCR
33
How we got operational in two data centers
Remember that TTL setting for the cached data?
Here’s some free advice
Couchbase at Cox AutomotiveStaying at the front of the pack
35
Wow; we’re maxed out again!
We have a fancy SQL Server*doing the heavy lifting
- 4 TB of primary data- Gigabytes of change per minute- 60 CPU cores!- 2TB RAM- 12TB Fusion IO cards!
*cluster of 3 machines
36
Data model migration
Forward plans for SQL and NoSQL
OPERATIONAL CACHEDCOMPUTED
RAW DATA
37
What functions should Couchbase serve?
• Easing the load for heavy-read objects from SQL Server• Applications where ‘eventually consistent’ is OK• Outbound ‘file’ storage for computed documents
• Novel queries against unstructured documents can leverage N1QL• When the query load is unpredictable, it can scale fast
38
Future steps: How to migrate production data?
Our traditional approach
PRODUCTION STAGINGRESTORE
QA ENV.
C# APP
39
Future steps: How to migrate production data?
With Couchbase 4.0
PRODUCTION STAGINGRESTORE
QA ENV.
XDCRFilter
Couchbase at Cox AutomotiveConclusion
41
How we began with Couchbase
“Start where you are.”- Chris Gardner
Couchbase at Cox AutomotiveQuestions
top related