Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Retail Reference Architecture: Real-Time, Geo-Distributed Inventory
Sep 08, 2014
Senior Consulting Engineer, MongoDB
Norman Graham
#MongoDBWorld
Retail Reference Architecture: Real-Time, Geo-Distributed Inventory
Inventory
Inventory
MongoDB
External Inventory
Internal Inventory
Regional Inventory
Purchase Orders
Fulfillment
Promotions
Inventory – Traditional Architecture
Relational DBSystem of Records
NightlyBatches
Analytics, Aggregations
,Reports
Caching Layer
Field Inventory
Internal & External
Apps
Point-in-time Loads
Inventory – Opportunities Missed
• Can’t reliably detect availability
• Can't redirect purchasers to in-store pickup
• Can’t do intra-day replenishment
• Degraded customer experience
• Higher internal expense
Inventory – Principles
• Single view of the inventory
• Used by most services and channels
• Read-dominated workload
• Local, real-time writes
• Bulk writes for refresh
• Geographically distributed
• Horizontally scalable
Requirement Challenge MongoDB
Single view of inventory
Ensure availability of inventory
information on all channels and
services
Developer-friendly, document-oriented
storage
High volume, low latency reads
Anytime, anywhere access to inventory
data without overloading the system of record
Fast, indexed readsLocal reads
Horizontal scaling
Bulk updates,intra-day deltas
Provide window-in-time consistency for
highly available services
Bulk writesFast, in-place
updatesHorizontal scaling
Rapid application development
cycles
Deliver new services rapidly to capture new opportunities
Flexible schemaRich query language
Agile-friendly iterations
Inventory – Requirements
Inventory – Target Architecture
Relational DBSystem of Records
Analytics, Aggregations
,Reports
Field Inventory
Internal & External
Apps
Inventory
Assortments
Shipments
Audits
Products
Stores
Point-in-time Loads
NightlyRefres
h
Real-timeUpdates
Horizontal Scaling
Inventory – Technical Decisions
Store
Inventory
Schema
Indexing
Inventory – Collections
Stores InventoryProducts
Audits AssortmentsShipments
> db.stores.findOne(){ "_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"), "className" : "catalog.Store", "storeId" : "store0", "name" : "Bessemer store", "address" : { "addr1" : "1st Main St", "city" : "Bessemer", "state" : "AL", "zip" : "12345", "country" : "US" }, "location" : [ -86.95444, 33.40178 ], ...}
Stores – Sample Document
Stores – Sample Queries
• Get a store by storeId
db.stores.find({ "storeId" : "store0" })
• Get a store by zip code
db.stores.find({ "address.zip" : "12345" })
What’s near me?
Stores – Sample Geo Queries
• Get nearby stores sorted by distance
db.runCommand({
geoNear : "stores",
near : {
type : "Point",
coordinates : [-82.8006, 40.0908] },
maxDistance : 10000.0,
spherical : true
})
Stores – Sample Geo Queries
• Get the five nearest stores within 10 km
db.stores.find({
location : {
$near : {
$geometry : {
type : "Point",
coordinates : [-82.80, 40.09] },
$maxDistance : 10000.0 } }
}).limit(5)
Stores – Indices
• { "storeId" : 1 }, { "unique" : true }
• { "name" : 1 }
• { "address.zip" : 1 }
• { "location" : "2dsphere" }
> db.inventory.findOne(){ "_id": "5354869f300487d20b2b011d", "storeId": "store0", "location": [-86.95444, 33.40178], "productId": "p0", "vars": [ { "sku": "sku1", "q": 14 }, { "sku": "sku3", "q": 7 }, { "sku": "sku7", "q": 32 }, { "sku": "sku14", "q": 65 }, ... ]}
Inventory – Sample Document
Inventory – Sample Queries
• Get all items in a store
db.inventory.find({ storeId : "store100" })
• Get quantity for an item at a store
db.inventory.find({
"storeId" : "store100",
"productId" : "p200"
})
Inventory – Sample Queries
• Get quantity for a sku at a store
db.inventory.find(
{
"storeId" : "store100",
"productId" : "p200",
"vars.sku" : "sku11736"
},
{ "vars.$" : 1 }
)
Inventory – Sample Update
• Increment / decrement inventory for an item at a store
db.inventory.update(
{
"storeId" : "store100",
"productId" : "p200",
"vars.sku" : "sku11736"
},
{ "$inc" : { "vars.$.q" : 20 } }
)
Inventory – Sample Aggregations
• Aggregate total quantity for a product
db.inventory.aggregate( [
{ $match : { productId : "p200" } },
{ $unwind : "$vars" },
{ $group : {
_id : "result",
count : { $sum : "$vars.q" } } } ] )
{ "_id" : "result", "count" : 101752 }
Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "store100" } },
{ $unwind : "$vars" },
{ $match : { "vars.q" : { $gt : 0 } } },
{ $group : {
_id : "result",
count : { $sum : 1 } } } ] )
{ "_id" : "result", "count" : 29347 }
Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "store100" } },
{ $unwind : "$vars" },
{ $group : {
_id : "result",
count : { $sum : "$vars.q" } } } ] )
{ "_id" : "result", "count" : 29347 }
Inventory – Sample Geo-Query
• Get inventory for an item near a point
db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true, limit : 10, query : { "productId" : "p200", "vars.sku" : "sku11736" } } )
Inventory – Sample Geo-Query
• Get closest store with available sku
db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.800672, 40.090844] }, maxDistance : 10000.0, spherical : true, limit : 1, query : { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : { $gt : 0 } } } } } )
Inventory – Sample Geo-Aggregation
• Get count of inventory for an item near a point
db.inventory.aggregate( [ { $geoNear: { near : { type : "Point", coordinates : [-82.800672, 40.090844] }, distanceField: "distance", maxDistance: 10000.0, spherical : true, query: { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : {$gt : 0} } } }, includeLocs: "dist.location", num: 5 } }, { $unwind: "$vars" }, { $match: { "vars.sku" : "sku11736" } }, { $group: { _id: "result", count: {$sum: "$vars.q"} } }])
Inventory – Sample Indices
• { storeId : 1 }
• { productId : 1, storeId : 1 }
• { productId : 1, location : "2dsphere" }
• Why not "vars.sku"?– { productId : 1, storeId : 1, "vars.sku" : 1 }
Horizontal Scaling
Inventory – Technical Decisions
Store
Inventory
Schema
Indexing
ShardEast
ShardCentr
al
ShardWest
East DC
Inventory – Sharding TopologyWest DC Central DC
LegacyInventor
y
Primary
Primary
Primary
Inventory – Shard Key
• Choose shard key– { storeId : 1 } ?– { productId : 1, storeId : 1 } ?– { storeId : 1, productId : 1 } ?
• Set up sharding– sh.enableSharding("inventoryDB")– sh.shardCollection( "inventoryDB.inventory", { storeId : 1, productId : 1 } )
Inventory – Shard Tags
• Set up shard tags– sh.addShardTag("shard0000", "west")– sh.addShardTag("shard0001", "central")– sh.addShardTag("shard0002", "east")
• Set up tag ranges– sh.addTagRange("inventoryDB.inventory", { storeId : 0 }, { storeId : 100}, "west" )– sh.addTagRange("inventoryDB.inventory", { storeId : 100 }, { storeId : 200 }, "central" )– sh.addTagRange("inventoryDB.inventory", { storeId : 200 }, { storeId : 300 }, "east" )
Senior Consulting Engineer, MongoDB
Norman Graham
#MongoDBWorld
Thank You