Data Management 3: Bulletproof Data Management

Post on 25-Jul-2015

155 Views

Category:

Technology

2 Downloads

Preview:

Click to see full reader

Transcript

Buzz MoschettiEnterprise Architect

buzzmoschettimongodbcombuzzmoschetti

BulletproofData Management

2

Part 3 In The Data Management Series

Validating Data

Software Best Practices

Safe Leverage

From RelationalTo MongoDB

ConqueringData Proliferation

BulletproofData Management

ccedilΩ

Part1

Part2

Part3

3

Congratulations At this Point Yoursquove

bull Created a Data Designbull Migrated Databull Built a PoC or maybe an Appbull Explored Operations

4

The Next Stage Defend amp Leverage

bull Document Validationbull Redactionbull Quality Of Service

5

MongoDB Doesnrsquot Have These Things

bull Document Validationbull Redactionbull Quality Of Service

YET^

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

  • Slide 1
  • Part 3 In The Data Management Series
  • Congratulations At this Point Yoursquove
  • The Next Stage Defend amp Leverage
  • MongoDB Doesnrsquot Have These Things
  • What Can We Do Today
  • Write Some Code
  • Starting Point The Data Access Layer
  • Document Validation
  • A Query Filters Outbound Data
  • How About Using It To Filter Inbounds
  • $exists And $type Already in MQL
  • hellip And MQL Goes Way Beyondhellip
  • A New MQL Validator Module Emerges
  • Migrating Capability into MongoDB
  • Code For The FuturehellipToday
  • But What About Today
  • Temporary Filling PQL
  • Bridge MQL to PQL
  • No PQL No Problem
  • MQL Is Easy To Navigate
  • Where Do Validations Come From
  • The Validations Collection
  • Various ldquoLevelsrdquo of Validation
  • Multiple Types Schema By Example
  • The Stack So Far
  • Representative Example
  • Redaction
  • Concept Post Query Operations (PQO)
  • Adopt MQL-like behavior
  • A New PostQuery Module Emerges
  • Where Do PQOs Come From
  • The Postquery Collection
  • The Stack Is Getting Rich
  • Representative Example (2)
  • Quality of Service
  • QOS In Action
  • Where Do We Store QOS Values
  • The QOS Collection
  • Coming Togetherhellip
  • Representative Example (3)
  • A Highly Leveragable Investment
  • Not Just Java Not A Problem
  • A Really Nice Stack
  • What Can We Do
  • Secure Access To Redacted Data for Testing
  • Get It Programmatically Too
  • Robust Validated Data Ingest
  • Concept The control_ Collection
  • Validation QOS and PQO via Web Services
  • Are We Excited Yet
  • Questions amp Answers
  • Slide 53
  • Concept DataProvider
  • The RESTful Provider

    2

    Part 3 In The Data Management Series

    Validating Data

    Software Best Practices

    Safe Leverage

    From RelationalTo MongoDB

    ConqueringData Proliferation

    BulletproofData Management

    ccedilΩ

    Part1

    Part2

    Part3

    3

    Congratulations At this Point Yoursquove

    bull Created a Data Designbull Migrated Databull Built a PoC or maybe an Appbull Explored Operations

    4

    The Next Stage Defend amp Leverage

    bull Document Validationbull Redactionbull Quality Of Service

    5

    MongoDB Doesnrsquot Have These Things

    bull Document Validationbull Redactionbull Quality Of Service

    YET^

    6

    What Can We Do Today

    Write Some Code

    1 Focus on interfaces2 Design for change3 Keep application data access layer

    data management logic and database io well-factored

    4 Minimize compile-time binding

    8

    Starting Point The Data Access Layer

    MongoDBJava Driver

    Data Access Layer

    Application

    class DataAccessLayer private String authenicatedID private String effectiveID private Role role

    init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

    List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

    DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

    while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

    9

    Document Validation

    10

    A Query Filters Outbound Data

    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

    11

    How About Using It To Filter Inbounds

    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

    12

    $exists And $type Already in MQL

    ldquonamerdquo$type2

    $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

    $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

    Ensure ldquonamerdquo exists (because not null) and is a string

    ldquoagerdquo optional but if exists must be a 32bit integer

    ldquonamerdquo required as string and weight and height both required integers or both not present

    13

    hellip And MQL Goes Way Beyondhellip

    $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

    14

    A New MQL Validator Module Emerges

    class MQLValidator ValidationResult validate(Map MQL Map data)

    MongoDBJava Driver

    Data Access Layer

    Application

    Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

    MQLValidator

    15

    MongoDBDB Engine

    Migrating Capability into MongoDB

    MongoDBJava Driver

    MQLValidatorJava

    Data Access Layer

    MongoDBDB Engine

    MongoDBJava Driver

    MQLValidator

    JavaData Access

    Layer

    bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

    and languages

    MongoDBPython Driver

    Application Application

    16

    Code For The FuturehellipToday

    class DataAccessLayer

    someWriteOperation(Map data)

    if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

    17

    But What About Today

    MQLValidator

    18

    Temporary Filling PQL

    bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

    PQL

    19

    Bridge MQL to PQL

    class MQLValidator private PQLFilter pqlfilter

    validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

    20

    No PQL No Problem

    class MQLValidator

    validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

    21

    MQL Is Easy To Navigate

    $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

    bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

    22

    Where Do Validations Come From

    The Database

    23

    The Validations Collection

    gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

    24

    Various ldquoLevelsrdquo of Validation

    gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

    25

    Multiple Types Schema By Example

    gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

    26

    The Stack So Far

    MongoDBJava Driver

    MQLValidator

    Data Access Layer

    Application

    ValidatorDBUtils

    ValidatorDBUtils populates an MQLValidator object from MongoDB

    PQLFilter

    27

    Representative Example

    class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

    someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

    28

    Redaction

    29

    Concept Post Query Operations (PQO)

    ssn $hash model birthdate null

    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

    30

    Adopt MQL-like behavior

    ldquossnrdquonull

    ldquoaddressrdquo ldquoXXXXrdquo

    ldquossnrdquo $substitute ldquossnmodelrdquo

    Remove field by setting to null

    Redact address with fixed value

    Substitute SSN with a different correct consistent value

    ldquocounterpartyrdquo $hash ldquoMD5rdquo

    Hash counterparty name to consistent value

    31

    A New PostQuery Module Emerges

    class PostQuery process(Map data Map operations)

    PostQuery

    MongoDBJava Driver

    MQLValidator

    Data Access Layer

    Application

    ValidatorDBUtils

    PQLFilter

    32

    Where Do PQOs Come From

    The Database

    33

    The Postquery Collection

    gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

    34

    The Stack Is Getting Rich

    PostQuery

    MongoDBJava Driver

    MQLValidator

    Data Access Layer

    Application

    ValidatorDBUtils

    PQLFilter

    PQODBUtils

    35

    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

    someWriteOperation(Map data) hellip

    someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

    36

    Quality of Service

    37

    QOS In Action

    class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

    Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

    38

    Where Do We Store QOS Values

    The Database

    39

    The QOS Collection

    gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

    ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

    40

    QOSDBUtils

    Coming Togetherhellip

    PostQuery

    MongoDBJava Driver

    MQLValidator

    Data Access Layer

    Application

    ValidatorDBUtils

    PQLFilter

    PQODBUtils

    QOS

    41

    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

    someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

    42

    QOSDBUtils

    A Highly Leveragable Investment

    PostQuery

    MQLValidator

    Data Access Layer 1

    Application1

    ValidatorDBUtils

    PQLFilter

    PQODBUtils

    QOS

    Application2

    Data Access Layer 2

    Application3

    Application4

    Data Access Layer 3

    Application5

    Application6Reusable For ALL Data Access Layer Logic

    43

    Not Just Java Not A Problem

    DAL operations have little or no statehellip

    Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

    Can you say hellip Web Service

    44

    A Really Nice Stack

    MongoDBJava Driver

    MQLValidator

    Data Access Layer

    JavaApplication

    ValidatorDBUtils

    PQLFilter

    HTTP Endpoint

    pythonApplication

    curl

    JSON over HTTP(S)

    JSONlt-gtJava Maps

    QOSDBUtils

    PostQuery

    PQODBUtils

    QOS

    45

    What Can We Do

    46

    Secure Access To Redacted Data for Testing

    $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

    $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

    $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

    47

    Get It Programmatically Too

    This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

    getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

    String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

    48

    Robust Validated Data Ingest

    $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

    $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

    49

    Concept The control_ Collection

    gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

    bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

    50

    Validation QOS and PQO via Web Services

    MongoDBJava Driver

    MQLValidator

    Data Access Layer

    JavaApplication

    ValidatorHTTPUtils

    PQLFilter

    pythonApplication

    curl

    JSON over HTTP(S)

    QOSHTTPUtils

    PostQuery

    PQOHTTPUtils

    QOS

    HTTP Service

    JSONlt-gtJava Maps

    51

    Are We Excited Yet

    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

    and other Java resources

    Questions amp Answers

    54

    Concept DataProvider

    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

    Class MongoProvider implements DataProvider hellip

    Class RESTfulProvider implements DataProvider hellip

    55

    The RESTful Provider

    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

    • Slide 1
    • Part 3 In The Data Management Series
    • Congratulations At this Point Yoursquove
    • The Next Stage Defend amp Leverage
    • MongoDB Doesnrsquot Have These Things
    • What Can We Do Today
    • Write Some Code
    • Starting Point The Data Access Layer
    • Document Validation
    • A Query Filters Outbound Data
    • How About Using It To Filter Inbounds
    • $exists And $type Already in MQL
    • hellip And MQL Goes Way Beyondhellip
    • A New MQL Validator Module Emerges
    • Migrating Capability into MongoDB
    • Code For The FuturehellipToday
    • But What About Today
    • Temporary Filling PQL
    • Bridge MQL to PQL
    • No PQL No Problem
    • MQL Is Easy To Navigate
    • Where Do Validations Come From
    • The Validations Collection
    • Various ldquoLevelsrdquo of Validation
    • Multiple Types Schema By Example
    • The Stack So Far
    • Representative Example
    • Redaction
    • Concept Post Query Operations (PQO)
    • Adopt MQL-like behavior
    • A New PostQuery Module Emerges
    • Where Do PQOs Come From
    • The Postquery Collection
    • The Stack Is Getting Rich
    • Representative Example (2)
    • Quality of Service
    • QOS In Action
    • Where Do We Store QOS Values
    • The QOS Collection
    • Coming Togetherhellip
    • Representative Example (3)
    • A Highly Leveragable Investment
    • Not Just Java Not A Problem
    • A Really Nice Stack
    • What Can We Do
    • Secure Access To Redacted Data for Testing
    • Get It Programmatically Too
    • Robust Validated Data Ingest
    • Concept The control_ Collection
    • Validation QOS and PQO via Web Services
    • Are We Excited Yet
    • Questions amp Answers
    • Slide 53
    • Concept DataProvider
    • The RESTful Provider

      3

      Congratulations At this Point Yoursquove

      bull Created a Data Designbull Migrated Databull Built a PoC or maybe an Appbull Explored Operations

      4

      The Next Stage Defend amp Leverage

      bull Document Validationbull Redactionbull Quality Of Service

      5

      MongoDB Doesnrsquot Have These Things

      bull Document Validationbull Redactionbull Quality Of Service

      YET^

      6

      What Can We Do Today

      Write Some Code

      1 Focus on interfaces2 Design for change3 Keep application data access layer

      data management logic and database io well-factored

      4 Minimize compile-time binding

      8

      Starting Point The Data Access Layer

      MongoDBJava Driver

      Data Access Layer

      Application

      class DataAccessLayer private String authenicatedID private String effectiveID private Role role

      init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

      List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

      DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

      while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

      9

      Document Validation

      10

      A Query Filters Outbound Data

      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

      11

      How About Using It To Filter Inbounds

      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

      12

      $exists And $type Already in MQL

      ldquonamerdquo$type2

      $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

      $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

      Ensure ldquonamerdquo exists (because not null) and is a string

      ldquoagerdquo optional but if exists must be a 32bit integer

      ldquonamerdquo required as string and weight and height both required integers or both not present

      13

      hellip And MQL Goes Way Beyondhellip

      $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

      14

      A New MQL Validator Module Emerges

      class MQLValidator ValidationResult validate(Map MQL Map data)

      MongoDBJava Driver

      Data Access Layer

      Application

      Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

      MQLValidator

      15

      MongoDBDB Engine

      Migrating Capability into MongoDB

      MongoDBJava Driver

      MQLValidatorJava

      Data Access Layer

      MongoDBDB Engine

      MongoDBJava Driver

      MQLValidator

      JavaData Access

      Layer

      bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

      and languages

      MongoDBPython Driver

      Application Application

      16

      Code For The FuturehellipToday

      class DataAccessLayer

      someWriteOperation(Map data)

      if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

      17

      But What About Today

      MQLValidator

      18

      Temporary Filling PQL

      bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

      PQL

      19

      Bridge MQL to PQL

      class MQLValidator private PQLFilter pqlfilter

      validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

      20

      No PQL No Problem

      class MQLValidator

      validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

      21

      MQL Is Easy To Navigate

      $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

      bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

      22

      Where Do Validations Come From

      The Database

      23

      The Validations Collection

      gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

      24

      Various ldquoLevelsrdquo of Validation

      gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

      25

      Multiple Types Schema By Example

      gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

      26

      The Stack So Far

      MongoDBJava Driver

      MQLValidator

      Data Access Layer

      Application

      ValidatorDBUtils

      ValidatorDBUtils populates an MQLValidator object from MongoDB

      PQLFilter

      27

      Representative Example

      class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

      someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

      28

      Redaction

      29

      Concept Post Query Operations (PQO)

      ssn $hash model birthdate null

      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

      30

      Adopt MQL-like behavior

      ldquossnrdquonull

      ldquoaddressrdquo ldquoXXXXrdquo

      ldquossnrdquo $substitute ldquossnmodelrdquo

      Remove field by setting to null

      Redact address with fixed value

      Substitute SSN with a different correct consistent value

      ldquocounterpartyrdquo $hash ldquoMD5rdquo

      Hash counterparty name to consistent value

      31

      A New PostQuery Module Emerges

      class PostQuery process(Map data Map operations)

      PostQuery

      MongoDBJava Driver

      MQLValidator

      Data Access Layer

      Application

      ValidatorDBUtils

      PQLFilter

      32

      Where Do PQOs Come From

      The Database

      33

      The Postquery Collection

      gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

      34

      The Stack Is Getting Rich

      PostQuery

      MongoDBJava Driver

      MQLValidator

      Data Access Layer

      Application

      ValidatorDBUtils

      PQLFilter

      PQODBUtils

      35

      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

      someWriteOperation(Map data) hellip

      someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

      36

      Quality of Service

      37

      QOS In Action

      class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

      Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

      38

      Where Do We Store QOS Values

      The Database

      39

      The QOS Collection

      gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

      ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

      40

      QOSDBUtils

      Coming Togetherhellip

      PostQuery

      MongoDBJava Driver

      MQLValidator

      Data Access Layer

      Application

      ValidatorDBUtils

      PQLFilter

      PQODBUtils

      QOS

      41

      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

      someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

      42

      QOSDBUtils

      A Highly Leveragable Investment

      PostQuery

      MQLValidator

      Data Access Layer 1

      Application1

      ValidatorDBUtils

      PQLFilter

      PQODBUtils

      QOS

      Application2

      Data Access Layer 2

      Application3

      Application4

      Data Access Layer 3

      Application5

      Application6Reusable For ALL Data Access Layer Logic

      43

      Not Just Java Not A Problem

      DAL operations have little or no statehellip

      Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

      Can you say hellip Web Service

      44

      A Really Nice Stack

      MongoDBJava Driver

      MQLValidator

      Data Access Layer

      JavaApplication

      ValidatorDBUtils

      PQLFilter

      HTTP Endpoint

      pythonApplication

      curl

      JSON over HTTP(S)

      JSONlt-gtJava Maps

      QOSDBUtils

      PostQuery

      PQODBUtils

      QOS

      45

      What Can We Do

      46

      Secure Access To Redacted Data for Testing

      $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

      $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

      $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

      47

      Get It Programmatically Too

      This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

      getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

      String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

      48

      Robust Validated Data Ingest

      $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

      $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

      49

      Concept The control_ Collection

      gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

      bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

      50

      Validation QOS and PQO via Web Services

      MongoDBJava Driver

      MQLValidator

      Data Access Layer

      JavaApplication

      ValidatorHTTPUtils

      PQLFilter

      pythonApplication

      curl

      JSON over HTTP(S)

      QOSHTTPUtils

      PostQuery

      PQOHTTPUtils

      QOS

      HTTP Service

      JSONlt-gtJava Maps

      51

      Are We Excited Yet

      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

      and other Java resources

      Questions amp Answers

      54

      Concept DataProvider

      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

      Class MongoProvider implements DataProvider hellip

      Class RESTfulProvider implements DataProvider hellip

      55

      The RESTful Provider

      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

      • Slide 1
      • Part 3 In The Data Management Series
      • Congratulations At this Point Yoursquove
      • The Next Stage Defend amp Leverage
      • MongoDB Doesnrsquot Have These Things
      • What Can We Do Today
      • Write Some Code
      • Starting Point The Data Access Layer
      • Document Validation
      • A Query Filters Outbound Data
      • How About Using It To Filter Inbounds
      • $exists And $type Already in MQL
      • hellip And MQL Goes Way Beyondhellip
      • A New MQL Validator Module Emerges
      • Migrating Capability into MongoDB
      • Code For The FuturehellipToday
      • But What About Today
      • Temporary Filling PQL
      • Bridge MQL to PQL
      • No PQL No Problem
      • MQL Is Easy To Navigate
      • Where Do Validations Come From
      • The Validations Collection
      • Various ldquoLevelsrdquo of Validation
      • Multiple Types Schema By Example
      • The Stack So Far
      • Representative Example
      • Redaction
      • Concept Post Query Operations (PQO)
      • Adopt MQL-like behavior
      • A New PostQuery Module Emerges
      • Where Do PQOs Come From
      • The Postquery Collection
      • The Stack Is Getting Rich
      • Representative Example (2)
      • Quality of Service
      • QOS In Action
      • Where Do We Store QOS Values
      • The QOS Collection
      • Coming Togetherhellip
      • Representative Example (3)
      • A Highly Leveragable Investment
      • Not Just Java Not A Problem
      • A Really Nice Stack
      • What Can We Do
      • Secure Access To Redacted Data for Testing
      • Get It Programmatically Too
      • Robust Validated Data Ingest
      • Concept The control_ Collection
      • Validation QOS and PQO via Web Services
      • Are We Excited Yet
      • Questions amp Answers
      • Slide 53
      • Concept DataProvider
      • The RESTful Provider

        4

        The Next Stage Defend amp Leverage

        bull Document Validationbull Redactionbull Quality Of Service

        5

        MongoDB Doesnrsquot Have These Things

        bull Document Validationbull Redactionbull Quality Of Service

        YET^

        6

        What Can We Do Today

        Write Some Code

        1 Focus on interfaces2 Design for change3 Keep application data access layer

        data management logic and database io well-factored

        4 Minimize compile-time binding

        8

        Starting Point The Data Access Layer

        MongoDBJava Driver

        Data Access Layer

        Application

        class DataAccessLayer private String authenicatedID private String effectiveID private Role role

        init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

        List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

        DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

        while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

        9

        Document Validation

        10

        A Query Filters Outbound Data

        $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

        11

        How About Using It To Filter Inbounds

        $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

        12

        $exists And $type Already in MQL

        ldquonamerdquo$type2

        $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

        $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

        Ensure ldquonamerdquo exists (because not null) and is a string

        ldquoagerdquo optional but if exists must be a 32bit integer

        ldquonamerdquo required as string and weight and height both required integers or both not present

        13

        hellip And MQL Goes Way Beyondhellip

        $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

        14

        A New MQL Validator Module Emerges

        class MQLValidator ValidationResult validate(Map MQL Map data)

        MongoDBJava Driver

        Data Access Layer

        Application

        Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

        MQLValidator

        15

        MongoDBDB Engine

        Migrating Capability into MongoDB

        MongoDBJava Driver

        MQLValidatorJava

        Data Access Layer

        MongoDBDB Engine

        MongoDBJava Driver

        MQLValidator

        JavaData Access

        Layer

        bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

        and languages

        MongoDBPython Driver

        Application Application

        16

        Code For The FuturehellipToday

        class DataAccessLayer

        someWriteOperation(Map data)

        if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

        17

        But What About Today

        MQLValidator

        18

        Temporary Filling PQL

        bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

        PQL

        19

        Bridge MQL to PQL

        class MQLValidator private PQLFilter pqlfilter

        validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

        20

        No PQL No Problem

        class MQLValidator

        validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

        21

        MQL Is Easy To Navigate

        $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

        bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

        22

        Where Do Validations Come From

        The Database

        23

        The Validations Collection

        gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

        24

        Various ldquoLevelsrdquo of Validation

        gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

        25

        Multiple Types Schema By Example

        gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

        26

        The Stack So Far

        MongoDBJava Driver

        MQLValidator

        Data Access Layer

        Application

        ValidatorDBUtils

        ValidatorDBUtils populates an MQLValidator object from MongoDB

        PQLFilter

        27

        Representative Example

        class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

        someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

        28

        Redaction

        29

        Concept Post Query Operations (PQO)

        ssn $hash model birthdate null

        $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

        30

        Adopt MQL-like behavior

        ldquossnrdquonull

        ldquoaddressrdquo ldquoXXXXrdquo

        ldquossnrdquo $substitute ldquossnmodelrdquo

        Remove field by setting to null

        Redact address with fixed value

        Substitute SSN with a different correct consistent value

        ldquocounterpartyrdquo $hash ldquoMD5rdquo

        Hash counterparty name to consistent value

        31

        A New PostQuery Module Emerges

        class PostQuery process(Map data Map operations)

        PostQuery

        MongoDBJava Driver

        MQLValidator

        Data Access Layer

        Application

        ValidatorDBUtils

        PQLFilter

        32

        Where Do PQOs Come From

        The Database

        33

        The Postquery Collection

        gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

        34

        The Stack Is Getting Rich

        PostQuery

        MongoDBJava Driver

        MQLValidator

        Data Access Layer

        Application

        ValidatorDBUtils

        PQLFilter

        PQODBUtils

        35

        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

        someWriteOperation(Map data) hellip

        someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

        36

        Quality of Service

        37

        QOS In Action

        class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

        Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

        38

        Where Do We Store QOS Values

        The Database

        39

        The QOS Collection

        gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

        ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

        40

        QOSDBUtils

        Coming Togetherhellip

        PostQuery

        MongoDBJava Driver

        MQLValidator

        Data Access Layer

        Application

        ValidatorDBUtils

        PQLFilter

        PQODBUtils

        QOS

        41

        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

        someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

        42

        QOSDBUtils

        A Highly Leveragable Investment

        PostQuery

        MQLValidator

        Data Access Layer 1

        Application1

        ValidatorDBUtils

        PQLFilter

        PQODBUtils

        QOS

        Application2

        Data Access Layer 2

        Application3

        Application4

        Data Access Layer 3

        Application5

        Application6Reusable For ALL Data Access Layer Logic

        43

        Not Just Java Not A Problem

        DAL operations have little or no statehellip

        Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

        Can you say hellip Web Service

        44

        A Really Nice Stack

        MongoDBJava Driver

        MQLValidator

        Data Access Layer

        JavaApplication

        ValidatorDBUtils

        PQLFilter

        HTTP Endpoint

        pythonApplication

        curl

        JSON over HTTP(S)

        JSONlt-gtJava Maps

        QOSDBUtils

        PostQuery

        PQODBUtils

        QOS

        45

        What Can We Do

        46

        Secure Access To Redacted Data for Testing

        $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

        $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

        $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

        47

        Get It Programmatically Too

        This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

        getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

        String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

        48

        Robust Validated Data Ingest

        $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

        $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

        49

        Concept The control_ Collection

        gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

        bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

        50

        Validation QOS and PQO via Web Services

        MongoDBJava Driver

        MQLValidator

        Data Access Layer

        JavaApplication

        ValidatorHTTPUtils

        PQLFilter

        pythonApplication

        curl

        JSON over HTTP(S)

        QOSHTTPUtils

        PostQuery

        PQOHTTPUtils

        QOS

        HTTP Service

        JSONlt-gtJava Maps

        51

        Are We Excited Yet

        Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

        and other Java resources

        Questions amp Answers

        54

        Concept DataProvider

        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

        Class MongoProvider implements DataProvider hellip

        Class RESTfulProvider implements DataProvider hellip

        55

        The RESTful Provider

        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

        • Slide 1
        • Part 3 In The Data Management Series
        • Congratulations At this Point Yoursquove
        • The Next Stage Defend amp Leverage
        • MongoDB Doesnrsquot Have These Things
        • What Can We Do Today
        • Write Some Code
        • Starting Point The Data Access Layer
        • Document Validation
        • A Query Filters Outbound Data
        • How About Using It To Filter Inbounds
        • $exists And $type Already in MQL
        • hellip And MQL Goes Way Beyondhellip
        • A New MQL Validator Module Emerges
        • Migrating Capability into MongoDB
        • Code For The FuturehellipToday
        • But What About Today
        • Temporary Filling PQL
        • Bridge MQL to PQL
        • No PQL No Problem
        • MQL Is Easy To Navigate
        • Where Do Validations Come From
        • The Validations Collection
        • Various ldquoLevelsrdquo of Validation
        • Multiple Types Schema By Example
        • The Stack So Far
        • Representative Example
        • Redaction
        • Concept Post Query Operations (PQO)
        • Adopt MQL-like behavior
        • A New PostQuery Module Emerges
        • Where Do PQOs Come From
        • The Postquery Collection
        • The Stack Is Getting Rich
        • Representative Example (2)
        • Quality of Service
        • QOS In Action
        • Where Do We Store QOS Values
        • The QOS Collection
        • Coming Togetherhellip
        • Representative Example (3)
        • A Highly Leveragable Investment
        • Not Just Java Not A Problem
        • A Really Nice Stack
        • What Can We Do
        • Secure Access To Redacted Data for Testing
        • Get It Programmatically Too
        • Robust Validated Data Ingest
        • Concept The control_ Collection
        • Validation QOS and PQO via Web Services
        • Are We Excited Yet
        • Questions amp Answers
        • Slide 53
        • Concept DataProvider
        • The RESTful Provider

          5

          MongoDB Doesnrsquot Have These Things

          bull Document Validationbull Redactionbull Quality Of Service

          YET^

          6

          What Can We Do Today

          Write Some Code

          1 Focus on interfaces2 Design for change3 Keep application data access layer

          data management logic and database io well-factored

          4 Minimize compile-time binding

          8

          Starting Point The Data Access Layer

          MongoDBJava Driver

          Data Access Layer

          Application

          class DataAccessLayer private String authenicatedID private String effectiveID private Role role

          init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

          List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

          DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

          while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

          9

          Document Validation

          10

          A Query Filters Outbound Data

          $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

          11

          How About Using It To Filter Inbounds

          $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

          12

          $exists And $type Already in MQL

          ldquonamerdquo$type2

          $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

          $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

          Ensure ldquonamerdquo exists (because not null) and is a string

          ldquoagerdquo optional but if exists must be a 32bit integer

          ldquonamerdquo required as string and weight and height both required integers or both not present

          13

          hellip And MQL Goes Way Beyondhellip

          $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

          14

          A New MQL Validator Module Emerges

          class MQLValidator ValidationResult validate(Map MQL Map data)

          MongoDBJava Driver

          Data Access Layer

          Application

          Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

          MQLValidator

          15

          MongoDBDB Engine

          Migrating Capability into MongoDB

          MongoDBJava Driver

          MQLValidatorJava

          Data Access Layer

          MongoDBDB Engine

          MongoDBJava Driver

          MQLValidator

          JavaData Access

          Layer

          bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

          and languages

          MongoDBPython Driver

          Application Application

          16

          Code For The FuturehellipToday

          class DataAccessLayer

          someWriteOperation(Map data)

          if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

          17

          But What About Today

          MQLValidator

          18

          Temporary Filling PQL

          bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

          PQL

          19

          Bridge MQL to PQL

          class MQLValidator private PQLFilter pqlfilter

          validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

          20

          No PQL No Problem

          class MQLValidator

          validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

          21

          MQL Is Easy To Navigate

          $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

          bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

          22

          Where Do Validations Come From

          The Database

          23

          The Validations Collection

          gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

          24

          Various ldquoLevelsrdquo of Validation

          gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

          25

          Multiple Types Schema By Example

          gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

          26

          The Stack So Far

          MongoDBJava Driver

          MQLValidator

          Data Access Layer

          Application

          ValidatorDBUtils

          ValidatorDBUtils populates an MQLValidator object from MongoDB

          PQLFilter

          27

          Representative Example

          class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

          someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

          28

          Redaction

          29

          Concept Post Query Operations (PQO)

          ssn $hash model birthdate null

          $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

          30

          Adopt MQL-like behavior

          ldquossnrdquonull

          ldquoaddressrdquo ldquoXXXXrdquo

          ldquossnrdquo $substitute ldquossnmodelrdquo

          Remove field by setting to null

          Redact address with fixed value

          Substitute SSN with a different correct consistent value

          ldquocounterpartyrdquo $hash ldquoMD5rdquo

          Hash counterparty name to consistent value

          31

          A New PostQuery Module Emerges

          class PostQuery process(Map data Map operations)

          PostQuery

          MongoDBJava Driver

          MQLValidator

          Data Access Layer

          Application

          ValidatorDBUtils

          PQLFilter

          32

          Where Do PQOs Come From

          The Database

          33

          The Postquery Collection

          gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

          34

          The Stack Is Getting Rich

          PostQuery

          MongoDBJava Driver

          MQLValidator

          Data Access Layer

          Application

          ValidatorDBUtils

          PQLFilter

          PQODBUtils

          35

          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

          someWriteOperation(Map data) hellip

          someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

          36

          Quality of Service

          37

          QOS In Action

          class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

          Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

          38

          Where Do We Store QOS Values

          The Database

          39

          The QOS Collection

          gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

          ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

          40

          QOSDBUtils

          Coming Togetherhellip

          PostQuery

          MongoDBJava Driver

          MQLValidator

          Data Access Layer

          Application

          ValidatorDBUtils

          PQLFilter

          PQODBUtils

          QOS

          41

          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

          someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

          42

          QOSDBUtils

          A Highly Leveragable Investment

          PostQuery

          MQLValidator

          Data Access Layer 1

          Application1

          ValidatorDBUtils

          PQLFilter

          PQODBUtils

          QOS

          Application2

          Data Access Layer 2

          Application3

          Application4

          Data Access Layer 3

          Application5

          Application6Reusable For ALL Data Access Layer Logic

          43

          Not Just Java Not A Problem

          DAL operations have little or no statehellip

          Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

          Can you say hellip Web Service

          44

          A Really Nice Stack

          MongoDBJava Driver

          MQLValidator

          Data Access Layer

          JavaApplication

          ValidatorDBUtils

          PQLFilter

          HTTP Endpoint

          pythonApplication

          curl

          JSON over HTTP(S)

          JSONlt-gtJava Maps

          QOSDBUtils

          PostQuery

          PQODBUtils

          QOS

          45

          What Can We Do

          46

          Secure Access To Redacted Data for Testing

          $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

          $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

          $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

          47

          Get It Programmatically Too

          This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

          getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

          String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

          48

          Robust Validated Data Ingest

          $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

          $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

          49

          Concept The control_ Collection

          gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

          bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

          50

          Validation QOS and PQO via Web Services

          MongoDBJava Driver

          MQLValidator

          Data Access Layer

          JavaApplication

          ValidatorHTTPUtils

          PQLFilter

          pythonApplication

          curl

          JSON over HTTP(S)

          QOSHTTPUtils

          PostQuery

          PQOHTTPUtils

          QOS

          HTTP Service

          JSONlt-gtJava Maps

          51

          Are We Excited Yet

          Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

          and other Java resources

          Questions amp Answers

          54

          Concept DataProvider

          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

          Class MongoProvider implements DataProvider hellip

          Class RESTfulProvider implements DataProvider hellip

          55

          The RESTful Provider

          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

          • Slide 1
          • Part 3 In The Data Management Series
          • Congratulations At this Point Yoursquove
          • The Next Stage Defend amp Leverage
          • MongoDB Doesnrsquot Have These Things
          • What Can We Do Today
          • Write Some Code
          • Starting Point The Data Access Layer
          • Document Validation
          • A Query Filters Outbound Data
          • How About Using It To Filter Inbounds
          • $exists And $type Already in MQL
          • hellip And MQL Goes Way Beyondhellip
          • A New MQL Validator Module Emerges
          • Migrating Capability into MongoDB
          • Code For The FuturehellipToday
          • But What About Today
          • Temporary Filling PQL
          • Bridge MQL to PQL
          • No PQL No Problem
          • MQL Is Easy To Navigate
          • Where Do Validations Come From
          • The Validations Collection
          • Various ldquoLevelsrdquo of Validation
          • Multiple Types Schema By Example
          • The Stack So Far
          • Representative Example
          • Redaction
          • Concept Post Query Operations (PQO)
          • Adopt MQL-like behavior
          • A New PostQuery Module Emerges
          • Where Do PQOs Come From
          • The Postquery Collection
          • The Stack Is Getting Rich
          • Representative Example (2)
          • Quality of Service
          • QOS In Action
          • Where Do We Store QOS Values
          • The QOS Collection
          • Coming Togetherhellip
          • Representative Example (3)
          • A Highly Leveragable Investment
          • Not Just Java Not A Problem
          • A Really Nice Stack
          • What Can We Do
          • Secure Access To Redacted Data for Testing
          • Get It Programmatically Too
          • Robust Validated Data Ingest
          • Concept The control_ Collection
          • Validation QOS and PQO via Web Services
          • Are We Excited Yet
          • Questions amp Answers
          • Slide 53
          • Concept DataProvider
          • The RESTful Provider

            6

            What Can We Do Today

            Write Some Code

            1 Focus on interfaces2 Design for change3 Keep application data access layer

            data management logic and database io well-factored

            4 Minimize compile-time binding

            8

            Starting Point The Data Access Layer

            MongoDBJava Driver

            Data Access Layer

            Application

            class DataAccessLayer private String authenicatedID private String effectiveID private Role role

            init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

            List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

            DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

            while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

            9

            Document Validation

            10

            A Query Filters Outbound Data

            $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

            11

            How About Using It To Filter Inbounds

            $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

            12

            $exists And $type Already in MQL

            ldquonamerdquo$type2

            $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

            $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

            Ensure ldquonamerdquo exists (because not null) and is a string

            ldquoagerdquo optional but if exists must be a 32bit integer

            ldquonamerdquo required as string and weight and height both required integers or both not present

            13

            hellip And MQL Goes Way Beyondhellip

            $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

            14

            A New MQL Validator Module Emerges

            class MQLValidator ValidationResult validate(Map MQL Map data)

            MongoDBJava Driver

            Data Access Layer

            Application

            Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

            MQLValidator

            15

            MongoDBDB Engine

            Migrating Capability into MongoDB

            MongoDBJava Driver

            MQLValidatorJava

            Data Access Layer

            MongoDBDB Engine

            MongoDBJava Driver

            MQLValidator

            JavaData Access

            Layer

            bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

            and languages

            MongoDBPython Driver

            Application Application

            16

            Code For The FuturehellipToday

            class DataAccessLayer

            someWriteOperation(Map data)

            if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

            17

            But What About Today

            MQLValidator

            18

            Temporary Filling PQL

            bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

            PQL

            19

            Bridge MQL to PQL

            class MQLValidator private PQLFilter pqlfilter

            validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

            20

            No PQL No Problem

            class MQLValidator

            validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

            21

            MQL Is Easy To Navigate

            $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

            bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

            22

            Where Do Validations Come From

            The Database

            23

            The Validations Collection

            gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

            24

            Various ldquoLevelsrdquo of Validation

            gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

            25

            Multiple Types Schema By Example

            gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

            26

            The Stack So Far

            MongoDBJava Driver

            MQLValidator

            Data Access Layer

            Application

            ValidatorDBUtils

            ValidatorDBUtils populates an MQLValidator object from MongoDB

            PQLFilter

            27

            Representative Example

            class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

            someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

            28

            Redaction

            29

            Concept Post Query Operations (PQO)

            ssn $hash model birthdate null

            $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

            30

            Adopt MQL-like behavior

            ldquossnrdquonull

            ldquoaddressrdquo ldquoXXXXrdquo

            ldquossnrdquo $substitute ldquossnmodelrdquo

            Remove field by setting to null

            Redact address with fixed value

            Substitute SSN with a different correct consistent value

            ldquocounterpartyrdquo $hash ldquoMD5rdquo

            Hash counterparty name to consistent value

            31

            A New PostQuery Module Emerges

            class PostQuery process(Map data Map operations)

            PostQuery

            MongoDBJava Driver

            MQLValidator

            Data Access Layer

            Application

            ValidatorDBUtils

            PQLFilter

            32

            Where Do PQOs Come From

            The Database

            33

            The Postquery Collection

            gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

            34

            The Stack Is Getting Rich

            PostQuery

            MongoDBJava Driver

            MQLValidator

            Data Access Layer

            Application

            ValidatorDBUtils

            PQLFilter

            PQODBUtils

            35

            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

            someWriteOperation(Map data) hellip

            someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

            36

            Quality of Service

            37

            QOS In Action

            class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

            Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

            38

            Where Do We Store QOS Values

            The Database

            39

            The QOS Collection

            gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

            ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

            40

            QOSDBUtils

            Coming Togetherhellip

            PostQuery

            MongoDBJava Driver

            MQLValidator

            Data Access Layer

            Application

            ValidatorDBUtils

            PQLFilter

            PQODBUtils

            QOS

            41

            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

            someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

            42

            QOSDBUtils

            A Highly Leveragable Investment

            PostQuery

            MQLValidator

            Data Access Layer 1

            Application1

            ValidatorDBUtils

            PQLFilter

            PQODBUtils

            QOS

            Application2

            Data Access Layer 2

            Application3

            Application4

            Data Access Layer 3

            Application5

            Application6Reusable For ALL Data Access Layer Logic

            43

            Not Just Java Not A Problem

            DAL operations have little or no statehellip

            Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

            Can you say hellip Web Service

            44

            A Really Nice Stack

            MongoDBJava Driver

            MQLValidator

            Data Access Layer

            JavaApplication

            ValidatorDBUtils

            PQLFilter

            HTTP Endpoint

            pythonApplication

            curl

            JSON over HTTP(S)

            JSONlt-gtJava Maps

            QOSDBUtils

            PostQuery

            PQODBUtils

            QOS

            45

            What Can We Do

            46

            Secure Access To Redacted Data for Testing

            $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

            $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

            $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

            47

            Get It Programmatically Too

            This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

            getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

            String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

            48

            Robust Validated Data Ingest

            $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

            $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

            49

            Concept The control_ Collection

            gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

            bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

            50

            Validation QOS and PQO via Web Services

            MongoDBJava Driver

            MQLValidator

            Data Access Layer

            JavaApplication

            ValidatorHTTPUtils

            PQLFilter

            pythonApplication

            curl

            JSON over HTTP(S)

            QOSHTTPUtils

            PostQuery

            PQOHTTPUtils

            QOS

            HTTP Service

            JSONlt-gtJava Maps

            51

            Are We Excited Yet

            Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

            and other Java resources

            Questions amp Answers

            54

            Concept DataProvider

            public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

            Class MongoProvider implements DataProvider hellip

            Class RESTfulProvider implements DataProvider hellip

            55

            The RESTful Provider

            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

            • Slide 1
            • Part 3 In The Data Management Series
            • Congratulations At this Point Yoursquove
            • The Next Stage Defend amp Leverage
            • MongoDB Doesnrsquot Have These Things
            • What Can We Do Today
            • Write Some Code
            • Starting Point The Data Access Layer
            • Document Validation
            • A Query Filters Outbound Data
            • How About Using It To Filter Inbounds
            • $exists And $type Already in MQL
            • hellip And MQL Goes Way Beyondhellip
            • A New MQL Validator Module Emerges
            • Migrating Capability into MongoDB
            • Code For The FuturehellipToday
            • But What About Today
            • Temporary Filling PQL
            • Bridge MQL to PQL
            • No PQL No Problem
            • MQL Is Easy To Navigate
            • Where Do Validations Come From
            • The Validations Collection
            • Various ldquoLevelsrdquo of Validation
            • Multiple Types Schema By Example
            • The Stack So Far
            • Representative Example
            • Redaction
            • Concept Post Query Operations (PQO)
            • Adopt MQL-like behavior
            • A New PostQuery Module Emerges
            • Where Do PQOs Come From
            • The Postquery Collection
            • The Stack Is Getting Rich
            • Representative Example (2)
            • Quality of Service
            • QOS In Action
            • Where Do We Store QOS Values
            • The QOS Collection
            • Coming Togetherhellip
            • Representative Example (3)
            • A Highly Leveragable Investment
            • Not Just Java Not A Problem
            • A Really Nice Stack
            • What Can We Do
            • Secure Access To Redacted Data for Testing
            • Get It Programmatically Too
            • Robust Validated Data Ingest
            • Concept The control_ Collection
            • Validation QOS and PQO via Web Services
            • Are We Excited Yet
            • Questions amp Answers
            • Slide 53
            • Concept DataProvider
            • The RESTful Provider

              Write Some Code

              1 Focus on interfaces2 Design for change3 Keep application data access layer

              data management logic and database io well-factored

              4 Minimize compile-time binding

              8

              Starting Point The Data Access Layer

              MongoDBJava Driver

              Data Access Layer

              Application

              class DataAccessLayer private String authenicatedID private String effectiveID private Role role

              init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

              List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

              DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

              while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

              9

              Document Validation

              10

              A Query Filters Outbound Data

              $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

              11

              How About Using It To Filter Inbounds

              $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

              12

              $exists And $type Already in MQL

              ldquonamerdquo$type2

              $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

              $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

              Ensure ldquonamerdquo exists (because not null) and is a string

              ldquoagerdquo optional but if exists must be a 32bit integer

              ldquonamerdquo required as string and weight and height both required integers or both not present

              13

              hellip And MQL Goes Way Beyondhellip

              $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

              14

              A New MQL Validator Module Emerges

              class MQLValidator ValidationResult validate(Map MQL Map data)

              MongoDBJava Driver

              Data Access Layer

              Application

              Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

              MQLValidator

              15

              MongoDBDB Engine

              Migrating Capability into MongoDB

              MongoDBJava Driver

              MQLValidatorJava

              Data Access Layer

              MongoDBDB Engine

              MongoDBJava Driver

              MQLValidator

              JavaData Access

              Layer

              bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

              and languages

              MongoDBPython Driver

              Application Application

              16

              Code For The FuturehellipToday

              class DataAccessLayer

              someWriteOperation(Map data)

              if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

              17

              But What About Today

              MQLValidator

              18

              Temporary Filling PQL

              bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

              PQL

              19

              Bridge MQL to PQL

              class MQLValidator private PQLFilter pqlfilter

              validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

              20

              No PQL No Problem

              class MQLValidator

              validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

              21

              MQL Is Easy To Navigate

              $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

              bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

              22

              Where Do Validations Come From

              The Database

              23

              The Validations Collection

              gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

              24

              Various ldquoLevelsrdquo of Validation

              gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

              25

              Multiple Types Schema By Example

              gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

              26

              The Stack So Far

              MongoDBJava Driver

              MQLValidator

              Data Access Layer

              Application

              ValidatorDBUtils

              ValidatorDBUtils populates an MQLValidator object from MongoDB

              PQLFilter

              27

              Representative Example

              class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

              someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

              28

              Redaction

              29

              Concept Post Query Operations (PQO)

              ssn $hash model birthdate null

              $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

              30

              Adopt MQL-like behavior

              ldquossnrdquonull

              ldquoaddressrdquo ldquoXXXXrdquo

              ldquossnrdquo $substitute ldquossnmodelrdquo

              Remove field by setting to null

              Redact address with fixed value

              Substitute SSN with a different correct consistent value

              ldquocounterpartyrdquo $hash ldquoMD5rdquo

              Hash counterparty name to consistent value

              31

              A New PostQuery Module Emerges

              class PostQuery process(Map data Map operations)

              PostQuery

              MongoDBJava Driver

              MQLValidator

              Data Access Layer

              Application

              ValidatorDBUtils

              PQLFilter

              32

              Where Do PQOs Come From

              The Database

              33

              The Postquery Collection

              gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

              34

              The Stack Is Getting Rich

              PostQuery

              MongoDBJava Driver

              MQLValidator

              Data Access Layer

              Application

              ValidatorDBUtils

              PQLFilter

              PQODBUtils

              35

              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

              someWriteOperation(Map data) hellip

              someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

              36

              Quality of Service

              37

              QOS In Action

              class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

              Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

              38

              Where Do We Store QOS Values

              The Database

              39

              The QOS Collection

              gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

              ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

              40

              QOSDBUtils

              Coming Togetherhellip

              PostQuery

              MongoDBJava Driver

              MQLValidator

              Data Access Layer

              Application

              ValidatorDBUtils

              PQLFilter

              PQODBUtils

              QOS

              41

              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

              someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

              42

              QOSDBUtils

              A Highly Leveragable Investment

              PostQuery

              MQLValidator

              Data Access Layer 1

              Application1

              ValidatorDBUtils

              PQLFilter

              PQODBUtils

              QOS

              Application2

              Data Access Layer 2

              Application3

              Application4

              Data Access Layer 3

              Application5

              Application6Reusable For ALL Data Access Layer Logic

              43

              Not Just Java Not A Problem

              DAL operations have little or no statehellip

              Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

              Can you say hellip Web Service

              44

              A Really Nice Stack

              MongoDBJava Driver

              MQLValidator

              Data Access Layer

              JavaApplication

              ValidatorDBUtils

              PQLFilter

              HTTP Endpoint

              pythonApplication

              curl

              JSON over HTTP(S)

              JSONlt-gtJava Maps

              QOSDBUtils

              PostQuery

              PQODBUtils

              QOS

              45

              What Can We Do

              46

              Secure Access To Redacted Data for Testing

              $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

              $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

              $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

              47

              Get It Programmatically Too

              This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

              getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

              String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

              48

              Robust Validated Data Ingest

              $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

              $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

              49

              Concept The control_ Collection

              gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

              bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

              50

              Validation QOS and PQO via Web Services

              MongoDBJava Driver

              MQLValidator

              Data Access Layer

              JavaApplication

              ValidatorHTTPUtils

              PQLFilter

              pythonApplication

              curl

              JSON over HTTP(S)

              QOSHTTPUtils

              PostQuery

              PQOHTTPUtils

              QOS

              HTTP Service

              JSONlt-gtJava Maps

              51

              Are We Excited Yet

              Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

              and other Java resources

              Questions amp Answers

              54

              Concept DataProvider

              public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

              Class MongoProvider implements DataProvider hellip

              Class RESTfulProvider implements DataProvider hellip

              55

              The RESTful Provider

              class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

              fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

              url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

              HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

              • Slide 1
              • Part 3 In The Data Management Series
              • Congratulations At this Point Yoursquove
              • The Next Stage Defend amp Leverage
              • MongoDB Doesnrsquot Have These Things
              • What Can We Do Today
              • Write Some Code
              • Starting Point The Data Access Layer
              • Document Validation
              • A Query Filters Outbound Data
              • How About Using It To Filter Inbounds
              • $exists And $type Already in MQL
              • hellip And MQL Goes Way Beyondhellip
              • A New MQL Validator Module Emerges
              • Migrating Capability into MongoDB
              • Code For The FuturehellipToday
              • But What About Today
              • Temporary Filling PQL
              • Bridge MQL to PQL
              • No PQL No Problem
              • MQL Is Easy To Navigate
              • Where Do Validations Come From
              • The Validations Collection
              • Various ldquoLevelsrdquo of Validation
              • Multiple Types Schema By Example
              • The Stack So Far
              • Representative Example
              • Redaction
              • Concept Post Query Operations (PQO)
              • Adopt MQL-like behavior
              • A New PostQuery Module Emerges
              • Where Do PQOs Come From
              • The Postquery Collection
              • The Stack Is Getting Rich
              • Representative Example (2)
              • Quality of Service
              • QOS In Action
              • Where Do We Store QOS Values
              • The QOS Collection
              • Coming Togetherhellip
              • Representative Example (3)
              • A Highly Leveragable Investment
              • Not Just Java Not A Problem
              • A Really Nice Stack
              • What Can We Do
              • Secure Access To Redacted Data for Testing
              • Get It Programmatically Too
              • Robust Validated Data Ingest
              • Concept The control_ Collection
              • Validation QOS and PQO via Web Services
              • Are We Excited Yet
              • Questions amp Answers
              • Slide 53
              • Concept DataProvider
              • The RESTful Provider

                8

                Starting Point The Data Access Layer

                MongoDBJava Driver

                Data Access Layer

                Application

                class DataAccessLayer private String authenicatedID private String effectiveID private Role role

                init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

                List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

                DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

                while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

                9

                Document Validation

                10

                A Query Filters Outbound Data

                $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                11

                How About Using It To Filter Inbounds

                $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                12

                $exists And $type Already in MQL

                ldquonamerdquo$type2

                $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

                $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

                Ensure ldquonamerdquo exists (because not null) and is a string

                ldquoagerdquo optional but if exists must be a 32bit integer

                ldquonamerdquo required as string and weight and height both required integers or both not present

                13

                hellip And MQL Goes Way Beyondhellip

                $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                14

                A New MQL Validator Module Emerges

                class MQLValidator ValidationResult validate(Map MQL Map data)

                MongoDBJava Driver

                Data Access Layer

                Application

                Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                MQLValidator

                15

                MongoDBDB Engine

                Migrating Capability into MongoDB

                MongoDBJava Driver

                MQLValidatorJava

                Data Access Layer

                MongoDBDB Engine

                MongoDBJava Driver

                MQLValidator

                JavaData Access

                Layer

                bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                and languages

                MongoDBPython Driver

                Application Application

                16

                Code For The FuturehellipToday

                class DataAccessLayer

                someWriteOperation(Map data)

                if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                17

                But What About Today

                MQLValidator

                18

                Temporary Filling PQL

                bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                PQL

                19

                Bridge MQL to PQL

                class MQLValidator private PQLFilter pqlfilter

                validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                20

                No PQL No Problem

                class MQLValidator

                validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                21

                MQL Is Easy To Navigate

                $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                22

                Where Do Validations Come From

                The Database

                23

                The Validations Collection

                gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                24

                Various ldquoLevelsrdquo of Validation

                gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                25

                Multiple Types Schema By Example

                gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                26

                The Stack So Far

                MongoDBJava Driver

                MQLValidator

                Data Access Layer

                Application

                ValidatorDBUtils

                ValidatorDBUtils populates an MQLValidator object from MongoDB

                PQLFilter

                27

                Representative Example

                class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                28

                Redaction

                29

                Concept Post Query Operations (PQO)

                ssn $hash model birthdate null

                $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                30

                Adopt MQL-like behavior

                ldquossnrdquonull

                ldquoaddressrdquo ldquoXXXXrdquo

                ldquossnrdquo $substitute ldquossnmodelrdquo

                Remove field by setting to null

                Redact address with fixed value

                Substitute SSN with a different correct consistent value

                ldquocounterpartyrdquo $hash ldquoMD5rdquo

                Hash counterparty name to consistent value

                31

                A New PostQuery Module Emerges

                class PostQuery process(Map data Map operations)

                PostQuery

                MongoDBJava Driver

                MQLValidator

                Data Access Layer

                Application

                ValidatorDBUtils

                PQLFilter

                32

                Where Do PQOs Come From

                The Database

                33

                The Postquery Collection

                gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                34

                The Stack Is Getting Rich

                PostQuery

                MongoDBJava Driver

                MQLValidator

                Data Access Layer

                Application

                ValidatorDBUtils

                PQLFilter

                PQODBUtils

                35

                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                someWriteOperation(Map data) hellip

                someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                36

                Quality of Service

                37

                QOS In Action

                class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                38

                Where Do We Store QOS Values

                The Database

                39

                The QOS Collection

                gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                40

                QOSDBUtils

                Coming Togetherhellip

                PostQuery

                MongoDBJava Driver

                MQLValidator

                Data Access Layer

                Application

                ValidatorDBUtils

                PQLFilter

                PQODBUtils

                QOS

                41

                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                42

                QOSDBUtils

                A Highly Leveragable Investment

                PostQuery

                MQLValidator

                Data Access Layer 1

                Application1

                ValidatorDBUtils

                PQLFilter

                PQODBUtils

                QOS

                Application2

                Data Access Layer 2

                Application3

                Application4

                Data Access Layer 3

                Application5

                Application6Reusable For ALL Data Access Layer Logic

                43

                Not Just Java Not A Problem

                DAL operations have little or no statehellip

                Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                Can you say hellip Web Service

                44

                A Really Nice Stack

                MongoDBJava Driver

                MQLValidator

                Data Access Layer

                JavaApplication

                ValidatorDBUtils

                PQLFilter

                HTTP Endpoint

                pythonApplication

                curl

                JSON over HTTP(S)

                JSONlt-gtJava Maps

                QOSDBUtils

                PostQuery

                PQODBUtils

                QOS

                45

                What Can We Do

                46

                Secure Access To Redacted Data for Testing

                $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                47

                Get It Programmatically Too

                This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                48

                Robust Validated Data Ingest

                $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                49

                Concept The control_ Collection

                gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                50

                Validation QOS and PQO via Web Services

                MongoDBJava Driver

                MQLValidator

                Data Access Layer

                JavaApplication

                ValidatorHTTPUtils

                PQLFilter

                pythonApplication

                curl

                JSON over HTTP(S)

                QOSHTTPUtils

                PostQuery

                PQOHTTPUtils

                QOS

                HTTP Service

                JSONlt-gtJava Maps

                51

                Are We Excited Yet

                Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                and other Java resources

                Questions amp Answers

                54

                Concept DataProvider

                public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                Class MongoProvider implements DataProvider hellip

                Class RESTfulProvider implements DataProvider hellip

                55

                The RESTful Provider

                class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                • Slide 1
                • Part 3 In The Data Management Series
                • Congratulations At this Point Yoursquove
                • The Next Stage Defend amp Leverage
                • MongoDB Doesnrsquot Have These Things
                • What Can We Do Today
                • Write Some Code
                • Starting Point The Data Access Layer
                • Document Validation
                • A Query Filters Outbound Data
                • How About Using It To Filter Inbounds
                • $exists And $type Already in MQL
                • hellip And MQL Goes Way Beyondhellip
                • A New MQL Validator Module Emerges
                • Migrating Capability into MongoDB
                • Code For The FuturehellipToday
                • But What About Today
                • Temporary Filling PQL
                • Bridge MQL to PQL
                • No PQL No Problem
                • MQL Is Easy To Navigate
                • Where Do Validations Come From
                • The Validations Collection
                • Various ldquoLevelsrdquo of Validation
                • Multiple Types Schema By Example
                • The Stack So Far
                • Representative Example
                • Redaction
                • Concept Post Query Operations (PQO)
                • Adopt MQL-like behavior
                • A New PostQuery Module Emerges
                • Where Do PQOs Come From
                • The Postquery Collection
                • The Stack Is Getting Rich
                • Representative Example (2)
                • Quality of Service
                • QOS In Action
                • Where Do We Store QOS Values
                • The QOS Collection
                • Coming Togetherhellip
                • Representative Example (3)
                • A Highly Leveragable Investment
                • Not Just Java Not A Problem
                • A Really Nice Stack
                • What Can We Do
                • Secure Access To Redacted Data for Testing
                • Get It Programmatically Too
                • Robust Validated Data Ingest
                • Concept The control_ Collection
                • Validation QOS and PQO via Web Services
                • Are We Excited Yet
                • Questions amp Answers
                • Slide 53
                • Concept DataProvider
                • The RESTful Provider

                  9

                  Document Validation

                  10

                  A Query Filters Outbound Data

                  $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                  11

                  How About Using It To Filter Inbounds

                  $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                  12

                  $exists And $type Already in MQL

                  ldquonamerdquo$type2

                  $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

                  $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

                  Ensure ldquonamerdquo exists (because not null) and is a string

                  ldquoagerdquo optional but if exists must be a 32bit integer

                  ldquonamerdquo required as string and weight and height both required integers or both not present

                  13

                  hellip And MQL Goes Way Beyondhellip

                  $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                  14

                  A New MQL Validator Module Emerges

                  class MQLValidator ValidationResult validate(Map MQL Map data)

                  MongoDBJava Driver

                  Data Access Layer

                  Application

                  Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                  MQLValidator

                  15

                  MongoDBDB Engine

                  Migrating Capability into MongoDB

                  MongoDBJava Driver

                  MQLValidatorJava

                  Data Access Layer

                  MongoDBDB Engine

                  MongoDBJava Driver

                  MQLValidator

                  JavaData Access

                  Layer

                  bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                  and languages

                  MongoDBPython Driver

                  Application Application

                  16

                  Code For The FuturehellipToday

                  class DataAccessLayer

                  someWriteOperation(Map data)

                  if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                  17

                  But What About Today

                  MQLValidator

                  18

                  Temporary Filling PQL

                  bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                  PQL

                  19

                  Bridge MQL to PQL

                  class MQLValidator private PQLFilter pqlfilter

                  validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                  20

                  No PQL No Problem

                  class MQLValidator

                  validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                  21

                  MQL Is Easy To Navigate

                  $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                  bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                  22

                  Where Do Validations Come From

                  The Database

                  23

                  The Validations Collection

                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                  24

                  Various ldquoLevelsrdquo of Validation

                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                  25

                  Multiple Types Schema By Example

                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                  26

                  The Stack So Far

                  MongoDBJava Driver

                  MQLValidator

                  Data Access Layer

                  Application

                  ValidatorDBUtils

                  ValidatorDBUtils populates an MQLValidator object from MongoDB

                  PQLFilter

                  27

                  Representative Example

                  class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                  someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                  28

                  Redaction

                  29

                  Concept Post Query Operations (PQO)

                  ssn $hash model birthdate null

                  $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                  30

                  Adopt MQL-like behavior

                  ldquossnrdquonull

                  ldquoaddressrdquo ldquoXXXXrdquo

                  ldquossnrdquo $substitute ldquossnmodelrdquo

                  Remove field by setting to null

                  Redact address with fixed value

                  Substitute SSN with a different correct consistent value

                  ldquocounterpartyrdquo $hash ldquoMD5rdquo

                  Hash counterparty name to consistent value

                  31

                  A New PostQuery Module Emerges

                  class PostQuery process(Map data Map operations)

                  PostQuery

                  MongoDBJava Driver

                  MQLValidator

                  Data Access Layer

                  Application

                  ValidatorDBUtils

                  PQLFilter

                  32

                  Where Do PQOs Come From

                  The Database

                  33

                  The Postquery Collection

                  gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                  34

                  The Stack Is Getting Rich

                  PostQuery

                  MongoDBJava Driver

                  MQLValidator

                  Data Access Layer

                  Application

                  ValidatorDBUtils

                  PQLFilter

                  PQODBUtils

                  35

                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                  someWriteOperation(Map data) hellip

                  someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                  36

                  Quality of Service

                  37

                  QOS In Action

                  class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                  Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                  38

                  Where Do We Store QOS Values

                  The Database

                  39

                  The QOS Collection

                  gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                  ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                  40

                  QOSDBUtils

                  Coming Togetherhellip

                  PostQuery

                  MongoDBJava Driver

                  MQLValidator

                  Data Access Layer

                  Application

                  ValidatorDBUtils

                  PQLFilter

                  PQODBUtils

                  QOS

                  41

                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                  someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                  42

                  QOSDBUtils

                  A Highly Leveragable Investment

                  PostQuery

                  MQLValidator

                  Data Access Layer 1

                  Application1

                  ValidatorDBUtils

                  PQLFilter

                  PQODBUtils

                  QOS

                  Application2

                  Data Access Layer 2

                  Application3

                  Application4

                  Data Access Layer 3

                  Application5

                  Application6Reusable For ALL Data Access Layer Logic

                  43

                  Not Just Java Not A Problem

                  DAL operations have little or no statehellip

                  Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                  Can you say hellip Web Service

                  44

                  A Really Nice Stack

                  MongoDBJava Driver

                  MQLValidator

                  Data Access Layer

                  JavaApplication

                  ValidatorDBUtils

                  PQLFilter

                  HTTP Endpoint

                  pythonApplication

                  curl

                  JSON over HTTP(S)

                  JSONlt-gtJava Maps

                  QOSDBUtils

                  PostQuery

                  PQODBUtils

                  QOS

                  45

                  What Can We Do

                  46

                  Secure Access To Redacted Data for Testing

                  $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                  $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                  $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                  47

                  Get It Programmatically Too

                  This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                  getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                  String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                  48

                  Robust Validated Data Ingest

                  $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                  $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                  49

                  Concept The control_ Collection

                  gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                  bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                  50

                  Validation QOS and PQO via Web Services

                  MongoDBJava Driver

                  MQLValidator

                  Data Access Layer

                  JavaApplication

                  ValidatorHTTPUtils

                  PQLFilter

                  pythonApplication

                  curl

                  JSON over HTTP(S)

                  QOSHTTPUtils

                  PostQuery

                  PQOHTTPUtils

                  QOS

                  HTTP Service

                  JSONlt-gtJava Maps

                  51

                  Are We Excited Yet

                  Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                  and other Java resources

                  Questions amp Answers

                  54

                  Concept DataProvider

                  public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                  Class MongoProvider implements DataProvider hellip

                  Class RESTfulProvider implements DataProvider hellip

                  55

                  The RESTful Provider

                  class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                  fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                  url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                  HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                  • Slide 1
                  • Part 3 In The Data Management Series
                  • Congratulations At this Point Yoursquove
                  • The Next Stage Defend amp Leverage
                  • MongoDB Doesnrsquot Have These Things
                  • What Can We Do Today
                  • Write Some Code
                  • Starting Point The Data Access Layer
                  • Document Validation
                  • A Query Filters Outbound Data
                  • How About Using It To Filter Inbounds
                  • $exists And $type Already in MQL
                  • hellip And MQL Goes Way Beyondhellip
                  • A New MQL Validator Module Emerges
                  • Migrating Capability into MongoDB
                  • Code For The FuturehellipToday
                  • But What About Today
                  • Temporary Filling PQL
                  • Bridge MQL to PQL
                  • No PQL No Problem
                  • MQL Is Easy To Navigate
                  • Where Do Validations Come From
                  • The Validations Collection
                  • Various ldquoLevelsrdquo of Validation
                  • Multiple Types Schema By Example
                  • The Stack So Far
                  • Representative Example
                  • Redaction
                  • Concept Post Query Operations (PQO)
                  • Adopt MQL-like behavior
                  • A New PostQuery Module Emerges
                  • Where Do PQOs Come From
                  • The Postquery Collection
                  • The Stack Is Getting Rich
                  • Representative Example (2)
                  • Quality of Service
                  • QOS In Action
                  • Where Do We Store QOS Values
                  • The QOS Collection
                  • Coming Togetherhellip
                  • Representative Example (3)
                  • A Highly Leveragable Investment
                  • Not Just Java Not A Problem
                  • A Really Nice Stack
                  • What Can We Do
                  • Secure Access To Redacted Data for Testing
                  • Get It Programmatically Too
                  • Robust Validated Data Ingest
                  • Concept The control_ Collection
                  • Validation QOS and PQO via Web Services
                  • Are We Excited Yet
                  • Questions amp Answers
                  • Slide 53
                  • Concept DataProvider
                  • The RESTful Provider

                    10

                    A Query Filters Outbound Data

                    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                    11

                    How About Using It To Filter Inbounds

                    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                    12

                    $exists And $type Already in MQL

                    ldquonamerdquo$type2

                    $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

                    $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

                    Ensure ldquonamerdquo exists (because not null) and is a string

                    ldquoagerdquo optional but if exists must be a 32bit integer

                    ldquonamerdquo required as string and weight and height both required integers or both not present

                    13

                    hellip And MQL Goes Way Beyondhellip

                    $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                    14

                    A New MQL Validator Module Emerges

                    class MQLValidator ValidationResult validate(Map MQL Map data)

                    MongoDBJava Driver

                    Data Access Layer

                    Application

                    Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                    MQLValidator

                    15

                    MongoDBDB Engine

                    Migrating Capability into MongoDB

                    MongoDBJava Driver

                    MQLValidatorJava

                    Data Access Layer

                    MongoDBDB Engine

                    MongoDBJava Driver

                    MQLValidator

                    JavaData Access

                    Layer

                    bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                    and languages

                    MongoDBPython Driver

                    Application Application

                    16

                    Code For The FuturehellipToday

                    class DataAccessLayer

                    someWriteOperation(Map data)

                    if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                    17

                    But What About Today

                    MQLValidator

                    18

                    Temporary Filling PQL

                    bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                    PQL

                    19

                    Bridge MQL to PQL

                    class MQLValidator private PQLFilter pqlfilter

                    validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                    20

                    No PQL No Problem

                    class MQLValidator

                    validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                    21

                    MQL Is Easy To Navigate

                    $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                    bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                    22

                    Where Do Validations Come From

                    The Database

                    23

                    The Validations Collection

                    gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                    24

                    Various ldquoLevelsrdquo of Validation

                    gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                    25

                    Multiple Types Schema By Example

                    gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                    26

                    The Stack So Far

                    MongoDBJava Driver

                    MQLValidator

                    Data Access Layer

                    Application

                    ValidatorDBUtils

                    ValidatorDBUtils populates an MQLValidator object from MongoDB

                    PQLFilter

                    27

                    Representative Example

                    class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                    someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                    28

                    Redaction

                    29

                    Concept Post Query Operations (PQO)

                    ssn $hash model birthdate null

                    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                    30

                    Adopt MQL-like behavior

                    ldquossnrdquonull

                    ldquoaddressrdquo ldquoXXXXrdquo

                    ldquossnrdquo $substitute ldquossnmodelrdquo

                    Remove field by setting to null

                    Redact address with fixed value

                    Substitute SSN with a different correct consistent value

                    ldquocounterpartyrdquo $hash ldquoMD5rdquo

                    Hash counterparty name to consistent value

                    31

                    A New PostQuery Module Emerges

                    class PostQuery process(Map data Map operations)

                    PostQuery

                    MongoDBJava Driver

                    MQLValidator

                    Data Access Layer

                    Application

                    ValidatorDBUtils

                    PQLFilter

                    32

                    Where Do PQOs Come From

                    The Database

                    33

                    The Postquery Collection

                    gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                    34

                    The Stack Is Getting Rich

                    PostQuery

                    MongoDBJava Driver

                    MQLValidator

                    Data Access Layer

                    Application

                    ValidatorDBUtils

                    PQLFilter

                    PQODBUtils

                    35

                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                    someWriteOperation(Map data) hellip

                    someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                    36

                    Quality of Service

                    37

                    QOS In Action

                    class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                    Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                    38

                    Where Do We Store QOS Values

                    The Database

                    39

                    The QOS Collection

                    gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                    ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                    40

                    QOSDBUtils

                    Coming Togetherhellip

                    PostQuery

                    MongoDBJava Driver

                    MQLValidator

                    Data Access Layer

                    Application

                    ValidatorDBUtils

                    PQLFilter

                    PQODBUtils

                    QOS

                    41

                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                    someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                    42

                    QOSDBUtils

                    A Highly Leveragable Investment

                    PostQuery

                    MQLValidator

                    Data Access Layer 1

                    Application1

                    ValidatorDBUtils

                    PQLFilter

                    PQODBUtils

                    QOS

                    Application2

                    Data Access Layer 2

                    Application3

                    Application4

                    Data Access Layer 3

                    Application5

                    Application6Reusable For ALL Data Access Layer Logic

                    43

                    Not Just Java Not A Problem

                    DAL operations have little or no statehellip

                    Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                    Can you say hellip Web Service

                    44

                    A Really Nice Stack

                    MongoDBJava Driver

                    MQLValidator

                    Data Access Layer

                    JavaApplication

                    ValidatorDBUtils

                    PQLFilter

                    HTTP Endpoint

                    pythonApplication

                    curl

                    JSON over HTTP(S)

                    JSONlt-gtJava Maps

                    QOSDBUtils

                    PostQuery

                    PQODBUtils

                    QOS

                    45

                    What Can We Do

                    46

                    Secure Access To Redacted Data for Testing

                    $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                    $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                    $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                    47

                    Get It Programmatically Too

                    This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                    getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                    String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                    48

                    Robust Validated Data Ingest

                    $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                    $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                    49

                    Concept The control_ Collection

                    gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                    bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                    50

                    Validation QOS and PQO via Web Services

                    MongoDBJava Driver

                    MQLValidator

                    Data Access Layer

                    JavaApplication

                    ValidatorHTTPUtils

                    PQLFilter

                    pythonApplication

                    curl

                    JSON over HTTP(S)

                    QOSHTTPUtils

                    PostQuery

                    PQOHTTPUtils

                    QOS

                    HTTP Service

                    JSONlt-gtJava Maps

                    51

                    Are We Excited Yet

                    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                    and other Java resources

                    Questions amp Answers

                    54

                    Concept DataProvider

                    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                    Class MongoProvider implements DataProvider hellip

                    Class RESTfulProvider implements DataProvider hellip

                    55

                    The RESTful Provider

                    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                    • Slide 1
                    • Part 3 In The Data Management Series
                    • Congratulations At this Point Yoursquove
                    • The Next Stage Defend amp Leverage
                    • MongoDB Doesnrsquot Have These Things
                    • What Can We Do Today
                    • Write Some Code
                    • Starting Point The Data Access Layer
                    • Document Validation
                    • A Query Filters Outbound Data
                    • How About Using It To Filter Inbounds
                    • $exists And $type Already in MQL
                    • hellip And MQL Goes Way Beyondhellip
                    • A New MQL Validator Module Emerges
                    • Migrating Capability into MongoDB
                    • Code For The FuturehellipToday
                    • But What About Today
                    • Temporary Filling PQL
                    • Bridge MQL to PQL
                    • No PQL No Problem
                    • MQL Is Easy To Navigate
                    • Where Do Validations Come From
                    • The Validations Collection
                    • Various ldquoLevelsrdquo of Validation
                    • Multiple Types Schema By Example
                    • The Stack So Far
                    • Representative Example
                    • Redaction
                    • Concept Post Query Operations (PQO)
                    • Adopt MQL-like behavior
                    • A New PostQuery Module Emerges
                    • Where Do PQOs Come From
                    • The Postquery Collection
                    • The Stack Is Getting Rich
                    • Representative Example (2)
                    • Quality of Service
                    • QOS In Action
                    • Where Do We Store QOS Values
                    • The QOS Collection
                    • Coming Togetherhellip
                    • Representative Example (3)
                    • A Highly Leveragable Investment
                    • Not Just Java Not A Problem
                    • A Really Nice Stack
                    • What Can We Do
                    • Secure Access To Redacted Data for Testing
                    • Get It Programmatically Too
                    • Robust Validated Data Ingest
                    • Concept The control_ Collection
                    • Validation QOS and PQO via Web Services
                    • Are We Excited Yet
                    • Questions amp Answers
                    • Slide 53
                    • Concept DataProvider
                    • The RESTful Provider

                      11

                      How About Using It To Filter Inbounds

                      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                      12

                      $exists And $type Already in MQL

                      ldquonamerdquo$type2

                      $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

                      $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

                      Ensure ldquonamerdquo exists (because not null) and is a string

                      ldquoagerdquo optional but if exists must be a 32bit integer

                      ldquonamerdquo required as string and weight and height both required integers or both not present

                      13

                      hellip And MQL Goes Way Beyondhellip

                      $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                      14

                      A New MQL Validator Module Emerges

                      class MQLValidator ValidationResult validate(Map MQL Map data)

                      MongoDBJava Driver

                      Data Access Layer

                      Application

                      Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                      MQLValidator

                      15

                      MongoDBDB Engine

                      Migrating Capability into MongoDB

                      MongoDBJava Driver

                      MQLValidatorJava

                      Data Access Layer

                      MongoDBDB Engine

                      MongoDBJava Driver

                      MQLValidator

                      JavaData Access

                      Layer

                      bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                      and languages

                      MongoDBPython Driver

                      Application Application

                      16

                      Code For The FuturehellipToday

                      class DataAccessLayer

                      someWriteOperation(Map data)

                      if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                      17

                      But What About Today

                      MQLValidator

                      18

                      Temporary Filling PQL

                      bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                      PQL

                      19

                      Bridge MQL to PQL

                      class MQLValidator private PQLFilter pqlfilter

                      validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                      20

                      No PQL No Problem

                      class MQLValidator

                      validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                      21

                      MQL Is Easy To Navigate

                      $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                      bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                      22

                      Where Do Validations Come From

                      The Database

                      23

                      The Validations Collection

                      gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                      24

                      Various ldquoLevelsrdquo of Validation

                      gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                      25

                      Multiple Types Schema By Example

                      gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                      26

                      The Stack So Far

                      MongoDBJava Driver

                      MQLValidator

                      Data Access Layer

                      Application

                      ValidatorDBUtils

                      ValidatorDBUtils populates an MQLValidator object from MongoDB

                      PQLFilter

                      27

                      Representative Example

                      class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                      someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                      28

                      Redaction

                      29

                      Concept Post Query Operations (PQO)

                      ssn $hash model birthdate null

                      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                      30

                      Adopt MQL-like behavior

                      ldquossnrdquonull

                      ldquoaddressrdquo ldquoXXXXrdquo

                      ldquossnrdquo $substitute ldquossnmodelrdquo

                      Remove field by setting to null

                      Redact address with fixed value

                      Substitute SSN with a different correct consistent value

                      ldquocounterpartyrdquo $hash ldquoMD5rdquo

                      Hash counterparty name to consistent value

                      31

                      A New PostQuery Module Emerges

                      class PostQuery process(Map data Map operations)

                      PostQuery

                      MongoDBJava Driver

                      MQLValidator

                      Data Access Layer

                      Application

                      ValidatorDBUtils

                      PQLFilter

                      32

                      Where Do PQOs Come From

                      The Database

                      33

                      The Postquery Collection

                      gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                      34

                      The Stack Is Getting Rich

                      PostQuery

                      MongoDBJava Driver

                      MQLValidator

                      Data Access Layer

                      Application

                      ValidatorDBUtils

                      PQLFilter

                      PQODBUtils

                      35

                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                      someWriteOperation(Map data) hellip

                      someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                      36

                      Quality of Service

                      37

                      QOS In Action

                      class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                      Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                      38

                      Where Do We Store QOS Values

                      The Database

                      39

                      The QOS Collection

                      gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                      ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                      40

                      QOSDBUtils

                      Coming Togetherhellip

                      PostQuery

                      MongoDBJava Driver

                      MQLValidator

                      Data Access Layer

                      Application

                      ValidatorDBUtils

                      PQLFilter

                      PQODBUtils

                      QOS

                      41

                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                      someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                      42

                      QOSDBUtils

                      A Highly Leveragable Investment

                      PostQuery

                      MQLValidator

                      Data Access Layer 1

                      Application1

                      ValidatorDBUtils

                      PQLFilter

                      PQODBUtils

                      QOS

                      Application2

                      Data Access Layer 2

                      Application3

                      Application4

                      Data Access Layer 3

                      Application5

                      Application6Reusable For ALL Data Access Layer Logic

                      43

                      Not Just Java Not A Problem

                      DAL operations have little or no statehellip

                      Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                      Can you say hellip Web Service

                      44

                      A Really Nice Stack

                      MongoDBJava Driver

                      MQLValidator

                      Data Access Layer

                      JavaApplication

                      ValidatorDBUtils

                      PQLFilter

                      HTTP Endpoint

                      pythonApplication

                      curl

                      JSON over HTTP(S)

                      JSONlt-gtJava Maps

                      QOSDBUtils

                      PostQuery

                      PQODBUtils

                      QOS

                      45

                      What Can We Do

                      46

                      Secure Access To Redacted Data for Testing

                      $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                      $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                      $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                      47

                      Get It Programmatically Too

                      This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                      getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                      String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                      48

                      Robust Validated Data Ingest

                      $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                      $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                      49

                      Concept The control_ Collection

                      gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                      bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                      50

                      Validation QOS and PQO via Web Services

                      MongoDBJava Driver

                      MQLValidator

                      Data Access Layer

                      JavaApplication

                      ValidatorHTTPUtils

                      PQLFilter

                      pythonApplication

                      curl

                      JSON over HTTP(S)

                      QOSHTTPUtils

                      PostQuery

                      PQOHTTPUtils

                      QOS

                      HTTP Service

                      JSONlt-gtJava Maps

                      51

                      Are We Excited Yet

                      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                      and other Java resources

                      Questions amp Answers

                      54

                      Concept DataProvider

                      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                      Class MongoProvider implements DataProvider hellip

                      Class RESTfulProvider implements DataProvider hellip

                      55

                      The RESTful Provider

                      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                      • Slide 1
                      • Part 3 In The Data Management Series
                      • Congratulations At this Point Yoursquove
                      • The Next Stage Defend amp Leverage
                      • MongoDB Doesnrsquot Have These Things
                      • What Can We Do Today
                      • Write Some Code
                      • Starting Point The Data Access Layer
                      • Document Validation
                      • A Query Filters Outbound Data
                      • How About Using It To Filter Inbounds
                      • $exists And $type Already in MQL
                      • hellip And MQL Goes Way Beyondhellip
                      • A New MQL Validator Module Emerges
                      • Migrating Capability into MongoDB
                      • Code For The FuturehellipToday
                      • But What About Today
                      • Temporary Filling PQL
                      • Bridge MQL to PQL
                      • No PQL No Problem
                      • MQL Is Easy To Navigate
                      • Where Do Validations Come From
                      • The Validations Collection
                      • Various ldquoLevelsrdquo of Validation
                      • Multiple Types Schema By Example
                      • The Stack So Far
                      • Representative Example
                      • Redaction
                      • Concept Post Query Operations (PQO)
                      • Adopt MQL-like behavior
                      • A New PostQuery Module Emerges
                      • Where Do PQOs Come From
                      • The Postquery Collection
                      • The Stack Is Getting Rich
                      • Representative Example (2)
                      • Quality of Service
                      • QOS In Action
                      • Where Do We Store QOS Values
                      • The QOS Collection
                      • Coming Togetherhellip
                      • Representative Example (3)
                      • A Highly Leveragable Investment
                      • Not Just Java Not A Problem
                      • A Really Nice Stack
                      • What Can We Do
                      • Secure Access To Redacted Data for Testing
                      • Get It Programmatically Too
                      • Robust Validated Data Ingest
                      • Concept The control_ Collection
                      • Validation QOS and PQO via Web Services
                      • Are We Excited Yet
                      • Questions amp Answers
                      • Slide 53
                      • Concept DataProvider
                      • The RESTful Provider

                        12

                        $exists And $type Already in MQL

                        ldquonamerdquo$type2

                        $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

                        $and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

                        Ensure ldquonamerdquo exists (because not null) and is a string

                        ldquoagerdquo optional but if exists must be a 32bit integer

                        ldquonamerdquo required as string and weight and height both required integers or both not present

                        13

                        hellip And MQL Goes Way Beyondhellip

                        $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                        14

                        A New MQL Validator Module Emerges

                        class MQLValidator ValidationResult validate(Map MQL Map data)

                        MongoDBJava Driver

                        Data Access Layer

                        Application

                        Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                        MQLValidator

                        15

                        MongoDBDB Engine

                        Migrating Capability into MongoDB

                        MongoDBJava Driver

                        MQLValidatorJava

                        Data Access Layer

                        MongoDBDB Engine

                        MongoDBJava Driver

                        MQLValidator

                        JavaData Access

                        Layer

                        bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                        and languages

                        MongoDBPython Driver

                        Application Application

                        16

                        Code For The FuturehellipToday

                        class DataAccessLayer

                        someWriteOperation(Map data)

                        if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                        17

                        But What About Today

                        MQLValidator

                        18

                        Temporary Filling PQL

                        bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                        PQL

                        19

                        Bridge MQL to PQL

                        class MQLValidator private PQLFilter pqlfilter

                        validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                        20

                        No PQL No Problem

                        class MQLValidator

                        validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                        21

                        MQL Is Easy To Navigate

                        $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                        bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                        22

                        Where Do Validations Come From

                        The Database

                        23

                        The Validations Collection

                        gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                        24

                        Various ldquoLevelsrdquo of Validation

                        gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                        25

                        Multiple Types Schema By Example

                        gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                        26

                        The Stack So Far

                        MongoDBJava Driver

                        MQLValidator

                        Data Access Layer

                        Application

                        ValidatorDBUtils

                        ValidatorDBUtils populates an MQLValidator object from MongoDB

                        PQLFilter

                        27

                        Representative Example

                        class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                        someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                        28

                        Redaction

                        29

                        Concept Post Query Operations (PQO)

                        ssn $hash model birthdate null

                        $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                        30

                        Adopt MQL-like behavior

                        ldquossnrdquonull

                        ldquoaddressrdquo ldquoXXXXrdquo

                        ldquossnrdquo $substitute ldquossnmodelrdquo

                        Remove field by setting to null

                        Redact address with fixed value

                        Substitute SSN with a different correct consistent value

                        ldquocounterpartyrdquo $hash ldquoMD5rdquo

                        Hash counterparty name to consistent value

                        31

                        A New PostQuery Module Emerges

                        class PostQuery process(Map data Map operations)

                        PostQuery

                        MongoDBJava Driver

                        MQLValidator

                        Data Access Layer

                        Application

                        ValidatorDBUtils

                        PQLFilter

                        32

                        Where Do PQOs Come From

                        The Database

                        33

                        The Postquery Collection

                        gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                        34

                        The Stack Is Getting Rich

                        PostQuery

                        MongoDBJava Driver

                        MQLValidator

                        Data Access Layer

                        Application

                        ValidatorDBUtils

                        PQLFilter

                        PQODBUtils

                        35

                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                        someWriteOperation(Map data) hellip

                        someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                        36

                        Quality of Service

                        37

                        QOS In Action

                        class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                        Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                        38

                        Where Do We Store QOS Values

                        The Database

                        39

                        The QOS Collection

                        gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                        ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                        40

                        QOSDBUtils

                        Coming Togetherhellip

                        PostQuery

                        MongoDBJava Driver

                        MQLValidator

                        Data Access Layer

                        Application

                        ValidatorDBUtils

                        PQLFilter

                        PQODBUtils

                        QOS

                        41

                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                        someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                        42

                        QOSDBUtils

                        A Highly Leveragable Investment

                        PostQuery

                        MQLValidator

                        Data Access Layer 1

                        Application1

                        ValidatorDBUtils

                        PQLFilter

                        PQODBUtils

                        QOS

                        Application2

                        Data Access Layer 2

                        Application3

                        Application4

                        Data Access Layer 3

                        Application5

                        Application6Reusable For ALL Data Access Layer Logic

                        43

                        Not Just Java Not A Problem

                        DAL operations have little or no statehellip

                        Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                        Can you say hellip Web Service

                        44

                        A Really Nice Stack

                        MongoDBJava Driver

                        MQLValidator

                        Data Access Layer

                        JavaApplication

                        ValidatorDBUtils

                        PQLFilter

                        HTTP Endpoint

                        pythonApplication

                        curl

                        JSON over HTTP(S)

                        JSONlt-gtJava Maps

                        QOSDBUtils

                        PostQuery

                        PQODBUtils

                        QOS

                        45

                        What Can We Do

                        46

                        Secure Access To Redacted Data for Testing

                        $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                        $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                        $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                        47

                        Get It Programmatically Too

                        This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                        getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                        String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                        48

                        Robust Validated Data Ingest

                        $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                        $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                        49

                        Concept The control_ Collection

                        gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                        bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                        50

                        Validation QOS and PQO via Web Services

                        MongoDBJava Driver

                        MQLValidator

                        Data Access Layer

                        JavaApplication

                        ValidatorHTTPUtils

                        PQLFilter

                        pythonApplication

                        curl

                        JSON over HTTP(S)

                        QOSHTTPUtils

                        PostQuery

                        PQOHTTPUtils

                        QOS

                        HTTP Service

                        JSONlt-gtJava Maps

                        51

                        Are We Excited Yet

                        Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                        and other Java resources

                        Questions amp Answers

                        54

                        Concept DataProvider

                        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                        Class MongoProvider implements DataProvider hellip

                        Class RESTfulProvider implements DataProvider hellip

                        55

                        The RESTful Provider

                        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                        • Slide 1
                        • Part 3 In The Data Management Series
                        • Congratulations At this Point Yoursquove
                        • The Next Stage Defend amp Leverage
                        • MongoDB Doesnrsquot Have These Things
                        • What Can We Do Today
                        • Write Some Code
                        • Starting Point The Data Access Layer
                        • Document Validation
                        • A Query Filters Outbound Data
                        • How About Using It To Filter Inbounds
                        • $exists And $type Already in MQL
                        • hellip And MQL Goes Way Beyondhellip
                        • A New MQL Validator Module Emerges
                        • Migrating Capability into MongoDB
                        • Code For The FuturehellipToday
                        • But What About Today
                        • Temporary Filling PQL
                        • Bridge MQL to PQL
                        • No PQL No Problem
                        • MQL Is Easy To Navigate
                        • Where Do Validations Come From
                        • The Validations Collection
                        • Various ldquoLevelsrdquo of Validation
                        • Multiple Types Schema By Example
                        • The Stack So Far
                        • Representative Example
                        • Redaction
                        • Concept Post Query Operations (PQO)
                        • Adopt MQL-like behavior
                        • A New PostQuery Module Emerges
                        • Where Do PQOs Come From
                        • The Postquery Collection
                        • The Stack Is Getting Rich
                        • Representative Example (2)
                        • Quality of Service
                        • QOS In Action
                        • Where Do We Store QOS Values
                        • The QOS Collection
                        • Coming Togetherhellip
                        • Representative Example (3)
                        • A Highly Leveragable Investment
                        • Not Just Java Not A Problem
                        • A Really Nice Stack
                        • What Can We Do
                        • Secure Access To Redacted Data for Testing
                        • Get It Programmatically Too
                        • Robust Validated Data Ingest
                        • Concept The control_ Collection
                        • Validation QOS and PQO via Web Services
                        • Are We Excited Yet
                        • Questions amp Answers
                        • Slide 53
                        • Concept DataProvider
                        • The RESTful Provider

                          13

                          hellip And MQL Goes Way Beyondhellip

                          $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                          14

                          A New MQL Validator Module Emerges

                          class MQLValidator ValidationResult validate(Map MQL Map data)

                          MongoDBJava Driver

                          Data Access Layer

                          Application

                          Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                          MQLValidator

                          15

                          MongoDBDB Engine

                          Migrating Capability into MongoDB

                          MongoDBJava Driver

                          MQLValidatorJava

                          Data Access Layer

                          MongoDBDB Engine

                          MongoDBJava Driver

                          MQLValidator

                          JavaData Access

                          Layer

                          bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                          and languages

                          MongoDBPython Driver

                          Application Application

                          16

                          Code For The FuturehellipToday

                          class DataAccessLayer

                          someWriteOperation(Map data)

                          if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                          17

                          But What About Today

                          MQLValidator

                          18

                          Temporary Filling PQL

                          bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                          PQL

                          19

                          Bridge MQL to PQL

                          class MQLValidator private PQLFilter pqlfilter

                          validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                          20

                          No PQL No Problem

                          class MQLValidator

                          validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                          21

                          MQL Is Easy To Navigate

                          $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                          bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                          22

                          Where Do Validations Come From

                          The Database

                          23

                          The Validations Collection

                          gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                          24

                          Various ldquoLevelsrdquo of Validation

                          gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                          25

                          Multiple Types Schema By Example

                          gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                          26

                          The Stack So Far

                          MongoDBJava Driver

                          MQLValidator

                          Data Access Layer

                          Application

                          ValidatorDBUtils

                          ValidatorDBUtils populates an MQLValidator object from MongoDB

                          PQLFilter

                          27

                          Representative Example

                          class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                          someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                          28

                          Redaction

                          29

                          Concept Post Query Operations (PQO)

                          ssn $hash model birthdate null

                          $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                          30

                          Adopt MQL-like behavior

                          ldquossnrdquonull

                          ldquoaddressrdquo ldquoXXXXrdquo

                          ldquossnrdquo $substitute ldquossnmodelrdquo

                          Remove field by setting to null

                          Redact address with fixed value

                          Substitute SSN with a different correct consistent value

                          ldquocounterpartyrdquo $hash ldquoMD5rdquo

                          Hash counterparty name to consistent value

                          31

                          A New PostQuery Module Emerges

                          class PostQuery process(Map data Map operations)

                          PostQuery

                          MongoDBJava Driver

                          MQLValidator

                          Data Access Layer

                          Application

                          ValidatorDBUtils

                          PQLFilter

                          32

                          Where Do PQOs Come From

                          The Database

                          33

                          The Postquery Collection

                          gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                          34

                          The Stack Is Getting Rich

                          PostQuery

                          MongoDBJava Driver

                          MQLValidator

                          Data Access Layer

                          Application

                          ValidatorDBUtils

                          PQLFilter

                          PQODBUtils

                          35

                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                          someWriteOperation(Map data) hellip

                          someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                          36

                          Quality of Service

                          37

                          QOS In Action

                          class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                          Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                          38

                          Where Do We Store QOS Values

                          The Database

                          39

                          The QOS Collection

                          gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                          ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                          40

                          QOSDBUtils

                          Coming Togetherhellip

                          PostQuery

                          MongoDBJava Driver

                          MQLValidator

                          Data Access Layer

                          Application

                          ValidatorDBUtils

                          PQLFilter

                          PQODBUtils

                          QOS

                          41

                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                          someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                          42

                          QOSDBUtils

                          A Highly Leveragable Investment

                          PostQuery

                          MQLValidator

                          Data Access Layer 1

                          Application1

                          ValidatorDBUtils

                          PQLFilter

                          PQODBUtils

                          QOS

                          Application2

                          Data Access Layer 2

                          Application3

                          Application4

                          Data Access Layer 3

                          Application5

                          Application6Reusable For ALL Data Access Layer Logic

                          43

                          Not Just Java Not A Problem

                          DAL operations have little or no statehellip

                          Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                          Can you say hellip Web Service

                          44

                          A Really Nice Stack

                          MongoDBJava Driver

                          MQLValidator

                          Data Access Layer

                          JavaApplication

                          ValidatorDBUtils

                          PQLFilter

                          HTTP Endpoint

                          pythonApplication

                          curl

                          JSON over HTTP(S)

                          JSONlt-gtJava Maps

                          QOSDBUtils

                          PostQuery

                          PQODBUtils

                          QOS

                          45

                          What Can We Do

                          46

                          Secure Access To Redacted Data for Testing

                          $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                          $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                          $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                          47

                          Get It Programmatically Too

                          This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                          getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                          String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                          48

                          Robust Validated Data Ingest

                          $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                          $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                          49

                          Concept The control_ Collection

                          gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                          bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                          50

                          Validation QOS and PQO via Web Services

                          MongoDBJava Driver

                          MQLValidator

                          Data Access Layer

                          JavaApplication

                          ValidatorHTTPUtils

                          PQLFilter

                          pythonApplication

                          curl

                          JSON over HTTP(S)

                          QOSHTTPUtils

                          PostQuery

                          PQOHTTPUtils

                          QOS

                          HTTP Service

                          JSONlt-gtJava Maps

                          51

                          Are We Excited Yet

                          Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                          and other Java resources

                          Questions amp Answers

                          54

                          Concept DataProvider

                          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                          Class MongoProvider implements DataProvider hellip

                          Class RESTfulProvider implements DataProvider hellip

                          55

                          The RESTful Provider

                          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                          • Slide 1
                          • Part 3 In The Data Management Series
                          • Congratulations At this Point Yoursquove
                          • The Next Stage Defend amp Leverage
                          • MongoDB Doesnrsquot Have These Things
                          • What Can We Do Today
                          • Write Some Code
                          • Starting Point The Data Access Layer
                          • Document Validation
                          • A Query Filters Outbound Data
                          • How About Using It To Filter Inbounds
                          • $exists And $type Already in MQL
                          • hellip And MQL Goes Way Beyondhellip
                          • A New MQL Validator Module Emerges
                          • Migrating Capability into MongoDB
                          • Code For The FuturehellipToday
                          • But What About Today
                          • Temporary Filling PQL
                          • Bridge MQL to PQL
                          • No PQL No Problem
                          • MQL Is Easy To Navigate
                          • Where Do Validations Come From
                          • The Validations Collection
                          • Various ldquoLevelsrdquo of Validation
                          • Multiple Types Schema By Example
                          • The Stack So Far
                          • Representative Example
                          • Redaction
                          • Concept Post Query Operations (PQO)
                          • Adopt MQL-like behavior
                          • A New PostQuery Module Emerges
                          • Where Do PQOs Come From
                          • The Postquery Collection
                          • The Stack Is Getting Rich
                          • Representative Example (2)
                          • Quality of Service
                          • QOS In Action
                          • Where Do We Store QOS Values
                          • The QOS Collection
                          • Coming Togetherhellip
                          • Representative Example (3)
                          • A Highly Leveragable Investment
                          • Not Just Java Not A Problem
                          • A Really Nice Stack
                          • What Can We Do
                          • Secure Access To Redacted Data for Testing
                          • Get It Programmatically Too
                          • Robust Validated Data Ingest
                          • Concept The control_ Collection
                          • Validation QOS and PQO via Web Services
                          • Are We Excited Yet
                          • Questions amp Answers
                          • Slide 53
                          • Concept DataProvider
                          • The RESTful Provider

                            14

                            A New MQL Validator Module Emerges

                            class MQLValidator ValidationResult validate(Map MQL Map data)

                            MongoDBJava Driver

                            Data Access Layer

                            Application

                            Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

                            MQLValidator

                            15

                            MongoDBDB Engine

                            Migrating Capability into MongoDB

                            MongoDBJava Driver

                            MQLValidatorJava

                            Data Access Layer

                            MongoDBDB Engine

                            MongoDBJava Driver

                            MQLValidator

                            JavaData Access

                            Layer

                            bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                            and languages

                            MongoDBPython Driver

                            Application Application

                            16

                            Code For The FuturehellipToday

                            class DataAccessLayer

                            someWriteOperation(Map data)

                            if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                            17

                            But What About Today

                            MQLValidator

                            18

                            Temporary Filling PQL

                            bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                            PQL

                            19

                            Bridge MQL to PQL

                            class MQLValidator private PQLFilter pqlfilter

                            validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                            20

                            No PQL No Problem

                            class MQLValidator

                            validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                            21

                            MQL Is Easy To Navigate

                            $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                            bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                            22

                            Where Do Validations Come From

                            The Database

                            23

                            The Validations Collection

                            gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                            24

                            Various ldquoLevelsrdquo of Validation

                            gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                            25

                            Multiple Types Schema By Example

                            gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                            26

                            The Stack So Far

                            MongoDBJava Driver

                            MQLValidator

                            Data Access Layer

                            Application

                            ValidatorDBUtils

                            ValidatorDBUtils populates an MQLValidator object from MongoDB

                            PQLFilter

                            27

                            Representative Example

                            class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                            someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                            28

                            Redaction

                            29

                            Concept Post Query Operations (PQO)

                            ssn $hash model birthdate null

                            $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                            30

                            Adopt MQL-like behavior

                            ldquossnrdquonull

                            ldquoaddressrdquo ldquoXXXXrdquo

                            ldquossnrdquo $substitute ldquossnmodelrdquo

                            Remove field by setting to null

                            Redact address with fixed value

                            Substitute SSN with a different correct consistent value

                            ldquocounterpartyrdquo $hash ldquoMD5rdquo

                            Hash counterparty name to consistent value

                            31

                            A New PostQuery Module Emerges

                            class PostQuery process(Map data Map operations)

                            PostQuery

                            MongoDBJava Driver

                            MQLValidator

                            Data Access Layer

                            Application

                            ValidatorDBUtils

                            PQLFilter

                            32

                            Where Do PQOs Come From

                            The Database

                            33

                            The Postquery Collection

                            gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                            34

                            The Stack Is Getting Rich

                            PostQuery

                            MongoDBJava Driver

                            MQLValidator

                            Data Access Layer

                            Application

                            ValidatorDBUtils

                            PQLFilter

                            PQODBUtils

                            35

                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                            someWriteOperation(Map data) hellip

                            someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                            36

                            Quality of Service

                            37

                            QOS In Action

                            class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                            Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                            38

                            Where Do We Store QOS Values

                            The Database

                            39

                            The QOS Collection

                            gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                            ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                            40

                            QOSDBUtils

                            Coming Togetherhellip

                            PostQuery

                            MongoDBJava Driver

                            MQLValidator

                            Data Access Layer

                            Application

                            ValidatorDBUtils

                            PQLFilter

                            PQODBUtils

                            QOS

                            41

                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                            someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                            42

                            QOSDBUtils

                            A Highly Leveragable Investment

                            PostQuery

                            MQLValidator

                            Data Access Layer 1

                            Application1

                            ValidatorDBUtils

                            PQLFilter

                            PQODBUtils

                            QOS

                            Application2

                            Data Access Layer 2

                            Application3

                            Application4

                            Data Access Layer 3

                            Application5

                            Application6Reusable For ALL Data Access Layer Logic

                            43

                            Not Just Java Not A Problem

                            DAL operations have little or no statehellip

                            Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                            Can you say hellip Web Service

                            44

                            A Really Nice Stack

                            MongoDBJava Driver

                            MQLValidator

                            Data Access Layer

                            JavaApplication

                            ValidatorDBUtils

                            PQLFilter

                            HTTP Endpoint

                            pythonApplication

                            curl

                            JSON over HTTP(S)

                            JSONlt-gtJava Maps

                            QOSDBUtils

                            PostQuery

                            PQODBUtils

                            QOS

                            45

                            What Can We Do

                            46

                            Secure Access To Redacted Data for Testing

                            $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                            $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                            $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                            47

                            Get It Programmatically Too

                            This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                            getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                            String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                            48

                            Robust Validated Data Ingest

                            $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                            $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                            49

                            Concept The control_ Collection

                            gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                            bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                            50

                            Validation QOS and PQO via Web Services

                            MongoDBJava Driver

                            MQLValidator

                            Data Access Layer

                            JavaApplication

                            ValidatorHTTPUtils

                            PQLFilter

                            pythonApplication

                            curl

                            JSON over HTTP(S)

                            QOSHTTPUtils

                            PostQuery

                            PQOHTTPUtils

                            QOS

                            HTTP Service

                            JSONlt-gtJava Maps

                            51

                            Are We Excited Yet

                            Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                            and other Java resources

                            Questions amp Answers

                            54

                            Concept DataProvider

                            public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                            Class MongoProvider implements DataProvider hellip

                            Class RESTfulProvider implements DataProvider hellip

                            55

                            The RESTful Provider

                            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                            • Slide 1
                            • Part 3 In The Data Management Series
                            • Congratulations At this Point Yoursquove
                            • The Next Stage Defend amp Leverage
                            • MongoDB Doesnrsquot Have These Things
                            • What Can We Do Today
                            • Write Some Code
                            • Starting Point The Data Access Layer
                            • Document Validation
                            • A Query Filters Outbound Data
                            • How About Using It To Filter Inbounds
                            • $exists And $type Already in MQL
                            • hellip And MQL Goes Way Beyondhellip
                            • A New MQL Validator Module Emerges
                            • Migrating Capability into MongoDB
                            • Code For The FuturehellipToday
                            • But What About Today
                            • Temporary Filling PQL
                            • Bridge MQL to PQL
                            • No PQL No Problem
                            • MQL Is Easy To Navigate
                            • Where Do Validations Come From
                            • The Validations Collection
                            • Various ldquoLevelsrdquo of Validation
                            • Multiple Types Schema By Example
                            • The Stack So Far
                            • Representative Example
                            • Redaction
                            • Concept Post Query Operations (PQO)
                            • Adopt MQL-like behavior
                            • A New PostQuery Module Emerges
                            • Where Do PQOs Come From
                            • The Postquery Collection
                            • The Stack Is Getting Rich
                            • Representative Example (2)
                            • Quality of Service
                            • QOS In Action
                            • Where Do We Store QOS Values
                            • The QOS Collection
                            • Coming Togetherhellip
                            • Representative Example (3)
                            • A Highly Leveragable Investment
                            • Not Just Java Not A Problem
                            • A Really Nice Stack
                            • What Can We Do
                            • Secure Access To Redacted Data for Testing
                            • Get It Programmatically Too
                            • Robust Validated Data Ingest
                            • Concept The control_ Collection
                            • Validation QOS and PQO via Web Services
                            • Are We Excited Yet
                            • Questions amp Answers
                            • Slide 53
                            • Concept DataProvider
                            • The RESTful Provider

                              15

                              MongoDBDB Engine

                              Migrating Capability into MongoDB

                              MongoDBJava Driver

                              MQLValidatorJava

                              Data Access Layer

                              MongoDBDB Engine

                              MongoDBJava Driver

                              MQLValidator

                              JavaData Access

                              Layer

                              bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

                              and languages

                              MongoDBPython Driver

                              Application Application

                              16

                              Code For The FuturehellipToday

                              class DataAccessLayer

                              someWriteOperation(Map data)

                              if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                              17

                              But What About Today

                              MQLValidator

                              18

                              Temporary Filling PQL

                              bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                              PQL

                              19

                              Bridge MQL to PQL

                              class MQLValidator private PQLFilter pqlfilter

                              validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                              20

                              No PQL No Problem

                              class MQLValidator

                              validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                              21

                              MQL Is Easy To Navigate

                              $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                              bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                              22

                              Where Do Validations Come From

                              The Database

                              23

                              The Validations Collection

                              gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                              24

                              Various ldquoLevelsrdquo of Validation

                              gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                              25

                              Multiple Types Schema By Example

                              gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                              26

                              The Stack So Far

                              MongoDBJava Driver

                              MQLValidator

                              Data Access Layer

                              Application

                              ValidatorDBUtils

                              ValidatorDBUtils populates an MQLValidator object from MongoDB

                              PQLFilter

                              27

                              Representative Example

                              class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                              someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                              28

                              Redaction

                              29

                              Concept Post Query Operations (PQO)

                              ssn $hash model birthdate null

                              $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                              30

                              Adopt MQL-like behavior

                              ldquossnrdquonull

                              ldquoaddressrdquo ldquoXXXXrdquo

                              ldquossnrdquo $substitute ldquossnmodelrdquo

                              Remove field by setting to null

                              Redact address with fixed value

                              Substitute SSN with a different correct consistent value

                              ldquocounterpartyrdquo $hash ldquoMD5rdquo

                              Hash counterparty name to consistent value

                              31

                              A New PostQuery Module Emerges

                              class PostQuery process(Map data Map operations)

                              PostQuery

                              MongoDBJava Driver

                              MQLValidator

                              Data Access Layer

                              Application

                              ValidatorDBUtils

                              PQLFilter

                              32

                              Where Do PQOs Come From

                              The Database

                              33

                              The Postquery Collection

                              gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                              34

                              The Stack Is Getting Rich

                              PostQuery

                              MongoDBJava Driver

                              MQLValidator

                              Data Access Layer

                              Application

                              ValidatorDBUtils

                              PQLFilter

                              PQODBUtils

                              35

                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                              someWriteOperation(Map data) hellip

                              someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                              36

                              Quality of Service

                              37

                              QOS In Action

                              class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                              Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                              38

                              Where Do We Store QOS Values

                              The Database

                              39

                              The QOS Collection

                              gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                              ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                              40

                              QOSDBUtils

                              Coming Togetherhellip

                              PostQuery

                              MongoDBJava Driver

                              MQLValidator

                              Data Access Layer

                              Application

                              ValidatorDBUtils

                              PQLFilter

                              PQODBUtils

                              QOS

                              41

                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                              someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                              42

                              QOSDBUtils

                              A Highly Leveragable Investment

                              PostQuery

                              MQLValidator

                              Data Access Layer 1

                              Application1

                              ValidatorDBUtils

                              PQLFilter

                              PQODBUtils

                              QOS

                              Application2

                              Data Access Layer 2

                              Application3

                              Application4

                              Data Access Layer 3

                              Application5

                              Application6Reusable For ALL Data Access Layer Logic

                              43

                              Not Just Java Not A Problem

                              DAL operations have little or no statehellip

                              Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                              Can you say hellip Web Service

                              44

                              A Really Nice Stack

                              MongoDBJava Driver

                              MQLValidator

                              Data Access Layer

                              JavaApplication

                              ValidatorDBUtils

                              PQLFilter

                              HTTP Endpoint

                              pythonApplication

                              curl

                              JSON over HTTP(S)

                              JSONlt-gtJava Maps

                              QOSDBUtils

                              PostQuery

                              PQODBUtils

                              QOS

                              45

                              What Can We Do

                              46

                              Secure Access To Redacted Data for Testing

                              $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                              $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                              $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                              47

                              Get It Programmatically Too

                              This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                              getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                              String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                              48

                              Robust Validated Data Ingest

                              $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                              $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                              49

                              Concept The control_ Collection

                              gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                              bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                              50

                              Validation QOS and PQO via Web Services

                              MongoDBJava Driver

                              MQLValidator

                              Data Access Layer

                              JavaApplication

                              ValidatorHTTPUtils

                              PQLFilter

                              pythonApplication

                              curl

                              JSON over HTTP(S)

                              QOSHTTPUtils

                              PostQuery

                              PQOHTTPUtils

                              QOS

                              HTTP Service

                              JSONlt-gtJava Maps

                              51

                              Are We Excited Yet

                              Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                              and other Java resources

                              Questions amp Answers

                              54

                              Concept DataProvider

                              public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                              Class MongoProvider implements DataProvider hellip

                              Class RESTfulProvider implements DataProvider hellip

                              55

                              The RESTful Provider

                              class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                              fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                              url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                              HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                              • Slide 1
                              • Part 3 In The Data Management Series
                              • Congratulations At this Point Yoursquove
                              • The Next Stage Defend amp Leverage
                              • MongoDB Doesnrsquot Have These Things
                              • What Can We Do Today
                              • Write Some Code
                              • Starting Point The Data Access Layer
                              • Document Validation
                              • A Query Filters Outbound Data
                              • How About Using It To Filter Inbounds
                              • $exists And $type Already in MQL
                              • hellip And MQL Goes Way Beyondhellip
                              • A New MQL Validator Module Emerges
                              • Migrating Capability into MongoDB
                              • Code For The FuturehellipToday
                              • But What About Today
                              • Temporary Filling PQL
                              • Bridge MQL to PQL
                              • No PQL No Problem
                              • MQL Is Easy To Navigate
                              • Where Do Validations Come From
                              • The Validations Collection
                              • Various ldquoLevelsrdquo of Validation
                              • Multiple Types Schema By Example
                              • The Stack So Far
                              • Representative Example
                              • Redaction
                              • Concept Post Query Operations (PQO)
                              • Adopt MQL-like behavior
                              • A New PostQuery Module Emerges
                              • Where Do PQOs Come From
                              • The Postquery Collection
                              • The Stack Is Getting Rich
                              • Representative Example (2)
                              • Quality of Service
                              • QOS In Action
                              • Where Do We Store QOS Values
                              • The QOS Collection
                              • Coming Togetherhellip
                              • Representative Example (3)
                              • A Highly Leveragable Investment
                              • Not Just Java Not A Problem
                              • A Really Nice Stack
                              • What Can We Do
                              • Secure Access To Redacted Data for Testing
                              • Get It Programmatically Too
                              • Robust Validated Data Ingest
                              • Concept The control_ Collection
                              • Validation QOS and PQO via Web Services
                              • Are We Excited Yet
                              • Questions amp Answers
                              • Slide 53
                              • Concept DataProvider
                              • The RESTful Provider

                                16

                                Code For The FuturehellipToday

                                class DataAccessLayer

                                someWriteOperation(Map data)

                                if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

                                17

                                But What About Today

                                MQLValidator

                                18

                                Temporary Filling PQL

                                bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                                PQL

                                19

                                Bridge MQL to PQL

                                class MQLValidator private PQLFilter pqlfilter

                                validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                                20

                                No PQL No Problem

                                class MQLValidator

                                validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                                21

                                MQL Is Easy To Navigate

                                $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                                bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                                22

                                Where Do Validations Come From

                                The Database

                                23

                                The Validations Collection

                                gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                24

                                Various ldquoLevelsrdquo of Validation

                                gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                25

                                Multiple Types Schema By Example

                                gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                26

                                The Stack So Far

                                MongoDBJava Driver

                                MQLValidator

                                Data Access Layer

                                Application

                                ValidatorDBUtils

                                ValidatorDBUtils populates an MQLValidator object from MongoDB

                                PQLFilter

                                27

                                Representative Example

                                class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                28

                                Redaction

                                29

                                Concept Post Query Operations (PQO)

                                ssn $hash model birthdate null

                                $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                30

                                Adopt MQL-like behavior

                                ldquossnrdquonull

                                ldquoaddressrdquo ldquoXXXXrdquo

                                ldquossnrdquo $substitute ldquossnmodelrdquo

                                Remove field by setting to null

                                Redact address with fixed value

                                Substitute SSN with a different correct consistent value

                                ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                Hash counterparty name to consistent value

                                31

                                A New PostQuery Module Emerges

                                class PostQuery process(Map data Map operations)

                                PostQuery

                                MongoDBJava Driver

                                MQLValidator

                                Data Access Layer

                                Application

                                ValidatorDBUtils

                                PQLFilter

                                32

                                Where Do PQOs Come From

                                The Database

                                33

                                The Postquery Collection

                                gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                34

                                The Stack Is Getting Rich

                                PostQuery

                                MongoDBJava Driver

                                MQLValidator

                                Data Access Layer

                                Application

                                ValidatorDBUtils

                                PQLFilter

                                PQODBUtils

                                35

                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                someWriteOperation(Map data) hellip

                                someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                36

                                Quality of Service

                                37

                                QOS In Action

                                class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                38

                                Where Do We Store QOS Values

                                The Database

                                39

                                The QOS Collection

                                gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                40

                                QOSDBUtils

                                Coming Togetherhellip

                                PostQuery

                                MongoDBJava Driver

                                MQLValidator

                                Data Access Layer

                                Application

                                ValidatorDBUtils

                                PQLFilter

                                PQODBUtils

                                QOS

                                41

                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                42

                                QOSDBUtils

                                A Highly Leveragable Investment

                                PostQuery

                                MQLValidator

                                Data Access Layer 1

                                Application1

                                ValidatorDBUtils

                                PQLFilter

                                PQODBUtils

                                QOS

                                Application2

                                Data Access Layer 2

                                Application3

                                Application4

                                Data Access Layer 3

                                Application5

                                Application6Reusable For ALL Data Access Layer Logic

                                43

                                Not Just Java Not A Problem

                                DAL operations have little or no statehellip

                                Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                Can you say hellip Web Service

                                44

                                A Really Nice Stack

                                MongoDBJava Driver

                                MQLValidator

                                Data Access Layer

                                JavaApplication

                                ValidatorDBUtils

                                PQLFilter

                                HTTP Endpoint

                                pythonApplication

                                curl

                                JSON over HTTP(S)

                                JSONlt-gtJava Maps

                                QOSDBUtils

                                PostQuery

                                PQODBUtils

                                QOS

                                45

                                What Can We Do

                                46

                                Secure Access To Redacted Data for Testing

                                $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                47

                                Get It Programmatically Too

                                This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                48

                                Robust Validated Data Ingest

                                $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                49

                                Concept The control_ Collection

                                gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                50

                                Validation QOS and PQO via Web Services

                                MongoDBJava Driver

                                MQLValidator

                                Data Access Layer

                                JavaApplication

                                ValidatorHTTPUtils

                                PQLFilter

                                pythonApplication

                                curl

                                JSON over HTTP(S)

                                QOSHTTPUtils

                                PostQuery

                                PQOHTTPUtils

                                QOS

                                HTTP Service

                                JSONlt-gtJava Maps

                                51

                                Are We Excited Yet

                                Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                and other Java resources

                                Questions amp Answers

                                54

                                Concept DataProvider

                                public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                Class MongoProvider implements DataProvider hellip

                                Class RESTfulProvider implements DataProvider hellip

                                55

                                The RESTful Provider

                                class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                • Slide 1
                                • Part 3 In The Data Management Series
                                • Congratulations At this Point Yoursquove
                                • The Next Stage Defend amp Leverage
                                • MongoDB Doesnrsquot Have These Things
                                • What Can We Do Today
                                • Write Some Code
                                • Starting Point The Data Access Layer
                                • Document Validation
                                • A Query Filters Outbound Data
                                • How About Using It To Filter Inbounds
                                • $exists And $type Already in MQL
                                • hellip And MQL Goes Way Beyondhellip
                                • A New MQL Validator Module Emerges
                                • Migrating Capability into MongoDB
                                • Code For The FuturehellipToday
                                • But What About Today
                                • Temporary Filling PQL
                                • Bridge MQL to PQL
                                • No PQL No Problem
                                • MQL Is Easy To Navigate
                                • Where Do Validations Come From
                                • The Validations Collection
                                • Various ldquoLevelsrdquo of Validation
                                • Multiple Types Schema By Example
                                • The Stack So Far
                                • Representative Example
                                • Redaction
                                • Concept Post Query Operations (PQO)
                                • Adopt MQL-like behavior
                                • A New PostQuery Module Emerges
                                • Where Do PQOs Come From
                                • The Postquery Collection
                                • The Stack Is Getting Rich
                                • Representative Example (2)
                                • Quality of Service
                                • QOS In Action
                                • Where Do We Store QOS Values
                                • The QOS Collection
                                • Coming Togetherhellip
                                • Representative Example (3)
                                • A Highly Leveragable Investment
                                • Not Just Java Not A Problem
                                • A Really Nice Stack
                                • What Can We Do
                                • Secure Access To Redacted Data for Testing
                                • Get It Programmatically Too
                                • Robust Validated Data Ingest
                                • Concept The control_ Collection
                                • Validation QOS and PQO via Web Services
                                • Are We Excited Yet
                                • Questions amp Answers
                                • Slide 53
                                • Concept DataProvider
                                • The RESTful Provider

                                  17

                                  But What About Today

                                  MQLValidator

                                  18

                                  Temporary Filling PQL

                                  bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                                  PQL

                                  19

                                  Bridge MQL to PQL

                                  class MQLValidator private PQLFilter pqlfilter

                                  validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                                  20

                                  No PQL No Problem

                                  class MQLValidator

                                  validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                                  21

                                  MQL Is Easy To Navigate

                                  $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                                  bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                                  22

                                  Where Do Validations Come From

                                  The Database

                                  23

                                  The Validations Collection

                                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                  24

                                  Various ldquoLevelsrdquo of Validation

                                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                  25

                                  Multiple Types Schema By Example

                                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                  26

                                  The Stack So Far

                                  MongoDBJava Driver

                                  MQLValidator

                                  Data Access Layer

                                  Application

                                  ValidatorDBUtils

                                  ValidatorDBUtils populates an MQLValidator object from MongoDB

                                  PQLFilter

                                  27

                                  Representative Example

                                  class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                  someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                  28

                                  Redaction

                                  29

                                  Concept Post Query Operations (PQO)

                                  ssn $hash model birthdate null

                                  $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                  30

                                  Adopt MQL-like behavior

                                  ldquossnrdquonull

                                  ldquoaddressrdquo ldquoXXXXrdquo

                                  ldquossnrdquo $substitute ldquossnmodelrdquo

                                  Remove field by setting to null

                                  Redact address with fixed value

                                  Substitute SSN with a different correct consistent value

                                  ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                  Hash counterparty name to consistent value

                                  31

                                  A New PostQuery Module Emerges

                                  class PostQuery process(Map data Map operations)

                                  PostQuery

                                  MongoDBJava Driver

                                  MQLValidator

                                  Data Access Layer

                                  Application

                                  ValidatorDBUtils

                                  PQLFilter

                                  32

                                  Where Do PQOs Come From

                                  The Database

                                  33

                                  The Postquery Collection

                                  gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                  34

                                  The Stack Is Getting Rich

                                  PostQuery

                                  MongoDBJava Driver

                                  MQLValidator

                                  Data Access Layer

                                  Application

                                  ValidatorDBUtils

                                  PQLFilter

                                  PQODBUtils

                                  35

                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                  someWriteOperation(Map data) hellip

                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                  36

                                  Quality of Service

                                  37

                                  QOS In Action

                                  class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                  Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                  38

                                  Where Do We Store QOS Values

                                  The Database

                                  39

                                  The QOS Collection

                                  gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                  ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                  40

                                  QOSDBUtils

                                  Coming Togetherhellip

                                  PostQuery

                                  MongoDBJava Driver

                                  MQLValidator

                                  Data Access Layer

                                  Application

                                  ValidatorDBUtils

                                  PQLFilter

                                  PQODBUtils

                                  QOS

                                  41

                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                  42

                                  QOSDBUtils

                                  A Highly Leveragable Investment

                                  PostQuery

                                  MQLValidator

                                  Data Access Layer 1

                                  Application1

                                  ValidatorDBUtils

                                  PQLFilter

                                  PQODBUtils

                                  QOS

                                  Application2

                                  Data Access Layer 2

                                  Application3

                                  Application4

                                  Data Access Layer 3

                                  Application5

                                  Application6Reusable For ALL Data Access Layer Logic

                                  43

                                  Not Just Java Not A Problem

                                  DAL operations have little or no statehellip

                                  Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                  Can you say hellip Web Service

                                  44

                                  A Really Nice Stack

                                  MongoDBJava Driver

                                  MQLValidator

                                  Data Access Layer

                                  JavaApplication

                                  ValidatorDBUtils

                                  PQLFilter

                                  HTTP Endpoint

                                  pythonApplication

                                  curl

                                  JSON over HTTP(S)

                                  JSONlt-gtJava Maps

                                  QOSDBUtils

                                  PostQuery

                                  PQODBUtils

                                  QOS

                                  45

                                  What Can We Do

                                  46

                                  Secure Access To Redacted Data for Testing

                                  $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                  $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                  $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                  47

                                  Get It Programmatically Too

                                  This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                  getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                  String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                  48

                                  Robust Validated Data Ingest

                                  $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                  $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                  49

                                  Concept The control_ Collection

                                  gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                  bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                  50

                                  Validation QOS and PQO via Web Services

                                  MongoDBJava Driver

                                  MQLValidator

                                  Data Access Layer

                                  JavaApplication

                                  ValidatorHTTPUtils

                                  PQLFilter

                                  pythonApplication

                                  curl

                                  JSON over HTTP(S)

                                  QOSHTTPUtils

                                  PostQuery

                                  PQOHTTPUtils

                                  QOS

                                  HTTP Service

                                  JSONlt-gtJava Maps

                                  51

                                  Are We Excited Yet

                                  Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                  and other Java resources

                                  Questions amp Answers

                                  54

                                  Concept DataProvider

                                  public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                  Class MongoProvider implements DataProvider hellip

                                  Class RESTfulProvider implements DataProvider hellip

                                  55

                                  The RESTful Provider

                                  class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                  fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                  url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                  HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                  • Slide 1
                                  • Part 3 In The Data Management Series
                                  • Congratulations At this Point Yoursquove
                                  • The Next Stage Defend amp Leverage
                                  • MongoDB Doesnrsquot Have These Things
                                  • What Can We Do Today
                                  • Write Some Code
                                  • Starting Point The Data Access Layer
                                  • Document Validation
                                  • A Query Filters Outbound Data
                                  • How About Using It To Filter Inbounds
                                  • $exists And $type Already in MQL
                                  • hellip And MQL Goes Way Beyondhellip
                                  • A New MQL Validator Module Emerges
                                  • Migrating Capability into MongoDB
                                  • Code For The FuturehellipToday
                                  • But What About Today
                                  • Temporary Filling PQL
                                  • Bridge MQL to PQL
                                  • No PQL No Problem
                                  • MQL Is Easy To Navigate
                                  • Where Do Validations Come From
                                  • The Validations Collection
                                  • Various ldquoLevelsrdquo of Validation
                                  • Multiple Types Schema By Example
                                  • The Stack So Far
                                  • Representative Example
                                  • Redaction
                                  • Concept Post Query Operations (PQO)
                                  • Adopt MQL-like behavior
                                  • A New PostQuery Module Emerges
                                  • Where Do PQOs Come From
                                  • The Postquery Collection
                                  • The Stack Is Getting Rich
                                  • Representative Example (2)
                                  • Quality of Service
                                  • QOS In Action
                                  • Where Do We Store QOS Values
                                  • The QOS Collection
                                  • Coming Togetherhellip
                                  • Representative Example (3)
                                  • A Highly Leveragable Investment
                                  • Not Just Java Not A Problem
                                  • A Really Nice Stack
                                  • What Can We Do
                                  • Secure Access To Redacted Data for Testing
                                  • Get It Programmatically Too
                                  • Robust Validated Data Ingest
                                  • Concept The control_ Collection
                                  • Validation QOS and PQO via Web Services
                                  • Are We Excited Yet
                                  • Questions amp Answers
                                  • Slide 53
                                  • Concept DataProvider
                                  • The RESTful Provider

                                    18

                                    Temporary Filling PQL

                                    bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

                                    PQL

                                    19

                                    Bridge MQL to PQL

                                    class MQLValidator private PQLFilter pqlfilter

                                    validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                                    20

                                    No PQL No Problem

                                    class MQLValidator

                                    validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                                    21

                                    MQL Is Easy To Navigate

                                    $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                                    bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                                    22

                                    Where Do Validations Come From

                                    The Database

                                    23

                                    The Validations Collection

                                    gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                    24

                                    Various ldquoLevelsrdquo of Validation

                                    gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                    25

                                    Multiple Types Schema By Example

                                    gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                    26

                                    The Stack So Far

                                    MongoDBJava Driver

                                    MQLValidator

                                    Data Access Layer

                                    Application

                                    ValidatorDBUtils

                                    ValidatorDBUtils populates an MQLValidator object from MongoDB

                                    PQLFilter

                                    27

                                    Representative Example

                                    class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                    someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                    28

                                    Redaction

                                    29

                                    Concept Post Query Operations (PQO)

                                    ssn $hash model birthdate null

                                    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                    30

                                    Adopt MQL-like behavior

                                    ldquossnrdquonull

                                    ldquoaddressrdquo ldquoXXXXrdquo

                                    ldquossnrdquo $substitute ldquossnmodelrdquo

                                    Remove field by setting to null

                                    Redact address with fixed value

                                    Substitute SSN with a different correct consistent value

                                    ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                    Hash counterparty name to consistent value

                                    31

                                    A New PostQuery Module Emerges

                                    class PostQuery process(Map data Map operations)

                                    PostQuery

                                    MongoDBJava Driver

                                    MQLValidator

                                    Data Access Layer

                                    Application

                                    ValidatorDBUtils

                                    PQLFilter

                                    32

                                    Where Do PQOs Come From

                                    The Database

                                    33

                                    The Postquery Collection

                                    gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                    34

                                    The Stack Is Getting Rich

                                    PostQuery

                                    MongoDBJava Driver

                                    MQLValidator

                                    Data Access Layer

                                    Application

                                    ValidatorDBUtils

                                    PQLFilter

                                    PQODBUtils

                                    35

                                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                    someWriteOperation(Map data) hellip

                                    someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                    36

                                    Quality of Service

                                    37

                                    QOS In Action

                                    class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                    Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                    38

                                    Where Do We Store QOS Values

                                    The Database

                                    39

                                    The QOS Collection

                                    gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                    ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                    40

                                    QOSDBUtils

                                    Coming Togetherhellip

                                    PostQuery

                                    MongoDBJava Driver

                                    MQLValidator

                                    Data Access Layer

                                    Application

                                    ValidatorDBUtils

                                    PQLFilter

                                    PQODBUtils

                                    QOS

                                    41

                                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                    someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                    42

                                    QOSDBUtils

                                    A Highly Leveragable Investment

                                    PostQuery

                                    MQLValidator

                                    Data Access Layer 1

                                    Application1

                                    ValidatorDBUtils

                                    PQLFilter

                                    PQODBUtils

                                    QOS

                                    Application2

                                    Data Access Layer 2

                                    Application3

                                    Application4

                                    Data Access Layer 3

                                    Application5

                                    Application6Reusable For ALL Data Access Layer Logic

                                    43

                                    Not Just Java Not A Problem

                                    DAL operations have little or no statehellip

                                    Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                    Can you say hellip Web Service

                                    44

                                    A Really Nice Stack

                                    MongoDBJava Driver

                                    MQLValidator

                                    Data Access Layer

                                    JavaApplication

                                    ValidatorDBUtils

                                    PQLFilter

                                    HTTP Endpoint

                                    pythonApplication

                                    curl

                                    JSON over HTTP(S)

                                    JSONlt-gtJava Maps

                                    QOSDBUtils

                                    PostQuery

                                    PQODBUtils

                                    QOS

                                    45

                                    What Can We Do

                                    46

                                    Secure Access To Redacted Data for Testing

                                    $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                    $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                    $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                    47

                                    Get It Programmatically Too

                                    This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                    getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                    String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                    48

                                    Robust Validated Data Ingest

                                    $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                    $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                    49

                                    Concept The control_ Collection

                                    gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                    bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                    50

                                    Validation QOS and PQO via Web Services

                                    MongoDBJava Driver

                                    MQLValidator

                                    Data Access Layer

                                    JavaApplication

                                    ValidatorHTTPUtils

                                    PQLFilter

                                    pythonApplication

                                    curl

                                    JSON over HTTP(S)

                                    QOSHTTPUtils

                                    PostQuery

                                    PQOHTTPUtils

                                    QOS

                                    HTTP Service

                                    JSONlt-gtJava Maps

                                    51

                                    Are We Excited Yet

                                    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                    and other Java resources

                                    Questions amp Answers

                                    54

                                    Concept DataProvider

                                    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                    Class MongoProvider implements DataProvider hellip

                                    Class RESTfulProvider implements DataProvider hellip

                                    55

                                    The RESTful Provider

                                    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                    • Slide 1
                                    • Part 3 In The Data Management Series
                                    • Congratulations At this Point Yoursquove
                                    • The Next Stage Defend amp Leverage
                                    • MongoDB Doesnrsquot Have These Things
                                    • What Can We Do Today
                                    • Write Some Code
                                    • Starting Point The Data Access Layer
                                    • Document Validation
                                    • A Query Filters Outbound Data
                                    • How About Using It To Filter Inbounds
                                    • $exists And $type Already in MQL
                                    • hellip And MQL Goes Way Beyondhellip
                                    • A New MQL Validator Module Emerges
                                    • Migrating Capability into MongoDB
                                    • Code For The FuturehellipToday
                                    • But What About Today
                                    • Temporary Filling PQL
                                    • Bridge MQL to PQL
                                    • No PQL No Problem
                                    • MQL Is Easy To Navigate
                                    • Where Do Validations Come From
                                    • The Validations Collection
                                    • Various ldquoLevelsrdquo of Validation
                                    • Multiple Types Schema By Example
                                    • The Stack So Far
                                    • Representative Example
                                    • Redaction
                                    • Concept Post Query Operations (PQO)
                                    • Adopt MQL-like behavior
                                    • A New PostQuery Module Emerges
                                    • Where Do PQOs Come From
                                    • The Postquery Collection
                                    • The Stack Is Getting Rich
                                    • Representative Example (2)
                                    • Quality of Service
                                    • QOS In Action
                                    • Where Do We Store QOS Values
                                    • The QOS Collection
                                    • Coming Togetherhellip
                                    • Representative Example (3)
                                    • A Highly Leveragable Investment
                                    • Not Just Java Not A Problem
                                    • A Really Nice Stack
                                    • What Can We Do
                                    • Secure Access To Redacted Data for Testing
                                    • Get It Programmatically Too
                                    • Robust Validated Data Ingest
                                    • Concept The control_ Collection
                                    • Validation QOS and PQO via Web Services
                                    • Are We Excited Yet
                                    • Questions amp Answers
                                    • Slide 53
                                    • Concept DataProvider
                                    • The RESTful Provider

                                      19

                                      Bridge MQL to PQL

                                      class MQLValidator private PQLFilter pqlfilter

                                      validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

                                      20

                                      No PQL No Problem

                                      class MQLValidator

                                      validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                                      21

                                      MQL Is Easy To Navigate

                                      $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                                      bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                                      22

                                      Where Do Validations Come From

                                      The Database

                                      23

                                      The Validations Collection

                                      gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                      24

                                      Various ldquoLevelsrdquo of Validation

                                      gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                      25

                                      Multiple Types Schema By Example

                                      gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                      26

                                      The Stack So Far

                                      MongoDBJava Driver

                                      MQLValidator

                                      Data Access Layer

                                      Application

                                      ValidatorDBUtils

                                      ValidatorDBUtils populates an MQLValidator object from MongoDB

                                      PQLFilter

                                      27

                                      Representative Example

                                      class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                      someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                      28

                                      Redaction

                                      29

                                      Concept Post Query Operations (PQO)

                                      ssn $hash model birthdate null

                                      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                      30

                                      Adopt MQL-like behavior

                                      ldquossnrdquonull

                                      ldquoaddressrdquo ldquoXXXXrdquo

                                      ldquossnrdquo $substitute ldquossnmodelrdquo

                                      Remove field by setting to null

                                      Redact address with fixed value

                                      Substitute SSN with a different correct consistent value

                                      ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                      Hash counterparty name to consistent value

                                      31

                                      A New PostQuery Module Emerges

                                      class PostQuery process(Map data Map operations)

                                      PostQuery

                                      MongoDBJava Driver

                                      MQLValidator

                                      Data Access Layer

                                      Application

                                      ValidatorDBUtils

                                      PQLFilter

                                      32

                                      Where Do PQOs Come From

                                      The Database

                                      33

                                      The Postquery Collection

                                      gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                      34

                                      The Stack Is Getting Rich

                                      PostQuery

                                      MongoDBJava Driver

                                      MQLValidator

                                      Data Access Layer

                                      Application

                                      ValidatorDBUtils

                                      PQLFilter

                                      PQODBUtils

                                      35

                                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                      someWriteOperation(Map data) hellip

                                      someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                      36

                                      Quality of Service

                                      37

                                      QOS In Action

                                      class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                      Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                      38

                                      Where Do We Store QOS Values

                                      The Database

                                      39

                                      The QOS Collection

                                      gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                      ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                      40

                                      QOSDBUtils

                                      Coming Togetherhellip

                                      PostQuery

                                      MongoDBJava Driver

                                      MQLValidator

                                      Data Access Layer

                                      Application

                                      ValidatorDBUtils

                                      PQLFilter

                                      PQODBUtils

                                      QOS

                                      41

                                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                      someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                      42

                                      QOSDBUtils

                                      A Highly Leveragable Investment

                                      PostQuery

                                      MQLValidator

                                      Data Access Layer 1

                                      Application1

                                      ValidatorDBUtils

                                      PQLFilter

                                      PQODBUtils

                                      QOS

                                      Application2

                                      Data Access Layer 2

                                      Application3

                                      Application4

                                      Data Access Layer 3

                                      Application5

                                      Application6Reusable For ALL Data Access Layer Logic

                                      43

                                      Not Just Java Not A Problem

                                      DAL operations have little or no statehellip

                                      Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                      Can you say hellip Web Service

                                      44

                                      A Really Nice Stack

                                      MongoDBJava Driver

                                      MQLValidator

                                      Data Access Layer

                                      JavaApplication

                                      ValidatorDBUtils

                                      PQLFilter

                                      HTTP Endpoint

                                      pythonApplication

                                      curl

                                      JSON over HTTP(S)

                                      JSONlt-gtJava Maps

                                      QOSDBUtils

                                      PostQuery

                                      PQODBUtils

                                      QOS

                                      45

                                      What Can We Do

                                      46

                                      Secure Access To Redacted Data for Testing

                                      $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                      $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                      $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                      47

                                      Get It Programmatically Too

                                      This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                      getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                      String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                      48

                                      Robust Validated Data Ingest

                                      $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                      $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                      49

                                      Concept The control_ Collection

                                      gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                      bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                      50

                                      Validation QOS and PQO via Web Services

                                      MongoDBJava Driver

                                      MQLValidator

                                      Data Access Layer

                                      JavaApplication

                                      ValidatorHTTPUtils

                                      PQLFilter

                                      pythonApplication

                                      curl

                                      JSON over HTTP(S)

                                      QOSHTTPUtils

                                      PostQuery

                                      PQOHTTPUtils

                                      QOS

                                      HTTP Service

                                      JSONlt-gtJava Maps

                                      51

                                      Are We Excited Yet

                                      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                      and other Java resources

                                      Questions amp Answers

                                      54

                                      Concept DataProvider

                                      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                      Class MongoProvider implements DataProvider hellip

                                      Class RESTfulProvider implements DataProvider hellip

                                      55

                                      The RESTful Provider

                                      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                      • Slide 1
                                      • Part 3 In The Data Management Series
                                      • Congratulations At this Point Yoursquove
                                      • The Next Stage Defend amp Leverage
                                      • MongoDB Doesnrsquot Have These Things
                                      • What Can We Do Today
                                      • Write Some Code
                                      • Starting Point The Data Access Layer
                                      • Document Validation
                                      • A Query Filters Outbound Data
                                      • How About Using It To Filter Inbounds
                                      • $exists And $type Already in MQL
                                      • hellip And MQL Goes Way Beyondhellip
                                      • A New MQL Validator Module Emerges
                                      • Migrating Capability into MongoDB
                                      • Code For The FuturehellipToday
                                      • But What About Today
                                      • Temporary Filling PQL
                                      • Bridge MQL to PQL
                                      • No PQL No Problem
                                      • MQL Is Easy To Navigate
                                      • Where Do Validations Come From
                                      • The Validations Collection
                                      • Various ldquoLevelsrdquo of Validation
                                      • Multiple Types Schema By Example
                                      • The Stack So Far
                                      • Representative Example
                                      • Redaction
                                      • Concept Post Query Operations (PQO)
                                      • Adopt MQL-like behavior
                                      • A New PostQuery Module Emerges
                                      • Where Do PQOs Come From
                                      • The Postquery Collection
                                      • The Stack Is Getting Rich
                                      • Representative Example (2)
                                      • Quality of Service
                                      • QOS In Action
                                      • Where Do We Store QOS Values
                                      • The QOS Collection
                                      • Coming Togetherhellip
                                      • Representative Example (3)
                                      • A Highly Leveragable Investment
                                      • Not Just Java Not A Problem
                                      • A Really Nice Stack
                                      • What Can We Do
                                      • Secure Access To Redacted Data for Testing
                                      • Get It Programmatically Too
                                      • Robust Validated Data Ingest
                                      • Concept The control_ Collection
                                      • Validation QOS and PQO via Web Services
                                      • Are We Excited Yet
                                      • Questions amp Answers
                                      • Slide 53
                                      • Concept DataProvider
                                      • The RESTful Provider

                                        20

                                        No PQL No Problem

                                        class MQLValidator

                                        validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

                                        21

                                        MQL Is Easy To Navigate

                                        $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                                        bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                                        22

                                        Where Do Validations Come From

                                        The Database

                                        23

                                        The Validations Collection

                                        gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                        24

                                        Various ldquoLevelsrdquo of Validation

                                        gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                        25

                                        Multiple Types Schema By Example

                                        gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                        26

                                        The Stack So Far

                                        MongoDBJava Driver

                                        MQLValidator

                                        Data Access Layer

                                        Application

                                        ValidatorDBUtils

                                        ValidatorDBUtils populates an MQLValidator object from MongoDB

                                        PQLFilter

                                        27

                                        Representative Example

                                        class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                        someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                        28

                                        Redaction

                                        29

                                        Concept Post Query Operations (PQO)

                                        ssn $hash model birthdate null

                                        $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                        30

                                        Adopt MQL-like behavior

                                        ldquossnrdquonull

                                        ldquoaddressrdquo ldquoXXXXrdquo

                                        ldquossnrdquo $substitute ldquossnmodelrdquo

                                        Remove field by setting to null

                                        Redact address with fixed value

                                        Substitute SSN with a different correct consistent value

                                        ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                        Hash counterparty name to consistent value

                                        31

                                        A New PostQuery Module Emerges

                                        class PostQuery process(Map data Map operations)

                                        PostQuery

                                        MongoDBJava Driver

                                        MQLValidator

                                        Data Access Layer

                                        Application

                                        ValidatorDBUtils

                                        PQLFilter

                                        32

                                        Where Do PQOs Come From

                                        The Database

                                        33

                                        The Postquery Collection

                                        gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                        34

                                        The Stack Is Getting Rich

                                        PostQuery

                                        MongoDBJava Driver

                                        MQLValidator

                                        Data Access Layer

                                        Application

                                        ValidatorDBUtils

                                        PQLFilter

                                        PQODBUtils

                                        35

                                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                        someWriteOperation(Map data) hellip

                                        someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                        36

                                        Quality of Service

                                        37

                                        QOS In Action

                                        class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                        Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                        38

                                        Where Do We Store QOS Values

                                        The Database

                                        39

                                        The QOS Collection

                                        gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                        ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                        40

                                        QOSDBUtils

                                        Coming Togetherhellip

                                        PostQuery

                                        MongoDBJava Driver

                                        MQLValidator

                                        Data Access Layer

                                        Application

                                        ValidatorDBUtils

                                        PQLFilter

                                        PQODBUtils

                                        QOS

                                        41

                                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                        someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                        42

                                        QOSDBUtils

                                        A Highly Leveragable Investment

                                        PostQuery

                                        MQLValidator

                                        Data Access Layer 1

                                        Application1

                                        ValidatorDBUtils

                                        PQLFilter

                                        PQODBUtils

                                        QOS

                                        Application2

                                        Data Access Layer 2

                                        Application3

                                        Application4

                                        Data Access Layer 3

                                        Application5

                                        Application6Reusable For ALL Data Access Layer Logic

                                        43

                                        Not Just Java Not A Problem

                                        DAL operations have little or no statehellip

                                        Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                        Can you say hellip Web Service

                                        44

                                        A Really Nice Stack

                                        MongoDBJava Driver

                                        MQLValidator

                                        Data Access Layer

                                        JavaApplication

                                        ValidatorDBUtils

                                        PQLFilter

                                        HTTP Endpoint

                                        pythonApplication

                                        curl

                                        JSON over HTTP(S)

                                        JSONlt-gtJava Maps

                                        QOSDBUtils

                                        PostQuery

                                        PQODBUtils

                                        QOS

                                        45

                                        What Can We Do

                                        46

                                        Secure Access To Redacted Data for Testing

                                        $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                        $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                        $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                        47

                                        Get It Programmatically Too

                                        This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                        getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                        String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                        48

                                        Robust Validated Data Ingest

                                        $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                        $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                        49

                                        Concept The control_ Collection

                                        gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                        bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                        50

                                        Validation QOS and PQO via Web Services

                                        MongoDBJava Driver

                                        MQLValidator

                                        Data Access Layer

                                        JavaApplication

                                        ValidatorHTTPUtils

                                        PQLFilter

                                        pythonApplication

                                        curl

                                        JSON over HTTP(S)

                                        QOSHTTPUtils

                                        PostQuery

                                        PQOHTTPUtils

                                        QOS

                                        HTTP Service

                                        JSONlt-gtJava Maps

                                        51

                                        Are We Excited Yet

                                        Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                        and other Java resources

                                        Questions amp Answers

                                        54

                                        Concept DataProvider

                                        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                        Class MongoProvider implements DataProvider hellip

                                        Class RESTfulProvider implements DataProvider hellip

                                        55

                                        The RESTful Provider

                                        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                        • Slide 1
                                        • Part 3 In The Data Management Series
                                        • Congratulations At this Point Yoursquove
                                        • The Next Stage Defend amp Leverage
                                        • MongoDB Doesnrsquot Have These Things
                                        • What Can We Do Today
                                        • Write Some Code
                                        • Starting Point The Data Access Layer
                                        • Document Validation
                                        • A Query Filters Outbound Data
                                        • How About Using It To Filter Inbounds
                                        • $exists And $type Already in MQL
                                        • hellip And MQL Goes Way Beyondhellip
                                        • A New MQL Validator Module Emerges
                                        • Migrating Capability into MongoDB
                                        • Code For The FuturehellipToday
                                        • But What About Today
                                        • Temporary Filling PQL
                                        • Bridge MQL to PQL
                                        • No PQL No Problem
                                        • MQL Is Easy To Navigate
                                        • Where Do Validations Come From
                                        • The Validations Collection
                                        • Various ldquoLevelsrdquo of Validation
                                        • Multiple Types Schema By Example
                                        • The Stack So Far
                                        • Representative Example
                                        • Redaction
                                        • Concept Post Query Operations (PQO)
                                        • Adopt MQL-like behavior
                                        • A New PostQuery Module Emerges
                                        • Where Do PQOs Come From
                                        • The Postquery Collection
                                        • The Stack Is Getting Rich
                                        • Representative Example (2)
                                        • Quality of Service
                                        • QOS In Action
                                        • Where Do We Store QOS Values
                                        • The QOS Collection
                                        • Coming Togetherhellip
                                        • Representative Example (3)
                                        • A Highly Leveragable Investment
                                        • Not Just Java Not A Problem
                                        • A Really Nice Stack
                                        • What Can We Do
                                        • Secure Access To Redacted Data for Testing
                                        • Get It Programmatically Too
                                        • Robust Validated Data Ingest
                                        • Concept The control_ Collection
                                        • Validation QOS and PQO via Web Services
                                        • Are We Excited Yet
                                        • Questions amp Answers
                                        • Slide 53
                                        • Concept DataProvider
                                        • The RESTful Provider

                                          21

                                          MQL Is Easy To Navigate

                                          $or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

                                          bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

                                          22

                                          Where Do Validations Come From

                                          The Database

                                          23

                                          The Validations Collection

                                          gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                          24

                                          Various ldquoLevelsrdquo of Validation

                                          gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                          25

                                          Multiple Types Schema By Example

                                          gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                          26

                                          The Stack So Far

                                          MongoDBJava Driver

                                          MQLValidator

                                          Data Access Layer

                                          Application

                                          ValidatorDBUtils

                                          ValidatorDBUtils populates an MQLValidator object from MongoDB

                                          PQLFilter

                                          27

                                          Representative Example

                                          class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                          someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                          28

                                          Redaction

                                          29

                                          Concept Post Query Operations (PQO)

                                          ssn $hash model birthdate null

                                          $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                          30

                                          Adopt MQL-like behavior

                                          ldquossnrdquonull

                                          ldquoaddressrdquo ldquoXXXXrdquo

                                          ldquossnrdquo $substitute ldquossnmodelrdquo

                                          Remove field by setting to null

                                          Redact address with fixed value

                                          Substitute SSN with a different correct consistent value

                                          ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                          Hash counterparty name to consistent value

                                          31

                                          A New PostQuery Module Emerges

                                          class PostQuery process(Map data Map operations)

                                          PostQuery

                                          MongoDBJava Driver

                                          MQLValidator

                                          Data Access Layer

                                          Application

                                          ValidatorDBUtils

                                          PQLFilter

                                          32

                                          Where Do PQOs Come From

                                          The Database

                                          33

                                          The Postquery Collection

                                          gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                          34

                                          The Stack Is Getting Rich

                                          PostQuery

                                          MongoDBJava Driver

                                          MQLValidator

                                          Data Access Layer

                                          Application

                                          ValidatorDBUtils

                                          PQLFilter

                                          PQODBUtils

                                          35

                                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                          someWriteOperation(Map data) hellip

                                          someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                          36

                                          Quality of Service

                                          37

                                          QOS In Action

                                          class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                          Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                          38

                                          Where Do We Store QOS Values

                                          The Database

                                          39

                                          The QOS Collection

                                          gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                          ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                          40

                                          QOSDBUtils

                                          Coming Togetherhellip

                                          PostQuery

                                          MongoDBJava Driver

                                          MQLValidator

                                          Data Access Layer

                                          Application

                                          ValidatorDBUtils

                                          PQLFilter

                                          PQODBUtils

                                          QOS

                                          41

                                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                          someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                          42

                                          QOSDBUtils

                                          A Highly Leveragable Investment

                                          PostQuery

                                          MQLValidator

                                          Data Access Layer 1

                                          Application1

                                          ValidatorDBUtils

                                          PQLFilter

                                          PQODBUtils

                                          QOS

                                          Application2

                                          Data Access Layer 2

                                          Application3

                                          Application4

                                          Data Access Layer 3

                                          Application5

                                          Application6Reusable For ALL Data Access Layer Logic

                                          43

                                          Not Just Java Not A Problem

                                          DAL operations have little or no statehellip

                                          Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                          Can you say hellip Web Service

                                          44

                                          A Really Nice Stack

                                          MongoDBJava Driver

                                          MQLValidator

                                          Data Access Layer

                                          JavaApplication

                                          ValidatorDBUtils

                                          PQLFilter

                                          HTTP Endpoint

                                          pythonApplication

                                          curl

                                          JSON over HTTP(S)

                                          JSONlt-gtJava Maps

                                          QOSDBUtils

                                          PostQuery

                                          PQODBUtils

                                          QOS

                                          45

                                          What Can We Do

                                          46

                                          Secure Access To Redacted Data for Testing

                                          $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                          $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                          $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                          47

                                          Get It Programmatically Too

                                          This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                          getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                          String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                          48

                                          Robust Validated Data Ingest

                                          $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                          $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                          49

                                          Concept The control_ Collection

                                          gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                          bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                          50

                                          Validation QOS and PQO via Web Services

                                          MongoDBJava Driver

                                          MQLValidator

                                          Data Access Layer

                                          JavaApplication

                                          ValidatorHTTPUtils

                                          PQLFilter

                                          pythonApplication

                                          curl

                                          JSON over HTTP(S)

                                          QOSHTTPUtils

                                          PostQuery

                                          PQOHTTPUtils

                                          QOS

                                          HTTP Service

                                          JSONlt-gtJava Maps

                                          51

                                          Are We Excited Yet

                                          Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                          and other Java resources

                                          Questions amp Answers

                                          54

                                          Concept DataProvider

                                          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                          Class MongoProvider implements DataProvider hellip

                                          Class RESTfulProvider implements DataProvider hellip

                                          55

                                          The RESTful Provider

                                          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                          • Slide 1
                                          • Part 3 In The Data Management Series
                                          • Congratulations At this Point Yoursquove
                                          • The Next Stage Defend amp Leverage
                                          • MongoDB Doesnrsquot Have These Things
                                          • What Can We Do Today
                                          • Write Some Code
                                          • Starting Point The Data Access Layer
                                          • Document Validation
                                          • A Query Filters Outbound Data
                                          • How About Using It To Filter Inbounds
                                          • $exists And $type Already in MQL
                                          • hellip And MQL Goes Way Beyondhellip
                                          • A New MQL Validator Module Emerges
                                          • Migrating Capability into MongoDB
                                          • Code For The FuturehellipToday
                                          • But What About Today
                                          • Temporary Filling PQL
                                          • Bridge MQL to PQL
                                          • No PQL No Problem
                                          • MQL Is Easy To Navigate
                                          • Where Do Validations Come From
                                          • The Validations Collection
                                          • Various ldquoLevelsrdquo of Validation
                                          • Multiple Types Schema By Example
                                          • The Stack So Far
                                          • Representative Example
                                          • Redaction
                                          • Concept Post Query Operations (PQO)
                                          • Adopt MQL-like behavior
                                          • A New PostQuery Module Emerges
                                          • Where Do PQOs Come From
                                          • The Postquery Collection
                                          • The Stack Is Getting Rich
                                          • Representative Example (2)
                                          • Quality of Service
                                          • QOS In Action
                                          • Where Do We Store QOS Values
                                          • The QOS Collection
                                          • Coming Togetherhellip
                                          • Representative Example (3)
                                          • A Highly Leveragable Investment
                                          • Not Just Java Not A Problem
                                          • A Really Nice Stack
                                          • What Can We Do
                                          • Secure Access To Redacted Data for Testing
                                          • Get It Programmatically Too
                                          • Robust Validated Data Ingest
                                          • Concept The control_ Collection
                                          • Validation QOS and PQO via Web Services
                                          • Are We Excited Yet
                                          • Questions amp Answers
                                          • Slide 53
                                          • Concept DataProvider
                                          • The RESTful Provider

                                            22

                                            Where Do Validations Come From

                                            The Database

                                            23

                                            The Validations Collection

                                            gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                            24

                                            Various ldquoLevelsrdquo of Validation

                                            gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                            25

                                            Multiple Types Schema By Example

                                            gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                            26

                                            The Stack So Far

                                            MongoDBJava Driver

                                            MQLValidator

                                            Data Access Layer

                                            Application

                                            ValidatorDBUtils

                                            ValidatorDBUtils populates an MQLValidator object from MongoDB

                                            PQLFilter

                                            27

                                            Representative Example

                                            class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                            someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                            28

                                            Redaction

                                            29

                                            Concept Post Query Operations (PQO)

                                            ssn $hash model birthdate null

                                            $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                            30

                                            Adopt MQL-like behavior

                                            ldquossnrdquonull

                                            ldquoaddressrdquo ldquoXXXXrdquo

                                            ldquossnrdquo $substitute ldquossnmodelrdquo

                                            Remove field by setting to null

                                            Redact address with fixed value

                                            Substitute SSN with a different correct consistent value

                                            ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                            Hash counterparty name to consistent value

                                            31

                                            A New PostQuery Module Emerges

                                            class PostQuery process(Map data Map operations)

                                            PostQuery

                                            MongoDBJava Driver

                                            MQLValidator

                                            Data Access Layer

                                            Application

                                            ValidatorDBUtils

                                            PQLFilter

                                            32

                                            Where Do PQOs Come From

                                            The Database

                                            33

                                            The Postquery Collection

                                            gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                            34

                                            The Stack Is Getting Rich

                                            PostQuery

                                            MongoDBJava Driver

                                            MQLValidator

                                            Data Access Layer

                                            Application

                                            ValidatorDBUtils

                                            PQLFilter

                                            PQODBUtils

                                            35

                                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                            someWriteOperation(Map data) hellip

                                            someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                            36

                                            Quality of Service

                                            37

                                            QOS In Action

                                            class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                            Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                            38

                                            Where Do We Store QOS Values

                                            The Database

                                            39

                                            The QOS Collection

                                            gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                            ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                            40

                                            QOSDBUtils

                                            Coming Togetherhellip

                                            PostQuery

                                            MongoDBJava Driver

                                            MQLValidator

                                            Data Access Layer

                                            Application

                                            ValidatorDBUtils

                                            PQLFilter

                                            PQODBUtils

                                            QOS

                                            41

                                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                            someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                            42

                                            QOSDBUtils

                                            A Highly Leveragable Investment

                                            PostQuery

                                            MQLValidator

                                            Data Access Layer 1

                                            Application1

                                            ValidatorDBUtils

                                            PQLFilter

                                            PQODBUtils

                                            QOS

                                            Application2

                                            Data Access Layer 2

                                            Application3

                                            Application4

                                            Data Access Layer 3

                                            Application5

                                            Application6Reusable For ALL Data Access Layer Logic

                                            43

                                            Not Just Java Not A Problem

                                            DAL operations have little or no statehellip

                                            Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                            Can you say hellip Web Service

                                            44

                                            A Really Nice Stack

                                            MongoDBJava Driver

                                            MQLValidator

                                            Data Access Layer

                                            JavaApplication

                                            ValidatorDBUtils

                                            PQLFilter

                                            HTTP Endpoint

                                            pythonApplication

                                            curl

                                            JSON over HTTP(S)

                                            JSONlt-gtJava Maps

                                            QOSDBUtils

                                            PostQuery

                                            PQODBUtils

                                            QOS

                                            45

                                            What Can We Do

                                            46

                                            Secure Access To Redacted Data for Testing

                                            $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                            $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                            $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                            47

                                            Get It Programmatically Too

                                            This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                            getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                            String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                            48

                                            Robust Validated Data Ingest

                                            $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                            $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                            49

                                            Concept The control_ Collection

                                            gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                            bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                            50

                                            Validation QOS and PQO via Web Services

                                            MongoDBJava Driver

                                            MQLValidator

                                            Data Access Layer

                                            JavaApplication

                                            ValidatorHTTPUtils

                                            PQLFilter

                                            pythonApplication

                                            curl

                                            JSON over HTTP(S)

                                            QOSHTTPUtils

                                            PostQuery

                                            PQOHTTPUtils

                                            QOS

                                            HTTP Service

                                            JSONlt-gtJava Maps

                                            51

                                            Are We Excited Yet

                                            Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                            and other Java resources

                                            Questions amp Answers

                                            54

                                            Concept DataProvider

                                            public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                            Class MongoProvider implements DataProvider hellip

                                            Class RESTfulProvider implements DataProvider hellip

                                            55

                                            The RESTful Provider

                                            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                            • Slide 1
                                            • Part 3 In The Data Management Series
                                            • Congratulations At this Point Yoursquove
                                            • The Next Stage Defend amp Leverage
                                            • MongoDB Doesnrsquot Have These Things
                                            • What Can We Do Today
                                            • Write Some Code
                                            • Starting Point The Data Access Layer
                                            • Document Validation
                                            • A Query Filters Outbound Data
                                            • How About Using It To Filter Inbounds
                                            • $exists And $type Already in MQL
                                            • hellip And MQL Goes Way Beyondhellip
                                            • A New MQL Validator Module Emerges
                                            • Migrating Capability into MongoDB
                                            • Code For The FuturehellipToday
                                            • But What About Today
                                            • Temporary Filling PQL
                                            • Bridge MQL to PQL
                                            • No PQL No Problem
                                            • MQL Is Easy To Navigate
                                            • Where Do Validations Come From
                                            • The Validations Collection
                                            • Various ldquoLevelsrdquo of Validation
                                            • Multiple Types Schema By Example
                                            • The Stack So Far
                                            • Representative Example
                                            • Redaction
                                            • Concept Post Query Operations (PQO)
                                            • Adopt MQL-like behavior
                                            • A New PostQuery Module Emerges
                                            • Where Do PQOs Come From
                                            • The Postquery Collection
                                            • The Stack Is Getting Rich
                                            • Representative Example (2)
                                            • Quality of Service
                                            • QOS In Action
                                            • Where Do We Store QOS Values
                                            • The QOS Collection
                                            • Coming Togetherhellip
                                            • Representative Example (3)
                                            • A Highly Leveragable Investment
                                            • Not Just Java Not A Problem
                                            • A Really Nice Stack
                                            • What Can We Do
                                            • Secure Access To Redacted Data for Testing
                                            • Get It Programmatically Too
                                            • Robust Validated Data Ingest
                                            • Concept The control_ Collection
                                            • Validation QOS and PQO via Web Services
                                            • Are We Excited Yet
                                            • Questions amp Answers
                                            • Slide 53
                                            • Concept DataProvider
                                            • The RESTful Provider

                                              23

                                              The Validations Collection

                                              gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

                                              24

                                              Various ldquoLevelsrdquo of Validation

                                              gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                              25

                                              Multiple Types Schema By Example

                                              gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                              26

                                              The Stack So Far

                                              MongoDBJava Driver

                                              MQLValidator

                                              Data Access Layer

                                              Application

                                              ValidatorDBUtils

                                              ValidatorDBUtils populates an MQLValidator object from MongoDB

                                              PQLFilter

                                              27

                                              Representative Example

                                              class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                              someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                              28

                                              Redaction

                                              29

                                              Concept Post Query Operations (PQO)

                                              ssn $hash model birthdate null

                                              $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                              30

                                              Adopt MQL-like behavior

                                              ldquossnrdquonull

                                              ldquoaddressrdquo ldquoXXXXrdquo

                                              ldquossnrdquo $substitute ldquossnmodelrdquo

                                              Remove field by setting to null

                                              Redact address with fixed value

                                              Substitute SSN with a different correct consistent value

                                              ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                              Hash counterparty name to consistent value

                                              31

                                              A New PostQuery Module Emerges

                                              class PostQuery process(Map data Map operations)

                                              PostQuery

                                              MongoDBJava Driver

                                              MQLValidator

                                              Data Access Layer

                                              Application

                                              ValidatorDBUtils

                                              PQLFilter

                                              32

                                              Where Do PQOs Come From

                                              The Database

                                              33

                                              The Postquery Collection

                                              gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                              34

                                              The Stack Is Getting Rich

                                              PostQuery

                                              MongoDBJava Driver

                                              MQLValidator

                                              Data Access Layer

                                              Application

                                              ValidatorDBUtils

                                              PQLFilter

                                              PQODBUtils

                                              35

                                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                              someWriteOperation(Map data) hellip

                                              someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                              36

                                              Quality of Service

                                              37

                                              QOS In Action

                                              class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                              Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                              38

                                              Where Do We Store QOS Values

                                              The Database

                                              39

                                              The QOS Collection

                                              gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                              ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                              40

                                              QOSDBUtils

                                              Coming Togetherhellip

                                              PostQuery

                                              MongoDBJava Driver

                                              MQLValidator

                                              Data Access Layer

                                              Application

                                              ValidatorDBUtils

                                              PQLFilter

                                              PQODBUtils

                                              QOS

                                              41

                                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                              someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                              42

                                              QOSDBUtils

                                              A Highly Leveragable Investment

                                              PostQuery

                                              MQLValidator

                                              Data Access Layer 1

                                              Application1

                                              ValidatorDBUtils

                                              PQLFilter

                                              PQODBUtils

                                              QOS

                                              Application2

                                              Data Access Layer 2

                                              Application3

                                              Application4

                                              Data Access Layer 3

                                              Application5

                                              Application6Reusable For ALL Data Access Layer Logic

                                              43

                                              Not Just Java Not A Problem

                                              DAL operations have little or no statehellip

                                              Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                              Can you say hellip Web Service

                                              44

                                              A Really Nice Stack

                                              MongoDBJava Driver

                                              MQLValidator

                                              Data Access Layer

                                              JavaApplication

                                              ValidatorDBUtils

                                              PQLFilter

                                              HTTP Endpoint

                                              pythonApplication

                                              curl

                                              JSON over HTTP(S)

                                              JSONlt-gtJava Maps

                                              QOSDBUtils

                                              PostQuery

                                              PQODBUtils

                                              QOS

                                              45

                                              What Can We Do

                                              46

                                              Secure Access To Redacted Data for Testing

                                              $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                              $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                              $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                              47

                                              Get It Programmatically Too

                                              This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                              getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                              String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                              48

                                              Robust Validated Data Ingest

                                              $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                              $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                              49

                                              Concept The control_ Collection

                                              gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                              bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                              50

                                              Validation QOS and PQO via Web Services

                                              MongoDBJava Driver

                                              MQLValidator

                                              Data Access Layer

                                              JavaApplication

                                              ValidatorHTTPUtils

                                              PQLFilter

                                              pythonApplication

                                              curl

                                              JSON over HTTP(S)

                                              QOSHTTPUtils

                                              PostQuery

                                              PQOHTTPUtils

                                              QOS

                                              HTTP Service

                                              JSONlt-gtJava Maps

                                              51

                                              Are We Excited Yet

                                              Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                              and other Java resources

                                              Questions amp Answers

                                              54

                                              Concept DataProvider

                                              public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                              Class MongoProvider implements DataProvider hellip

                                              Class RESTfulProvider implements DataProvider hellip

                                              55

                                              The RESTful Provider

                                              class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                              fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                              url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                              HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                              • Slide 1
                                              • Part 3 In The Data Management Series
                                              • Congratulations At this Point Yoursquove
                                              • The Next Stage Defend amp Leverage
                                              • MongoDB Doesnrsquot Have These Things
                                              • What Can We Do Today
                                              • Write Some Code
                                              • Starting Point The Data Access Layer
                                              • Document Validation
                                              • A Query Filters Outbound Data
                                              • How About Using It To Filter Inbounds
                                              • $exists And $type Already in MQL
                                              • hellip And MQL Goes Way Beyondhellip
                                              • A New MQL Validator Module Emerges
                                              • Migrating Capability into MongoDB
                                              • Code For The FuturehellipToday
                                              • But What About Today
                                              • Temporary Filling PQL
                                              • Bridge MQL to PQL
                                              • No PQL No Problem
                                              • MQL Is Easy To Navigate
                                              • Where Do Validations Come From
                                              • The Validations Collection
                                              • Various ldquoLevelsrdquo of Validation
                                              • Multiple Types Schema By Example
                                              • The Stack So Far
                                              • Representative Example
                                              • Redaction
                                              • Concept Post Query Operations (PQO)
                                              • Adopt MQL-like behavior
                                              • A New PostQuery Module Emerges
                                              • Where Do PQOs Come From
                                              • The Postquery Collection
                                              • The Stack Is Getting Rich
                                              • Representative Example (2)
                                              • Quality of Service
                                              • QOS In Action
                                              • Where Do We Store QOS Values
                                              • The QOS Collection
                                              • Coming Togetherhellip
                                              • Representative Example (3)
                                              • A Highly Leveragable Investment
                                              • Not Just Java Not A Problem
                                              • A Really Nice Stack
                                              • What Can We Do
                                              • Secure Access To Redacted Data for Testing
                                              • Get It Programmatically Too
                                              • Robust Validated Data Ingest
                                              • Concept The control_ Collection
                                              • Validation QOS and PQO via Web Services
                                              • Are We Excited Yet
                                              • Questions amp Answers
                                              • Slide 53
                                              • Concept DataProvider
                                              • The RESTful Provider

                                                24

                                                Various ldquoLevelsrdquo of Validation

                                                gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

                                                25

                                                Multiple Types Schema By Example

                                                gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                                26

                                                The Stack So Far

                                                MongoDBJava Driver

                                                MQLValidator

                                                Data Access Layer

                                                Application

                                                ValidatorDBUtils

                                                ValidatorDBUtils populates an MQLValidator object from MongoDB

                                                PQLFilter

                                                27

                                                Representative Example

                                                class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                                someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                                28

                                                Redaction

                                                29

                                                Concept Post Query Operations (PQO)

                                                ssn $hash model birthdate null

                                                $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                                30

                                                Adopt MQL-like behavior

                                                ldquossnrdquonull

                                                ldquoaddressrdquo ldquoXXXXrdquo

                                                ldquossnrdquo $substitute ldquossnmodelrdquo

                                                Remove field by setting to null

                                                Redact address with fixed value

                                                Substitute SSN with a different correct consistent value

                                                ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                Hash counterparty name to consistent value

                                                31

                                                A New PostQuery Module Emerges

                                                class PostQuery process(Map data Map operations)

                                                PostQuery

                                                MongoDBJava Driver

                                                MQLValidator

                                                Data Access Layer

                                                Application

                                                ValidatorDBUtils

                                                PQLFilter

                                                32

                                                Where Do PQOs Come From

                                                The Database

                                                33

                                                The Postquery Collection

                                                gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                34

                                                The Stack Is Getting Rich

                                                PostQuery

                                                MongoDBJava Driver

                                                MQLValidator

                                                Data Access Layer

                                                Application

                                                ValidatorDBUtils

                                                PQLFilter

                                                PQODBUtils

                                                35

                                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                someWriteOperation(Map data) hellip

                                                someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                36

                                                Quality of Service

                                                37

                                                QOS In Action

                                                class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                38

                                                Where Do We Store QOS Values

                                                The Database

                                                39

                                                The QOS Collection

                                                gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                40

                                                QOSDBUtils

                                                Coming Togetherhellip

                                                PostQuery

                                                MongoDBJava Driver

                                                MQLValidator

                                                Data Access Layer

                                                Application

                                                ValidatorDBUtils

                                                PQLFilter

                                                PQODBUtils

                                                QOS

                                                41

                                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                42

                                                QOSDBUtils

                                                A Highly Leveragable Investment

                                                PostQuery

                                                MQLValidator

                                                Data Access Layer 1

                                                Application1

                                                ValidatorDBUtils

                                                PQLFilter

                                                PQODBUtils

                                                QOS

                                                Application2

                                                Data Access Layer 2

                                                Application3

                                                Application4

                                                Data Access Layer 3

                                                Application5

                                                Application6Reusable For ALL Data Access Layer Logic

                                                43

                                                Not Just Java Not A Problem

                                                DAL operations have little or no statehellip

                                                Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                Can you say hellip Web Service

                                                44

                                                A Really Nice Stack

                                                MongoDBJava Driver

                                                MQLValidator

                                                Data Access Layer

                                                JavaApplication

                                                ValidatorDBUtils

                                                PQLFilter

                                                HTTP Endpoint

                                                pythonApplication

                                                curl

                                                JSON over HTTP(S)

                                                JSONlt-gtJava Maps

                                                QOSDBUtils

                                                PostQuery

                                                PQODBUtils

                                                QOS

                                                45

                                                What Can We Do

                                                46

                                                Secure Access To Redacted Data for Testing

                                                $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                47

                                                Get It Programmatically Too

                                                This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                48

                                                Robust Validated Data Ingest

                                                $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                49

                                                Concept The control_ Collection

                                                gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                50

                                                Validation QOS and PQO via Web Services

                                                MongoDBJava Driver

                                                MQLValidator

                                                Data Access Layer

                                                JavaApplication

                                                ValidatorHTTPUtils

                                                PQLFilter

                                                pythonApplication

                                                curl

                                                JSON over HTTP(S)

                                                QOSHTTPUtils

                                                PostQuery

                                                PQOHTTPUtils

                                                QOS

                                                HTTP Service

                                                JSONlt-gtJava Maps

                                                51

                                                Are We Excited Yet

                                                Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                and other Java resources

                                                Questions amp Answers

                                                54

                                                Concept DataProvider

                                                public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                Class MongoProvider implements DataProvider hellip

                                                Class RESTfulProvider implements DataProvider hellip

                                                55

                                                The RESTful Provider

                                                class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                • Slide 1
                                                • Part 3 In The Data Management Series
                                                • Congratulations At this Point Yoursquove
                                                • The Next Stage Defend amp Leverage
                                                • MongoDB Doesnrsquot Have These Things
                                                • What Can We Do Today
                                                • Write Some Code
                                                • Starting Point The Data Access Layer
                                                • Document Validation
                                                • A Query Filters Outbound Data
                                                • How About Using It To Filter Inbounds
                                                • $exists And $type Already in MQL
                                                • hellip And MQL Goes Way Beyondhellip
                                                • A New MQL Validator Module Emerges
                                                • Migrating Capability into MongoDB
                                                • Code For The FuturehellipToday
                                                • But What About Today
                                                • Temporary Filling PQL
                                                • Bridge MQL to PQL
                                                • No PQL No Problem
                                                • MQL Is Easy To Navigate
                                                • Where Do Validations Come From
                                                • The Validations Collection
                                                • Various ldquoLevelsrdquo of Validation
                                                • Multiple Types Schema By Example
                                                • The Stack So Far
                                                • Representative Example
                                                • Redaction
                                                • Concept Post Query Operations (PQO)
                                                • Adopt MQL-like behavior
                                                • A New PostQuery Module Emerges
                                                • Where Do PQOs Come From
                                                • The Postquery Collection
                                                • The Stack Is Getting Rich
                                                • Representative Example (2)
                                                • Quality of Service
                                                • QOS In Action
                                                • Where Do We Store QOS Values
                                                • The QOS Collection
                                                • Coming Togetherhellip
                                                • Representative Example (3)
                                                • A Highly Leveragable Investment
                                                • Not Just Java Not A Problem
                                                • A Really Nice Stack
                                                • What Can We Do
                                                • Secure Access To Redacted Data for Testing
                                                • Get It Programmatically Too
                                                • Robust Validated Data Ingest
                                                • Concept The control_ Collection
                                                • Validation QOS and PQO via Web Services
                                                • Are We Excited Yet
                                                • Questions amp Answers
                                                • Slide 53
                                                • Concept DataProvider
                                                • The RESTful Provider

                                                  25

                                                  Multiple Types Schema By Example

                                                  gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

                                                  26

                                                  The Stack So Far

                                                  MongoDBJava Driver

                                                  MQLValidator

                                                  Data Access Layer

                                                  Application

                                                  ValidatorDBUtils

                                                  ValidatorDBUtils populates an MQLValidator object from MongoDB

                                                  PQLFilter

                                                  27

                                                  Representative Example

                                                  class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                                  someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                                  28

                                                  Redaction

                                                  29

                                                  Concept Post Query Operations (PQO)

                                                  ssn $hash model birthdate null

                                                  $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                                  30

                                                  Adopt MQL-like behavior

                                                  ldquossnrdquonull

                                                  ldquoaddressrdquo ldquoXXXXrdquo

                                                  ldquossnrdquo $substitute ldquossnmodelrdquo

                                                  Remove field by setting to null

                                                  Redact address with fixed value

                                                  Substitute SSN with a different correct consistent value

                                                  ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                  Hash counterparty name to consistent value

                                                  31

                                                  A New PostQuery Module Emerges

                                                  class PostQuery process(Map data Map operations)

                                                  PostQuery

                                                  MongoDBJava Driver

                                                  MQLValidator

                                                  Data Access Layer

                                                  Application

                                                  ValidatorDBUtils

                                                  PQLFilter

                                                  32

                                                  Where Do PQOs Come From

                                                  The Database

                                                  33

                                                  The Postquery Collection

                                                  gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                  34

                                                  The Stack Is Getting Rich

                                                  PostQuery

                                                  MongoDBJava Driver

                                                  MQLValidator

                                                  Data Access Layer

                                                  Application

                                                  ValidatorDBUtils

                                                  PQLFilter

                                                  PQODBUtils

                                                  35

                                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                  someWriteOperation(Map data) hellip

                                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                  36

                                                  Quality of Service

                                                  37

                                                  QOS In Action

                                                  class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                  Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                  38

                                                  Where Do We Store QOS Values

                                                  The Database

                                                  39

                                                  The QOS Collection

                                                  gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                  ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                  40

                                                  QOSDBUtils

                                                  Coming Togetherhellip

                                                  PostQuery

                                                  MongoDBJava Driver

                                                  MQLValidator

                                                  Data Access Layer

                                                  Application

                                                  ValidatorDBUtils

                                                  PQLFilter

                                                  PQODBUtils

                                                  QOS

                                                  41

                                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                  42

                                                  QOSDBUtils

                                                  A Highly Leveragable Investment

                                                  PostQuery

                                                  MQLValidator

                                                  Data Access Layer 1

                                                  Application1

                                                  ValidatorDBUtils

                                                  PQLFilter

                                                  PQODBUtils

                                                  QOS

                                                  Application2

                                                  Data Access Layer 2

                                                  Application3

                                                  Application4

                                                  Data Access Layer 3

                                                  Application5

                                                  Application6Reusable For ALL Data Access Layer Logic

                                                  43

                                                  Not Just Java Not A Problem

                                                  DAL operations have little or no statehellip

                                                  Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                  Can you say hellip Web Service

                                                  44

                                                  A Really Nice Stack

                                                  MongoDBJava Driver

                                                  MQLValidator

                                                  Data Access Layer

                                                  JavaApplication

                                                  ValidatorDBUtils

                                                  PQLFilter

                                                  HTTP Endpoint

                                                  pythonApplication

                                                  curl

                                                  JSON over HTTP(S)

                                                  JSONlt-gtJava Maps

                                                  QOSDBUtils

                                                  PostQuery

                                                  PQODBUtils

                                                  QOS

                                                  45

                                                  What Can We Do

                                                  46

                                                  Secure Access To Redacted Data for Testing

                                                  $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                  $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                  $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                  47

                                                  Get It Programmatically Too

                                                  This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                  getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                  String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                  48

                                                  Robust Validated Data Ingest

                                                  $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                  $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                  49

                                                  Concept The control_ Collection

                                                  gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                  bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                  50

                                                  Validation QOS and PQO via Web Services

                                                  MongoDBJava Driver

                                                  MQLValidator

                                                  Data Access Layer

                                                  JavaApplication

                                                  ValidatorHTTPUtils

                                                  PQLFilter

                                                  pythonApplication

                                                  curl

                                                  JSON over HTTP(S)

                                                  QOSHTTPUtils

                                                  PostQuery

                                                  PQOHTTPUtils

                                                  QOS

                                                  HTTP Service

                                                  JSONlt-gtJava Maps

                                                  51

                                                  Are We Excited Yet

                                                  Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                  and other Java resources

                                                  Questions amp Answers

                                                  54

                                                  Concept DataProvider

                                                  public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                  Class MongoProvider implements DataProvider hellip

                                                  Class RESTfulProvider implements DataProvider hellip

                                                  55

                                                  The RESTful Provider

                                                  class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                  fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                  url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                  HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                  • Slide 1
                                                  • Part 3 In The Data Management Series
                                                  • Congratulations At this Point Yoursquove
                                                  • The Next Stage Defend amp Leverage
                                                  • MongoDB Doesnrsquot Have These Things
                                                  • What Can We Do Today
                                                  • Write Some Code
                                                  • Starting Point The Data Access Layer
                                                  • Document Validation
                                                  • A Query Filters Outbound Data
                                                  • How About Using It To Filter Inbounds
                                                  • $exists And $type Already in MQL
                                                  • hellip And MQL Goes Way Beyondhellip
                                                  • A New MQL Validator Module Emerges
                                                  • Migrating Capability into MongoDB
                                                  • Code For The FuturehellipToday
                                                  • But What About Today
                                                  • Temporary Filling PQL
                                                  • Bridge MQL to PQL
                                                  • No PQL No Problem
                                                  • MQL Is Easy To Navigate
                                                  • Where Do Validations Come From
                                                  • The Validations Collection
                                                  • Various ldquoLevelsrdquo of Validation
                                                  • Multiple Types Schema By Example
                                                  • The Stack So Far
                                                  • Representative Example
                                                  • Redaction
                                                  • Concept Post Query Operations (PQO)
                                                  • Adopt MQL-like behavior
                                                  • A New PostQuery Module Emerges
                                                  • Where Do PQOs Come From
                                                  • The Postquery Collection
                                                  • The Stack Is Getting Rich
                                                  • Representative Example (2)
                                                  • Quality of Service
                                                  • QOS In Action
                                                  • Where Do We Store QOS Values
                                                  • The QOS Collection
                                                  • Coming Togetherhellip
                                                  • Representative Example (3)
                                                  • A Highly Leveragable Investment
                                                  • Not Just Java Not A Problem
                                                  • A Really Nice Stack
                                                  • What Can We Do
                                                  • Secure Access To Redacted Data for Testing
                                                  • Get It Programmatically Too
                                                  • Robust Validated Data Ingest
                                                  • Concept The control_ Collection
                                                  • Validation QOS and PQO via Web Services
                                                  • Are We Excited Yet
                                                  • Questions amp Answers
                                                  • Slide 53
                                                  • Concept DataProvider
                                                  • The RESTful Provider

                                                    26

                                                    The Stack So Far

                                                    MongoDBJava Driver

                                                    MQLValidator

                                                    Data Access Layer

                                                    Application

                                                    ValidatorDBUtils

                                                    ValidatorDBUtils populates an MQLValidator object from MongoDB

                                                    PQLFilter

                                                    27

                                                    Representative Example

                                                    class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                                    someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                                    28

                                                    Redaction

                                                    29

                                                    Concept Post Query Operations (PQO)

                                                    ssn $hash model birthdate null

                                                    $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                                    30

                                                    Adopt MQL-like behavior

                                                    ldquossnrdquonull

                                                    ldquoaddressrdquo ldquoXXXXrdquo

                                                    ldquossnrdquo $substitute ldquossnmodelrdquo

                                                    Remove field by setting to null

                                                    Redact address with fixed value

                                                    Substitute SSN with a different correct consistent value

                                                    ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                    Hash counterparty name to consistent value

                                                    31

                                                    A New PostQuery Module Emerges

                                                    class PostQuery process(Map data Map operations)

                                                    PostQuery

                                                    MongoDBJava Driver

                                                    MQLValidator

                                                    Data Access Layer

                                                    Application

                                                    ValidatorDBUtils

                                                    PQLFilter

                                                    32

                                                    Where Do PQOs Come From

                                                    The Database

                                                    33

                                                    The Postquery Collection

                                                    gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                    34

                                                    The Stack Is Getting Rich

                                                    PostQuery

                                                    MongoDBJava Driver

                                                    MQLValidator

                                                    Data Access Layer

                                                    Application

                                                    ValidatorDBUtils

                                                    PQLFilter

                                                    PQODBUtils

                                                    35

                                                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                    someWriteOperation(Map data) hellip

                                                    someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                    36

                                                    Quality of Service

                                                    37

                                                    QOS In Action

                                                    class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                    Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                    38

                                                    Where Do We Store QOS Values

                                                    The Database

                                                    39

                                                    The QOS Collection

                                                    gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                    ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                    40

                                                    QOSDBUtils

                                                    Coming Togetherhellip

                                                    PostQuery

                                                    MongoDBJava Driver

                                                    MQLValidator

                                                    Data Access Layer

                                                    Application

                                                    ValidatorDBUtils

                                                    PQLFilter

                                                    PQODBUtils

                                                    QOS

                                                    41

                                                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                    someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                    42

                                                    QOSDBUtils

                                                    A Highly Leveragable Investment

                                                    PostQuery

                                                    MQLValidator

                                                    Data Access Layer 1

                                                    Application1

                                                    ValidatorDBUtils

                                                    PQLFilter

                                                    PQODBUtils

                                                    QOS

                                                    Application2

                                                    Data Access Layer 2

                                                    Application3

                                                    Application4

                                                    Data Access Layer 3

                                                    Application5

                                                    Application6Reusable For ALL Data Access Layer Logic

                                                    43

                                                    Not Just Java Not A Problem

                                                    DAL operations have little or no statehellip

                                                    Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                    Can you say hellip Web Service

                                                    44

                                                    A Really Nice Stack

                                                    MongoDBJava Driver

                                                    MQLValidator

                                                    Data Access Layer

                                                    JavaApplication

                                                    ValidatorDBUtils

                                                    PQLFilter

                                                    HTTP Endpoint

                                                    pythonApplication

                                                    curl

                                                    JSON over HTTP(S)

                                                    JSONlt-gtJava Maps

                                                    QOSDBUtils

                                                    PostQuery

                                                    PQODBUtils

                                                    QOS

                                                    45

                                                    What Can We Do

                                                    46

                                                    Secure Access To Redacted Data for Testing

                                                    $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                    $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                    $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                    47

                                                    Get It Programmatically Too

                                                    This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                    getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                    String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                    48

                                                    Robust Validated Data Ingest

                                                    $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                    $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                    49

                                                    Concept The control_ Collection

                                                    gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                    bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                    50

                                                    Validation QOS and PQO via Web Services

                                                    MongoDBJava Driver

                                                    MQLValidator

                                                    Data Access Layer

                                                    JavaApplication

                                                    ValidatorHTTPUtils

                                                    PQLFilter

                                                    pythonApplication

                                                    curl

                                                    JSON over HTTP(S)

                                                    QOSHTTPUtils

                                                    PostQuery

                                                    PQOHTTPUtils

                                                    QOS

                                                    HTTP Service

                                                    JSONlt-gtJava Maps

                                                    51

                                                    Are We Excited Yet

                                                    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                    and other Java resources

                                                    Questions amp Answers

                                                    54

                                                    Concept DataProvider

                                                    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                    Class MongoProvider implements DataProvider hellip

                                                    Class RESTfulProvider implements DataProvider hellip

                                                    55

                                                    The RESTful Provider

                                                    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                    • Slide 1
                                                    • Part 3 In The Data Management Series
                                                    • Congratulations At this Point Yoursquove
                                                    • The Next Stage Defend amp Leverage
                                                    • MongoDB Doesnrsquot Have These Things
                                                    • What Can We Do Today
                                                    • Write Some Code
                                                    • Starting Point The Data Access Layer
                                                    • Document Validation
                                                    • A Query Filters Outbound Data
                                                    • How About Using It To Filter Inbounds
                                                    • $exists And $type Already in MQL
                                                    • hellip And MQL Goes Way Beyondhellip
                                                    • A New MQL Validator Module Emerges
                                                    • Migrating Capability into MongoDB
                                                    • Code For The FuturehellipToday
                                                    • But What About Today
                                                    • Temporary Filling PQL
                                                    • Bridge MQL to PQL
                                                    • No PQL No Problem
                                                    • MQL Is Easy To Navigate
                                                    • Where Do Validations Come From
                                                    • The Validations Collection
                                                    • Various ldquoLevelsrdquo of Validation
                                                    • Multiple Types Schema By Example
                                                    • The Stack So Far
                                                    • Representative Example
                                                    • Redaction
                                                    • Concept Post Query Operations (PQO)
                                                    • Adopt MQL-like behavior
                                                    • A New PostQuery Module Emerges
                                                    • Where Do PQOs Come From
                                                    • The Postquery Collection
                                                    • The Stack Is Getting Rich
                                                    • Representative Example (2)
                                                    • Quality of Service
                                                    • QOS In Action
                                                    • Where Do We Store QOS Values
                                                    • The QOS Collection
                                                    • Coming Togetherhellip
                                                    • Representative Example (3)
                                                    • A Highly Leveragable Investment
                                                    • Not Just Java Not A Problem
                                                    • A Really Nice Stack
                                                    • What Can We Do
                                                    • Secure Access To Redacted Data for Testing
                                                    • Get It Programmatically Too
                                                    • Robust Validated Data Ingest
                                                    • Concept The control_ Collection
                                                    • Validation QOS and PQO via Web Services
                                                    • Are We Excited Yet
                                                    • Questions amp Answers
                                                    • Slide 53
                                                    • Concept DataProvider
                                                    • The RESTful Provider

                                                      27

                                                      Representative Example

                                                      class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

                                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

                                                      someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

                                                      28

                                                      Redaction

                                                      29

                                                      Concept Post Query Operations (PQO)

                                                      ssn $hash model birthdate null

                                                      $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                                      30

                                                      Adopt MQL-like behavior

                                                      ldquossnrdquonull

                                                      ldquoaddressrdquo ldquoXXXXrdquo

                                                      ldquossnrdquo $substitute ldquossnmodelrdquo

                                                      Remove field by setting to null

                                                      Redact address with fixed value

                                                      Substitute SSN with a different correct consistent value

                                                      ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                      Hash counterparty name to consistent value

                                                      31

                                                      A New PostQuery Module Emerges

                                                      class PostQuery process(Map data Map operations)

                                                      PostQuery

                                                      MongoDBJava Driver

                                                      MQLValidator

                                                      Data Access Layer

                                                      Application

                                                      ValidatorDBUtils

                                                      PQLFilter

                                                      32

                                                      Where Do PQOs Come From

                                                      The Database

                                                      33

                                                      The Postquery Collection

                                                      gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                      34

                                                      The Stack Is Getting Rich

                                                      PostQuery

                                                      MongoDBJava Driver

                                                      MQLValidator

                                                      Data Access Layer

                                                      Application

                                                      ValidatorDBUtils

                                                      PQLFilter

                                                      PQODBUtils

                                                      35

                                                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                      someWriteOperation(Map data) hellip

                                                      someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                      36

                                                      Quality of Service

                                                      37

                                                      QOS In Action

                                                      class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                      Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                      38

                                                      Where Do We Store QOS Values

                                                      The Database

                                                      39

                                                      The QOS Collection

                                                      gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                      ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                      40

                                                      QOSDBUtils

                                                      Coming Togetherhellip

                                                      PostQuery

                                                      MongoDBJava Driver

                                                      MQLValidator

                                                      Data Access Layer

                                                      Application

                                                      ValidatorDBUtils

                                                      PQLFilter

                                                      PQODBUtils

                                                      QOS

                                                      41

                                                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                      someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                      42

                                                      QOSDBUtils

                                                      A Highly Leveragable Investment

                                                      PostQuery

                                                      MQLValidator

                                                      Data Access Layer 1

                                                      Application1

                                                      ValidatorDBUtils

                                                      PQLFilter

                                                      PQODBUtils

                                                      QOS

                                                      Application2

                                                      Data Access Layer 2

                                                      Application3

                                                      Application4

                                                      Data Access Layer 3

                                                      Application5

                                                      Application6Reusable For ALL Data Access Layer Logic

                                                      43

                                                      Not Just Java Not A Problem

                                                      DAL operations have little or no statehellip

                                                      Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                      Can you say hellip Web Service

                                                      44

                                                      A Really Nice Stack

                                                      MongoDBJava Driver

                                                      MQLValidator

                                                      Data Access Layer

                                                      JavaApplication

                                                      ValidatorDBUtils

                                                      PQLFilter

                                                      HTTP Endpoint

                                                      pythonApplication

                                                      curl

                                                      JSON over HTTP(S)

                                                      JSONlt-gtJava Maps

                                                      QOSDBUtils

                                                      PostQuery

                                                      PQODBUtils

                                                      QOS

                                                      45

                                                      What Can We Do

                                                      46

                                                      Secure Access To Redacted Data for Testing

                                                      $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                      $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                      $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                      47

                                                      Get It Programmatically Too

                                                      This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                      getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                      String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                      48

                                                      Robust Validated Data Ingest

                                                      $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                      $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                      49

                                                      Concept The control_ Collection

                                                      gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                      bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                      50

                                                      Validation QOS and PQO via Web Services

                                                      MongoDBJava Driver

                                                      MQLValidator

                                                      Data Access Layer

                                                      JavaApplication

                                                      ValidatorHTTPUtils

                                                      PQLFilter

                                                      pythonApplication

                                                      curl

                                                      JSON over HTTP(S)

                                                      QOSHTTPUtils

                                                      PostQuery

                                                      PQOHTTPUtils

                                                      QOS

                                                      HTTP Service

                                                      JSONlt-gtJava Maps

                                                      51

                                                      Are We Excited Yet

                                                      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                      and other Java resources

                                                      Questions amp Answers

                                                      54

                                                      Concept DataProvider

                                                      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                      Class MongoProvider implements DataProvider hellip

                                                      Class RESTfulProvider implements DataProvider hellip

                                                      55

                                                      The RESTful Provider

                                                      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                      • Slide 1
                                                      • Part 3 In The Data Management Series
                                                      • Congratulations At this Point Yoursquove
                                                      • The Next Stage Defend amp Leverage
                                                      • MongoDB Doesnrsquot Have These Things
                                                      • What Can We Do Today
                                                      • Write Some Code
                                                      • Starting Point The Data Access Layer
                                                      • Document Validation
                                                      • A Query Filters Outbound Data
                                                      • How About Using It To Filter Inbounds
                                                      • $exists And $type Already in MQL
                                                      • hellip And MQL Goes Way Beyondhellip
                                                      • A New MQL Validator Module Emerges
                                                      • Migrating Capability into MongoDB
                                                      • Code For The FuturehellipToday
                                                      • But What About Today
                                                      • Temporary Filling PQL
                                                      • Bridge MQL to PQL
                                                      • No PQL No Problem
                                                      • MQL Is Easy To Navigate
                                                      • Where Do Validations Come From
                                                      • The Validations Collection
                                                      • Various ldquoLevelsrdquo of Validation
                                                      • Multiple Types Schema By Example
                                                      • The Stack So Far
                                                      • Representative Example
                                                      • Redaction
                                                      • Concept Post Query Operations (PQO)
                                                      • Adopt MQL-like behavior
                                                      • A New PostQuery Module Emerges
                                                      • Where Do PQOs Come From
                                                      • The Postquery Collection
                                                      • The Stack Is Getting Rich
                                                      • Representative Example (2)
                                                      • Quality of Service
                                                      • QOS In Action
                                                      • Where Do We Store QOS Values
                                                      • The QOS Collection
                                                      • Coming Togetherhellip
                                                      • Representative Example (3)
                                                      • A Highly Leveragable Investment
                                                      • Not Just Java Not A Problem
                                                      • A Really Nice Stack
                                                      • What Can We Do
                                                      • Secure Access To Redacted Data for Testing
                                                      • Get It Programmatically Too
                                                      • Robust Validated Data Ingest
                                                      • Concept The control_ Collection
                                                      • Validation QOS and PQO via Web Services
                                                      • Are We Excited Yet
                                                      • Questions amp Answers
                                                      • Slide 53
                                                      • Concept DataProvider
                                                      • The RESTful Provider

                                                        28

                                                        Redaction

                                                        29

                                                        Concept Post Query Operations (PQO)

                                                        ssn $hash model birthdate null

                                                        $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                                        30

                                                        Adopt MQL-like behavior

                                                        ldquossnrdquonull

                                                        ldquoaddressrdquo ldquoXXXXrdquo

                                                        ldquossnrdquo $substitute ldquossnmodelrdquo

                                                        Remove field by setting to null

                                                        Redact address with fixed value

                                                        Substitute SSN with a different correct consistent value

                                                        ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                        Hash counterparty name to consistent value

                                                        31

                                                        A New PostQuery Module Emerges

                                                        class PostQuery process(Map data Map operations)

                                                        PostQuery

                                                        MongoDBJava Driver

                                                        MQLValidator

                                                        Data Access Layer

                                                        Application

                                                        ValidatorDBUtils

                                                        PQLFilter

                                                        32

                                                        Where Do PQOs Come From

                                                        The Database

                                                        33

                                                        The Postquery Collection

                                                        gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                        34

                                                        The Stack Is Getting Rich

                                                        PostQuery

                                                        MongoDBJava Driver

                                                        MQLValidator

                                                        Data Access Layer

                                                        Application

                                                        ValidatorDBUtils

                                                        PQLFilter

                                                        PQODBUtils

                                                        35

                                                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                        someWriteOperation(Map data) hellip

                                                        someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                        36

                                                        Quality of Service

                                                        37

                                                        QOS In Action

                                                        class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                        Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                        38

                                                        Where Do We Store QOS Values

                                                        The Database

                                                        39

                                                        The QOS Collection

                                                        gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                        ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                        40

                                                        QOSDBUtils

                                                        Coming Togetherhellip

                                                        PostQuery

                                                        MongoDBJava Driver

                                                        MQLValidator

                                                        Data Access Layer

                                                        Application

                                                        ValidatorDBUtils

                                                        PQLFilter

                                                        PQODBUtils

                                                        QOS

                                                        41

                                                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                        someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                        42

                                                        QOSDBUtils

                                                        A Highly Leveragable Investment

                                                        PostQuery

                                                        MQLValidator

                                                        Data Access Layer 1

                                                        Application1

                                                        ValidatorDBUtils

                                                        PQLFilter

                                                        PQODBUtils

                                                        QOS

                                                        Application2

                                                        Data Access Layer 2

                                                        Application3

                                                        Application4

                                                        Data Access Layer 3

                                                        Application5

                                                        Application6Reusable For ALL Data Access Layer Logic

                                                        43

                                                        Not Just Java Not A Problem

                                                        DAL operations have little or no statehellip

                                                        Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                        Can you say hellip Web Service

                                                        44

                                                        A Really Nice Stack

                                                        MongoDBJava Driver

                                                        MQLValidator

                                                        Data Access Layer

                                                        JavaApplication

                                                        ValidatorDBUtils

                                                        PQLFilter

                                                        HTTP Endpoint

                                                        pythonApplication

                                                        curl

                                                        JSON over HTTP(S)

                                                        JSONlt-gtJava Maps

                                                        QOSDBUtils

                                                        PostQuery

                                                        PQODBUtils

                                                        QOS

                                                        45

                                                        What Can We Do

                                                        46

                                                        Secure Access To Redacted Data for Testing

                                                        $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                        $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                        $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                        47

                                                        Get It Programmatically Too

                                                        This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                        getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                        String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                        48

                                                        Robust Validated Data Ingest

                                                        $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                        $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                        49

                                                        Concept The control_ Collection

                                                        gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                        bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                        50

                                                        Validation QOS and PQO via Web Services

                                                        MongoDBJava Driver

                                                        MQLValidator

                                                        Data Access Layer

                                                        JavaApplication

                                                        ValidatorHTTPUtils

                                                        PQLFilter

                                                        pythonApplication

                                                        curl

                                                        JSON over HTTP(S)

                                                        QOSHTTPUtils

                                                        PostQuery

                                                        PQOHTTPUtils

                                                        QOS

                                                        HTTP Service

                                                        JSONlt-gtJava Maps

                                                        51

                                                        Are We Excited Yet

                                                        Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                        and other Java resources

                                                        Questions amp Answers

                                                        54

                                                        Concept DataProvider

                                                        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                        Class MongoProvider implements DataProvider hellip

                                                        Class RESTfulProvider implements DataProvider hellip

                                                        55

                                                        The RESTful Provider

                                                        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                        • Slide 1
                                                        • Part 3 In The Data Management Series
                                                        • Congratulations At this Point Yoursquove
                                                        • The Next Stage Defend amp Leverage
                                                        • MongoDB Doesnrsquot Have These Things
                                                        • What Can We Do Today
                                                        • Write Some Code
                                                        • Starting Point The Data Access Layer
                                                        • Document Validation
                                                        • A Query Filters Outbound Data
                                                        • How About Using It To Filter Inbounds
                                                        • $exists And $type Already in MQL
                                                        • hellip And MQL Goes Way Beyondhellip
                                                        • A New MQL Validator Module Emerges
                                                        • Migrating Capability into MongoDB
                                                        • Code For The FuturehellipToday
                                                        • But What About Today
                                                        • Temporary Filling PQL
                                                        • Bridge MQL to PQL
                                                        • No PQL No Problem
                                                        • MQL Is Easy To Navigate
                                                        • Where Do Validations Come From
                                                        • The Validations Collection
                                                        • Various ldquoLevelsrdquo of Validation
                                                        • Multiple Types Schema By Example
                                                        • The Stack So Far
                                                        • Representative Example
                                                        • Redaction
                                                        • Concept Post Query Operations (PQO)
                                                        • Adopt MQL-like behavior
                                                        • A New PostQuery Module Emerges
                                                        • Where Do PQOs Come From
                                                        • The Postquery Collection
                                                        • The Stack Is Getting Rich
                                                        • Representative Example (2)
                                                        • Quality of Service
                                                        • QOS In Action
                                                        • Where Do We Store QOS Values
                                                        • The QOS Collection
                                                        • Coming Togetherhellip
                                                        • Representative Example (3)
                                                        • A Highly Leveragable Investment
                                                        • Not Just Java Not A Problem
                                                        • A Really Nice Stack
                                                        • What Can We Do
                                                        • Secure Access To Redacted Data for Testing
                                                        • Get It Programmatically Too
                                                        • Robust Validated Data Ingest
                                                        • Concept The control_ Collection
                                                        • Validation QOS and PQO via Web Services
                                                        • Are We Excited Yet
                                                        • Questions amp Answers
                                                        • Slide 53
                                                        • Concept DataProvider
                                                        • The RESTful Provider

                                                          29

                                                          Concept Post Query Operations (PQO)

                                                          ssn $hash model birthdate null

                                                          $and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

                                                          30

                                                          Adopt MQL-like behavior

                                                          ldquossnrdquonull

                                                          ldquoaddressrdquo ldquoXXXXrdquo

                                                          ldquossnrdquo $substitute ldquossnmodelrdquo

                                                          Remove field by setting to null

                                                          Redact address with fixed value

                                                          Substitute SSN with a different correct consistent value

                                                          ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                          Hash counterparty name to consistent value

                                                          31

                                                          A New PostQuery Module Emerges

                                                          class PostQuery process(Map data Map operations)

                                                          PostQuery

                                                          MongoDBJava Driver

                                                          MQLValidator

                                                          Data Access Layer

                                                          Application

                                                          ValidatorDBUtils

                                                          PQLFilter

                                                          32

                                                          Where Do PQOs Come From

                                                          The Database

                                                          33

                                                          The Postquery Collection

                                                          gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                          34

                                                          The Stack Is Getting Rich

                                                          PostQuery

                                                          MongoDBJava Driver

                                                          MQLValidator

                                                          Data Access Layer

                                                          Application

                                                          ValidatorDBUtils

                                                          PQLFilter

                                                          PQODBUtils

                                                          35

                                                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                          someWriteOperation(Map data) hellip

                                                          someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                          36

                                                          Quality of Service

                                                          37

                                                          QOS In Action

                                                          class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                          Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                          38

                                                          Where Do We Store QOS Values

                                                          The Database

                                                          39

                                                          The QOS Collection

                                                          gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                          ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                          40

                                                          QOSDBUtils

                                                          Coming Togetherhellip

                                                          PostQuery

                                                          MongoDBJava Driver

                                                          MQLValidator

                                                          Data Access Layer

                                                          Application

                                                          ValidatorDBUtils

                                                          PQLFilter

                                                          PQODBUtils

                                                          QOS

                                                          41

                                                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                          someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                          42

                                                          QOSDBUtils

                                                          A Highly Leveragable Investment

                                                          PostQuery

                                                          MQLValidator

                                                          Data Access Layer 1

                                                          Application1

                                                          ValidatorDBUtils

                                                          PQLFilter

                                                          PQODBUtils

                                                          QOS

                                                          Application2

                                                          Data Access Layer 2

                                                          Application3

                                                          Application4

                                                          Data Access Layer 3

                                                          Application5

                                                          Application6Reusable For ALL Data Access Layer Logic

                                                          43

                                                          Not Just Java Not A Problem

                                                          DAL operations have little or no statehellip

                                                          Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                          Can you say hellip Web Service

                                                          44

                                                          A Really Nice Stack

                                                          MongoDBJava Driver

                                                          MQLValidator

                                                          Data Access Layer

                                                          JavaApplication

                                                          ValidatorDBUtils

                                                          PQLFilter

                                                          HTTP Endpoint

                                                          pythonApplication

                                                          curl

                                                          JSON over HTTP(S)

                                                          JSONlt-gtJava Maps

                                                          QOSDBUtils

                                                          PostQuery

                                                          PQODBUtils

                                                          QOS

                                                          45

                                                          What Can We Do

                                                          46

                                                          Secure Access To Redacted Data for Testing

                                                          $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                          $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                          $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                          47

                                                          Get It Programmatically Too

                                                          This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                          getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                          String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                          48

                                                          Robust Validated Data Ingest

                                                          $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                          $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                          49

                                                          Concept The control_ Collection

                                                          gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                          bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                          50

                                                          Validation QOS and PQO via Web Services

                                                          MongoDBJava Driver

                                                          MQLValidator

                                                          Data Access Layer

                                                          JavaApplication

                                                          ValidatorHTTPUtils

                                                          PQLFilter

                                                          pythonApplication

                                                          curl

                                                          JSON over HTTP(S)

                                                          QOSHTTPUtils

                                                          PostQuery

                                                          PQOHTTPUtils

                                                          QOS

                                                          HTTP Service

                                                          JSONlt-gtJava Maps

                                                          51

                                                          Are We Excited Yet

                                                          Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                          and other Java resources

                                                          Questions amp Answers

                                                          54

                                                          Concept DataProvider

                                                          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                          Class MongoProvider implements DataProvider hellip

                                                          Class RESTfulProvider implements DataProvider hellip

                                                          55

                                                          The RESTful Provider

                                                          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                          • Slide 1
                                                          • Part 3 In The Data Management Series
                                                          • Congratulations At this Point Yoursquove
                                                          • The Next Stage Defend amp Leverage
                                                          • MongoDB Doesnrsquot Have These Things
                                                          • What Can We Do Today
                                                          • Write Some Code
                                                          • Starting Point The Data Access Layer
                                                          • Document Validation
                                                          • A Query Filters Outbound Data
                                                          • How About Using It To Filter Inbounds
                                                          • $exists And $type Already in MQL
                                                          • hellip And MQL Goes Way Beyondhellip
                                                          • A New MQL Validator Module Emerges
                                                          • Migrating Capability into MongoDB
                                                          • Code For The FuturehellipToday
                                                          • But What About Today
                                                          • Temporary Filling PQL
                                                          • Bridge MQL to PQL
                                                          • No PQL No Problem
                                                          • MQL Is Easy To Navigate
                                                          • Where Do Validations Come From
                                                          • The Validations Collection
                                                          • Various ldquoLevelsrdquo of Validation
                                                          • Multiple Types Schema By Example
                                                          • The Stack So Far
                                                          • Representative Example
                                                          • Redaction
                                                          • Concept Post Query Operations (PQO)
                                                          • Adopt MQL-like behavior
                                                          • A New PostQuery Module Emerges
                                                          • Where Do PQOs Come From
                                                          • The Postquery Collection
                                                          • The Stack Is Getting Rich
                                                          • Representative Example (2)
                                                          • Quality of Service
                                                          • QOS In Action
                                                          • Where Do We Store QOS Values
                                                          • The QOS Collection
                                                          • Coming Togetherhellip
                                                          • Representative Example (3)
                                                          • A Highly Leveragable Investment
                                                          • Not Just Java Not A Problem
                                                          • A Really Nice Stack
                                                          • What Can We Do
                                                          • Secure Access To Redacted Data for Testing
                                                          • Get It Programmatically Too
                                                          • Robust Validated Data Ingest
                                                          • Concept The control_ Collection
                                                          • Validation QOS and PQO via Web Services
                                                          • Are We Excited Yet
                                                          • Questions amp Answers
                                                          • Slide 53
                                                          • Concept DataProvider
                                                          • The RESTful Provider

                                                            30

                                                            Adopt MQL-like behavior

                                                            ldquossnrdquonull

                                                            ldquoaddressrdquo ldquoXXXXrdquo

                                                            ldquossnrdquo $substitute ldquossnmodelrdquo

                                                            Remove field by setting to null

                                                            Redact address with fixed value

                                                            Substitute SSN with a different correct consistent value

                                                            ldquocounterpartyrdquo $hash ldquoMD5rdquo

                                                            Hash counterparty name to consistent value

                                                            31

                                                            A New PostQuery Module Emerges

                                                            class PostQuery process(Map data Map operations)

                                                            PostQuery

                                                            MongoDBJava Driver

                                                            MQLValidator

                                                            Data Access Layer

                                                            Application

                                                            ValidatorDBUtils

                                                            PQLFilter

                                                            32

                                                            Where Do PQOs Come From

                                                            The Database

                                                            33

                                                            The Postquery Collection

                                                            gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                            34

                                                            The Stack Is Getting Rich

                                                            PostQuery

                                                            MongoDBJava Driver

                                                            MQLValidator

                                                            Data Access Layer

                                                            Application

                                                            ValidatorDBUtils

                                                            PQLFilter

                                                            PQODBUtils

                                                            35

                                                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                            someWriteOperation(Map data) hellip

                                                            someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                            36

                                                            Quality of Service

                                                            37

                                                            QOS In Action

                                                            class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                            Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                            38

                                                            Where Do We Store QOS Values

                                                            The Database

                                                            39

                                                            The QOS Collection

                                                            gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                            ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                            40

                                                            QOSDBUtils

                                                            Coming Togetherhellip

                                                            PostQuery

                                                            MongoDBJava Driver

                                                            MQLValidator

                                                            Data Access Layer

                                                            Application

                                                            ValidatorDBUtils

                                                            PQLFilter

                                                            PQODBUtils

                                                            QOS

                                                            41

                                                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                            someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                            42

                                                            QOSDBUtils

                                                            A Highly Leveragable Investment

                                                            PostQuery

                                                            MQLValidator

                                                            Data Access Layer 1

                                                            Application1

                                                            ValidatorDBUtils

                                                            PQLFilter

                                                            PQODBUtils

                                                            QOS

                                                            Application2

                                                            Data Access Layer 2

                                                            Application3

                                                            Application4

                                                            Data Access Layer 3

                                                            Application5

                                                            Application6Reusable For ALL Data Access Layer Logic

                                                            43

                                                            Not Just Java Not A Problem

                                                            DAL operations have little or no statehellip

                                                            Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                            Can you say hellip Web Service

                                                            44

                                                            A Really Nice Stack

                                                            MongoDBJava Driver

                                                            MQLValidator

                                                            Data Access Layer

                                                            JavaApplication

                                                            ValidatorDBUtils

                                                            PQLFilter

                                                            HTTP Endpoint

                                                            pythonApplication

                                                            curl

                                                            JSON over HTTP(S)

                                                            JSONlt-gtJava Maps

                                                            QOSDBUtils

                                                            PostQuery

                                                            PQODBUtils

                                                            QOS

                                                            45

                                                            What Can We Do

                                                            46

                                                            Secure Access To Redacted Data for Testing

                                                            $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                            $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                            $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                            47

                                                            Get It Programmatically Too

                                                            This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                            getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                            String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                            48

                                                            Robust Validated Data Ingest

                                                            $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                            $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                            49

                                                            Concept The control_ Collection

                                                            gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                            bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                            50

                                                            Validation QOS and PQO via Web Services

                                                            MongoDBJava Driver

                                                            MQLValidator

                                                            Data Access Layer

                                                            JavaApplication

                                                            ValidatorHTTPUtils

                                                            PQLFilter

                                                            pythonApplication

                                                            curl

                                                            JSON over HTTP(S)

                                                            QOSHTTPUtils

                                                            PostQuery

                                                            PQOHTTPUtils

                                                            QOS

                                                            HTTP Service

                                                            JSONlt-gtJava Maps

                                                            51

                                                            Are We Excited Yet

                                                            Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                            and other Java resources

                                                            Questions amp Answers

                                                            54

                                                            Concept DataProvider

                                                            public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                            Class MongoProvider implements DataProvider hellip

                                                            Class RESTfulProvider implements DataProvider hellip

                                                            55

                                                            The RESTful Provider

                                                            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                            • Slide 1
                                                            • Part 3 In The Data Management Series
                                                            • Congratulations At this Point Yoursquove
                                                            • The Next Stage Defend amp Leverage
                                                            • MongoDB Doesnrsquot Have These Things
                                                            • What Can We Do Today
                                                            • Write Some Code
                                                            • Starting Point The Data Access Layer
                                                            • Document Validation
                                                            • A Query Filters Outbound Data
                                                            • How About Using It To Filter Inbounds
                                                            • $exists And $type Already in MQL
                                                            • hellip And MQL Goes Way Beyondhellip
                                                            • A New MQL Validator Module Emerges
                                                            • Migrating Capability into MongoDB
                                                            • Code For The FuturehellipToday
                                                            • But What About Today
                                                            • Temporary Filling PQL
                                                            • Bridge MQL to PQL
                                                            • No PQL No Problem
                                                            • MQL Is Easy To Navigate
                                                            • Where Do Validations Come From
                                                            • The Validations Collection
                                                            • Various ldquoLevelsrdquo of Validation
                                                            • Multiple Types Schema By Example
                                                            • The Stack So Far
                                                            • Representative Example
                                                            • Redaction
                                                            • Concept Post Query Operations (PQO)
                                                            • Adopt MQL-like behavior
                                                            • A New PostQuery Module Emerges
                                                            • Where Do PQOs Come From
                                                            • The Postquery Collection
                                                            • The Stack Is Getting Rich
                                                            • Representative Example (2)
                                                            • Quality of Service
                                                            • QOS In Action
                                                            • Where Do We Store QOS Values
                                                            • The QOS Collection
                                                            • Coming Togetherhellip
                                                            • Representative Example (3)
                                                            • A Highly Leveragable Investment
                                                            • Not Just Java Not A Problem
                                                            • A Really Nice Stack
                                                            • What Can We Do
                                                            • Secure Access To Redacted Data for Testing
                                                            • Get It Programmatically Too
                                                            • Robust Validated Data Ingest
                                                            • Concept The control_ Collection
                                                            • Validation QOS and PQO via Web Services
                                                            • Are We Excited Yet
                                                            • Questions amp Answers
                                                            • Slide 53
                                                            • Concept DataProvider
                                                            • The RESTful Provider

                                                              31

                                                              A New PostQuery Module Emerges

                                                              class PostQuery process(Map data Map operations)

                                                              PostQuery

                                                              MongoDBJava Driver

                                                              MQLValidator

                                                              Data Access Layer

                                                              Application

                                                              ValidatorDBUtils

                                                              PQLFilter

                                                              32

                                                              Where Do PQOs Come From

                                                              The Database

                                                              33

                                                              The Postquery Collection

                                                              gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                              34

                                                              The Stack Is Getting Rich

                                                              PostQuery

                                                              MongoDBJava Driver

                                                              MQLValidator

                                                              Data Access Layer

                                                              Application

                                                              ValidatorDBUtils

                                                              PQLFilter

                                                              PQODBUtils

                                                              35

                                                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                              someWriteOperation(Map data) hellip

                                                              someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                              36

                                                              Quality of Service

                                                              37

                                                              QOS In Action

                                                              class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                              Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                              38

                                                              Where Do We Store QOS Values

                                                              The Database

                                                              39

                                                              The QOS Collection

                                                              gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                              ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                              40

                                                              QOSDBUtils

                                                              Coming Togetherhellip

                                                              PostQuery

                                                              MongoDBJava Driver

                                                              MQLValidator

                                                              Data Access Layer

                                                              Application

                                                              ValidatorDBUtils

                                                              PQLFilter

                                                              PQODBUtils

                                                              QOS

                                                              41

                                                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                              someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                              42

                                                              QOSDBUtils

                                                              A Highly Leveragable Investment

                                                              PostQuery

                                                              MQLValidator

                                                              Data Access Layer 1

                                                              Application1

                                                              ValidatorDBUtils

                                                              PQLFilter

                                                              PQODBUtils

                                                              QOS

                                                              Application2

                                                              Data Access Layer 2

                                                              Application3

                                                              Application4

                                                              Data Access Layer 3

                                                              Application5

                                                              Application6Reusable For ALL Data Access Layer Logic

                                                              43

                                                              Not Just Java Not A Problem

                                                              DAL operations have little or no statehellip

                                                              Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                              Can you say hellip Web Service

                                                              44

                                                              A Really Nice Stack

                                                              MongoDBJava Driver

                                                              MQLValidator

                                                              Data Access Layer

                                                              JavaApplication

                                                              ValidatorDBUtils

                                                              PQLFilter

                                                              HTTP Endpoint

                                                              pythonApplication

                                                              curl

                                                              JSON over HTTP(S)

                                                              JSONlt-gtJava Maps

                                                              QOSDBUtils

                                                              PostQuery

                                                              PQODBUtils

                                                              QOS

                                                              45

                                                              What Can We Do

                                                              46

                                                              Secure Access To Redacted Data for Testing

                                                              $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                              $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                              $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                              47

                                                              Get It Programmatically Too

                                                              This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                              getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                              String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                              48

                                                              Robust Validated Data Ingest

                                                              $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                              $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                              49

                                                              Concept The control_ Collection

                                                              gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                              bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                              50

                                                              Validation QOS and PQO via Web Services

                                                              MongoDBJava Driver

                                                              MQLValidator

                                                              Data Access Layer

                                                              JavaApplication

                                                              ValidatorHTTPUtils

                                                              PQLFilter

                                                              pythonApplication

                                                              curl

                                                              JSON over HTTP(S)

                                                              QOSHTTPUtils

                                                              PostQuery

                                                              PQOHTTPUtils

                                                              QOS

                                                              HTTP Service

                                                              JSONlt-gtJava Maps

                                                              51

                                                              Are We Excited Yet

                                                              Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                              and other Java resources

                                                              Questions amp Answers

                                                              54

                                                              Concept DataProvider

                                                              public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                              Class MongoProvider implements DataProvider hellip

                                                              Class RESTfulProvider implements DataProvider hellip

                                                              55

                                                              The RESTful Provider

                                                              class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                              fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                              url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                              HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                              • Slide 1
                                                              • Part 3 In The Data Management Series
                                                              • Congratulations At this Point Yoursquove
                                                              • The Next Stage Defend amp Leverage
                                                              • MongoDB Doesnrsquot Have These Things
                                                              • What Can We Do Today
                                                              • Write Some Code
                                                              • Starting Point The Data Access Layer
                                                              • Document Validation
                                                              • A Query Filters Outbound Data
                                                              • How About Using It To Filter Inbounds
                                                              • $exists And $type Already in MQL
                                                              • hellip And MQL Goes Way Beyondhellip
                                                              • A New MQL Validator Module Emerges
                                                              • Migrating Capability into MongoDB
                                                              • Code For The FuturehellipToday
                                                              • But What About Today
                                                              • Temporary Filling PQL
                                                              • Bridge MQL to PQL
                                                              • No PQL No Problem
                                                              • MQL Is Easy To Navigate
                                                              • Where Do Validations Come From
                                                              • The Validations Collection
                                                              • Various ldquoLevelsrdquo of Validation
                                                              • Multiple Types Schema By Example
                                                              • The Stack So Far
                                                              • Representative Example
                                                              • Redaction
                                                              • Concept Post Query Operations (PQO)
                                                              • Adopt MQL-like behavior
                                                              • A New PostQuery Module Emerges
                                                              • Where Do PQOs Come From
                                                              • The Postquery Collection
                                                              • The Stack Is Getting Rich
                                                              • Representative Example (2)
                                                              • Quality of Service
                                                              • QOS In Action
                                                              • Where Do We Store QOS Values
                                                              • The QOS Collection
                                                              • Coming Togetherhellip
                                                              • Representative Example (3)
                                                              • A Highly Leveragable Investment
                                                              • Not Just Java Not A Problem
                                                              • A Really Nice Stack
                                                              • What Can We Do
                                                              • Secure Access To Redacted Data for Testing
                                                              • Get It Programmatically Too
                                                              • Robust Validated Data Ingest
                                                              • Concept The control_ Collection
                                                              • Validation QOS and PQO via Web Services
                                                              • Are We Excited Yet
                                                              • Questions amp Answers
                                                              • Slide 53
                                                              • Concept DataProvider
                                                              • The RESTful Provider

                                                                32

                                                                Where Do PQOs Come From

                                                                The Database

                                                                33

                                                                The Postquery Collection

                                                                gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                                34

                                                                The Stack Is Getting Rich

                                                                PostQuery

                                                                MongoDBJava Driver

                                                                MQLValidator

                                                                Data Access Layer

                                                                Application

                                                                ValidatorDBUtils

                                                                PQLFilter

                                                                PQODBUtils

                                                                35

                                                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                                someWriteOperation(Map data) hellip

                                                                someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                                36

                                                                Quality of Service

                                                                37

                                                                QOS In Action

                                                                class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                                Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                38

                                                                Where Do We Store QOS Values

                                                                The Database

                                                                39

                                                                The QOS Collection

                                                                gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                40

                                                                QOSDBUtils

                                                                Coming Togetherhellip

                                                                PostQuery

                                                                MongoDBJava Driver

                                                                MQLValidator

                                                                Data Access Layer

                                                                Application

                                                                ValidatorDBUtils

                                                                PQLFilter

                                                                PQODBUtils

                                                                QOS

                                                                41

                                                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                42

                                                                QOSDBUtils

                                                                A Highly Leveragable Investment

                                                                PostQuery

                                                                MQLValidator

                                                                Data Access Layer 1

                                                                Application1

                                                                ValidatorDBUtils

                                                                PQLFilter

                                                                PQODBUtils

                                                                QOS

                                                                Application2

                                                                Data Access Layer 2

                                                                Application3

                                                                Application4

                                                                Data Access Layer 3

                                                                Application5

                                                                Application6Reusable For ALL Data Access Layer Logic

                                                                43

                                                                Not Just Java Not A Problem

                                                                DAL operations have little or no statehellip

                                                                Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                Can you say hellip Web Service

                                                                44

                                                                A Really Nice Stack

                                                                MongoDBJava Driver

                                                                MQLValidator

                                                                Data Access Layer

                                                                JavaApplication

                                                                ValidatorDBUtils

                                                                PQLFilter

                                                                HTTP Endpoint

                                                                pythonApplication

                                                                curl

                                                                JSON over HTTP(S)

                                                                JSONlt-gtJava Maps

                                                                QOSDBUtils

                                                                PostQuery

                                                                PQODBUtils

                                                                QOS

                                                                45

                                                                What Can We Do

                                                                46

                                                                Secure Access To Redacted Data for Testing

                                                                $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                47

                                                                Get It Programmatically Too

                                                                This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                48

                                                                Robust Validated Data Ingest

                                                                $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                49

                                                                Concept The control_ Collection

                                                                gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                50

                                                                Validation QOS and PQO via Web Services

                                                                MongoDBJava Driver

                                                                MQLValidator

                                                                Data Access Layer

                                                                JavaApplication

                                                                ValidatorHTTPUtils

                                                                PQLFilter

                                                                pythonApplication

                                                                curl

                                                                JSON over HTTP(S)

                                                                QOSHTTPUtils

                                                                PostQuery

                                                                PQOHTTPUtils

                                                                QOS

                                                                HTTP Service

                                                                JSONlt-gtJava Maps

                                                                51

                                                                Are We Excited Yet

                                                                Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                and other Java resources

                                                                Questions amp Answers

                                                                54

                                                                Concept DataProvider

                                                                public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                Class MongoProvider implements DataProvider hellip

                                                                Class RESTfulProvider implements DataProvider hellip

                                                                55

                                                                The RESTful Provider

                                                                class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                • Slide 1
                                                                • Part 3 In The Data Management Series
                                                                • Congratulations At this Point Yoursquove
                                                                • The Next Stage Defend amp Leverage
                                                                • MongoDB Doesnrsquot Have These Things
                                                                • What Can We Do Today
                                                                • Write Some Code
                                                                • Starting Point The Data Access Layer
                                                                • Document Validation
                                                                • A Query Filters Outbound Data
                                                                • How About Using It To Filter Inbounds
                                                                • $exists And $type Already in MQL
                                                                • hellip And MQL Goes Way Beyondhellip
                                                                • A New MQL Validator Module Emerges
                                                                • Migrating Capability into MongoDB
                                                                • Code For The FuturehellipToday
                                                                • But What About Today
                                                                • Temporary Filling PQL
                                                                • Bridge MQL to PQL
                                                                • No PQL No Problem
                                                                • MQL Is Easy To Navigate
                                                                • Where Do Validations Come From
                                                                • The Validations Collection
                                                                • Various ldquoLevelsrdquo of Validation
                                                                • Multiple Types Schema By Example
                                                                • The Stack So Far
                                                                • Representative Example
                                                                • Redaction
                                                                • Concept Post Query Operations (PQO)
                                                                • Adopt MQL-like behavior
                                                                • A New PostQuery Module Emerges
                                                                • Where Do PQOs Come From
                                                                • The Postquery Collection
                                                                • The Stack Is Getting Rich
                                                                • Representative Example (2)
                                                                • Quality of Service
                                                                • QOS In Action
                                                                • Where Do We Store QOS Values
                                                                • The QOS Collection
                                                                • Coming Togetherhellip
                                                                • Representative Example (3)
                                                                • A Highly Leveragable Investment
                                                                • Not Just Java Not A Problem
                                                                • A Really Nice Stack
                                                                • What Can We Do
                                                                • Secure Access To Redacted Data for Testing
                                                                • Get It Programmatically Too
                                                                • Robust Validated Data Ingest
                                                                • Concept The control_ Collection
                                                                • Validation QOS and PQO via Web Services
                                                                • Are We Excited Yet
                                                                • Questions amp Answers
                                                                • Slide 53
                                                                • Concept DataProvider
                                                                • The RESTful Provider

                                                                  33

                                                                  The Postquery Collection

                                                                  gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

                                                                  34

                                                                  The Stack Is Getting Rich

                                                                  PostQuery

                                                                  MongoDBJava Driver

                                                                  MQLValidator

                                                                  Data Access Layer

                                                                  Application

                                                                  ValidatorDBUtils

                                                                  PQLFilter

                                                                  PQODBUtils

                                                                  35

                                                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                                  someWriteOperation(Map data) hellip

                                                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                                  36

                                                                  Quality of Service

                                                                  37

                                                                  QOS In Action

                                                                  class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                                  Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                  38

                                                                  Where Do We Store QOS Values

                                                                  The Database

                                                                  39

                                                                  The QOS Collection

                                                                  gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                  ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                  40

                                                                  QOSDBUtils

                                                                  Coming Togetherhellip

                                                                  PostQuery

                                                                  MongoDBJava Driver

                                                                  MQLValidator

                                                                  Data Access Layer

                                                                  Application

                                                                  ValidatorDBUtils

                                                                  PQLFilter

                                                                  PQODBUtils

                                                                  QOS

                                                                  41

                                                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                  42

                                                                  QOSDBUtils

                                                                  A Highly Leveragable Investment

                                                                  PostQuery

                                                                  MQLValidator

                                                                  Data Access Layer 1

                                                                  Application1

                                                                  ValidatorDBUtils

                                                                  PQLFilter

                                                                  PQODBUtils

                                                                  QOS

                                                                  Application2

                                                                  Data Access Layer 2

                                                                  Application3

                                                                  Application4

                                                                  Data Access Layer 3

                                                                  Application5

                                                                  Application6Reusable For ALL Data Access Layer Logic

                                                                  43

                                                                  Not Just Java Not A Problem

                                                                  DAL operations have little or no statehellip

                                                                  Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                  Can you say hellip Web Service

                                                                  44

                                                                  A Really Nice Stack

                                                                  MongoDBJava Driver

                                                                  MQLValidator

                                                                  Data Access Layer

                                                                  JavaApplication

                                                                  ValidatorDBUtils

                                                                  PQLFilter

                                                                  HTTP Endpoint

                                                                  pythonApplication

                                                                  curl

                                                                  JSON over HTTP(S)

                                                                  JSONlt-gtJava Maps

                                                                  QOSDBUtils

                                                                  PostQuery

                                                                  PQODBUtils

                                                                  QOS

                                                                  45

                                                                  What Can We Do

                                                                  46

                                                                  Secure Access To Redacted Data for Testing

                                                                  $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                  $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                  $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                  47

                                                                  Get It Programmatically Too

                                                                  This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                  getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                  String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                  48

                                                                  Robust Validated Data Ingest

                                                                  $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                  $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                  49

                                                                  Concept The control_ Collection

                                                                  gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                  bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                  50

                                                                  Validation QOS and PQO via Web Services

                                                                  MongoDBJava Driver

                                                                  MQLValidator

                                                                  Data Access Layer

                                                                  JavaApplication

                                                                  ValidatorHTTPUtils

                                                                  PQLFilter

                                                                  pythonApplication

                                                                  curl

                                                                  JSON over HTTP(S)

                                                                  QOSHTTPUtils

                                                                  PostQuery

                                                                  PQOHTTPUtils

                                                                  QOS

                                                                  HTTP Service

                                                                  JSONlt-gtJava Maps

                                                                  51

                                                                  Are We Excited Yet

                                                                  Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                  and other Java resources

                                                                  Questions amp Answers

                                                                  54

                                                                  Concept DataProvider

                                                                  public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                  Class MongoProvider implements DataProvider hellip

                                                                  Class RESTfulProvider implements DataProvider hellip

                                                                  55

                                                                  The RESTful Provider

                                                                  class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                  fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                  url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                  HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                  • Slide 1
                                                                  • Part 3 In The Data Management Series
                                                                  • Congratulations At this Point Yoursquove
                                                                  • The Next Stage Defend amp Leverage
                                                                  • MongoDB Doesnrsquot Have These Things
                                                                  • What Can We Do Today
                                                                  • Write Some Code
                                                                  • Starting Point The Data Access Layer
                                                                  • Document Validation
                                                                  • A Query Filters Outbound Data
                                                                  • How About Using It To Filter Inbounds
                                                                  • $exists And $type Already in MQL
                                                                  • hellip And MQL Goes Way Beyondhellip
                                                                  • A New MQL Validator Module Emerges
                                                                  • Migrating Capability into MongoDB
                                                                  • Code For The FuturehellipToday
                                                                  • But What About Today
                                                                  • Temporary Filling PQL
                                                                  • Bridge MQL to PQL
                                                                  • No PQL No Problem
                                                                  • MQL Is Easy To Navigate
                                                                  • Where Do Validations Come From
                                                                  • The Validations Collection
                                                                  • Various ldquoLevelsrdquo of Validation
                                                                  • Multiple Types Schema By Example
                                                                  • The Stack So Far
                                                                  • Representative Example
                                                                  • Redaction
                                                                  • Concept Post Query Operations (PQO)
                                                                  • Adopt MQL-like behavior
                                                                  • A New PostQuery Module Emerges
                                                                  • Where Do PQOs Come From
                                                                  • The Postquery Collection
                                                                  • The Stack Is Getting Rich
                                                                  • Representative Example (2)
                                                                  • Quality of Service
                                                                  • QOS In Action
                                                                  • Where Do We Store QOS Values
                                                                  • The QOS Collection
                                                                  • Coming Togetherhellip
                                                                  • Representative Example (3)
                                                                  • A Highly Leveragable Investment
                                                                  • Not Just Java Not A Problem
                                                                  • A Really Nice Stack
                                                                  • What Can We Do
                                                                  • Secure Access To Redacted Data for Testing
                                                                  • Get It Programmatically Too
                                                                  • Robust Validated Data Ingest
                                                                  • Concept The control_ Collection
                                                                  • Validation QOS and PQO via Web Services
                                                                  • Are We Excited Yet
                                                                  • Questions amp Answers
                                                                  • Slide 53
                                                                  • Concept DataProvider
                                                                  • The RESTful Provider

                                                                    34

                                                                    The Stack Is Getting Rich

                                                                    PostQuery

                                                                    MongoDBJava Driver

                                                                    MQLValidator

                                                                    Data Access Layer

                                                                    Application

                                                                    ValidatorDBUtils

                                                                    PQLFilter

                                                                    PQODBUtils

                                                                    35

                                                                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                                    someWriteOperation(Map data) hellip

                                                                    someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                                    36

                                                                    Quality of Service

                                                                    37

                                                                    QOS In Action

                                                                    class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                                    Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                    38

                                                                    Where Do We Store QOS Values

                                                                    The Database

                                                                    39

                                                                    The QOS Collection

                                                                    gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                    ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                    40

                                                                    QOSDBUtils

                                                                    Coming Togetherhellip

                                                                    PostQuery

                                                                    MongoDBJava Driver

                                                                    MQLValidator

                                                                    Data Access Layer

                                                                    Application

                                                                    ValidatorDBUtils

                                                                    PQLFilter

                                                                    PQODBUtils

                                                                    QOS

                                                                    41

                                                                    Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                    init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                    someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                    42

                                                                    QOSDBUtils

                                                                    A Highly Leveragable Investment

                                                                    PostQuery

                                                                    MQLValidator

                                                                    Data Access Layer 1

                                                                    Application1

                                                                    ValidatorDBUtils

                                                                    PQLFilter

                                                                    PQODBUtils

                                                                    QOS

                                                                    Application2

                                                                    Data Access Layer 2

                                                                    Application3

                                                                    Application4

                                                                    Data Access Layer 3

                                                                    Application5

                                                                    Application6Reusable For ALL Data Access Layer Logic

                                                                    43

                                                                    Not Just Java Not A Problem

                                                                    DAL operations have little or no statehellip

                                                                    Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                    Can you say hellip Web Service

                                                                    44

                                                                    A Really Nice Stack

                                                                    MongoDBJava Driver

                                                                    MQLValidator

                                                                    Data Access Layer

                                                                    JavaApplication

                                                                    ValidatorDBUtils

                                                                    PQLFilter

                                                                    HTTP Endpoint

                                                                    pythonApplication

                                                                    curl

                                                                    JSON over HTTP(S)

                                                                    JSONlt-gtJava Maps

                                                                    QOSDBUtils

                                                                    PostQuery

                                                                    PQODBUtils

                                                                    QOS

                                                                    45

                                                                    What Can We Do

                                                                    46

                                                                    Secure Access To Redacted Data for Testing

                                                                    $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                    $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                    $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                    47

                                                                    Get It Programmatically Too

                                                                    This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                    getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                    String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                    48

                                                                    Robust Validated Data Ingest

                                                                    $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                    $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                    49

                                                                    Concept The control_ Collection

                                                                    gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                    bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                    50

                                                                    Validation QOS and PQO via Web Services

                                                                    MongoDBJava Driver

                                                                    MQLValidator

                                                                    Data Access Layer

                                                                    JavaApplication

                                                                    ValidatorHTTPUtils

                                                                    PQLFilter

                                                                    pythonApplication

                                                                    curl

                                                                    JSON over HTTP(S)

                                                                    QOSHTTPUtils

                                                                    PostQuery

                                                                    PQOHTTPUtils

                                                                    QOS

                                                                    HTTP Service

                                                                    JSONlt-gtJava Maps

                                                                    51

                                                                    Are We Excited Yet

                                                                    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                    and other Java resources

                                                                    Questions amp Answers

                                                                    54

                                                                    Concept DataProvider

                                                                    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                    Class MongoProvider implements DataProvider hellip

                                                                    Class RESTfulProvider implements DataProvider hellip

                                                                    55

                                                                    The RESTful Provider

                                                                    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                    • Slide 1
                                                                    • Part 3 In The Data Management Series
                                                                    • Congratulations At this Point Yoursquove
                                                                    • The Next Stage Defend amp Leverage
                                                                    • MongoDB Doesnrsquot Have These Things
                                                                    • What Can We Do Today
                                                                    • Write Some Code
                                                                    • Starting Point The Data Access Layer
                                                                    • Document Validation
                                                                    • A Query Filters Outbound Data
                                                                    • How About Using It To Filter Inbounds
                                                                    • $exists And $type Already in MQL
                                                                    • hellip And MQL Goes Way Beyondhellip
                                                                    • A New MQL Validator Module Emerges
                                                                    • Migrating Capability into MongoDB
                                                                    • Code For The FuturehellipToday
                                                                    • But What About Today
                                                                    • Temporary Filling PQL
                                                                    • Bridge MQL to PQL
                                                                    • No PQL No Problem
                                                                    • MQL Is Easy To Navigate
                                                                    • Where Do Validations Come From
                                                                    • The Validations Collection
                                                                    • Various ldquoLevelsrdquo of Validation
                                                                    • Multiple Types Schema By Example
                                                                    • The Stack So Far
                                                                    • Representative Example
                                                                    • Redaction
                                                                    • Concept Post Query Operations (PQO)
                                                                    • Adopt MQL-like behavior
                                                                    • A New PostQuery Module Emerges
                                                                    • Where Do PQOs Come From
                                                                    • The Postquery Collection
                                                                    • The Stack Is Getting Rich
                                                                    • Representative Example (2)
                                                                    • Quality of Service
                                                                    • QOS In Action
                                                                    • Where Do We Store QOS Values
                                                                    • The QOS Collection
                                                                    • Coming Togetherhellip
                                                                    • Representative Example (3)
                                                                    • A Highly Leveragable Investment
                                                                    • Not Just Java Not A Problem
                                                                    • A Really Nice Stack
                                                                    • What Can We Do
                                                                    • Secure Access To Redacted Data for Testing
                                                                    • Get It Programmatically Too
                                                                    • Robust Validated Data Ingest
                                                                    • Concept The control_ Collection
                                                                    • Validation QOS and PQO via Web Services
                                                                    • Are We Excited Yet
                                                                    • Questions amp Answers
                                                                    • Slide 53
                                                                    • Concept DataProvider
                                                                    • The RESTful Provider

                                                                      35

                                                                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

                                                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

                                                                      someWriteOperation(Map data) hellip

                                                                      someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

                                                                      36

                                                                      Quality of Service

                                                                      37

                                                                      QOS In Action

                                                                      class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                                      Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                      38

                                                                      Where Do We Store QOS Values

                                                                      The Database

                                                                      39

                                                                      The QOS Collection

                                                                      gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                      ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                      40

                                                                      QOSDBUtils

                                                                      Coming Togetherhellip

                                                                      PostQuery

                                                                      MongoDBJava Driver

                                                                      MQLValidator

                                                                      Data Access Layer

                                                                      Application

                                                                      ValidatorDBUtils

                                                                      PQLFilter

                                                                      PQODBUtils

                                                                      QOS

                                                                      41

                                                                      Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                      init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                      someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                      42

                                                                      QOSDBUtils

                                                                      A Highly Leveragable Investment

                                                                      PostQuery

                                                                      MQLValidator

                                                                      Data Access Layer 1

                                                                      Application1

                                                                      ValidatorDBUtils

                                                                      PQLFilter

                                                                      PQODBUtils

                                                                      QOS

                                                                      Application2

                                                                      Data Access Layer 2

                                                                      Application3

                                                                      Application4

                                                                      Data Access Layer 3

                                                                      Application5

                                                                      Application6Reusable For ALL Data Access Layer Logic

                                                                      43

                                                                      Not Just Java Not A Problem

                                                                      DAL operations have little or no statehellip

                                                                      Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                      Can you say hellip Web Service

                                                                      44

                                                                      A Really Nice Stack

                                                                      MongoDBJava Driver

                                                                      MQLValidator

                                                                      Data Access Layer

                                                                      JavaApplication

                                                                      ValidatorDBUtils

                                                                      PQLFilter

                                                                      HTTP Endpoint

                                                                      pythonApplication

                                                                      curl

                                                                      JSON over HTTP(S)

                                                                      JSONlt-gtJava Maps

                                                                      QOSDBUtils

                                                                      PostQuery

                                                                      PQODBUtils

                                                                      QOS

                                                                      45

                                                                      What Can We Do

                                                                      46

                                                                      Secure Access To Redacted Data for Testing

                                                                      $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                      $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                      $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                      47

                                                                      Get It Programmatically Too

                                                                      This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                      getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                      String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                      48

                                                                      Robust Validated Data Ingest

                                                                      $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                      $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                      49

                                                                      Concept The control_ Collection

                                                                      gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                      bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                      50

                                                                      Validation QOS and PQO via Web Services

                                                                      MongoDBJava Driver

                                                                      MQLValidator

                                                                      Data Access Layer

                                                                      JavaApplication

                                                                      ValidatorHTTPUtils

                                                                      PQLFilter

                                                                      pythonApplication

                                                                      curl

                                                                      JSON over HTTP(S)

                                                                      QOSHTTPUtils

                                                                      PostQuery

                                                                      PQOHTTPUtils

                                                                      QOS

                                                                      HTTP Service

                                                                      JSONlt-gtJava Maps

                                                                      51

                                                                      Are We Excited Yet

                                                                      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                      and other Java resources

                                                                      Questions amp Answers

                                                                      54

                                                                      Concept DataProvider

                                                                      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                      Class MongoProvider implements DataProvider hellip

                                                                      Class RESTfulProvider implements DataProvider hellip

                                                                      55

                                                                      The RESTful Provider

                                                                      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                      • Slide 1
                                                                      • Part 3 In The Data Management Series
                                                                      • Congratulations At this Point Yoursquove
                                                                      • The Next Stage Defend amp Leverage
                                                                      • MongoDB Doesnrsquot Have These Things
                                                                      • What Can We Do Today
                                                                      • Write Some Code
                                                                      • Starting Point The Data Access Layer
                                                                      • Document Validation
                                                                      • A Query Filters Outbound Data
                                                                      • How About Using It To Filter Inbounds
                                                                      • $exists And $type Already in MQL
                                                                      • hellip And MQL Goes Way Beyondhellip
                                                                      • A New MQL Validator Module Emerges
                                                                      • Migrating Capability into MongoDB
                                                                      • Code For The FuturehellipToday
                                                                      • But What About Today
                                                                      • Temporary Filling PQL
                                                                      • Bridge MQL to PQL
                                                                      • No PQL No Problem
                                                                      • MQL Is Easy To Navigate
                                                                      • Where Do Validations Come From
                                                                      • The Validations Collection
                                                                      • Various ldquoLevelsrdquo of Validation
                                                                      • Multiple Types Schema By Example
                                                                      • The Stack So Far
                                                                      • Representative Example
                                                                      • Redaction
                                                                      • Concept Post Query Operations (PQO)
                                                                      • Adopt MQL-like behavior
                                                                      • A New PostQuery Module Emerges
                                                                      • Where Do PQOs Come From
                                                                      • The Postquery Collection
                                                                      • The Stack Is Getting Rich
                                                                      • Representative Example (2)
                                                                      • Quality of Service
                                                                      • QOS In Action
                                                                      • Where Do We Store QOS Values
                                                                      • The QOS Collection
                                                                      • Coming Togetherhellip
                                                                      • Representative Example (3)
                                                                      • A Highly Leveragable Investment
                                                                      • Not Just Java Not A Problem
                                                                      • A Really Nice Stack
                                                                      • What Can We Do
                                                                      • Secure Access To Redacted Data for Testing
                                                                      • Get It Programmatically Too
                                                                      • Robust Validated Data Ingest
                                                                      • Concept The control_ Collection
                                                                      • Validation QOS and PQO via Web Services
                                                                      • Are We Excited Yet
                                                                      • Questions amp Answers
                                                                      • Slide 53
                                                                      • Concept DataProvider
                                                                      • The RESTful Provider

                                                                        36

                                                                        Quality of Service

                                                                        37

                                                                        QOS In Action

                                                                        class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                                        Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                        38

                                                                        Where Do We Store QOS Values

                                                                        The Database

                                                                        39

                                                                        The QOS Collection

                                                                        gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                        ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                        40

                                                                        QOSDBUtils

                                                                        Coming Togetherhellip

                                                                        PostQuery

                                                                        MongoDBJava Driver

                                                                        MQLValidator

                                                                        Data Access Layer

                                                                        Application

                                                                        ValidatorDBUtils

                                                                        PQLFilter

                                                                        PQODBUtils

                                                                        QOS

                                                                        41

                                                                        Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                        init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                        someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                        42

                                                                        QOSDBUtils

                                                                        A Highly Leveragable Investment

                                                                        PostQuery

                                                                        MQLValidator

                                                                        Data Access Layer 1

                                                                        Application1

                                                                        ValidatorDBUtils

                                                                        PQLFilter

                                                                        PQODBUtils

                                                                        QOS

                                                                        Application2

                                                                        Data Access Layer 2

                                                                        Application3

                                                                        Application4

                                                                        Data Access Layer 3

                                                                        Application5

                                                                        Application6Reusable For ALL Data Access Layer Logic

                                                                        43

                                                                        Not Just Java Not A Problem

                                                                        DAL operations have little or no statehellip

                                                                        Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                        Can you say hellip Web Service

                                                                        44

                                                                        A Really Nice Stack

                                                                        MongoDBJava Driver

                                                                        MQLValidator

                                                                        Data Access Layer

                                                                        JavaApplication

                                                                        ValidatorDBUtils

                                                                        PQLFilter

                                                                        HTTP Endpoint

                                                                        pythonApplication

                                                                        curl

                                                                        JSON over HTTP(S)

                                                                        JSONlt-gtJava Maps

                                                                        QOSDBUtils

                                                                        PostQuery

                                                                        PQODBUtils

                                                                        QOS

                                                                        45

                                                                        What Can We Do

                                                                        46

                                                                        Secure Access To Redacted Data for Testing

                                                                        $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                        $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                        $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                        47

                                                                        Get It Programmatically Too

                                                                        This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                        getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                        String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                        48

                                                                        Robust Validated Data Ingest

                                                                        $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                        $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                        49

                                                                        Concept The control_ Collection

                                                                        gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                        bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                        50

                                                                        Validation QOS and PQO via Web Services

                                                                        MongoDBJava Driver

                                                                        MQLValidator

                                                                        Data Access Layer

                                                                        JavaApplication

                                                                        ValidatorHTTPUtils

                                                                        PQLFilter

                                                                        pythonApplication

                                                                        curl

                                                                        JSON over HTTP(S)

                                                                        QOSHTTPUtils

                                                                        PostQuery

                                                                        PQOHTTPUtils

                                                                        QOS

                                                                        HTTP Service

                                                                        JSONlt-gtJava Maps

                                                                        51

                                                                        Are We Excited Yet

                                                                        Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                        and other Java resources

                                                                        Questions amp Answers

                                                                        54

                                                                        Concept DataProvider

                                                                        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                        Class MongoProvider implements DataProvider hellip

                                                                        Class RESTfulProvider implements DataProvider hellip

                                                                        55

                                                                        The RESTful Provider

                                                                        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                        • Slide 1
                                                                        • Part 3 In The Data Management Series
                                                                        • Congratulations At this Point Yoursquove
                                                                        • The Next Stage Defend amp Leverage
                                                                        • MongoDB Doesnrsquot Have These Things
                                                                        • What Can We Do Today
                                                                        • Write Some Code
                                                                        • Starting Point The Data Access Layer
                                                                        • Document Validation
                                                                        • A Query Filters Outbound Data
                                                                        • How About Using It To Filter Inbounds
                                                                        • $exists And $type Already in MQL
                                                                        • hellip And MQL Goes Way Beyondhellip
                                                                        • A New MQL Validator Module Emerges
                                                                        • Migrating Capability into MongoDB
                                                                        • Code For The FuturehellipToday
                                                                        • But What About Today
                                                                        • Temporary Filling PQL
                                                                        • Bridge MQL to PQL
                                                                        • No PQL No Problem
                                                                        • MQL Is Easy To Navigate
                                                                        • Where Do Validations Come From
                                                                        • The Validations Collection
                                                                        • Various ldquoLevelsrdquo of Validation
                                                                        • Multiple Types Schema By Example
                                                                        • The Stack So Far
                                                                        • Representative Example
                                                                        • Redaction
                                                                        • Concept Post Query Operations (PQO)
                                                                        • Adopt MQL-like behavior
                                                                        • A New PostQuery Module Emerges
                                                                        • Where Do PQOs Come From
                                                                        • The Postquery Collection
                                                                        • The Stack Is Getting Rich
                                                                        • Representative Example (2)
                                                                        • Quality of Service
                                                                        • QOS In Action
                                                                        • Where Do We Store QOS Values
                                                                        • The QOS Collection
                                                                        • Coming Togetherhellip
                                                                        • Representative Example (3)
                                                                        • A Highly Leveragable Investment
                                                                        • Not Just Java Not A Problem
                                                                        • A Really Nice Stack
                                                                        • What Can We Do
                                                                        • Secure Access To Redacted Data for Testing
                                                                        • Get It Programmatically Too
                                                                        • Robust Validated Data Ingest
                                                                        • Concept The control_ Collection
                                                                        • Validation QOS and PQO via Web Services
                                                                        • Are We Excited Yet
                                                                        • Questions amp Answers
                                                                        • Slide 53
                                                                        • Concept DataProvider
                                                                        • The RESTful Provider

                                                                          37

                                                                          QOS In Action

                                                                          class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

                                                                          Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                          38

                                                                          Where Do We Store QOS Values

                                                                          The Database

                                                                          39

                                                                          The QOS Collection

                                                                          gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                          ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                          40

                                                                          QOSDBUtils

                                                                          Coming Togetherhellip

                                                                          PostQuery

                                                                          MongoDBJava Driver

                                                                          MQLValidator

                                                                          Data Access Layer

                                                                          Application

                                                                          ValidatorDBUtils

                                                                          PQLFilter

                                                                          PQODBUtils

                                                                          QOS

                                                                          41

                                                                          Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                          init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                          someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                          42

                                                                          QOSDBUtils

                                                                          A Highly Leveragable Investment

                                                                          PostQuery

                                                                          MQLValidator

                                                                          Data Access Layer 1

                                                                          Application1

                                                                          ValidatorDBUtils

                                                                          PQLFilter

                                                                          PQODBUtils

                                                                          QOS

                                                                          Application2

                                                                          Data Access Layer 2

                                                                          Application3

                                                                          Application4

                                                                          Data Access Layer 3

                                                                          Application5

                                                                          Application6Reusable For ALL Data Access Layer Logic

                                                                          43

                                                                          Not Just Java Not A Problem

                                                                          DAL operations have little or no statehellip

                                                                          Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                          Can you say hellip Web Service

                                                                          44

                                                                          A Really Nice Stack

                                                                          MongoDBJava Driver

                                                                          MQLValidator

                                                                          Data Access Layer

                                                                          JavaApplication

                                                                          ValidatorDBUtils

                                                                          PQLFilter

                                                                          HTTP Endpoint

                                                                          pythonApplication

                                                                          curl

                                                                          JSON over HTTP(S)

                                                                          JSONlt-gtJava Maps

                                                                          QOSDBUtils

                                                                          PostQuery

                                                                          PQODBUtils

                                                                          QOS

                                                                          45

                                                                          What Can We Do

                                                                          46

                                                                          Secure Access To Redacted Data for Testing

                                                                          $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                          $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                          $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                          47

                                                                          Get It Programmatically Too

                                                                          This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                          getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                          String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                          48

                                                                          Robust Validated Data Ingest

                                                                          $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                          $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                          49

                                                                          Concept The control_ Collection

                                                                          gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                          bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                          50

                                                                          Validation QOS and PQO via Web Services

                                                                          MongoDBJava Driver

                                                                          MQLValidator

                                                                          Data Access Layer

                                                                          JavaApplication

                                                                          ValidatorHTTPUtils

                                                                          PQLFilter

                                                                          pythonApplication

                                                                          curl

                                                                          JSON over HTTP(S)

                                                                          QOSHTTPUtils

                                                                          PostQuery

                                                                          PQOHTTPUtils

                                                                          QOS

                                                                          HTTP Service

                                                                          JSONlt-gtJava Maps

                                                                          51

                                                                          Are We Excited Yet

                                                                          Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                          and other Java resources

                                                                          Questions amp Answers

                                                                          54

                                                                          Concept DataProvider

                                                                          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                          Class MongoProvider implements DataProvider hellip

                                                                          Class RESTfulProvider implements DataProvider hellip

                                                                          55

                                                                          The RESTful Provider

                                                                          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                          • Slide 1
                                                                          • Part 3 In The Data Management Series
                                                                          • Congratulations At this Point Yoursquove
                                                                          • The Next Stage Defend amp Leverage
                                                                          • MongoDB Doesnrsquot Have These Things
                                                                          • What Can We Do Today
                                                                          • Write Some Code
                                                                          • Starting Point The Data Access Layer
                                                                          • Document Validation
                                                                          • A Query Filters Outbound Data
                                                                          • How About Using It To Filter Inbounds
                                                                          • $exists And $type Already in MQL
                                                                          • hellip And MQL Goes Way Beyondhellip
                                                                          • A New MQL Validator Module Emerges
                                                                          • Migrating Capability into MongoDB
                                                                          • Code For The FuturehellipToday
                                                                          • But What About Today
                                                                          • Temporary Filling PQL
                                                                          • Bridge MQL to PQL
                                                                          • No PQL No Problem
                                                                          • MQL Is Easy To Navigate
                                                                          • Where Do Validations Come From
                                                                          • The Validations Collection
                                                                          • Various ldquoLevelsrdquo of Validation
                                                                          • Multiple Types Schema By Example
                                                                          • The Stack So Far
                                                                          • Representative Example
                                                                          • Redaction
                                                                          • Concept Post Query Operations (PQO)
                                                                          • Adopt MQL-like behavior
                                                                          • A New PostQuery Module Emerges
                                                                          • Where Do PQOs Come From
                                                                          • The Postquery Collection
                                                                          • The Stack Is Getting Rich
                                                                          • Representative Example (2)
                                                                          • Quality of Service
                                                                          • QOS In Action
                                                                          • Where Do We Store QOS Values
                                                                          • The QOS Collection
                                                                          • Coming Togetherhellip
                                                                          • Representative Example (3)
                                                                          • A Highly Leveragable Investment
                                                                          • Not Just Java Not A Problem
                                                                          • A Really Nice Stack
                                                                          • What Can We Do
                                                                          • Secure Access To Redacted Data for Testing
                                                                          • Get It Programmatically Too
                                                                          • Robust Validated Data Ingest
                                                                          • Concept The control_ Collection
                                                                          • Validation QOS and PQO via Web Services
                                                                          • Are We Excited Yet
                                                                          • Questions amp Answers
                                                                          • Slide 53
                                                                          • Concept DataProvider
                                                                          • The RESTful Provider

                                                                            38

                                                                            Where Do We Store QOS Values

                                                                            The Database

                                                                            39

                                                                            The QOS Collection

                                                                            gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                            ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                            40

                                                                            QOSDBUtils

                                                                            Coming Togetherhellip

                                                                            PostQuery

                                                                            MongoDBJava Driver

                                                                            MQLValidator

                                                                            Data Access Layer

                                                                            Application

                                                                            ValidatorDBUtils

                                                                            PQLFilter

                                                                            PQODBUtils

                                                                            QOS

                                                                            41

                                                                            Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                            init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                            someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                            42

                                                                            QOSDBUtils

                                                                            A Highly Leveragable Investment

                                                                            PostQuery

                                                                            MQLValidator

                                                                            Data Access Layer 1

                                                                            Application1

                                                                            ValidatorDBUtils

                                                                            PQLFilter

                                                                            PQODBUtils

                                                                            QOS

                                                                            Application2

                                                                            Data Access Layer 2

                                                                            Application3

                                                                            Application4

                                                                            Data Access Layer 3

                                                                            Application5

                                                                            Application6Reusable For ALL Data Access Layer Logic

                                                                            43

                                                                            Not Just Java Not A Problem

                                                                            DAL operations have little or no statehellip

                                                                            Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                            Can you say hellip Web Service

                                                                            44

                                                                            A Really Nice Stack

                                                                            MongoDBJava Driver

                                                                            MQLValidator

                                                                            Data Access Layer

                                                                            JavaApplication

                                                                            ValidatorDBUtils

                                                                            PQLFilter

                                                                            HTTP Endpoint

                                                                            pythonApplication

                                                                            curl

                                                                            JSON over HTTP(S)

                                                                            JSONlt-gtJava Maps

                                                                            QOSDBUtils

                                                                            PostQuery

                                                                            PQODBUtils

                                                                            QOS

                                                                            45

                                                                            What Can We Do

                                                                            46

                                                                            Secure Access To Redacted Data for Testing

                                                                            $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                            $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                            $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                            47

                                                                            Get It Programmatically Too

                                                                            This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                            getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                            String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                            48

                                                                            Robust Validated Data Ingest

                                                                            $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                            $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                            49

                                                                            Concept The control_ Collection

                                                                            gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                            bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                            50

                                                                            Validation QOS and PQO via Web Services

                                                                            MongoDBJava Driver

                                                                            MQLValidator

                                                                            Data Access Layer

                                                                            JavaApplication

                                                                            ValidatorHTTPUtils

                                                                            PQLFilter

                                                                            pythonApplication

                                                                            curl

                                                                            JSON over HTTP(S)

                                                                            QOSHTTPUtils

                                                                            PostQuery

                                                                            PQOHTTPUtils

                                                                            QOS

                                                                            HTTP Service

                                                                            JSONlt-gtJava Maps

                                                                            51

                                                                            Are We Excited Yet

                                                                            Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                            and other Java resources

                                                                            Questions amp Answers

                                                                            54

                                                                            Concept DataProvider

                                                                            public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                            Class MongoProvider implements DataProvider hellip

                                                                            Class RESTfulProvider implements DataProvider hellip

                                                                            55

                                                                            The RESTful Provider

                                                                            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                            • Slide 1
                                                                            • Part 3 In The Data Management Series
                                                                            • Congratulations At this Point Yoursquove
                                                                            • The Next Stage Defend amp Leverage
                                                                            • MongoDB Doesnrsquot Have These Things
                                                                            • What Can We Do Today
                                                                            • Write Some Code
                                                                            • Starting Point The Data Access Layer
                                                                            • Document Validation
                                                                            • A Query Filters Outbound Data
                                                                            • How About Using It To Filter Inbounds
                                                                            • $exists And $type Already in MQL
                                                                            • hellip And MQL Goes Way Beyondhellip
                                                                            • A New MQL Validator Module Emerges
                                                                            • Migrating Capability into MongoDB
                                                                            • Code For The FuturehellipToday
                                                                            • But What About Today
                                                                            • Temporary Filling PQL
                                                                            • Bridge MQL to PQL
                                                                            • No PQL No Problem
                                                                            • MQL Is Easy To Navigate
                                                                            • Where Do Validations Come From
                                                                            • The Validations Collection
                                                                            • Various ldquoLevelsrdquo of Validation
                                                                            • Multiple Types Schema By Example
                                                                            • The Stack So Far
                                                                            • Representative Example
                                                                            • Redaction
                                                                            • Concept Post Query Operations (PQO)
                                                                            • Adopt MQL-like behavior
                                                                            • A New PostQuery Module Emerges
                                                                            • Where Do PQOs Come From
                                                                            • The Postquery Collection
                                                                            • The Stack Is Getting Rich
                                                                            • Representative Example (2)
                                                                            • Quality of Service
                                                                            • QOS In Action
                                                                            • Where Do We Store QOS Values
                                                                            • The QOS Collection
                                                                            • Coming Togetherhellip
                                                                            • Representative Example (3)
                                                                            • A Highly Leveragable Investment
                                                                            • Not Just Java Not A Problem
                                                                            • A Really Nice Stack
                                                                            • What Can We Do
                                                                            • Secure Access To Redacted Data for Testing
                                                                            • Get It Programmatically Too
                                                                            • Robust Validated Data Ingest
                                                                            • Concept The control_ Collection
                                                                            • Validation QOS and PQO via Web Services
                                                                            • Are We Excited Yet
                                                                            • Questions amp Answers
                                                                            • Slide 53
                                                                            • Concept DataProvider
                                                                            • The RESTful Provider

                                                                              39

                                                                              The QOS Collection

                                                                              gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

                                                                              ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

                                                                              40

                                                                              QOSDBUtils

                                                                              Coming Togetherhellip

                                                                              PostQuery

                                                                              MongoDBJava Driver

                                                                              MQLValidator

                                                                              Data Access Layer

                                                                              Application

                                                                              ValidatorDBUtils

                                                                              PQLFilter

                                                                              PQODBUtils

                                                                              QOS

                                                                              41

                                                                              Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                              init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                              someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                              42

                                                                              QOSDBUtils

                                                                              A Highly Leveragable Investment

                                                                              PostQuery

                                                                              MQLValidator

                                                                              Data Access Layer 1

                                                                              Application1

                                                                              ValidatorDBUtils

                                                                              PQLFilter

                                                                              PQODBUtils

                                                                              QOS

                                                                              Application2

                                                                              Data Access Layer 2

                                                                              Application3

                                                                              Application4

                                                                              Data Access Layer 3

                                                                              Application5

                                                                              Application6Reusable For ALL Data Access Layer Logic

                                                                              43

                                                                              Not Just Java Not A Problem

                                                                              DAL operations have little or no statehellip

                                                                              Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                              Can you say hellip Web Service

                                                                              44

                                                                              A Really Nice Stack

                                                                              MongoDBJava Driver

                                                                              MQLValidator

                                                                              Data Access Layer

                                                                              JavaApplication

                                                                              ValidatorDBUtils

                                                                              PQLFilter

                                                                              HTTP Endpoint

                                                                              pythonApplication

                                                                              curl

                                                                              JSON over HTTP(S)

                                                                              JSONlt-gtJava Maps

                                                                              QOSDBUtils

                                                                              PostQuery

                                                                              PQODBUtils

                                                                              QOS

                                                                              45

                                                                              What Can We Do

                                                                              46

                                                                              Secure Access To Redacted Data for Testing

                                                                              $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                              $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                              $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                              47

                                                                              Get It Programmatically Too

                                                                              This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                              getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                              String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                              48

                                                                              Robust Validated Data Ingest

                                                                              $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                              $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                              49

                                                                              Concept The control_ Collection

                                                                              gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                              bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                              50

                                                                              Validation QOS and PQO via Web Services

                                                                              MongoDBJava Driver

                                                                              MQLValidator

                                                                              Data Access Layer

                                                                              JavaApplication

                                                                              ValidatorHTTPUtils

                                                                              PQLFilter

                                                                              pythonApplication

                                                                              curl

                                                                              JSON over HTTP(S)

                                                                              QOSHTTPUtils

                                                                              PostQuery

                                                                              PQOHTTPUtils

                                                                              QOS

                                                                              HTTP Service

                                                                              JSONlt-gtJava Maps

                                                                              51

                                                                              Are We Excited Yet

                                                                              Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                              and other Java resources

                                                                              Questions amp Answers

                                                                              54

                                                                              Concept DataProvider

                                                                              public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                              Class MongoProvider implements DataProvider hellip

                                                                              Class RESTfulProvider implements DataProvider hellip

                                                                              55

                                                                              The RESTful Provider

                                                                              class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                              fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                              url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                              HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                              • Slide 1
                                                                              • Part 3 In The Data Management Series
                                                                              • Congratulations At this Point Yoursquove
                                                                              • The Next Stage Defend amp Leverage
                                                                              • MongoDB Doesnrsquot Have These Things
                                                                              • What Can We Do Today
                                                                              • Write Some Code
                                                                              • Starting Point The Data Access Layer
                                                                              • Document Validation
                                                                              • A Query Filters Outbound Data
                                                                              • How About Using It To Filter Inbounds
                                                                              • $exists And $type Already in MQL
                                                                              • hellip And MQL Goes Way Beyondhellip
                                                                              • A New MQL Validator Module Emerges
                                                                              • Migrating Capability into MongoDB
                                                                              • Code For The FuturehellipToday
                                                                              • But What About Today
                                                                              • Temporary Filling PQL
                                                                              • Bridge MQL to PQL
                                                                              • No PQL No Problem
                                                                              • MQL Is Easy To Navigate
                                                                              • Where Do Validations Come From
                                                                              • The Validations Collection
                                                                              • Various ldquoLevelsrdquo of Validation
                                                                              • Multiple Types Schema By Example
                                                                              • The Stack So Far
                                                                              • Representative Example
                                                                              • Redaction
                                                                              • Concept Post Query Operations (PQO)
                                                                              • Adopt MQL-like behavior
                                                                              • A New PostQuery Module Emerges
                                                                              • Where Do PQOs Come From
                                                                              • The Postquery Collection
                                                                              • The Stack Is Getting Rich
                                                                              • Representative Example (2)
                                                                              • Quality of Service
                                                                              • QOS In Action
                                                                              • Where Do We Store QOS Values
                                                                              • The QOS Collection
                                                                              • Coming Togetherhellip
                                                                              • Representative Example (3)
                                                                              • A Highly Leveragable Investment
                                                                              • Not Just Java Not A Problem
                                                                              • A Really Nice Stack
                                                                              • What Can We Do
                                                                              • Secure Access To Redacted Data for Testing
                                                                              • Get It Programmatically Too
                                                                              • Robust Validated Data Ingest
                                                                              • Concept The control_ Collection
                                                                              • Validation QOS and PQO via Web Services
                                                                              • Are We Excited Yet
                                                                              • Questions amp Answers
                                                                              • Slide 53
                                                                              • Concept DataProvider
                                                                              • The RESTful Provider

                                                                                40

                                                                                QOSDBUtils

                                                                                Coming Togetherhellip

                                                                                PostQuery

                                                                                MongoDBJava Driver

                                                                                MQLValidator

                                                                                Data Access Layer

                                                                                Application

                                                                                ValidatorDBUtils

                                                                                PQLFilter

                                                                                PQODBUtils

                                                                                QOS

                                                                                41

                                                                                Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                                init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                                someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                                42

                                                                                QOSDBUtils

                                                                                A Highly Leveragable Investment

                                                                                PostQuery

                                                                                MQLValidator

                                                                                Data Access Layer 1

                                                                                Application1

                                                                                ValidatorDBUtils

                                                                                PQLFilter

                                                                                PQODBUtils

                                                                                QOS

                                                                                Application2

                                                                                Data Access Layer 2

                                                                                Application3

                                                                                Application4

                                                                                Data Access Layer 3

                                                                                Application5

                                                                                Application6Reusable For ALL Data Access Layer Logic

                                                                                43

                                                                                Not Just Java Not A Problem

                                                                                DAL operations have little or no statehellip

                                                                                Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                                Can you say hellip Web Service

                                                                                44

                                                                                A Really Nice Stack

                                                                                MongoDBJava Driver

                                                                                MQLValidator

                                                                                Data Access Layer

                                                                                JavaApplication

                                                                                ValidatorDBUtils

                                                                                PQLFilter

                                                                                HTTP Endpoint

                                                                                pythonApplication

                                                                                curl

                                                                                JSON over HTTP(S)

                                                                                JSONlt-gtJava Maps

                                                                                QOSDBUtils

                                                                                PostQuery

                                                                                PQODBUtils

                                                                                QOS

                                                                                45

                                                                                What Can We Do

                                                                                46

                                                                                Secure Access To Redacted Data for Testing

                                                                                $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                47

                                                                                Get It Programmatically Too

                                                                                This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                48

                                                                                Robust Validated Data Ingest

                                                                                $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                49

                                                                                Concept The control_ Collection

                                                                                gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                50

                                                                                Validation QOS and PQO via Web Services

                                                                                MongoDBJava Driver

                                                                                MQLValidator

                                                                                Data Access Layer

                                                                                JavaApplication

                                                                                ValidatorHTTPUtils

                                                                                PQLFilter

                                                                                pythonApplication

                                                                                curl

                                                                                JSON over HTTP(S)

                                                                                QOSHTTPUtils

                                                                                PostQuery

                                                                                PQOHTTPUtils

                                                                                QOS

                                                                                HTTP Service

                                                                                JSONlt-gtJava Maps

                                                                                51

                                                                                Are We Excited Yet

                                                                                Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                and other Java resources

                                                                                Questions amp Answers

                                                                                54

                                                                                Concept DataProvider

                                                                                public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                Class MongoProvider implements DataProvider hellip

                                                                                Class RESTfulProvider implements DataProvider hellip

                                                                                55

                                                                                The RESTful Provider

                                                                                class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                • Slide 1
                                                                                • Part 3 In The Data Management Series
                                                                                • Congratulations At this Point Yoursquove
                                                                                • The Next Stage Defend amp Leverage
                                                                                • MongoDB Doesnrsquot Have These Things
                                                                                • What Can We Do Today
                                                                                • Write Some Code
                                                                                • Starting Point The Data Access Layer
                                                                                • Document Validation
                                                                                • A Query Filters Outbound Data
                                                                                • How About Using It To Filter Inbounds
                                                                                • $exists And $type Already in MQL
                                                                                • hellip And MQL Goes Way Beyondhellip
                                                                                • A New MQL Validator Module Emerges
                                                                                • Migrating Capability into MongoDB
                                                                                • Code For The FuturehellipToday
                                                                                • But What About Today
                                                                                • Temporary Filling PQL
                                                                                • Bridge MQL to PQL
                                                                                • No PQL No Problem
                                                                                • MQL Is Easy To Navigate
                                                                                • Where Do Validations Come From
                                                                                • The Validations Collection
                                                                                • Various ldquoLevelsrdquo of Validation
                                                                                • Multiple Types Schema By Example
                                                                                • The Stack So Far
                                                                                • Representative Example
                                                                                • Redaction
                                                                                • Concept Post Query Operations (PQO)
                                                                                • Adopt MQL-like behavior
                                                                                • A New PostQuery Module Emerges
                                                                                • Where Do PQOs Come From
                                                                                • The Postquery Collection
                                                                                • The Stack Is Getting Rich
                                                                                • Representative Example (2)
                                                                                • Quality of Service
                                                                                • QOS In Action
                                                                                • Where Do We Store QOS Values
                                                                                • The QOS Collection
                                                                                • Coming Togetherhellip
                                                                                • Representative Example (3)
                                                                                • A Highly Leveragable Investment
                                                                                • Not Just Java Not A Problem
                                                                                • A Really Nice Stack
                                                                                • What Can We Do
                                                                                • Secure Access To Redacted Data for Testing
                                                                                • Get It Programmatically Too
                                                                                • Robust Validated Data Ingest
                                                                                • Concept The control_ Collection
                                                                                • Validation QOS and PQO via Web Services
                                                                                • Are We Excited Yet
                                                                                • Questions amp Answers
                                                                                • Slide 53
                                                                                • Concept DataProvider
                                                                                • The RESTful Provider

                                                                                  41

                                                                                  Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

                                                                                  init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

                                                                                  someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

                                                                                  42

                                                                                  QOSDBUtils

                                                                                  A Highly Leveragable Investment

                                                                                  PostQuery

                                                                                  MQLValidator

                                                                                  Data Access Layer 1

                                                                                  Application1

                                                                                  ValidatorDBUtils

                                                                                  PQLFilter

                                                                                  PQODBUtils

                                                                                  QOS

                                                                                  Application2

                                                                                  Data Access Layer 2

                                                                                  Application3

                                                                                  Application4

                                                                                  Data Access Layer 3

                                                                                  Application5

                                                                                  Application6Reusable For ALL Data Access Layer Logic

                                                                                  43

                                                                                  Not Just Java Not A Problem

                                                                                  DAL operations have little or no statehellip

                                                                                  Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                                  Can you say hellip Web Service

                                                                                  44

                                                                                  A Really Nice Stack

                                                                                  MongoDBJava Driver

                                                                                  MQLValidator

                                                                                  Data Access Layer

                                                                                  JavaApplication

                                                                                  ValidatorDBUtils

                                                                                  PQLFilter

                                                                                  HTTP Endpoint

                                                                                  pythonApplication

                                                                                  curl

                                                                                  JSON over HTTP(S)

                                                                                  JSONlt-gtJava Maps

                                                                                  QOSDBUtils

                                                                                  PostQuery

                                                                                  PQODBUtils

                                                                                  QOS

                                                                                  45

                                                                                  What Can We Do

                                                                                  46

                                                                                  Secure Access To Redacted Data for Testing

                                                                                  $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                  $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                  $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                  47

                                                                                  Get It Programmatically Too

                                                                                  This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                  getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                  String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                  48

                                                                                  Robust Validated Data Ingest

                                                                                  $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                  $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                  49

                                                                                  Concept The control_ Collection

                                                                                  gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                  bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                  50

                                                                                  Validation QOS and PQO via Web Services

                                                                                  MongoDBJava Driver

                                                                                  MQLValidator

                                                                                  Data Access Layer

                                                                                  JavaApplication

                                                                                  ValidatorHTTPUtils

                                                                                  PQLFilter

                                                                                  pythonApplication

                                                                                  curl

                                                                                  JSON over HTTP(S)

                                                                                  QOSHTTPUtils

                                                                                  PostQuery

                                                                                  PQOHTTPUtils

                                                                                  QOS

                                                                                  HTTP Service

                                                                                  JSONlt-gtJava Maps

                                                                                  51

                                                                                  Are We Excited Yet

                                                                                  Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                  and other Java resources

                                                                                  Questions amp Answers

                                                                                  54

                                                                                  Concept DataProvider

                                                                                  public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                  Class MongoProvider implements DataProvider hellip

                                                                                  Class RESTfulProvider implements DataProvider hellip

                                                                                  55

                                                                                  The RESTful Provider

                                                                                  class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                  fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                  url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                  HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                  • Slide 1
                                                                                  • Part 3 In The Data Management Series
                                                                                  • Congratulations At this Point Yoursquove
                                                                                  • The Next Stage Defend amp Leverage
                                                                                  • MongoDB Doesnrsquot Have These Things
                                                                                  • What Can We Do Today
                                                                                  • Write Some Code
                                                                                  • Starting Point The Data Access Layer
                                                                                  • Document Validation
                                                                                  • A Query Filters Outbound Data
                                                                                  • How About Using It To Filter Inbounds
                                                                                  • $exists And $type Already in MQL
                                                                                  • hellip And MQL Goes Way Beyondhellip
                                                                                  • A New MQL Validator Module Emerges
                                                                                  • Migrating Capability into MongoDB
                                                                                  • Code For The FuturehellipToday
                                                                                  • But What About Today
                                                                                  • Temporary Filling PQL
                                                                                  • Bridge MQL to PQL
                                                                                  • No PQL No Problem
                                                                                  • MQL Is Easy To Navigate
                                                                                  • Where Do Validations Come From
                                                                                  • The Validations Collection
                                                                                  • Various ldquoLevelsrdquo of Validation
                                                                                  • Multiple Types Schema By Example
                                                                                  • The Stack So Far
                                                                                  • Representative Example
                                                                                  • Redaction
                                                                                  • Concept Post Query Operations (PQO)
                                                                                  • Adopt MQL-like behavior
                                                                                  • A New PostQuery Module Emerges
                                                                                  • Where Do PQOs Come From
                                                                                  • The Postquery Collection
                                                                                  • The Stack Is Getting Rich
                                                                                  • Representative Example (2)
                                                                                  • Quality of Service
                                                                                  • QOS In Action
                                                                                  • Where Do We Store QOS Values
                                                                                  • The QOS Collection
                                                                                  • Coming Togetherhellip
                                                                                  • Representative Example (3)
                                                                                  • A Highly Leveragable Investment
                                                                                  • Not Just Java Not A Problem
                                                                                  • A Really Nice Stack
                                                                                  • What Can We Do
                                                                                  • Secure Access To Redacted Data for Testing
                                                                                  • Get It Programmatically Too
                                                                                  • Robust Validated Data Ingest
                                                                                  • Concept The control_ Collection
                                                                                  • Validation QOS and PQO via Web Services
                                                                                  • Are We Excited Yet
                                                                                  • Questions amp Answers
                                                                                  • Slide 53
                                                                                  • Concept DataProvider
                                                                                  • The RESTful Provider

                                                                                    42

                                                                                    QOSDBUtils

                                                                                    A Highly Leveragable Investment

                                                                                    PostQuery

                                                                                    MQLValidator

                                                                                    Data Access Layer 1

                                                                                    Application1

                                                                                    ValidatorDBUtils

                                                                                    PQLFilter

                                                                                    PQODBUtils

                                                                                    QOS

                                                                                    Application2

                                                                                    Data Access Layer 2

                                                                                    Application3

                                                                                    Application4

                                                                                    Data Access Layer 3

                                                                                    Application5

                                                                                    Application6Reusable For ALL Data Access Layer Logic

                                                                                    43

                                                                                    Not Just Java Not A Problem

                                                                                    DAL operations have little or no statehellip

                                                                                    Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                                    Can you say hellip Web Service

                                                                                    44

                                                                                    A Really Nice Stack

                                                                                    MongoDBJava Driver

                                                                                    MQLValidator

                                                                                    Data Access Layer

                                                                                    JavaApplication

                                                                                    ValidatorDBUtils

                                                                                    PQLFilter

                                                                                    HTTP Endpoint

                                                                                    pythonApplication

                                                                                    curl

                                                                                    JSON over HTTP(S)

                                                                                    JSONlt-gtJava Maps

                                                                                    QOSDBUtils

                                                                                    PostQuery

                                                                                    PQODBUtils

                                                                                    QOS

                                                                                    45

                                                                                    What Can We Do

                                                                                    46

                                                                                    Secure Access To Redacted Data for Testing

                                                                                    $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                    $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                    $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                    47

                                                                                    Get It Programmatically Too

                                                                                    This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                    getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                    String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                    48

                                                                                    Robust Validated Data Ingest

                                                                                    $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                    $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                    49

                                                                                    Concept The control_ Collection

                                                                                    gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                    bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                    50

                                                                                    Validation QOS and PQO via Web Services

                                                                                    MongoDBJava Driver

                                                                                    MQLValidator

                                                                                    Data Access Layer

                                                                                    JavaApplication

                                                                                    ValidatorHTTPUtils

                                                                                    PQLFilter

                                                                                    pythonApplication

                                                                                    curl

                                                                                    JSON over HTTP(S)

                                                                                    QOSHTTPUtils

                                                                                    PostQuery

                                                                                    PQOHTTPUtils

                                                                                    QOS

                                                                                    HTTP Service

                                                                                    JSONlt-gtJava Maps

                                                                                    51

                                                                                    Are We Excited Yet

                                                                                    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                    and other Java resources

                                                                                    Questions amp Answers

                                                                                    54

                                                                                    Concept DataProvider

                                                                                    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                    Class MongoProvider implements DataProvider hellip

                                                                                    Class RESTfulProvider implements DataProvider hellip

                                                                                    55

                                                                                    The RESTful Provider

                                                                                    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                    • Slide 1
                                                                                    • Part 3 In The Data Management Series
                                                                                    • Congratulations At this Point Yoursquove
                                                                                    • The Next Stage Defend amp Leverage
                                                                                    • MongoDB Doesnrsquot Have These Things
                                                                                    • What Can We Do Today
                                                                                    • Write Some Code
                                                                                    • Starting Point The Data Access Layer
                                                                                    • Document Validation
                                                                                    • A Query Filters Outbound Data
                                                                                    • How About Using It To Filter Inbounds
                                                                                    • $exists And $type Already in MQL
                                                                                    • hellip And MQL Goes Way Beyondhellip
                                                                                    • A New MQL Validator Module Emerges
                                                                                    • Migrating Capability into MongoDB
                                                                                    • Code For The FuturehellipToday
                                                                                    • But What About Today
                                                                                    • Temporary Filling PQL
                                                                                    • Bridge MQL to PQL
                                                                                    • No PQL No Problem
                                                                                    • MQL Is Easy To Navigate
                                                                                    • Where Do Validations Come From
                                                                                    • The Validations Collection
                                                                                    • Various ldquoLevelsrdquo of Validation
                                                                                    • Multiple Types Schema By Example
                                                                                    • The Stack So Far
                                                                                    • Representative Example
                                                                                    • Redaction
                                                                                    • Concept Post Query Operations (PQO)
                                                                                    • Adopt MQL-like behavior
                                                                                    • A New PostQuery Module Emerges
                                                                                    • Where Do PQOs Come From
                                                                                    • The Postquery Collection
                                                                                    • The Stack Is Getting Rich
                                                                                    • Representative Example (2)
                                                                                    • Quality of Service
                                                                                    • QOS In Action
                                                                                    • Where Do We Store QOS Values
                                                                                    • The QOS Collection
                                                                                    • Coming Togetherhellip
                                                                                    • Representative Example (3)
                                                                                    • A Highly Leveragable Investment
                                                                                    • Not Just Java Not A Problem
                                                                                    • A Really Nice Stack
                                                                                    • What Can We Do
                                                                                    • Secure Access To Redacted Data for Testing
                                                                                    • Get It Programmatically Too
                                                                                    • Robust Validated Data Ingest
                                                                                    • Concept The control_ Collection
                                                                                    • Validation QOS and PQO via Web Services
                                                                                    • Are We Excited Yet
                                                                                    • Questions amp Answers
                                                                                    • Slide 53
                                                                                    • Concept DataProvider
                                                                                    • The RESTful Provider

                                                                                      43

                                                                                      Not Just Java Not A Problem

                                                                                      DAL operations have little or no statehellip

                                                                                      Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

                                                                                      Can you say hellip Web Service

                                                                                      44

                                                                                      A Really Nice Stack

                                                                                      MongoDBJava Driver

                                                                                      MQLValidator

                                                                                      Data Access Layer

                                                                                      JavaApplication

                                                                                      ValidatorDBUtils

                                                                                      PQLFilter

                                                                                      HTTP Endpoint

                                                                                      pythonApplication

                                                                                      curl

                                                                                      JSON over HTTP(S)

                                                                                      JSONlt-gtJava Maps

                                                                                      QOSDBUtils

                                                                                      PostQuery

                                                                                      PQODBUtils

                                                                                      QOS

                                                                                      45

                                                                                      What Can We Do

                                                                                      46

                                                                                      Secure Access To Redacted Data for Testing

                                                                                      $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                      $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                      $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                      47

                                                                                      Get It Programmatically Too

                                                                                      This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                      getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                      String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                      48

                                                                                      Robust Validated Data Ingest

                                                                                      $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                      $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                      49

                                                                                      Concept The control_ Collection

                                                                                      gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                      bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                      50

                                                                                      Validation QOS and PQO via Web Services

                                                                                      MongoDBJava Driver

                                                                                      MQLValidator

                                                                                      Data Access Layer

                                                                                      JavaApplication

                                                                                      ValidatorHTTPUtils

                                                                                      PQLFilter

                                                                                      pythonApplication

                                                                                      curl

                                                                                      JSON over HTTP(S)

                                                                                      QOSHTTPUtils

                                                                                      PostQuery

                                                                                      PQOHTTPUtils

                                                                                      QOS

                                                                                      HTTP Service

                                                                                      JSONlt-gtJava Maps

                                                                                      51

                                                                                      Are We Excited Yet

                                                                                      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                      and other Java resources

                                                                                      Questions amp Answers

                                                                                      54

                                                                                      Concept DataProvider

                                                                                      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                      Class MongoProvider implements DataProvider hellip

                                                                                      Class RESTfulProvider implements DataProvider hellip

                                                                                      55

                                                                                      The RESTful Provider

                                                                                      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                      • Slide 1
                                                                                      • Part 3 In The Data Management Series
                                                                                      • Congratulations At this Point Yoursquove
                                                                                      • The Next Stage Defend amp Leverage
                                                                                      • MongoDB Doesnrsquot Have These Things
                                                                                      • What Can We Do Today
                                                                                      • Write Some Code
                                                                                      • Starting Point The Data Access Layer
                                                                                      • Document Validation
                                                                                      • A Query Filters Outbound Data
                                                                                      • How About Using It To Filter Inbounds
                                                                                      • $exists And $type Already in MQL
                                                                                      • hellip And MQL Goes Way Beyondhellip
                                                                                      • A New MQL Validator Module Emerges
                                                                                      • Migrating Capability into MongoDB
                                                                                      • Code For The FuturehellipToday
                                                                                      • But What About Today
                                                                                      • Temporary Filling PQL
                                                                                      • Bridge MQL to PQL
                                                                                      • No PQL No Problem
                                                                                      • MQL Is Easy To Navigate
                                                                                      • Where Do Validations Come From
                                                                                      • The Validations Collection
                                                                                      • Various ldquoLevelsrdquo of Validation
                                                                                      • Multiple Types Schema By Example
                                                                                      • The Stack So Far
                                                                                      • Representative Example
                                                                                      • Redaction
                                                                                      • Concept Post Query Operations (PQO)
                                                                                      • Adopt MQL-like behavior
                                                                                      • A New PostQuery Module Emerges
                                                                                      • Where Do PQOs Come From
                                                                                      • The Postquery Collection
                                                                                      • The Stack Is Getting Rich
                                                                                      • Representative Example (2)
                                                                                      • Quality of Service
                                                                                      • QOS In Action
                                                                                      • Where Do We Store QOS Values
                                                                                      • The QOS Collection
                                                                                      • Coming Togetherhellip
                                                                                      • Representative Example (3)
                                                                                      • A Highly Leveragable Investment
                                                                                      • Not Just Java Not A Problem
                                                                                      • A Really Nice Stack
                                                                                      • What Can We Do
                                                                                      • Secure Access To Redacted Data for Testing
                                                                                      • Get It Programmatically Too
                                                                                      • Robust Validated Data Ingest
                                                                                      • Concept The control_ Collection
                                                                                      • Validation QOS and PQO via Web Services
                                                                                      • Are We Excited Yet
                                                                                      • Questions amp Answers
                                                                                      • Slide 53
                                                                                      • Concept DataProvider
                                                                                      • The RESTful Provider

                                                                                        44

                                                                                        A Really Nice Stack

                                                                                        MongoDBJava Driver

                                                                                        MQLValidator

                                                                                        Data Access Layer

                                                                                        JavaApplication

                                                                                        ValidatorDBUtils

                                                                                        PQLFilter

                                                                                        HTTP Endpoint

                                                                                        pythonApplication

                                                                                        curl

                                                                                        JSON over HTTP(S)

                                                                                        JSONlt-gtJava Maps

                                                                                        QOSDBUtils

                                                                                        PostQuery

                                                                                        PQODBUtils

                                                                                        QOS

                                                                                        45

                                                                                        What Can We Do

                                                                                        46

                                                                                        Secure Access To Redacted Data for Testing

                                                                                        $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                        $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                        $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                        47

                                                                                        Get It Programmatically Too

                                                                                        This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                        getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                        String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                        48

                                                                                        Robust Validated Data Ingest

                                                                                        $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                        $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                        49

                                                                                        Concept The control_ Collection

                                                                                        gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                        bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                        50

                                                                                        Validation QOS and PQO via Web Services

                                                                                        MongoDBJava Driver

                                                                                        MQLValidator

                                                                                        Data Access Layer

                                                                                        JavaApplication

                                                                                        ValidatorHTTPUtils

                                                                                        PQLFilter

                                                                                        pythonApplication

                                                                                        curl

                                                                                        JSON over HTTP(S)

                                                                                        QOSHTTPUtils

                                                                                        PostQuery

                                                                                        PQOHTTPUtils

                                                                                        QOS

                                                                                        HTTP Service

                                                                                        JSONlt-gtJava Maps

                                                                                        51

                                                                                        Are We Excited Yet

                                                                                        Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                        and other Java resources

                                                                                        Questions amp Answers

                                                                                        54

                                                                                        Concept DataProvider

                                                                                        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                        Class MongoProvider implements DataProvider hellip

                                                                                        Class RESTfulProvider implements DataProvider hellip

                                                                                        55

                                                                                        The RESTful Provider

                                                                                        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                        • Slide 1
                                                                                        • Part 3 In The Data Management Series
                                                                                        • Congratulations At this Point Yoursquove
                                                                                        • The Next Stage Defend amp Leverage
                                                                                        • MongoDB Doesnrsquot Have These Things
                                                                                        • What Can We Do Today
                                                                                        • Write Some Code
                                                                                        • Starting Point The Data Access Layer
                                                                                        • Document Validation
                                                                                        • A Query Filters Outbound Data
                                                                                        • How About Using It To Filter Inbounds
                                                                                        • $exists And $type Already in MQL
                                                                                        • hellip And MQL Goes Way Beyondhellip
                                                                                        • A New MQL Validator Module Emerges
                                                                                        • Migrating Capability into MongoDB
                                                                                        • Code For The FuturehellipToday
                                                                                        • But What About Today
                                                                                        • Temporary Filling PQL
                                                                                        • Bridge MQL to PQL
                                                                                        • No PQL No Problem
                                                                                        • MQL Is Easy To Navigate
                                                                                        • Where Do Validations Come From
                                                                                        • The Validations Collection
                                                                                        • Various ldquoLevelsrdquo of Validation
                                                                                        • Multiple Types Schema By Example
                                                                                        • The Stack So Far
                                                                                        • Representative Example
                                                                                        • Redaction
                                                                                        • Concept Post Query Operations (PQO)
                                                                                        • Adopt MQL-like behavior
                                                                                        • A New PostQuery Module Emerges
                                                                                        • Where Do PQOs Come From
                                                                                        • The Postquery Collection
                                                                                        • The Stack Is Getting Rich
                                                                                        • Representative Example (2)
                                                                                        • Quality of Service
                                                                                        • QOS In Action
                                                                                        • Where Do We Store QOS Values
                                                                                        • The QOS Collection
                                                                                        • Coming Togetherhellip
                                                                                        • Representative Example (3)
                                                                                        • A Highly Leveragable Investment
                                                                                        • Not Just Java Not A Problem
                                                                                        • A Really Nice Stack
                                                                                        • What Can We Do
                                                                                        • Secure Access To Redacted Data for Testing
                                                                                        • Get It Programmatically Too
                                                                                        • Robust Validated Data Ingest
                                                                                        • Concept The control_ Collection
                                                                                        • Validation QOS and PQO via Web Services
                                                                                        • Are We Excited Yet
                                                                                        • Questions amp Answers
                                                                                        • Slide 53
                                                                                        • Concept DataProvider
                                                                                        • The RESTful Provider

                                                                                          45

                                                                                          What Can We Do

                                                                                          46

                                                                                          Secure Access To Redacted Data for Testing

                                                                                          $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                          $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                          $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                          47

                                                                                          Get It Programmatically Too

                                                                                          This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                          getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                          String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                          48

                                                                                          Robust Validated Data Ingest

                                                                                          $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                          $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                          49

                                                                                          Concept The control_ Collection

                                                                                          gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                          bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                          50

                                                                                          Validation QOS and PQO via Web Services

                                                                                          MongoDBJava Driver

                                                                                          MQLValidator

                                                                                          Data Access Layer

                                                                                          JavaApplication

                                                                                          ValidatorHTTPUtils

                                                                                          PQLFilter

                                                                                          pythonApplication

                                                                                          curl

                                                                                          JSON over HTTP(S)

                                                                                          QOSHTTPUtils

                                                                                          PostQuery

                                                                                          PQOHTTPUtils

                                                                                          QOS

                                                                                          HTTP Service

                                                                                          JSONlt-gtJava Maps

                                                                                          51

                                                                                          Are We Excited Yet

                                                                                          Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                          and other Java resources

                                                                                          Questions amp Answers

                                                                                          54

                                                                                          Concept DataProvider

                                                                                          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                          Class MongoProvider implements DataProvider hellip

                                                                                          Class RESTfulProvider implements DataProvider hellip

                                                                                          55

                                                                                          The RESTful Provider

                                                                                          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                          • Slide 1
                                                                                          • Part 3 In The Data Management Series
                                                                                          • Congratulations At this Point Yoursquove
                                                                                          • The Next Stage Defend amp Leverage
                                                                                          • MongoDB Doesnrsquot Have These Things
                                                                                          • What Can We Do Today
                                                                                          • Write Some Code
                                                                                          • Starting Point The Data Access Layer
                                                                                          • Document Validation
                                                                                          • A Query Filters Outbound Data
                                                                                          • How About Using It To Filter Inbounds
                                                                                          • $exists And $type Already in MQL
                                                                                          • hellip And MQL Goes Way Beyondhellip
                                                                                          • A New MQL Validator Module Emerges
                                                                                          • Migrating Capability into MongoDB
                                                                                          • Code For The FuturehellipToday
                                                                                          • But What About Today
                                                                                          • Temporary Filling PQL
                                                                                          • Bridge MQL to PQL
                                                                                          • No PQL No Problem
                                                                                          • MQL Is Easy To Navigate
                                                                                          • Where Do Validations Come From
                                                                                          • The Validations Collection
                                                                                          • Various ldquoLevelsrdquo of Validation
                                                                                          • Multiple Types Schema By Example
                                                                                          • The Stack So Far
                                                                                          • Representative Example
                                                                                          • Redaction
                                                                                          • Concept Post Query Operations (PQO)
                                                                                          • Adopt MQL-like behavior
                                                                                          • A New PostQuery Module Emerges
                                                                                          • Where Do PQOs Come From
                                                                                          • The Postquery Collection
                                                                                          • The Stack Is Getting Rich
                                                                                          • Representative Example (2)
                                                                                          • Quality of Service
                                                                                          • QOS In Action
                                                                                          • Where Do We Store QOS Values
                                                                                          • The QOS Collection
                                                                                          • Coming Togetherhellip
                                                                                          • Representative Example (3)
                                                                                          • A Highly Leveragable Investment
                                                                                          • Not Just Java Not A Problem
                                                                                          • A Really Nice Stack
                                                                                          • What Can We Do
                                                                                          • Secure Access To Redacted Data for Testing
                                                                                          • Get It Programmatically Too
                                                                                          • Robust Validated Data Ingest
                                                                                          • Concept The control_ Collection
                                                                                          • Validation QOS and PQO via Web Services
                                                                                          • Are We Excited Yet
                                                                                          • Questions amp Answers
                                                                                          • Slide 53
                                                                                          • Concept DataProvider
                                                                                          • The RESTful Provider

                                                                                            46

                                                                                            Secure Access To Redacted Data for Testing

                                                                                            $ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

                                                                                            $ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

                                                                                            $ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

                                                                                            47

                                                                                            Get It Programmatically Too

                                                                                            This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                            getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                            String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                            48

                                                                                            Robust Validated Data Ingest

                                                                                            $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                            $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                            49

                                                                                            Concept The control_ Collection

                                                                                            gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                            bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                            50

                                                                                            Validation QOS and PQO via Web Services

                                                                                            MongoDBJava Driver

                                                                                            MQLValidator

                                                                                            Data Access Layer

                                                                                            JavaApplication

                                                                                            ValidatorHTTPUtils

                                                                                            PQLFilter

                                                                                            pythonApplication

                                                                                            curl

                                                                                            JSON over HTTP(S)

                                                                                            QOSHTTPUtils

                                                                                            PostQuery

                                                                                            PQOHTTPUtils

                                                                                            QOS

                                                                                            HTTP Service

                                                                                            JSONlt-gtJava Maps

                                                                                            51

                                                                                            Are We Excited Yet

                                                                                            Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                            and other Java resources

                                                                                            Questions amp Answers

                                                                                            54

                                                                                            Concept DataProvider

                                                                                            public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                            Class MongoProvider implements DataProvider hellip

                                                                                            Class RESTfulProvider implements DataProvider hellip

                                                                                            55

                                                                                            The RESTful Provider

                                                                                            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                            • Slide 1
                                                                                            • Part 3 In The Data Management Series
                                                                                            • Congratulations At this Point Yoursquove
                                                                                            • The Next Stage Defend amp Leverage
                                                                                            • MongoDB Doesnrsquot Have These Things
                                                                                            • What Can We Do Today
                                                                                            • Write Some Code
                                                                                            • Starting Point The Data Access Layer
                                                                                            • Document Validation
                                                                                            • A Query Filters Outbound Data
                                                                                            • How About Using It To Filter Inbounds
                                                                                            • $exists And $type Already in MQL
                                                                                            • hellip And MQL Goes Way Beyondhellip
                                                                                            • A New MQL Validator Module Emerges
                                                                                            • Migrating Capability into MongoDB
                                                                                            • Code For The FuturehellipToday
                                                                                            • But What About Today
                                                                                            • Temporary Filling PQL
                                                                                            • Bridge MQL to PQL
                                                                                            • No PQL No Problem
                                                                                            • MQL Is Easy To Navigate
                                                                                            • Where Do Validations Come From
                                                                                            • The Validations Collection
                                                                                            • Various ldquoLevelsrdquo of Validation
                                                                                            • Multiple Types Schema By Example
                                                                                            • The Stack So Far
                                                                                            • Representative Example
                                                                                            • Redaction
                                                                                            • Concept Post Query Operations (PQO)
                                                                                            • Adopt MQL-like behavior
                                                                                            • A New PostQuery Module Emerges
                                                                                            • Where Do PQOs Come From
                                                                                            • The Postquery Collection
                                                                                            • The Stack Is Getting Rich
                                                                                            • Representative Example (2)
                                                                                            • Quality of Service
                                                                                            • QOS In Action
                                                                                            • Where Do We Store QOS Values
                                                                                            • The QOS Collection
                                                                                            • Coming Togetherhellip
                                                                                            • Representative Example (3)
                                                                                            • A Highly Leveragable Investment
                                                                                            • Not Just Java Not A Problem
                                                                                            • A Really Nice Stack
                                                                                            • What Can We Do
                                                                                            • Secure Access To Redacted Data for Testing
                                                                                            • Get It Programmatically Too
                                                                                            • Robust Validated Data Ingest
                                                                                            • Concept The control_ Collection
                                                                                            • Validation QOS and PQO via Web Services
                                                                                            • Are We Excited Yet
                                                                                            • Questions amp Answers
                                                                                            • Slide 53
                                                                                            • Concept DataProvider
                                                                                            • The RESTful Provider

                                                                                              47

                                                                                              Get It Programmatically Too

                                                                                              This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

                                                                                              getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

                                                                                              String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

                                                                                              48

                                                                                              Robust Validated Data Ingest

                                                                                              $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                              $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                              49

                                                                                              Concept The control_ Collection

                                                                                              gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                              bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                              50

                                                                                              Validation QOS and PQO via Web Services

                                                                                              MongoDBJava Driver

                                                                                              MQLValidator

                                                                                              Data Access Layer

                                                                                              JavaApplication

                                                                                              ValidatorHTTPUtils

                                                                                              PQLFilter

                                                                                              pythonApplication

                                                                                              curl

                                                                                              JSON over HTTP(S)

                                                                                              QOSHTTPUtils

                                                                                              PostQuery

                                                                                              PQOHTTPUtils

                                                                                              QOS

                                                                                              HTTP Service

                                                                                              JSONlt-gtJava Maps

                                                                                              51

                                                                                              Are We Excited Yet

                                                                                              Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                              and other Java resources

                                                                                              Questions amp Answers

                                                                                              54

                                                                                              Concept DataProvider

                                                                                              public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                              Class MongoProvider implements DataProvider hellip

                                                                                              Class RESTfulProvider implements DataProvider hellip

                                                                                              55

                                                                                              The RESTful Provider

                                                                                              class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                              fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                              url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                              HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                              • Slide 1
                                                                                              • Part 3 In The Data Management Series
                                                                                              • Congratulations At this Point Yoursquove
                                                                                              • The Next Stage Defend amp Leverage
                                                                                              • MongoDB Doesnrsquot Have These Things
                                                                                              • What Can We Do Today
                                                                                              • Write Some Code
                                                                                              • Starting Point The Data Access Layer
                                                                                              • Document Validation
                                                                                              • A Query Filters Outbound Data
                                                                                              • How About Using It To Filter Inbounds
                                                                                              • $exists And $type Already in MQL
                                                                                              • hellip And MQL Goes Way Beyondhellip
                                                                                              • A New MQL Validator Module Emerges
                                                                                              • Migrating Capability into MongoDB
                                                                                              • Code For The FuturehellipToday
                                                                                              • But What About Today
                                                                                              • Temporary Filling PQL
                                                                                              • Bridge MQL to PQL
                                                                                              • No PQL No Problem
                                                                                              • MQL Is Easy To Navigate
                                                                                              • Where Do Validations Come From
                                                                                              • The Validations Collection
                                                                                              • Various ldquoLevelsrdquo of Validation
                                                                                              • Multiple Types Schema By Example
                                                                                              • The Stack So Far
                                                                                              • Representative Example
                                                                                              • Redaction
                                                                                              • Concept Post Query Operations (PQO)
                                                                                              • Adopt MQL-like behavior
                                                                                              • A New PostQuery Module Emerges
                                                                                              • Where Do PQOs Come From
                                                                                              • The Postquery Collection
                                                                                              • The Stack Is Getting Rich
                                                                                              • Representative Example (2)
                                                                                              • Quality of Service
                                                                                              • QOS In Action
                                                                                              • Where Do We Store QOS Values
                                                                                              • The QOS Collection
                                                                                              • Coming Togetherhellip
                                                                                              • Representative Example (3)
                                                                                              • A Highly Leveragable Investment
                                                                                              • Not Just Java Not A Problem
                                                                                              • A Really Nice Stack
                                                                                              • What Can We Do
                                                                                              • Secure Access To Redacted Data for Testing
                                                                                              • Get It Programmatically Too
                                                                                              • Robust Validated Data Ingest
                                                                                              • Concept The control_ Collection
                                                                                              • Validation QOS and PQO via Web Services
                                                                                              • Are We Excited Yet
                                                                                              • Questions amp Answers
                                                                                              • Slide 53
                                                                                              • Concept DataProvider
                                                                                              • The RESTful Provider

                                                                                                48

                                                                                                Robust Validated Data Ingest

                                                                                                $ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

                                                                                                $ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

                                                                                                49

                                                                                                Concept The control_ Collection

                                                                                                gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                                bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                                50

                                                                                                Validation QOS and PQO via Web Services

                                                                                                MongoDBJava Driver

                                                                                                MQLValidator

                                                                                                Data Access Layer

                                                                                                JavaApplication

                                                                                                ValidatorHTTPUtils

                                                                                                PQLFilter

                                                                                                pythonApplication

                                                                                                curl

                                                                                                JSON over HTTP(S)

                                                                                                QOSHTTPUtils

                                                                                                PostQuery

                                                                                                PQOHTTPUtils

                                                                                                QOS

                                                                                                HTTP Service

                                                                                                JSONlt-gtJava Maps

                                                                                                51

                                                                                                Are We Excited Yet

                                                                                                Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                                and other Java resources

                                                                                                Questions amp Answers

                                                                                                54

                                                                                                Concept DataProvider

                                                                                                public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                                Class MongoProvider implements DataProvider hellip

                                                                                                Class RESTfulProvider implements DataProvider hellip

                                                                                                55

                                                                                                The RESTful Provider

                                                                                                class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                • Slide 1
                                                                                                • Part 3 In The Data Management Series
                                                                                                • Congratulations At this Point Yoursquove
                                                                                                • The Next Stage Defend amp Leverage
                                                                                                • MongoDB Doesnrsquot Have These Things
                                                                                                • What Can We Do Today
                                                                                                • Write Some Code
                                                                                                • Starting Point The Data Access Layer
                                                                                                • Document Validation
                                                                                                • A Query Filters Outbound Data
                                                                                                • How About Using It To Filter Inbounds
                                                                                                • $exists And $type Already in MQL
                                                                                                • hellip And MQL Goes Way Beyondhellip
                                                                                                • A New MQL Validator Module Emerges
                                                                                                • Migrating Capability into MongoDB
                                                                                                • Code For The FuturehellipToday
                                                                                                • But What About Today
                                                                                                • Temporary Filling PQL
                                                                                                • Bridge MQL to PQL
                                                                                                • No PQL No Problem
                                                                                                • MQL Is Easy To Navigate
                                                                                                • Where Do Validations Come From
                                                                                                • The Validations Collection
                                                                                                • Various ldquoLevelsrdquo of Validation
                                                                                                • Multiple Types Schema By Example
                                                                                                • The Stack So Far
                                                                                                • Representative Example
                                                                                                • Redaction
                                                                                                • Concept Post Query Operations (PQO)
                                                                                                • Adopt MQL-like behavior
                                                                                                • A New PostQuery Module Emerges
                                                                                                • Where Do PQOs Come From
                                                                                                • The Postquery Collection
                                                                                                • The Stack Is Getting Rich
                                                                                                • Representative Example (2)
                                                                                                • Quality of Service
                                                                                                • QOS In Action
                                                                                                • Where Do We Store QOS Values
                                                                                                • The QOS Collection
                                                                                                • Coming Togetherhellip
                                                                                                • Representative Example (3)
                                                                                                • A Highly Leveragable Investment
                                                                                                • Not Just Java Not A Problem
                                                                                                • A Really Nice Stack
                                                                                                • What Can We Do
                                                                                                • Secure Access To Redacted Data for Testing
                                                                                                • Get It Programmatically Too
                                                                                                • Robust Validated Data Ingest
                                                                                                • Concept The control_ Collection
                                                                                                • Validation QOS and PQO via Web Services
                                                                                                • Are We Excited Yet
                                                                                                • Questions amp Answers
                                                                                                • Slide 53
                                                                                                • Concept DataProvider
                                                                                                • The RESTful Provider

                                                                                                  49

                                                                                                  Concept The control_ Collection

                                                                                                  gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

                                                                                                  bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

                                                                                                  50

                                                                                                  Validation QOS and PQO via Web Services

                                                                                                  MongoDBJava Driver

                                                                                                  MQLValidator

                                                                                                  Data Access Layer

                                                                                                  JavaApplication

                                                                                                  ValidatorHTTPUtils

                                                                                                  PQLFilter

                                                                                                  pythonApplication

                                                                                                  curl

                                                                                                  JSON over HTTP(S)

                                                                                                  QOSHTTPUtils

                                                                                                  PostQuery

                                                                                                  PQOHTTPUtils

                                                                                                  QOS

                                                                                                  HTTP Service

                                                                                                  JSONlt-gtJava Maps

                                                                                                  51

                                                                                                  Are We Excited Yet

                                                                                                  Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                                  and other Java resources

                                                                                                  Questions amp Answers

                                                                                                  54

                                                                                                  Concept DataProvider

                                                                                                  public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                                  Class MongoProvider implements DataProvider hellip

                                                                                                  Class RESTfulProvider implements DataProvider hellip

                                                                                                  55

                                                                                                  The RESTful Provider

                                                                                                  class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                  fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                  url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                  HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                  • Slide 1
                                                                                                  • Part 3 In The Data Management Series
                                                                                                  • Congratulations At this Point Yoursquove
                                                                                                  • The Next Stage Defend amp Leverage
                                                                                                  • MongoDB Doesnrsquot Have These Things
                                                                                                  • What Can We Do Today
                                                                                                  • Write Some Code
                                                                                                  • Starting Point The Data Access Layer
                                                                                                  • Document Validation
                                                                                                  • A Query Filters Outbound Data
                                                                                                  • How About Using It To Filter Inbounds
                                                                                                  • $exists And $type Already in MQL
                                                                                                  • hellip And MQL Goes Way Beyondhellip
                                                                                                  • A New MQL Validator Module Emerges
                                                                                                  • Migrating Capability into MongoDB
                                                                                                  • Code For The FuturehellipToday
                                                                                                  • But What About Today
                                                                                                  • Temporary Filling PQL
                                                                                                  • Bridge MQL to PQL
                                                                                                  • No PQL No Problem
                                                                                                  • MQL Is Easy To Navigate
                                                                                                  • Where Do Validations Come From
                                                                                                  • The Validations Collection
                                                                                                  • Various ldquoLevelsrdquo of Validation
                                                                                                  • Multiple Types Schema By Example
                                                                                                  • The Stack So Far
                                                                                                  • Representative Example
                                                                                                  • Redaction
                                                                                                  • Concept Post Query Operations (PQO)
                                                                                                  • Adopt MQL-like behavior
                                                                                                  • A New PostQuery Module Emerges
                                                                                                  • Where Do PQOs Come From
                                                                                                  • The Postquery Collection
                                                                                                  • The Stack Is Getting Rich
                                                                                                  • Representative Example (2)
                                                                                                  • Quality of Service
                                                                                                  • QOS In Action
                                                                                                  • Where Do We Store QOS Values
                                                                                                  • The QOS Collection
                                                                                                  • Coming Togetherhellip
                                                                                                  • Representative Example (3)
                                                                                                  • A Highly Leveragable Investment
                                                                                                  • Not Just Java Not A Problem
                                                                                                  • A Really Nice Stack
                                                                                                  • What Can We Do
                                                                                                  • Secure Access To Redacted Data for Testing
                                                                                                  • Get It Programmatically Too
                                                                                                  • Robust Validated Data Ingest
                                                                                                  • Concept The control_ Collection
                                                                                                  • Validation QOS and PQO via Web Services
                                                                                                  • Are We Excited Yet
                                                                                                  • Questions amp Answers
                                                                                                  • Slide 53
                                                                                                  • Concept DataProvider
                                                                                                  • The RESTful Provider

                                                                                                    50

                                                                                                    Validation QOS and PQO via Web Services

                                                                                                    MongoDBJava Driver

                                                                                                    MQLValidator

                                                                                                    Data Access Layer

                                                                                                    JavaApplication

                                                                                                    ValidatorHTTPUtils

                                                                                                    PQLFilter

                                                                                                    pythonApplication

                                                                                                    curl

                                                                                                    JSON over HTTP(S)

                                                                                                    QOSHTTPUtils

                                                                                                    PostQuery

                                                                                                    PQOHTTPUtils

                                                                                                    QOS

                                                                                                    HTTP Service

                                                                                                    JSONlt-gtJava Maps

                                                                                                    51

                                                                                                    Are We Excited Yet

                                                                                                    Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                                    and other Java resources

                                                                                                    Questions amp Answers

                                                                                                    54

                                                                                                    Concept DataProvider

                                                                                                    public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                                    Class MongoProvider implements DataProvider hellip

                                                                                                    Class RESTfulProvider implements DataProvider hellip

                                                                                                    55

                                                                                                    The RESTful Provider

                                                                                                    class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                    fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                    url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                    HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                    • Slide 1
                                                                                                    • Part 3 In The Data Management Series
                                                                                                    • Congratulations At this Point Yoursquove
                                                                                                    • The Next Stage Defend amp Leverage
                                                                                                    • MongoDB Doesnrsquot Have These Things
                                                                                                    • What Can We Do Today
                                                                                                    • Write Some Code
                                                                                                    • Starting Point The Data Access Layer
                                                                                                    • Document Validation
                                                                                                    • A Query Filters Outbound Data
                                                                                                    • How About Using It To Filter Inbounds
                                                                                                    • $exists And $type Already in MQL
                                                                                                    • hellip And MQL Goes Way Beyondhellip
                                                                                                    • A New MQL Validator Module Emerges
                                                                                                    • Migrating Capability into MongoDB
                                                                                                    • Code For The FuturehellipToday
                                                                                                    • But What About Today
                                                                                                    • Temporary Filling PQL
                                                                                                    • Bridge MQL to PQL
                                                                                                    • No PQL No Problem
                                                                                                    • MQL Is Easy To Navigate
                                                                                                    • Where Do Validations Come From
                                                                                                    • The Validations Collection
                                                                                                    • Various ldquoLevelsrdquo of Validation
                                                                                                    • Multiple Types Schema By Example
                                                                                                    • The Stack So Far
                                                                                                    • Representative Example
                                                                                                    • Redaction
                                                                                                    • Concept Post Query Operations (PQO)
                                                                                                    • Adopt MQL-like behavior
                                                                                                    • A New PostQuery Module Emerges
                                                                                                    • Where Do PQOs Come From
                                                                                                    • The Postquery Collection
                                                                                                    • The Stack Is Getting Rich
                                                                                                    • Representative Example (2)
                                                                                                    • Quality of Service
                                                                                                    • QOS In Action
                                                                                                    • Where Do We Store QOS Values
                                                                                                    • The QOS Collection
                                                                                                    • Coming Togetherhellip
                                                                                                    • Representative Example (3)
                                                                                                    • A Highly Leveragable Investment
                                                                                                    • Not Just Java Not A Problem
                                                                                                    • A Really Nice Stack
                                                                                                    • What Can We Do
                                                                                                    • Secure Access To Redacted Data for Testing
                                                                                                    • Get It Programmatically Too
                                                                                                    • Robust Validated Data Ingest
                                                                                                    • Concept The control_ Collection
                                                                                                    • Validation QOS and PQO via Web Services
                                                                                                    • Are We Excited Yet
                                                                                                    • Questions amp Answers
                                                                                                    • Slide 53
                                                                                                    • Concept DataProvider
                                                                                                    • The RESTful Provider

                                                                                                      51

                                                                                                      Are We Excited Yet

                                                                                                      Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

                                                                                                      and other Java resources

                                                                                                      Questions amp Answers

                                                                                                      54

                                                                                                      Concept DataProvider

                                                                                                      public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                                      Class MongoProvider implements DataProvider hellip

                                                                                                      Class RESTfulProvider implements DataProvider hellip

                                                                                                      55

                                                                                                      The RESTful Provider

                                                                                                      class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                      fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                      url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                      HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                      • Slide 1
                                                                                                      • Part 3 In The Data Management Series
                                                                                                      • Congratulations At this Point Yoursquove
                                                                                                      • The Next Stage Defend amp Leverage
                                                                                                      • MongoDB Doesnrsquot Have These Things
                                                                                                      • What Can We Do Today
                                                                                                      • Write Some Code
                                                                                                      • Starting Point The Data Access Layer
                                                                                                      • Document Validation
                                                                                                      • A Query Filters Outbound Data
                                                                                                      • How About Using It To Filter Inbounds
                                                                                                      • $exists And $type Already in MQL
                                                                                                      • hellip And MQL Goes Way Beyondhellip
                                                                                                      • A New MQL Validator Module Emerges
                                                                                                      • Migrating Capability into MongoDB
                                                                                                      • Code For The FuturehellipToday
                                                                                                      • But What About Today
                                                                                                      • Temporary Filling PQL
                                                                                                      • Bridge MQL to PQL
                                                                                                      • No PQL No Problem
                                                                                                      • MQL Is Easy To Navigate
                                                                                                      • Where Do Validations Come From
                                                                                                      • The Validations Collection
                                                                                                      • Various ldquoLevelsrdquo of Validation
                                                                                                      • Multiple Types Schema By Example
                                                                                                      • The Stack So Far
                                                                                                      • Representative Example
                                                                                                      • Redaction
                                                                                                      • Concept Post Query Operations (PQO)
                                                                                                      • Adopt MQL-like behavior
                                                                                                      • A New PostQuery Module Emerges
                                                                                                      • Where Do PQOs Come From
                                                                                                      • The Postquery Collection
                                                                                                      • The Stack Is Getting Rich
                                                                                                      • Representative Example (2)
                                                                                                      • Quality of Service
                                                                                                      • QOS In Action
                                                                                                      • Where Do We Store QOS Values
                                                                                                      • The QOS Collection
                                                                                                      • Coming Togetherhellip
                                                                                                      • Representative Example (3)
                                                                                                      • A Highly Leveragable Investment
                                                                                                      • Not Just Java Not A Problem
                                                                                                      • A Really Nice Stack
                                                                                                      • What Can We Do
                                                                                                      • Secure Access To Redacted Data for Testing
                                                                                                      • Get It Programmatically Too
                                                                                                      • Robust Validated Data Ingest
                                                                                                      • Concept The control_ Collection
                                                                                                      • Validation QOS and PQO via Web Services
                                                                                                      • Are We Excited Yet
                                                                                                      • Questions amp Answers
                                                                                                      • Slide 53
                                                                                                      • Concept DataProvider
                                                                                                      • The RESTful Provider

                                                                                                        Questions amp Answers

                                                                                                        54

                                                                                                        Concept DataProvider

                                                                                                        public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                                        Class MongoProvider implements DataProvider hellip

                                                                                                        Class RESTfulProvider implements DataProvider hellip

                                                                                                        55

                                                                                                        The RESTful Provider

                                                                                                        class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                        fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                        url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                        HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                        • Slide 1
                                                                                                        • Part 3 In The Data Management Series
                                                                                                        • Congratulations At this Point Yoursquove
                                                                                                        • The Next Stage Defend amp Leverage
                                                                                                        • MongoDB Doesnrsquot Have These Things
                                                                                                        • What Can We Do Today
                                                                                                        • Write Some Code
                                                                                                        • Starting Point The Data Access Layer
                                                                                                        • Document Validation
                                                                                                        • A Query Filters Outbound Data
                                                                                                        • How About Using It To Filter Inbounds
                                                                                                        • $exists And $type Already in MQL
                                                                                                        • hellip And MQL Goes Way Beyondhellip
                                                                                                        • A New MQL Validator Module Emerges
                                                                                                        • Migrating Capability into MongoDB
                                                                                                        • Code For The FuturehellipToday
                                                                                                        • But What About Today
                                                                                                        • Temporary Filling PQL
                                                                                                        • Bridge MQL to PQL
                                                                                                        • No PQL No Problem
                                                                                                        • MQL Is Easy To Navigate
                                                                                                        • Where Do Validations Come From
                                                                                                        • The Validations Collection
                                                                                                        • Various ldquoLevelsrdquo of Validation
                                                                                                        • Multiple Types Schema By Example
                                                                                                        • The Stack So Far
                                                                                                        • Representative Example
                                                                                                        • Redaction
                                                                                                        • Concept Post Query Operations (PQO)
                                                                                                        • Adopt MQL-like behavior
                                                                                                        • A New PostQuery Module Emerges
                                                                                                        • Where Do PQOs Come From
                                                                                                        • The Postquery Collection
                                                                                                        • The Stack Is Getting Rich
                                                                                                        • Representative Example (2)
                                                                                                        • Quality of Service
                                                                                                        • QOS In Action
                                                                                                        • Where Do We Store QOS Values
                                                                                                        • The QOS Collection
                                                                                                        • Coming Togetherhellip
                                                                                                        • Representative Example (3)
                                                                                                        • A Highly Leveragable Investment
                                                                                                        • Not Just Java Not A Problem
                                                                                                        • A Really Nice Stack
                                                                                                        • What Can We Do
                                                                                                        • Secure Access To Redacted Data for Testing
                                                                                                        • Get It Programmatically Too
                                                                                                        • Robust Validated Data Ingest
                                                                                                        • Concept The control_ Collection
                                                                                                        • Validation QOS and PQO via Web Services
                                                                                                        • Are We Excited Yet
                                                                                                        • Questions amp Answers
                                                                                                        • Slide 53
                                                                                                        • Concept DataProvider
                                                                                                        • The RESTful Provider

                                                                                                          54

                                                                                                          Concept DataProvider

                                                                                                          public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

                                                                                                          Class MongoProvider implements DataProvider hellip

                                                                                                          Class RESTfulProvider implements DataProvider hellip

                                                                                                          55

                                                                                                          The RESTful Provider

                                                                                                          class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                          fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                          url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                          HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                          • Slide 1
                                                                                                          • Part 3 In The Data Management Series
                                                                                                          • Congratulations At this Point Yoursquove
                                                                                                          • The Next Stage Defend amp Leverage
                                                                                                          • MongoDB Doesnrsquot Have These Things
                                                                                                          • What Can We Do Today
                                                                                                          • Write Some Code
                                                                                                          • Starting Point The Data Access Layer
                                                                                                          • Document Validation
                                                                                                          • A Query Filters Outbound Data
                                                                                                          • How About Using It To Filter Inbounds
                                                                                                          • $exists And $type Already in MQL
                                                                                                          • hellip And MQL Goes Way Beyondhellip
                                                                                                          • A New MQL Validator Module Emerges
                                                                                                          • Migrating Capability into MongoDB
                                                                                                          • Code For The FuturehellipToday
                                                                                                          • But What About Today
                                                                                                          • Temporary Filling PQL
                                                                                                          • Bridge MQL to PQL
                                                                                                          • No PQL No Problem
                                                                                                          • MQL Is Easy To Navigate
                                                                                                          • Where Do Validations Come From
                                                                                                          • The Validations Collection
                                                                                                          • Various ldquoLevelsrdquo of Validation
                                                                                                          • Multiple Types Schema By Example
                                                                                                          • The Stack So Far
                                                                                                          • Representative Example
                                                                                                          • Redaction
                                                                                                          • Concept Post Query Operations (PQO)
                                                                                                          • Adopt MQL-like behavior
                                                                                                          • A New PostQuery Module Emerges
                                                                                                          • Where Do PQOs Come From
                                                                                                          • The Postquery Collection
                                                                                                          • The Stack Is Getting Rich
                                                                                                          • Representative Example (2)
                                                                                                          • Quality of Service
                                                                                                          • QOS In Action
                                                                                                          • Where Do We Store QOS Values
                                                                                                          • The QOS Collection
                                                                                                          • Coming Togetherhellip
                                                                                                          • Representative Example (3)
                                                                                                          • A Highly Leveragable Investment
                                                                                                          • Not Just Java Not A Problem
                                                                                                          • A Really Nice Stack
                                                                                                          • What Can We Do
                                                                                                          • Secure Access To Redacted Data for Testing
                                                                                                          • Get It Programmatically Too
                                                                                                          • Robust Validated Data Ingest
                                                                                                          • Concept The control_ Collection
                                                                                                          • Validation QOS and PQO via Web Services
                                                                                                          • Are We Excited Yet
                                                                                                          • Questions amp Answers
                                                                                                          • Slide 53
                                                                                                          • Concept DataProvider
                                                                                                          • The RESTful Provider

                                                                                                            55

                                                                                                            The RESTful Provider

                                                                                                            class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

                                                                                                            fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

                                                                                                            url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

                                                                                                            HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

                                                                                                            • Slide 1
                                                                                                            • Part 3 In The Data Management Series
                                                                                                            • Congratulations At this Point Yoursquove
                                                                                                            • The Next Stage Defend amp Leverage
                                                                                                            • MongoDB Doesnrsquot Have These Things
                                                                                                            • What Can We Do Today
                                                                                                            • Write Some Code
                                                                                                            • Starting Point The Data Access Layer
                                                                                                            • Document Validation
                                                                                                            • A Query Filters Outbound Data
                                                                                                            • How About Using It To Filter Inbounds
                                                                                                            • $exists And $type Already in MQL
                                                                                                            • hellip And MQL Goes Way Beyondhellip
                                                                                                            • A New MQL Validator Module Emerges
                                                                                                            • Migrating Capability into MongoDB
                                                                                                            • Code For The FuturehellipToday
                                                                                                            • But What About Today
                                                                                                            • Temporary Filling PQL
                                                                                                            • Bridge MQL to PQL
                                                                                                            • No PQL No Problem
                                                                                                            • MQL Is Easy To Navigate
                                                                                                            • Where Do Validations Come From
                                                                                                            • The Validations Collection
                                                                                                            • Various ldquoLevelsrdquo of Validation
                                                                                                            • Multiple Types Schema By Example
                                                                                                            • The Stack So Far
                                                                                                            • Representative Example
                                                                                                            • Redaction
                                                                                                            • Concept Post Query Operations (PQO)
                                                                                                            • Adopt MQL-like behavior
                                                                                                            • A New PostQuery Module Emerges
                                                                                                            • Where Do PQOs Come From
                                                                                                            • The Postquery Collection
                                                                                                            • The Stack Is Getting Rich
                                                                                                            • Representative Example (2)
                                                                                                            • Quality of Service
                                                                                                            • QOS In Action
                                                                                                            • Where Do We Store QOS Values
                                                                                                            • The QOS Collection
                                                                                                            • Coming Togetherhellip
                                                                                                            • Representative Example (3)
                                                                                                            • A Highly Leveragable Investment
                                                                                                            • Not Just Java Not A Problem
                                                                                                            • A Really Nice Stack
                                                                                                            • What Can We Do
                                                                                                            • Secure Access To Redacted Data for Testing
                                                                                                            • Get It Programmatically Too
                                                                                                            • Robust Validated Data Ingest
                                                                                                            • Concept The control_ Collection
                                                                                                            • Validation QOS and PQO via Web Services
                                                                                                            • Are We Excited Yet
                                                                                                            • Questions amp Answers
                                                                                                            • Slide 53
                                                                                                            • Concept DataProvider
                                                                                                            • The RESTful Provider

                                                                                                              top related

                                                                                                              How to Bulletproof Your Data Defenses Locally & In the Cloud
                                                                                                              Business
                                                                                                              Bulletproof Presentations Handouts.pdf
                                                                                                              Documents
                                                                                                              Bulletproof Bumpers
                                                                                                              Documents
                                                                                                              Bulletproof Vest
                                                                                                              Education
                                                                                                              Bulletproof WebSphere Architectures
                                                                                                              Documents
                                                                                                              Write bulletproof trigger code
                                                                                                              Documents
                                                                                                              Bulletproof Diet Shopping List...
                                                                                                              Documents
                                                                                                              BULLETPROOF GLASS
                                                                                                              Documents
                                                                                                              Bulletproof Project Management
                                                                                                              Documents
                                                                                                              Den Bulletproof- Problemlösungsansatz erlernen
                                                                                                              Documents
                                                                                                              Andy Nyman Bulletproof
                                                                                                              Documents
                                                                                                              Bulletproof helmets
                                                                                                              Documents
                                                                                                              Bulletproof Ajax
                                                                                                              Technology
                                                                                                              Bulletproof Jackets
                                                                                                              Documents
                                                                                                              IT Courseware - Bulletproof Academy
                                                                                                              Documents
                                                                                                              Bulletproof Diet Infographic Vector
                                                                                                              Documents