Assessing the performance of RDF Engines: Discussing RDF Benchmarks

AssessingtheperformanceofRDFEngines:DiscussingRDFBenchmarks

IriniFundulakiInstituteofComputerScience–FORTH,Greece

AnastasiosKementsietsidisGoogleResearch,USA

6/15/16 ESWC2016:AssessingtheperformanceofRDFEngines-DiscussingRDFBenchmarks 1

TraditionalWeb:WebofDocuments•  singleinformationspace:globalfilesystem•  designedforhumanconsumption•  documentsaretheprimaryobjectswithaloosestructure•  URLsarethegloballyuniqueIDsandpartoftheretrieval

mechanism•  cannotaskexpressivequeries


©Hartig

,Cyg

aniac,Bizer,H

ausenb

las,Hea

th

How

toPub

lishLink

edDataon

theWeb

HTML HTML HTML

WebBrowsers WebBrowsers

hyperlinks hyperlinks

GoingfromtheWebofDocumentstotheWebofData•  Aglobaldatabase•  Designedformachinesfirst,humanslater•  Thingsareprimaryobjectswithawelldefinedstructure•  Typedlinksbetweenthings•  Abilitytoexpressstructuredqueries


Thing

Thing

Thing

Thing

Thing

Thing

Don’tlinkthedocuments,linkthethings

typedlinks typedlinks

©The

Web

ofL

inke

dData:Tom

Hea

th,

AnIntrod

uctio

ntoLinke

dData

LinkingOpenDatasets(LOD)•  PublishopendataasLinkedDataontheWeb•  Interlinkentitiesbetweenheterogeneousdatasources


StatusoftheLinkedOpenDataCloud,2007






Media

Government

Geographic

Publications

User-generated

Lifesciences

Cross-domain

RDF,acommondatamodel

Morethan31BtriplesinLOD

Links(external):500M

LinkedDatainnumbers(2014)•  StateoftheLODCloud2014,UniversityofManheim


Domain Datasets % Any SPARQL Dump

Government 183 18.05 61(32.80%) 30.11% 30.65%

Publications 96 9.47 10(10.58%) 9.62% 3.85%

LifeSciences 83 8.19 19(21.35%) 20.22% 16.85%

User-generatedcontent

48 4.73 3(5.4%5) 5.45%

1.82%

Cross-domain 41 4.04 4(9.09%) 4.55% 6.82%

Media 22 2.17 1(2.70%) 0.00% 2.70%

Geographic 21 2.07 8(19.51%) 12.20% 12.20%

SocialWeb 520 51.28 6(1.16%%) 1.16% 0.39%

Total 1014 - 48(5.89%) 4.54% 3.80%

AccessMethods

ProliferationofBigDataStores


Many(notalot)RDFStores


TheQuestion(s)•  WhicharetheproblemsthatIwishtosolve?•  Whicharetherelevantkeyperformanceindicators?•  Whichisthebehavioroftheexistingenginesw.r.t.thekey

performanceindicators?


Whicharethetool(s)thatIshoulduseformydataandformyusecase?

TheAnswer:Benchmarkyourengines!


•  QueryingBenchmarkcomprisesof

–  datasets(syntheticorreal)–  setofsoftwaretools

•  syntheticdatagenerators•  querygenerators

–  performancemetrics,and

–  setofclearexecutionrules•  Standardizedapplicationscenario(s)thatserveasabasisfor

testingsystems

•  Mustincludeaclearsetoffactorstobemeasuredandtheconditionsunderwhichthesystemsshouldbemeasured


•  Benchmarksexist–  Toallowadequatemeasurementsofsystems–  Toprovideevaluationofenginesforreal(orclosetoreal)usecases

•  Providehelp–  DesignersandDeveloperstoassesstheperformanceoftheirtools

–  Userstocomparethedifferentavailabletoolsandevaluatesuitabilityfortheirneeds

–  Researcherstocomparetheirworktoothers•  Leadstoimprovements:–  Vendorscanimprovetheirtechnology–  Researcherscanaddressnewchallenges–  Currentbenchmarkdesigncanbeimprovedtocovernewnecessitiesandapplicationdomains

ImportanceofBenchmarking

TutorialObjective&Benefits•  Objectives:–  Discussasetofprinciplesandbestpracticesforbenchmarkdevelopment

–  PresentanoverviewofthecurrentworkonbenchmarksforRDFqueryengines

–  Focusonidentifyingresearchchallenges&unexploredresearchdirections

•  Benefitsfortheaudience–  Academic:Obtainasolidbackground,discovernewresearchdirections

–  Practitioner:findoutwhataretheavailablebenchmarks,advantagesandlimitationsthereof


PurposeoftheTutorial

•  Stimulatediscussionsonthefollowingtopics:1.  Howcanonecomeupwiththerightbenchmarkthat

accuratelycapturesusecasesofinterest?

2.  HowcanabenchmarkcapturethefactthatRDFdataoriginatefromamultitudeofformats

! Structured:relationaland/orXMLdatatoRDF

! Unstructured3.  Howcanabenchmarkcapturethedifferentdataandquery

patternsandprovideaconsistentpictureforsystembehavioracrossdifferentapplicationsettings?

4.  Howcanoneselecttherightbenchmarkforhersystem,dataandworkload?


Overview•  IntroducingBenchmarks•  AshortdiscussionaboutLinkedData–  ResourceDescriptionFramework(DataModel)–  SPARQL(QueryLanguage)

•  BenchmarkingPrinciples&ChokePoints•  Benchmarks–  Synthetic–  Real–  BenchmarkGenerators

•  Sumup:whatdidwelearntoday?


AshortdiscussionaboutLinkedData-ResourceDescriptionFramework(DataModel)-SPARQL(QueryLanguage)


ResourceDescriptionFramework(RDF)•  W3CstandardtorepresentWebdataandmetadata•  genericandsimplegraphbasedmodel•  informationfromheterogeneoussourcesmergesnaturally:–  resourceswiththesameURIdenotethesamenon-informationresource(leadingtotheLinkedDataCloud)

•  structureisaddedusingschemalanguagesandisrepresentedasRDFtriples

•  WebbrowsersuseURIstoretrieveinformation


ResourceDescriptionFramework(RDF)•  AnRDFtripleisoftheform(s,p,o)where–  sisthesubject:theURIidentifyingthedescribedresource–  oistheobject:caneitherbeasimpleliteralvalueortheURIofanotherresource

–  pisthepredicate:theURIindicatingtherelationbetweensubjectandobject

•  AnRDFgraphisasetoftriples–  Canbeviewedasanodeandedge-labeleddirectedgraph–  Itispublishedindifferentformats

•  RDF-XML,turtle,n3triples,…


(dbpedia:Good_Day_Sunshine,dbpedia-owl:artist,dbpedia:The_Beatles)

Closetohowpeopleseetheworld(asagraph)!

AddingSemanticstoRDF•  RDFisageneric,abstractdatamodelfordescribingresources

intheformoftriples•  RDFdoesnotprovidewaysofdefiningclasses,properties,

constraints•  W3CStandardSchemaLanguages– RDFVocabularyDescriptionLanguage(RDFSchema-RDFS)todefineschemavocabularies

– OntologyWebLanguage(OWL)todefineontologies


AddingSemanticstoRDF

•  RDFVocabulariesaresetsoftermsusedtodescribenotionsinadomainofinterest

•  AnRDFtermiseitheraClassoraProperty– Objectpropertiesdenoterelationshipsbetweenobjects– Datatypepropertiesdenoteattributesofresources

•  RDFSdesignedtointroduceusefulsemanticstoRDFtriples•  RDFSSchemasarerepresentedasRDFtriples


"AnRDFVocabularyisaschemacomprisingofclasses,propertiesandrelationshipswhichcanbeusedfor

describingdataandmetadata"

RDFVocabularyDescriptionLanguage(RDFS)•  Typing:definingclasses,properties,instances•  Relationshipsbetweenclassesandproperties:subsumption•  Constraints:domainandrangeofproperties•  Inferencerulestoentailnew,inferredknowledge


Subject Predicate Object

t1 dbo:MusicalWork rdfs:subClassOf dbo:Album

t2 dbo:MusicalWork rdfs:domain dbo:artist

t3 dbo:MusicalWork rdfs:range dbo:march

t4 dbr:Seven_Seas_Of_Rye rdf:type dbo:MusicalWork

t5 dbo:Album rdf:type rdf:Class

RDFSInference


•  Usedtoentailnewinformationfromtheonethatisexplicitlystatedinthedataset–  Transitiveclosureacrossclassandpropertyhierarchies

–  Transitiveclosurealongthetypeandclass/propertyrelations

•  Twowaystoimplementit:Forward&BackwardReasoning–  ForwardReasoning:closureiscomputedatloadingtime–  BackwardReasoning:closureiscomputedontheflywhenneeded

(P1,rdfs:subPropertyOf,P2),(P2,rdfs:subPropertyOf,P3)

(P1,rdfs:subPropertyOf,P3)R1:

(C1,rdfs:subClassOf,C2),(C2,rdfs:subClassOf,C3)

(C1,rdfs:subClassOf,C3)R2:

(C1,rdfs:subClassOf,C2),(r1,rdf:type,C1)

(r1,rdf:type,C2)R2:

(P1,rdfs:subPropertyOf,P2),(r1,P1,r2)

(r1,P2,r2)R3:

RDFSInference

•  Transitiveclosurealongthetypeandclass/propertyrelations


(C1,rdfs:subClassOf,C2),(r1,rdf:type,C1)

(r1,rdf:type,C2)R2:


t1 dbo:MusicalWork rdfs:subClassOf dbo:Album

t2 dbo:MusicalWork rdfs:domain dbo:artist

t3 dbo:MusicalWork rdfs:range dbo:march


t5 dbo:Album rdf:type rdf:Class

t6 dbo:MusicalWork rdf:type rdf:Class

SPARQL:QueryingRDFData•  SPARQL:W3CStandardLanguageforQueryingLinked

Data•  SPARQL1.0(2008)onlyallowsaccessingthedata(query)•  SPARQL1.1(2013)introduces:–  QueryExtensions:aggregates,sub-queries,negation,expressionsintheSELECTclause,propertypaths,assignment,shortformforCONSTRUCT,expandedsetoffunctionsandoperators

–  Updates:•  Datamanagement:Insert,Delete,Delete/Insert•  Graphmanagement:Create,Load,Clear,Drop,Copy,Move,Add

–  Federationextension:Service,values,servicevariables(informative)


SPARQLQueries(1)•  BuildingBlockistheTriplePattern–  RDFtriplewithvariables

•  GroupGraphPatterns–  BuiltthroughinductiveconstructioncombiningsmallerpatternsintomorecomplexonesusingSPARQLoperators

•  Join-similartorelationaljoin

•  Union(UNION)–similartorelationalunion

•  Optional(OPTIONAL)operatorsontriplepatterns–similartorelationalleftouterjoin(introducesnegationinthelanguage)

•  Filteringconditions(FILTER)•  PatternsonNamedGraphs6/21/16 ESWC2016:AssessingtheperformanceofRDFEngines-DiscussingRDFBenchmarks 26

SPARQLQueries(2)•  Aggregates–  specifyexpressionsovergroupsofsolutions–  Asinstandardsettingsusedwhentheresultiscomputedoveragroupofsolutionsratherthanasinglesolution•  Example:averagevalueofasetofvalues,sumofaset

–  AggregatesdefinedinSPARQL1.1areCOUNT,SUM,MIN,MAX,AVG,GROUP_CONCAT,andSAMPLE.

–  SolutionsaregroupedusingtheGROUPBYclause–  PruningatgrouplevelisperformedwiththeHAVINGclause

•  AdditionalFeatures–  duplicateelimination(DISTINCT)–  orderingresults(ORDERBY)withanoptionalLIMITclause


SPARQLSemantics•  SPARQLsemanticsbasedonPatternMatching

– Queriesdescribesubgraphsofthequeriedgraph

–  SPARQLgraphpatternsdescribethesubgraphstomatch


IntuitivelyatriplepatterndenotesthetriplesinanRDFgraphthatareofaspecificform

TP1=(?album,dbpedia-owl:artist,dbpedia:The_Beatles)

TP2=(dbpedia_The_Beatles,?property,?object)

matchesallalbumsoftheBeatles

matchesallinformationaboutTheBeatles

SPARQLTypesofQueries•  SELECTreturnsorderedmulti-setofvariablebindings–  Bindings:mappingsofvariablestoRDFtermsinthedataset

–  SQL-LikeSyntax

•  ASKcheckswhetheragraphpatternhasatleastonesolution-returnsaBooleanvalue(true/false)

•  CONSTRUCTreturnsanewRDFgraphasspecifiedbythegraphtemplateoftheCONSTRUCTclauseusingthecomputedbindingsfromthequery’sWHEREclause

•  DESCRIBEreturnstheRDFgraphcontainingtheRDFdataabouttherequestedresource


SELECT?v1,?v2,…WHEREGraphPattern

QueryingRDFDatawithSPARQL(1)


PREFIXdc:<http://purl.org/dc/elements/1.1/>SELECT?titleWHERE{<http://example.org/book/book1>dc:title?title}

SimpleSELECTquery

PREFIXfoaf:<http://xmlns.com/foaf/0.1/>SELECT?name?mboxWHERE{?xfoaf:name?name.?xfoaf:mbox?mbox.}

JOINQuery

PREFIXfoaf:<http://xmlns.com/foaf/0.1/>SELECT?name?mboxWHERE{?xfoaf:name?name.OPTIONAL{?xfoaf:mbox?mbox}}

OPTIONALOperator

PREFIXdc:<http://purl.org/dc/elements/1.1/>SELECT?titleWHERE{?xdc:title?title.FILTERregex(?title,"^SPARQL")}

REGEXinFILTER

QueryingRDFDatawithSPARQL(2)


PREFIXfoaf:<http://xmlns.com/foaf/0.1/>PREFIXorg:<http://example.com/ns#>CONSTRUCT{?xfoaf:name?name}WHERE{?xorg:employeeName?name}

PREFIXfoaf:<http://xmlns.com/foaf/0.1/>ASK{?xfoaf:name"Alice"}

“Findthepeoplewholivein“PaloAlto” andhavefoundedorareboardmembersofcompaniesinthesoftwareindustry.Foreachsuchcompany,findtheproductsthatweredevelopedbyit,itsrevenue,andoptionallyitsnumberofemployees.“SELECT*WHERE{?xhome“PaloAlto” .

{?xfounder?y}UNION{?xmember?y}{

?yindustry“Software” .?zdeveloper?y.?yrevenue?n.OPTIONAL{?yemployees?m}.

}}

SPARQL1.1:SPARQLplusAggregates,Sub-

queries,Propertypaths,Negationandmore!

StoringandQueryingRDFdata•  Schemaagnostic–  triplesarestoredinalargetripletablewheretheattributesare(subject,predicateandobject)-“Monolithic”triple-stores

–  Butitcangetabitmoreefficient




t2 dbr:Starman_(song) rdf:type dbo:MusicalWork

t3 dbr:Seven_Seas_Of_Rye dbo:artist dbo:Queen

id URI/Literal

1 dbr:Seven_Seas_Of_Rye

2 dbr:Starman_(song)

3 dbo:MusicalWork

4 dbo:Queen

5 dbo:artist

6 rdf:type


1 6 3

2 6 3

1 5 4

RDF-3Xmaintains6indexes,namely,SPO,SOP,OSP,OPS,PSO,POS.Toavoidstorageoverhead,indexesarecompressed![NW09]

StoringandQueryingRDFdata•  schemaaware:

–  onetableiscreatedperpropertywithsubjectandobjectattributes(PropertyTables[Wilkinson06])



ID1 type BookType

ID1 title “XYZ”

ID1 author “Fox,Joe”

ID1 copyright “2001”

ID2 type CDType

ID2 title “ABC”

ID2 artist “Orr,Tim”


ID2 language “French”

ID3 type BookType

ID3 title “MNO”

ID3 language “English”

ID4 type DVDType

ID4 title “DEF”

ID5 type CDType

ID5 title “GHI”


ID6 type BookType


Subject Type Title copyright

ID1 BookType “XYZ” “2001”

ID2 CDType “ABC” “1985”

ID3 BookType “MNO” NULL

ID4 DVDType “DEF” NULL

ID5 CDType “GHI” “1995”

ID6 BookType NULL “2004”






Subject Title Author copyright

ID1 “XYZ” “Fox,Joe” “2001”

ID3 “MNO” NULL NULL

ID6 NULL NULL “2004”

Subject Title artist copyright

ID2 “ABC” “Orr,Tim” “1985”

ID5 “GHI” NULL “1985”




ID4 type DVDType

ID4 title “DEF”

Booktype

CDType

Property-classTable

Subject Object

… …

… …

ClusteredPropertyTable

Multi-ValueP

StoringandQueryingRDFdata•  VerticallypartitionedRDF[AMM+07]



ID1 type BookType

ID1 title “XYZ”



ID2 type CDType

ID2 title “ABC”




ID3 type BookType

ID3 title “MNO”


ID4 type DVDType

ID4 title “DEF”

ID5 type CDType

ID5 title “GHI”


ID6 type BookType


Subject Object

ID1 BookType

ID2 CDType

ID3 BookType

ID4 DVDType

ID5 CDType

ID6 BookType

Subject Object

ID1 “XYZ”

ID2 “ABC”

ID3 “MNO”

ID4 “DEF”

ID5 “GHI”

Subject Object

ID1 “2001”

ID2 “1985”

ID5 “1995”

ID6 “2004”

Subject Object

ID2 “Orr,Tim”

Subject Object

ID1 “Fox,Joe”

Subject Object

ID2 “French”

ID3 “English”

type

title

copyright

author

artist

language

Togetth

emosto

utofthisp

ar0cular

decompo

si0on

,acolum

n-oriented

DB

MSisrecommen

ded.

ComparisonofStorageTechniques[BDK+13]


company released

Google Android

Apple iPhonesubject object

Google Android

Google developer Android

subject predicate object

LarryPage born “1973”

LarryPage founder Google

Google HQ “MTV”

Google employees 50,000

Google industry Internet

Google industry Software

Google industry Hardware

Triplestore

person born founder

LarryPage “1973 Google

Type-orientedstore

company HQ employees

Google “MTV” 50,000


Google industry Internet

Google industry Software

Google industry Hardware

subject object

LarryPage “1973”

Predicate-orientedstore

subject object

Google “MTV”

subject object

Google Internet

Google Software

Google Hardware

subject object

LarryPage Google

subject object

Google 50,000

born

founder

HQ

employees

industry

industtry

LarryPage

“1973”

Google

Internet

Software

Hardware

“MTV”HQ

50,000employee

s

samplegraphColumnsareoverloaded

Traditionalrelationalcolumntreatment

Staticmixofoverloadedandnormalcolumns

developer

Schemadoesnotchangeonupdates

Schemamightchangeonupdates

StoringLinkedData:QueryProcessing•  SchemaAgnostic–  algebraicplanobtainedforaqueryinvolvesalargenumberofselfjoins

–  queriesarefavorablewhenthepredicateisavariable

•  HybridApproachandSchema-aware–  algebraicplancontainsoperationsovertheappropriateproperty/classtables(moreinthespiritofexistingrelationalschemas)

–  savesmanyself-joinsovertripletables–  ifthepredicateisavariable,thenonequeryperproperty/classmustbeexpressed


PurposeofanRDFQueryingBenchmark

•  TesttheperformanceofRDFstores–  Independentlyofunderlyingstorageengine–  Independentlyofunderlyinglogicalandphysicalschema–  Independentlyofthequeryactuallyexecutedintheengine•  SPARQLfornativestores•  SQL(SPARQLtranslatedtoSQL)forrelationalstores






BenchmarkingPrinciples&ChokePoints


WhyBenchmarks?•  PerformanceEvaluation–  Thereisnonosinglerecipeonhowtodoitright–  Therearemanywayshowtodoitwrong–  Thereareanumberofbestpracticesbutnobroadlyacceptedstandardonhowtodesignanddevelopabenchmark

•  Questionsasked:– Whatdata/datasetsshouldweuse?– Whichworkload/queriesshouldweconsider?– Whattomeasureandhowtomeasure?


BenchmarkCategories

•  Micro-benchmarks•  Standardbenchmarks•  Real-lifeapplications


MicroBenchmarks

•  Specialized,stand-alonepieceofsoftware•  Isolateoneparticularfunctionalityofalargersystem•  Indatabasesamicrobenchmarktestsasingledatabase

operator–  Selection,Join(andalltypesthereof),Projection,Aggregates,Sub-Queries,…


MicroBenchmarks:Advantages•  Veryfocused

–  Testaspecificoperatorofthesystem•  Controllabledata&workload

–  SyntheticandRealDatasets•  Differentvaluerangesandvaluedistributionandcorrelations(mostlyapplicabletostructureddata)

–  Variousdatasizestotacklescalabilityconcerns•  Queries

–  Workloadsofdifferentcomplexity&size•  Complexity:astothetypesofqueryoperatorsandpatterns•  Size:astothenumberofqueryoperatorsinvolved

–  Allowbroadparameterrange(s)


! Usefulfordetailed,in-depthanalysis! Lowsetupthreshold;! Easytorun

MicroBenchmarks:Disadvantages•  Neglectlargerpicturesincetheydonottestthewholesystem•  Donotconsidertheflowofcostsofspecificoperationstothe

costofthesystem•  Donotmeasuretheimpactofmicro-benchmarkonreal-life

applications•  Difficulttogeneralizetheresults•  Theresultsofmicro-benchmarkscannotbeappliedina

straightforwardmanner•  Micro-benchmarksdonotusestandardizedmetrics


StandardBenchmarks

•  Relational,ObjectOriented,ObjectRelationalDatabaseManagementSystems–  FamilyofTPCBenchmarksforrelationaldatabases

•  XML,XPath,XQuery,– Mbench,XBench,XMach-1,XMark,

•  GeneralComputing–  SPEC


StandardBenchmarks:Advantages&Disadvantages•  Advantages

–  Mimicreal-lifescenarios(respondtorealneeds)•  E.g.,TPCisabusinessorientedbenchmark

–  Publiclyavailable–  Welldefined–  Providescalabledatasetsandworkloads–  Metricsarewelldefined

•  Disadvantages–  Outdated(standardizationisalengthyprocess)

•  XQuerytookaround7yearstobecomeastandard•  TPCbenchmarkdefinitionisstillanongoingprocess

–  Verylargeandcomplicatedtorun–  Limiteddatasetvariation(targetaspecifictypeofdata)–  LimitedWorkload(focusesontheapplicationinmind)–  Systemsareoftenoptimizedforthebenchmark(s)


•  Managementandmethodologicalactivitiesperformedbyagroupofpeople– Management:Organizationalprotocolstocontroltheprocess– Methodological:principles,methodsandstepsforbenchmarkcreation

•  BenchmarkDevelopment–  Rolesandbodies:people/groupsinvolvedinthedevelopment–  Designprinciples:fundamentalrulesthatdirectthedevelopmentofabenchmark

–  Developmentprocess:seriesofstepstodevelopabenchmarkbasedonChokePoints

BenchmarkDevelopmentMethodology


ChokePoints:thesetoftechnicaldifficultiesthatforcesystemstoimprovetheirperformance

TheExampleStandardBenchmark:TPC•  TransactionProcessingCouncil(TPC)

–  non-profitcorporationfocusedondevelopingdata-centricbenchmarkstandardsanddisseminatingobjective,verifiableperformancedatatotheindustry

–  goalisto«create,manageandmaintainasetoffairandcomprehensivebenchmarksthatenableend-usersandvendorstoobjectivelyevaluatesystemperformanceunderwelldefinedconsistentandcomparableworkloads»[NPM+12]


Benchmark Explanation

TPC-C Focusesontransactions.

TPC-DI FocusesonETLprocesses

TPC-DS Decisionsupportsolutionsfor,butnotlimitedto,BigData.

TPC-E On-LineTransactionProcessing(OLTP)workload

TPC-H Decisionsupportbenchmark,adhocqueriesandconcurrentdatamodifications

TPC-VMS VirtualMeasurementSingleSystemSpecificationforrunningandreportingperformancemetricsforvirtualizeddatabases

TPC-xHS measureofhardware,operatingsystemandcommercialApacheHadoopFileSystemAPI

TPX-xV measuretheperformanceofserversrunningdatabaseworkloadsinvirtualmachines.

Acti

veT

PCB

enchm

ark

s(

2016)

BenchmarkDevelopmentProcess(1)•  DesignPrinciples[L97]


Principle Comment

Relevant Thebenchmarkismeaningfulforthetargetdomain

Understandable Thebenchmarkiseasytounderstandanduse

GoodMetrics Themetricsdefinedbythebenchmarkarelinear,orthogonalandmonotonic

Scalable Thebenchmarkisapplicabletoabroadspectrumofhardwareandsoftwareconfigurations

Coverage Thebenchmarkworkloaddoesnotoversimplifythetypicalenvironment

Acceptance Thebenchmarkisrecognizedasrelevantbythemajorityofvendorsandusers

BenchmarkDevelopmentProcess(2)•  BenchmarkingMetrics

–  Performance–  Price/Performance–  Energy/PerformanceMetrics:Energymetrictomeasuretheenergy

consumptionofsystemcomponents

•  TPCPricingspecification–  Providesconsistentmethodologiesforcomputingthepriceofthe

benchmarkedsystem,licensingofsoftware,maintenance,…


Benchmark Metrics

TPC-C TransactionRate(tpmC),PriceperTransaction($/tmpC)

TPC-E TransactionsperSecond(tpS)

TPC-H CompositeQueryperHourPerformanceMetric(QpH@Size),PriceperCompositeQueryperHourPerformanceMetric($/QpH@Size)

DesirableAttributesofaBenchmark:

•  “Agoodbenchmarkiswritteninahigh-levellanguage,makingitportableacrossdifferentmachines;isrepresentativeofsomeprogrammingstyleorapplication;canbemeasuredeasily;haswidedistribution[W90]”

•  “adomainspecificbenchmarkmustmeetfourimportantcriteria:relevance,portability,simplicity,scalability[G93]”

•  SixdesirableattributesforTPC-C[L97]:relevance,understandability,goodmetrics,scalability,coverage,acceptance

•  FivedesirableattributesinHuppler[H09]:relevance,repeatability,fairness,verifiability,economy

•  BigDataBenchmarking[1]:“asuccessfulbenchmarkshouldbesimpletoimplementandexecute,costeffective,timelyandverifiable”.


DesirableAttributesofaBenchmark:


DesignPrinciples:DesirableAttributesofaBenchmark


•  Relevant/Representative:basedonrealisticusecasescenariosandmustreflecttheneedsoftheusecase

•  Understandable/Simple:theresultsandworkloadareeasilyunderstandablebyusers

•  Portable/Fair/Repeatable:nosystembenefitsfromthebenchmark.Mustbedeterministicandprovidea«goldstandard»

•  Metrics:shouldbewelldefinedtobeabletoassessandcomparethesystems.

•  Scalable:datasetsshouldbeintheorderofbillionsof«objects»

•  Verifiable:allowverifiableresultsineachexecution

BenchmarkAttributes

relevant

representative

understandable

simple

portable

fair

repeatable

metrics

scalable

verifiable

DesignofBenchmarkWorkload[Grey93]

•  Designthequeriestotestspecificfeaturesofthequerylanguageortotestspecificdatamanagementapproaches

•  Basethequerymixonspecificrequirementsofrealworldusecases–  Leadstocomplexqueriesthatinvolvemany(different)languagefeatures


Micro-benchmarks

Domainspecificandstandardbenchmarks

DevelopmentProcess:ChokePoints•  Abenchmarkexposesasystemtoaworkloadandshouldidentify

thetechnicaldifficultiesofthesystemundertest

•  ChokePoints[BNE14]arethosetechnologicalchallengeswhoseresolutionwillsignificantlyimprovetheperformanceofaproduct

•  TPC-H:a20yearsoldbenchmark(supersededbyTPC-DS)butstillinfluentialusingbusiness-orientedqueriesandconcurrentmodifications

•  22queriescapturing(mostof)theaspectsofrelationalqueryprocessing

•  [BNE14]performedananalysisoftheTPC-Hworkloadandidentified28chokepointsgroupedinto6categories


ChokePointsàlaTPC-H•  CP1:AggregationPerformance

–  Orderedaggregation,smallgroup-bykeys,interestingorders,dependentgroup-bykeys

•  CP2:JoinPerformance –  Largejoins,sparseforeignkeys,richjoinorderoptimization,lateprojection

•  CP3:DataAccessLocality(materializedviews)–  Columnarlocality,physicallocalitybykey,detectingcorrelation

•  CP4:ExpressionCalculation–  RawExpressionArithmetic,ComplexBooleanExpressionsinJoinsand

Selections,StringMatchingPerformance

•  CP5:CorrelatedSub-queries–  Flatteningsub-queries,movingpredicatestoasub-query,overlapbetween

outer-andsub-query

•  CP6:ParallelismandConcurrency–  Queryplanparallelization,workloadmanagement,resultre-use


ChokePointsàlaRDF

ChokePoint Description

CP1:JOINORDERING

1.  Testsiftheenginecanevaluatethetrade-offsbetweenthetimespenttofindthebestexecutionplanandthequalityoftheoutputplan

2.  Teststheabilityoftheenginetoconsidercardinalityconstraintsexpressedbythedifferentkindsofschemaconstraints(e.g.,functionalandinversefunctionalproperties)

CP2:AGGREGATION

Aggregationsareimplementedwiththeuseofsub-selectsintheSPARQLquery;theoptimizershouldrecognizetheoperationsincludedinthesub-selectsandevaluatethemfirst.

CP3:OPTIONAL&NESTEDOPTIONALCLAUSES

Teststheabilityoftheoptimizertoproduceaplanwheretheexecutionoftheoptionaltriplepatternsisthelasttobeperformedsinceoptionalclausesdonotreducethesizeofintermediateresults.


ChokePointsinRDFBenchmarks


CP4:REASONINGTeststheabilityoftheenginetohandleefficientlyRDFSandOWLconstructsexpressedintheschema

CP5:PARALLELEXECUTIONOFUNIONS

Teststheabilityoftheoptimizertoproduceplanswhereunionsareexecutedinparallel

CP6:FILTERSTeststheabilityoftheenginestoexecuteasearlyaspossiblethosefilterexpressionstoeliminateapossiblylargenumberofintermediateresults

CP7:ORDERINGTeststheabilityoftheenginetochoosequeryplan(s)thatfacilitatetheorderingofresults

CP8:GEO-SPATIALPREDICATES

Teststheabilityofthesystemtohandlequeriesforgeospatialdata


ChokePointsinRDFBenchmarks


CP9:FULLTEXT Queriesthatinvolvetheevaluationofregularexpressionsondatavaluepropertiesofresources

CP10:DUPLICATEELIMINATION

Teststheabilityofthesystemtoidentifyduplicateentriesandeliminatethemduringthecreationofintermediateresults

CP11:COMPLEXFILTERCONDITIONS

Teststheabilityoftheenginetodealwithnegation,conjunctionanddisjunctionefficiently(i.e.,breakingthefiltersintoconjunctionoffiltersandexecutetheminparallel).


QueryCharacteristics


Characteristics

SimplefiltersUnboundpredicates

LIMIT REGEX CONSTRUCT

Complexfilters Negation ORDERBY UNION ASK

>=9TPs OPTIONAL DISTINCT DESCRIBE





ASurveyofRDFBenchmarksSyntheticBenchmarksRealBenchmarksBenchmarkGenerators


BenchmarkComponents•  Datasets•  Therawmaterialofthebenchmarkagainstwhichtheworkload

willbeevaluated•  Synthetic&RealDatasets

!  Synthetic:Producedwithadatagenerator(thathopefullyproducesdatawithinterestingcharacteristics)

!  Real:Widelyuseddatasetsfromadomainofinterest

•  QueryWorkload•  Setsofqueriesand/orupdatestoevaluatethesystemwith

•  Metrics•  Theperformancemetric(s)thatdeterminethesystemsbehavior


SyntheticRDFBenchmarks


LehighUniversityBenchmark(LUBM)[GPH05]•  BenchmarkintendedtofacilitatetheevaluationofSemantic

Webrepositories•  WidelyadoptedbythedataengineeringandSemanticWeb

communities

•  FocusesonevaluatingtheperformanceofqueryoptimizersandnotontologyreasoningasinDLsystems

•  Components:–  ScalableSyntheticdatagenerator– Ontologyofmoderatesizeandcomplexity–  Supportsextensionalqueries(i.e.,queriesthatrequestinstancesandnotonlyschemainformation)

– ProposesPerformancemetrics


LUBMUniv-BenchOntology•  Describesuniversitiesanddepartmentsandrelatedactivities•  ExpressedinOWLLite(tookintoconsiderationthe

limitationsofreasoningsystemsreg.completeness)


Statistics:!  43Classes!  32ObjectTypeProperties!  7DataTypeProperties! OWLLiteinverseOf,TransitiveProperty,

someValuesFrom,intersectionOf

LUBMDataGeneration(1)•  Syntheticallyproducedextensionaldatathatconformtothe

LUBMOntology•  DataaregeneratedusingtheUBA(Univ-BenchArtificialData

Generator)•  RandomandRepeatableDataGeneration•  Minimumunitofdatageneration:Universitythathas

departments,employees,courses•  Instancesofclassesandpropertiesarerandomlyproduced•  Tomakedatamorerealisticrestrictionsareapplied:–  «Minimum15andmaximum25departmentsperuniversity»–  «Undergraduatestudent/facultyratiobetween8and14inclusive»


LUBMDataGeneration(2)•  AssignmentofIdentifiersisdoneusingzero-basedindexes– University0,Department0,…

•  Datageneratedbythetoolarerepeatablefortheuniversities– Userentersaseedfortherandomnumbergeneratoremployedinthedatagenerationprocess

•  DatacreatedarerepresentedinOWLLite•  Configurableserializationandrepresentationmodel(RDF/

XMLin.owlfiles,DAML)


LUBMQueries(1)•  14RealisticQueries•  WritteninSPARQL1.0•  QueryDesigncriteria–  InputSize:•  proportionoftheclassinstancesinvolvedandentailedinthequerytothetotalinstancesinthedataset

– Selectivity:•  estimatedproportionoftheclassinstancesthatsatisfythequerycriteria•  dependsontheinputdatasetsize


LUBMQueries(2)– Complexity:• measuredonthebasisofthenumberofclassesandpropertiesinvolvedinthequery•  differentcomplexityforthesamequeryandfordifferentimplementations:relationalvsRDF

– Hierarchyinformation:•  classandpropertyhierarchiesareusedtoobtainallqueryanswers

–  Logicalinference:•  inferenceisrequiredtoobtainallqueryanswers


LUBMQueries(3):Characteristics


Characteristic Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14

Simplefilters

Complexfilters

>=9TPs

Unboundpredicates

Negation

OPTIONAL

LIMIT

ORDERBY

DISTINCT

REGEX

UNION

DESCRIBE

CONSTRUCT

ASK

SimpleSPARQLSELECTQueries

LUBMQueries(4):ChokePoints

# CP1 CP2 CP3 CP4 CP5 CP6 CP7 CP8 CP9 CP10 CP11

Q1

Q2 ✓

Q3 ✓

Q4 ✓ ✓

Q5 ✓

Q6 ✓

Q7 ✓

Q8 ✓

Q9 ✓

Q10 ✓

Q11 ✓

Q12 ✓ ✓

Q13 ✓

Q14


JoinOrderingMostcomplexquerycontains5joins

ReasoningFocusonsubClassandsubProperty

hierarchies

LUBMPerformanceMetrics(1)•  LoadTime:–  Timeneededtoparse,loadandreasonforadataset–  Focusesonpersistentstores

•  RepositorySize:–  Forpersistentstorageonly–  Thesizeofallfilesthatconstitutetherepository

•  QueryResponseTime:– Averagetimeforexecutingaquery10times(warmrun)


LUBMPerformanceMetrics(2)•  QueryCompletenessandSoundness:– Measuresthedegreeofcompletenessofaqueryanswerasthepercentageofentaileduniqueanswers

•  CombinedMetric:– Combinesqueryresponsetimewithanswercompletenessandanswersoundness

– Measuresthetrade-offbetweenqueryresponsetimeandcompletenessofresults•  Seehowreasoningaffectsqueryperformance

– Providesanabsoluterankingofsystems– Buthidesdetails!


SP2Bench[SHM+09]•  Proposesalanguagespecificbenchmarktotestthemost

commonSPARQLconstructs,operatorconstellationsandRDFaccesspatterns

•  Components:–  Scalablesyntheticdatagenerator•  CreationofDBLPdocumentsinRDFmimickingkeycharacteristicsoftheoriginalDBLPdataset•  ProduceddatasetscontainblanknodesandRDFcontainers

–  Supportsextensionalqueries(i.e.,queriesthatrequestinstancesandnotschemainformation)

– Proposesperformancemetrics


SP2BenchSchemaDBLP(1)

•  StudyofDBLPrealdata–  Determinetheprobabilitydistributionforselectedattributesperdocumentclassesthatformsthebasisforgeneratingclassinstances

–  Revealsthatonlyfewoftheattributesarerepeatedforthesameclass


<!ELEMENTdblp(article|inproceedings|proceedings|book|incollection|phdthesis|masterthesis|www)*><!ENTITY%field“author|editor|title|booktitle|pages|year|address|journal|volume|number|month|url|ee|cdrom|cite|publisher|note|crossref|isbn|series|school|chapter”><!ELEMENTarticle(%field)*><!ELEMENTinproceedings(%field)*>

ExtractDBLPDTD2008

SP2BenchSchemaDBLP(2)•  Probabilitydistributionforselectedattributesperdocument

classes

•  Additionalassumptionisthatattributesarenotdependent–  Existenceofanattributedoesnotdependonanother

•  UseBell-shapedGaussiancurvestoapproximateinputdata–  Typicallyusedtomodelnormaldistributions

•  Studiedthenumberofclassinstancesovertimeandmodeledthosewithapowerlawdistribution


Article Inproc. Proc. Book WWW

author 0.9895 0.9970 0.0001 0.8937 0.9973

cite 0.0048 0.0104 0.0001 0.0079 0.0000

editor 0.0000 0.0000 0.7992 0.1040 0.0004

isbn 0.0000 0.0000 0.8592 0.9294 0.0000

… … … … … …

SP2BenchDataGeneration


•  SyntheticallyproducedextensionaldatathatconformtotheDBLPSchema

•  Useofexistingexternalvocabulariestodescriberesourcesinauniformway–  FOAF(persons)–FriendofAFriend[FOAF],SWRC-SemanticWeb

forResearchCommunities(scientificpublications)[SWRC],DC–DublinCore[DC]

•  IntroduceblanknodesandRDFcontainers(rdf:Bag)tocaptureallaspectsoftheRDFdatamodel

•  DatagenerationtakesintoaccountdataapproximationasreflectedintheGaussiancurves

•  Datageneratortakesasinputeitherthetriplecount,oryearuptowhichthedataisgenerated–  Alwaysendingupinaconsistentstate!

•  Randomfunctionsarebasedonafixedseedmakingdatagenerationdeterministic

SP2BenchQueries(1):Characteristics•  17queries–  12mainqueriesandmodificationsthereof

•  Providedinnaturallanguage,inSPARQL1.0andSQLtranslationsarealsoavailable

•  Querydesigncriteria–  FocusonSELECTandASKSPARQLforms– AimatcoveringthemajorityofSPARQLconstructs(includingDISTINCT,ORDERBy,LIMIT,OFFSET)


SP2BenchQueries(2):Characteristics


Characteristic Q1 Q2 Q3abc Q4 Q5ab Q6 Q7 Q8 Q9 Q10 Q11 Q12abc

Simplefilters ✔ ✔ ✔ ✔Complexfilters ✔ ✔ ✔>=9TPs ✔ ✔ ✔ ✔ ✔Unboundpredicates

✔ ✔

Negation ✔ ✔

OPTIONAL ✔ ✔ ✔LIMIT ✔ORDERBY ✔ ✔DISTINCT ✔ ✔ ✔ ✔ ✔ ✔REGEX

UNION ✔ ✔ ✔DESCRIBE

CONSTRUCT

ASK ✔

SP2BenchQueries(3):ChokePoints


Q1 ✓

Q2 ✓ ✓

Q3 ✓

Q4 ✓ ✓ ✓

Q5 ✓ ✓ ✓

Q6 ✓ ✓ ✓ ✓

Q7 ✓ ✓ ✓ ✓

Q8 ✓ ✓ ✓ ✓ ✓

Q9 ✓ ✓

Q10

Q11 ✓

Q12 ✓ ✓ ✓


JoinOrdering:mostcomplexquerycontains8joins

Filters:mostcomplexquerycontains2filters

DuplicateElimination

SP2BenchPerformanceMetrics•  LoadingTime:–  timeneededtoparse,loadandreasonusingthetestedsystemforadataset

–  Focusesonpersistentstores•  «Per-query»performance:–  Performanceofeachquery

•  «Global»performance:–  Listthearithmeticandgeometricmeanofqueries

1.  Multiplytheexecutiontimeofall17queries2.  Penalizequeriesthatfailwith3600spenalty3.  Computethe17throotoftheresult

•  Memoryconsumption–  Highwatermarkofmainmemoryconsumption–  Averagememoryconsumptionofallqueries


BerlinSPARQLBenchmark(BSBM)[BS09][BSBM]•  Builtaroundane-commerceusecase•  Querymixemulatesthesearchandnavigationpatternsofauser

lookingforaproductofinterest•  Goals–  AllowthecomparisonofSPARQLenginesacrossdifferentarchitectures(relationaland/orRDF)

–  Challengeforwardandbackwardchainreasoningengines–  Focusesonanenterprisesettingwheremultipleclientsconcurrentlyexecuteworkloads

– MeasuresSPARQLqueryperformanceandnot(somuch)reasoning

•  Components–  Datagenerator:supportsthecreationofarbitrarilylargedatasets

–  TestDriver:executessequencesofSPARQLqueries6/21/16 ESWC2016:AssessingtheperformanceofRDFEngines-DiscussingRDFBenchmarks 82

BSBMSchema(1)•  E-commerceusecase:productsareofferedbyseveralvendors

andconsumerspostreviewsforthoseproducts

6/21/16 ESWC2016:AssessingtheperformanceofRDFEngines-DiscussingRDFBenchmarks 839..22

Reviewbsbm:reviewForrev:reviewerbsbm:reviewDatedc:titlerev:textbsbm:rating1[0..1]bsbm:rating2[0..1]bsbm:rating3[0..1]bsbm:rating4[0..1]

Producerrdfs:labelrdfs:commentrdf:typefoaf:homepagebsbm:country

ProductTyperdfs:labelrdfs:commentrdf:typerdfs:subClassOf[1..0]

ProductFeaturerdfs:labelrdfs:commentrdf:type

Productrdfs:labelrdfs:commentrdf:typebsbm:producerbsbm:productFeature[9..22]bsbm:productPropertyTextual1bsbm:productPropertyTextual2bsbm:productPropertyTextual3bsbm:productPropertyTextual4[0..1]bsbm:productPropertyTextual5[0..1]bsbm:productPropertyNumeric1bsbm:productPropertyNumeric2bsbm:productPropertyNumeric3bsbm:productPropertyNumeric4[0..1]bsbm:productPropertyNumeric5[0..1]

Offerbsbm:productbsbm:vendorbsbm:pricebsbm:validFrombsbm:validTobsbm:deliveryDaysbsbm:offerWebpage

Personfoaf:namefoaf:mbox_sha1sumbsbm:country

Vendorrdfs:labelrdfs:commentrdf:typefoaf:homepagebsbm:country

1..89

1

1..*

1..*

1..*

1

2..16

14..32

1

280..3730

2..37

1

BSBMSchema&DataCharacteristics(1)•  Everyproducthasatypefromaproducthierarchy•  ProductHierarchyisnotfixed(dependsonthedatasetsize)–  It’sdepthandwidthdependsonthechosenscalefactor–  Hierarchydepth–  Branchingfactorfor

•  rootlevel•  allotherlevelsis8

•  Producttypesareassignedavariablenumberofproductfeatures–  computedaslowerBoundandupperBoundwith

•  aa–  Setofpossiblefeaturesforagivenproducttypeistheunionofthetypeandallits“super-types”.


d =1+round(log10(n)) / 2n

bfr =1+ round(log10(n))

lowerBound = 35* i / (d *(d +1) / 2−1),upperBound = 75* i / (d *(d +1) / 2−1)

BSBMSchema&DataCharacteristics(2)•  Products,Vendors,Offers–  Productsthatsharethesametype,havealsothesamesetoffeatures

–  Foragivenproduct,itsfeaturesarechosenfromthesetofpossiblefeatureswithahard-codedprobabilityof25%

–  Normaldistributionwithameanofμ=50andstandarddeviationσ=16.6isemployedtoassociateproductswithproducers

–  Vendorsareassociatedtocountriesfollowinghard-codeddistributions

–  Sizeofoffersisn*20 aredistributedoverproductsfollowinganormaldistributionwith«fixedparameters»μ=n/2andσ=n/4

–  Offersaredistributedovervendorsfollowinganormaldistributionwith«fixedparameters»μ=2000andσ=667


BSBMSchema&DataCharacteristics(3)•  Reviews–  10timesthescalefactorn –  Datatypepropertyvalues(titleandtext)between50–300words

–  Upto4ratings,eachratingisarandomintegerbetween1and10

–  Eachratingismissingwithhard-codedprobability10%–  Distributedoverproductswithanormaldistributiondependingondatasetsizeandfollowingμ=n/2andσ=n/4

–  Numberofreviewsperreviewerfollowsnormaldistributionwithμ=20andσ=6.6

–  Reviewsaregenerateduntilallreviewsareassignedareviewer–  Reviewercountriesfollowthesamedistributionasvendorcountries


BSBMDataGeneration(1)•  SyntheticallyproducesinstancesofclassProductthatconformto

theBSBMSchema


Total#triples 250K 1M 2M 100M

#products 666 2,785 70,812 284,826

#productfeatures

2,860 4,745 23,833 47,884

#producttypes 55 151 731 2011

#producers 14 60 1422 5,618

#vendors 8 34 722 2,854

#offers 13,320 55,700 1,416,240 5,696,520

#reviewers 339 1432 36,249 146,054

#reviews 6,660 27,850 708,120 2,848,260

Total#instances 23,922 92,757 2,258,129 9,034,027

Indicativenumberofinstancesfordifferentdatasetsizes

BSBMQueries(1)•  12Queries•  Querymixisemulatessearchandnavigationpatternsofacustomer

lookingforaproduct•  BSBMqueriesaregiveninnaturallanguage,SPARQLandSQL


Query Description

Q1 Findproductsforagivensetofgenericfeatures

Q2 Retrievebasicinformationaboutaspecificproductfordisplaypurposes

Q3 Findproductshavingsomespecificfeaturesandnothavingonefeature

Q4 Findproductsmatchingtwodifferentsetsoffeatures

Q5 Findproductsthataresimilartoagivenproduct

Q6 Findproductshavingalabelthatcontainsaspecificstring

Q7 Retrievein-depthinformationaboutaproductincludingoffersandreviews

Q8 Givemerecentlanguagereviewsforaspecificproduct

Q9 Getinformationaboutareviewer

Q10 Getcheapofferswhichfulfilltheconsumer’sdeliveryrequirements

Q11 Getallinformationaboutanoffer

Q12 Exportinformationaboutanofferintoanotherschema

Characteristic Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12

Simplefilters ✔ ✔ ✔ ✔ ✔ ✔ ✔

Complexfilters ✔ ✔

>9TPs ✔ ✔ ✔ ✔ ✔

Unboundpredicates

✔

✔

Negation ✔

OPTIONAL ✔ ✔ ✔ ✔

LIMIT ✔ ✔ ✔ ✔ ✔ ✔

ORDERBY ✔ ✔ ✔ ✔ ✔ ✔

DISTINCT ✔ ✔ ✔

REGEX ✔

UNION ✔ ✔

DESCRIBE ✔

CONSTRUCT ✔

ASK

BSBMQueries(2):Characteristics


11JOINs,3OPTIONALclauses,

3Filters,1Unboundvariable

4OPTIONALclauses

BSBMQueries(3):ChokePoints



Q1 ✔ ✔ ✔ ✔

Q2 ✔Q3 ✔ ✔ ✔

Q4 ✔ ✔ ✔ ✔Q5 ✔ ✔ ✔ ✔Q6 ✔ ✔Q7 ✔ ✔ ✔Q8 ✔ ✔ ✔

Q9 ✔Q10 ✔ ✔ ✔ ✔Q11 ✔

Q12 ✔


Filters:mostcomplexquerycontains3filtersandmostcomplexfiltercontainsarithmeticexpressions

ResultOrdering

BSBM:PerformanceMetrics•  QueryMixesperHour(QMpH)– MeasuresthenumberofcompleteBSBMquerymixesansweredbyasystemundertestandforaspecificnumberofclientsrunningconcurrentlyagainstthesystemundertest

•  QueriesperSecond(QpS)– Measuresthenumberofqueriesofaspecifictypehandledbythesystemundertestinasecond

–  Calculatedbydividingthenumberofqueriesofaspecifictypewithinabenchmarkrunbythetotalexecutiontimeofthosequeries

•  LoadTime:–  TimetoloadthedatasetintheRDForrelationalrepositories

•  Includesthetimetocreatetheappropriatedatastructures&indices


SemanticPublishingBenchmark(SPB)•  DevelopedinthecontextofFP7EUProjectLDBC(2012-2015)•  LDBC’sgoals:– Developqueryingbenchmarksthatwillspurresearch&industryprogressinlarge-scalegraphandRDFdatamanagement•  scalability,storage,indexingandqueryoptimizationtechniquesforRDFandgraphdatabasesolutions•  quantitativelyandqualitativelyassessdifferentsolutionsforRDFdataintegration

–  Toestablishanindustry-neutralentity-LDBCfoundation-àlatheTransactionProcessingCouncil(TPC)


SemanticPublishingBenchmark(SPB)•  Industry-motivatedbenchmark–  Thescenarioinvolvesamedia/publisherorganizationthatmaintainssemanticmetadataaboutitsJournalisticassets

•  Components–  ScalableSyntheticDataGenerator•  CreationofinstancesofBBContologiesmimickingcharacteristicsoftheoriginalrealinputdatasets

–  Supportsextensionalqueries(i.e.,queriesthatrequestinstancesandnotschemainformation)

– WorkloadsimulatesconsumptionofRDFmetadata•  Concurrentreadandupdatequeries

– Proposesperformancemetrics6/21/16 ESWC2016:AssessingtheperformanceofRDFEngines-DiscussingRDFBenchmarks 93

SPBDesign:Requirements•  StoringandprocessingRDFdata–  StoringandisolatingdatainseparateRDFgraphs–  SupportingfollowingSPARQLstandards:

•  SPARQL1.1Protocol,Query,Update

•  SupportforSchemaLanguages–  SupportforRDFStoobtainthecorrectanswers–  OptionalsupportfortheRLprofileofWebOntologyLanguage(OWL2RL)inordertopasstheconformancetestsuite

•  LoadingdatafromRDFserializationformats–  N-Quads,TRIG,Turtle,etc.


SPBSchema:BBCOntologies(1)•  CoreOntologies:7ontologiesdescribebasicconceptsabout

entitiesandrelationshipsinthedomainofinterest–  BasicConcepts:CreativeWorks,Places,Persons,ProvenanceInformation,CompanyInformation,etc.


Thing CreativeWork

String

cwork:title

owl:Thing owl:sameAs

Theme Organisation

Event PlacePerson Programme

NewsItemBlogPost

cwork:tag

cwork:shortTitle

String

cwork:categoryxsd:Any

cwork:description

String

Audience

International Audience National Audience

cwork:audience

cwork:Format

Textual Format

VideoFormat

Interactive Format

Image Format Audio Format

PictureGallery Format

cwork:primaryFormat

xsd:dateTime

xsd:dateTime

cwork:dateModifiedcwork:dateCreated

cwork:Thumbnail

cwork:thumbnail

Thumbnail ThumbnailTypethumbnailType

StandardThumbnail

FixedSize66Thumbnail

CloseUpThumbnail

FixedSize266Thumbnail

FixedSize466Thumbnailp

rdfs:subClassOf rdfs:subPropertyOf

rdf:type

tag

about mentions

Stringcwork:altText

SchemaBBCSchema(2)•  DomainOntologies:3ontologiesdescribeconceptsand

propertiesrelatedtoaspecificdomain–  sports(competitions,events)–  politicsentities–  news(conceptsthatjournaliststagannotationswith)

•  Statistics–  74classes–  88datatypeproperties,28objecttypeproperties–  60rdfs:subClassOf(maximumdepth3),17rdfs:subPropertyOf(maximumdepth1)hierarchies

–  105rdfs:domainand115rdfs:rangeRDFSproperties–  8owl:oneOfclassaxioms,1oneowl:TransitivePropertyproperty.


SPB:Referencedatasets•  Collectionsofentitiesdescribingvariousdomains–  SnapshotsoftherealdatasetsofBBC

•  Footballcompetitionsandteams•  FormulaOnecompetitionsandteams•  UKParliamentMembers

–  Additionaldatasets•  GeoNames-Places,namesandcoordinates•  DBPedia–Persondata

–  ReferenceDatasetSize:25Mtriples


SPBDataGeneration(1):Process

1.   Loader–  Ontology&ReferenceData

2.   DataGeneratora.  Retrievesinstances

fromReferenceDatasetsb.  GeneratesCreativeWorks

accordingtopre-definedallocationsandmodels

c.  Writesgenerateddatatodisk


RDFRepository

BBCOntologies

ReferenceDatasets

Ontology&ReferenceDataSetLoader

CreativeWorksGenerator

SPARQLEndpoint

SPBDataGenerator

Datagenerationparameters

(1) (1)

(2.a)

GeneratedCWs

(2.c)

(1)

(2.d)

SPBDataGeneration(2)•  Producessyntheticdatathatmimicmostofthecharacteristicsofreal

worlddataprovidedbyBBC•  Input:Core&DomainOntologiesandReferencedatasets•  Output:

–  InstancesthatconformtoBBCcoreontologies(classCreativeWork)–  Instancesrefertoentitiesinthereferencedatasetsusingtheabout&

mentionsschemaproperties–  followsthe(user)pre-defineddistributionsofSPB’sDataGenerator

Tagg

edentities

01/2012 12/2012

clustering

correla1onsrandomdistribu1on


SPBOperationalPhases•  DataLoading

1.  Initialloadingofreferencedatasets•  BBCdatasetsenrichedwithDBPediaPersonandGeoNamesplacedata

2.  GenerationofCreativeWorks•  Parallelgeneration(multi-threadedandmulti-process)

3.  LoadingofCreativeWorksintheRDFrepository

•  RunningtheBenchmark1.  Warm-upphrase2.  RunthebenchmarkusingtheTestDriver3.  Runconformancetests(OWL2RL)[optional]


BenchmarkConfiguration•  DataGenerator–  AllocationoftagsinCreativeWorks

•  Correlationsofcreativeworkswithimportantentities(persons,places,events)•  ClusteringofCreativeWorksaroundmajor/minorevents

–  Sizeofgenerateddata(triples)–  Paralleldatageneration

•  TestDriver–  Distributionofqueriesinthequery-mix

•  editorialoperations(deletion/additionofRDFtriples)•  aggregateoperations(complexSPARQLqueries)

–  Numberofeditorial/aggregationagents–  DurationofWarm-upandBenchmarkphases–  Eachoperationalphasecanbeenabledordisabled


SPBBaseWorkloadQueries(2)


Characteristic Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12

Simplefilters ✔

Complexfilters ✔ ✔ ✔>9TPs ✔ ✔ ✔ ✔Unboundpredicates

✔

Negation

OPTIONAL ✔ ✔ ✔ ✔LIMIT ✔ ✔ ✔ ✔ ✔ ✔ORDERBY ✔ ✔ ✔ ✔ ✔ ✔ ✔DISTINCT ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔COUNT ✔

REGEX

UNION ✔ ✔ ✔GROYPBY ✔CONSTRUCT ✔ ✔ ✔ ✔ ✔

Evaluate(partsofthe)queryongraphs

SPBQueries(1)•  BaseandAdvancedWorkloads– BaseWorkload:12queries&updateoperations– AdvancedWorkload:24queries

•  WorkloadsbasedonrealqueriesusedbyBBCjournalistsduringtheireditorialoperations

•  Editorialagents–simulateeditorialworkperformedbyjournalists:–  Insert,Update,Delete

•  Aggregationagents–simulateretrievaloperationsperformedbyend-users


SPBBaseWorkloadQueries(3):ChokePoints



Q1 ✔ ✔ ✔ ✔ ✔Q2 ✔ ✔ ✔Q3 ✔ ✔ ✔ ✔ ✔ ✔Q4 ✔ ✔ ✔ ✔ ✔Q5 ✔ ✔ ✔ ✔ ✔Q6 ✔ ✔ ✔ ✔Q7 ✔ ✔Q8 ✔ ✔ ✔Q9 ✔ ✔ ✔Q10 ✔ ✔ ✔ ✔Q11 ✔ ✔ ✔ ✔ ✔Q12 ✔ ✔

Reasoningreg.class&propertyhierarchies

JoinOrdering

Ordering&DuplicateElimination

SPBPerformanceMetrics•  SPBPrimaryMetrics

•  QueryExecutionReport(1)

•  QueryExecutionReport(2)


QueryRateInteractivemix

(Queriespersecond)

QueryRateAnalyticalMix

(Queriespersecond)

UpdateRate(Operationsper

second)

DurationofBulkLoad(inms)

DurationofMeasurement

Window(inminutes)

#CompleteAnalyticalmixes

(persecond)

#CompleteInteractivemixes

(persecond)

#CompleteUpdate

Operations

Query ArithmeticMeanExecutionTime

MinimumExecutionTime

90th%AverageExecutionTime

#Executions

RealRDFBenchmarks


UniProt[RU09][UniprotKB]•  Comprehensive,high-qualityandfreelyaccessibleresourceof

proteinsequenceandfunctionalinformation•  UniProtSchema–  UniProtCoreVocabulary,BIBO(journals),ECO(evidencecodes),DublinCore(metadata)

–  UniProtCoreVocabulary:124classes,113Properties•  Datasetcontainsapproximately–  13billiontriples–  2.5billiondistinctsubjects–  2billiondistinctobjects

•  Queries–  Norepresentativesetofqueriesisoffered.–  [NW09]offersasetof8queriestotesttheRDF-3Xengine


UniProtQueries(1)[NW09]:Characteristics


Characteristic Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

Simplefilters

Complexfilters

>9TPs ✔ ✔ ✔ ✔ ✔ ✔Unboundpredicates

Negation

OPTIONAL

LIMIT

ORDERBY

DISTINCT

REGEX

UNION

DESCRIBE

CONSTRUCT

ASK

JoinOrderingRDF-3XaimsatoptimizingjoinprocessingforRDFdata

UniProtQueries(2)[NW09]:ChokePoints•  Focusondiscoveringoptimalorclosetooptimaljoinorders



Q1 ✔Q2 ✔Q3 ✔Q4 ✔Q5 ✔Q6 ✔Q7 ✔Q8 ✔ JoinOrdering:most

complexquerycontains12joins7queriescontainmorethan7joins

YAGO(YetAnotherGreatOntology)[SKW07]•  Highqualitymultilingualknowledgebasedderivedfrom

Wikipedia,WordNetandGeoNames•  Schema– WikipediaEntities,WordNetandGeoNamesConceptsandRelationships:associatesWordNettaxonomywithWikipediaCategorySystem

–  10millionschemaentities•  Dataset–  120milliontriplesaboutschemaentities–  2.625millionlinkstoDBPedia

•  Queries–  NorepresentativesetofqueriesisofferedbyYAGO–  [NW10]providesarepresentativesetof8queriesforRDF-3XEvaluation


YAGOQueries(1)[NW10]:Characteristics•  SimpleSELECTqueriesthatfocusonJoinordering,negation

andduplicateelimination


Characteristic A1 A2 A3 B1 B2 B3 C1 C2

Simplefilters ✔

Complexfilters

>9TPs ✔

Unboundpredicates

Negation ✔ ✔ ✔

OPTIONAL

LIMIT

ORDERBY

DISTINCT ✔ ✔ ✔ ✔ ✔

REGEX

UNION ✔

YAGOQueries(2)[NW10]:ChokePoints•  Queriesfocusmostlyondiscoveringoptimalorclosetoquery

evaluationplans,includingnegationinfiltersandduplicateelimination.

• 



A1 ✔

A2 ✔

A3 ✔ ✔ ✔ ✔

B1 ✔ ✔ ✔

B2 ✔ ✔

B3 ✔ ✔ ✔

C1 ✔ ✔ ✔ ✔

C2 ✔ ✔JoinOrdering:mostcomplexquerycontains8joins

allqueriescontainmorethan5joins

BartonLibrary[Barton]•  DatafromtheMITSimileProjectthatdevelopstoolsforlibrarydata

management–  containsrecordsthatcomposeanRDF-formatteddumpoftheMIT

LibrariesBartoncatalog–  convertedfromrawdatastoredinanoldlibraryformatstandard

calledMARC(MachineReadableCatalog).•  Schema

–  CommontypesincludeRecordandItem,thelatterbeingassociatedwithinstancesoftypePersonandwithinstancesofDescription.

–  PrimitivetypesincludeTitleandDate.•  Dataset

–  Approximately45millionRDFtriples•  Queries

–  NorepresentativequeriesprovidedwiththeBartonLibraryDataset–  [Abadi07]providesaworkloadof7queries([NW10]inSPARQL)


BartonQueries(1)[NW10]:Characteristics


Characteristic Q1 Q2 Q3 Q4 Q5 Q6 Q7

Simplefilters ✔ ✔ ✔ ✔

Complexfilters

>9TPs

Unboundpredicates

Negation ✔

OPTIONAL

LIMIT

ORDERBY

DISTINCT ✔ ✔ ✔

REGEX

UNION ✔

BartonQueries(2)[NW10]:ChokePoints•  Queriesfocusmostlyondiscoveringoptimalorcloseto

optimalqueryevaluationplans,includingnegationinfiltersandduplicateelimination.

• 



Q1 ✔

Q2 ✔ ✔

Q3 ✔

Q4 ✔

Q5 ✔ ✔

Q6 ✔ ✔ ✔ ✔

Q7 ✔


LinkedSensorDataset[PHS10]•  Expressivedescriptionsofapproximately20,000weather

stationsintheUS•  dividedupintomultiplesubsets,thatreflectweatherdatafor

specifichurricanesorblizzardsfromthepast(focusonhurricaneIke)

•  Schema–  Containsinformationabouttemperature,precipitation,pressure,wind,speed,humidity

–  ContainslinkstoGeoNamesandlinkstoobservationsprovidedbyMesoWest(meteorologicalserviceintheUS)

•  Dataset– morethan1billiontriples

•  Queries–  Norepresentativesetofqueriesisoffered.


WordNet[WordNet]•  LargelexicaldatabaseofEnglish,developedunderthe

directionofGeorgeA.Miller(Emeritus).•  Schema–  Nouns,verbs,adjectivesandadverbsaregroupedintosetsofcognitivesynonyms(synsets),eachexpressingadistinctconcept.

–  Synsetsareinterlinkedbymeansofconceptual-semanticandlexicalrelations.Theresultingnetworkofmeaningfullyrelatedwordsandconceptscanbenavigatedwiththebrowser.

•  Dataset–  Approximately1.9milliontriples(300MB).

•  Queries–  Norepresentativequeryworkload


PublishingTPC-HasRDF[TPC-H]•  Benchmarkcanbeusedbydecisionsupportsystemsthat

examine

–  largevolumesofdata,executequerieswithahighdegreeofcomplexity,andprovideanswerstocriticalbusinessquestions

•  Benchmarkprovidesasuiteofbusinessorientedad-hocqueriesandconcurrentdatamodifications

•  Queriesandthedatapopulatingthedatabasehavebeenchosentohavebroadindustry-widerelevance

•  UsetheDBGENTPC-HgeneratortogenerateaTPC-Hrelationaldataset

•  UsetheD2RtoolorotherrelationaltoRDFtooltoconverttherelationaldatasettotheequivalentRDFone.

•  TPCSQLqueriesaretranslatedtoequivalentSPARQLqueries6/21/16 ESWC2016:AssessingtheperformanceofRDFEngines-DiscussingRDFBenchmarks 118

BenchmarkGenerators


DBPediaSPARQLBenchmark(DBSB)[MLA+14]•  GenericMethodologyforSPARQLBenchmarkCreation•  Basedon–  Flexibledatagenerationthatmimicsaninputdatasource–  Query-logmining–  Clusteringofqueries–  SPARQLqueriesfeatureanalysis

•  Methodologyisschemaagnostic–  DemonstratedusingDBPediaKB

•  ProposedapproachappliedonvarioussizesoftheDBPediaKnowledgeBase

•  BenchmarkproposesqueryworkloadbasedonrealqueriesexpressedagainstDBPedia


DBSBDataGeneration(1)•  Workingassumptions

1.  Outputdatasetshouldhavesimilarcharacteristicsasinputdataset

• Numberclasses,properties,valuedistributions,taxonomicstructures(hierarchies)

2.  Varyingoutputdatasetsizes3.  Characteristicssuchasin-,out-degreeofnodesin

datasetsofvaryingsizesshouldbesimilar

4.  Easilyrepeatabledatagenerationprocess


DBSBDataGeneration(2)•  Idea

1.   Largedatasetsproducedby•  Duplicatingalltriplesandchangingtheirnamespace

2.   Smallerdatasetsproducedby•  Removingtriplesinawaythatwouldpreservethe

propertiesoftheoriginalgraph•  Usingaseedbasedmethodbasedontheassumptionthata

representativesetofresourcesisobtainedbysamplingacrossclasses1.  Foreachselectedelementinthedataset,itsconcise

bounddescription(CBD)isretrievedandaddedinthequeue

2.  Processisrepeateduntilthenumberoftriplesisreached


DBSBQueryAnalysis(1)•  Goalistodetectprototypicalqueriesthatweresenttoa

DBPediaSPARQLendpointusingsimilaritymeasures–  Stringsimilarityandgraphsimilarity

•  Idea:4-stepqueryanalysisandclusteringapproach1.  Selectqueriesexecutedfrequentlyontheinputdata2.  Stripcommonsyntacticconstructs(namespace,prefixes)3.  Computequerysimilarityusingstringmatching4.  Computequeryclustersusingasoftgraphclustering

algorithm•  Clustersusedtodevisethebenchmarkquerygenerationpatterns


DBSBQueryAnalysis(2)•  QuerySelection

1.  UseDBPediaSPARQLQuerylog(31.5millionqueriesina3monthperiod)

2.  Reducetheinitialsetofqueriesbyconsidering•  QueryVariations:useastandardwaytonamevariablestoreducedifferencesamongqueries(promotingqueryconstructssuchasDISTINCT,REGEX)•  QueryFrequency:discardquerieswithlowfrequencysincetheydonotcontributetotheoverallqueryperformance

–  Result:35,965queries3.  StringStripping:removeallSPARQLkeywordsandcommon

prefixes4.  SimilarityComputation:computethesimilarityofthestripped

queries


DBSBQueryAnalysis(3)•  QuerySelection(cont’d)

4.  SimilarityComputation•  Reducethetimeofbenchmarkcompilation,useLIMES[NS11]framework

•  UsetheLevenshteinstringsimilaritymeasure,0.9threshold•  Reduceby16.6%thenumberofcomputationsrequiredbycomputingtheCartesianproductofqueries

5.  Clustering•  Applygraphclusteringtothequerysimilaritygraphof(4)•  Goalistoidentifysimilargroupsofqueriesoutofwhichprototypicalquerieswillbegenerated

•  UseBorderFlow[NS09]algorithmthatfollowsaseed-basedapproach

•  Obtain12272clusters,24%containasinglequery•  Selecttheclusterswith>5queries


DBSBQueryGeneration(1)•  SelectthemostinterestingSPARQLqueries

–  WhicharethemostfrequentlyaskedSPARQLqueries–  WhichofthosequeriescoverthemostSPARQLfeatures

•  SPARQLFeatures–  Overallnumberoftriplepatterns

•  Testtheefficiencyofjoinoperations(CP1)–  SPARQLpatternconstructors(UNION&OPTIONAL)

•  HandleparallelexecutionofUnions(CP5)•  PerformOPTIONALsaslateaspossibleinthequeryplan(CP3)

–  Solutionsequences&modifiers(DISTINCT)•  Efficiencyofduplicationelimination(CP10)

–  Filterconditionsandoperators(FILTER,LANG,REGEX,STR)•  Efficiencyofenginestoexecutefiltersasearlyaspossible(CP6)


DBSBQueryGeneration(2)•  25queriesareselected–  Foreachofthefeatures,manuallyselectthepartofthequerytobevaried(IRIorfiltercondition)

–  Variabilityofquerytemplate(s)forthechosenvaluesissufficientlyhigh(>=1000perquerytemplate)


Methodensuresthat•  Executedqueriesduringthebenchmarkdiffer

•  Alwaysreturnnonemptyresults

ApplesandOranges[DKS+11]•  Proposestructurednesstocharacterizedatasets

–  ThelevelofstructurednessofadatasetD,withrespecttoatype(class)T,isdeterminedbyhowwelltheinstancesofT,conformtotypeT

–  IfeachinstanceofThasthepropertiesdefinedinT,thenthedatasethashighstructurednesswithrespecttoT


0

1

2

3

4

5

6

name office ext major GPA

OC(p,I(T,D))

OC(p,T)foreachpropertypofT

0

1

2

3

4

5

6

name office ext major GPA

Highlystructureddataset

•  allinstanceshavethenameattribute•  ext&GPApropertiesencounteredin

50%oftheinstances•  οfficepropertyfoundin20%oftheinstances•  majorpropertyin10%oftheinstances

•  allinstanceshaveallattributes

ApplesandOranges[DKS+11]•  Oneofthekeyconsiderationswhiledeciding:–  appropriatedatarepresentationformat(e.g.,relationalforstructuredandXMLforsemi-structureddata)

–  organizationofdata(e.g.,dependencytheoryandnormalformsfortherelationalmodel,andXML).

–  dataindexes(e.g.,B+-treeindexesforrelationalandnumberingscheme-basedindexesforXML).

–  dataquerying(e.g.,usingSQLfortherelationalandXPath/XQueryforXML).


Inotherwords,structurednesspermeateseveryaspectofdatamanagement

ApplesandOranges[DKS+11]


0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Highlystructured

datasets

(relationa

llike)

Lessstruc

turedda

tasets

SyntheticDatasets

RealDatasets

ApplesandOranges[DKS+11]

Someimportantobservations:•  SinceTPC-Hisarelational

dataset,itshouldhavehighstructuredness.

•  Thereisadifferencebetweensyntheticandandrealdatasets.

•  Syntheticarefairlystructuredandrelational-like

•  Realdatasetscoverthewholespectrumofstructuredness.


0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Structuredness of datasets

ExistingRDFstoresaretestedandcomparedagainsteachotherwithrespecttodatasetsthatarenotrepresentativeof

mostrealRDFdata.

ApplesandOranges[DKS+11]•  Nothingcanbetterrepresentdatathanthedataitself!•  Idea:Turneverydatasetintoabenchmark

1.  Noneedtosyntheticallygeneratevalues•  Usetheactualdatavaluesinthedataset

2.  Noneedtosyntheticallygeneratequeries.•  Thequeriesthatareknowntoruninyourdatacanbe

usedinthebenchmark.3.  Butweneedtocoverthestructurednessspectrum•  togetdataascloseaspossibletotherealworlddata•  toseehowthesystemsperformwhendatagoesfrom

verystructuredtolessstructured


CountingCoins[DKS+11]

•  StartwithadatasetwithsizeSandCH=0.5

•  AimforadatasetwithsizeS’andCH’,whereS>S’andCH>CH’.

Process:

•  Assignacointoeachtriple(s,p,o)andcomputetheimpactinCHofitsremoval

–  Theremovalwillimpactthesizeby1.Example:Consider(person1,ext,x5304).RemovingthetriplefromDgivesadatasetwithCH(T,D)=0.467.Thereforethecoin(person1,ext,x5304)=0.5–0.467=0.033.

•  Formulate(automatically)anintegerprogrammingproblemwhosesolutionswilltellushowmanycoinstoremovetoachievethedesiredcoherenceCH’andsizeS’.



person0 name Eric

person0 office BA7430

person0 ext x4401

person1 name Kenny



person1 ext x5304

person2 name Kyle

person2 ext x6281

person3 name Timmy

person3 major C.S.

person3 GPA 3.4

person4 name Stan

person4 GPA 3.8

person5 name Jimmy

person5 GPA 3.7 Oneofthefewoccasionsinlifewherehavingtoomanycoinsisundesirable…

Technicalchallengesinproblemformulation•  Computecoinswhichrepresenttheimpactonstructuredness

ofremovingalltripleswithsubjectsthatareinstancesofatypeTwithpropertiesequaltop–  Thereforeonecoinforeachtype/propertycombination.

•  Addconstraintsthatsetlowerandupperboundsonthenumberofcoinsthatcanberemovedsoasnottocompletelyremoveapropertyfromatype.

•  Addconstraintswhichguaranteethatnotallinstancesofatypeareremoved.

•  Todealwemulti-valuedproperties,weaddconstraintsthatintroducearelaxationparameterρ–  requiredbecauseoftheapproximationbyusingtheaveragenumberoftriplespercoin.


WaterlooSPARQLDiversityTestSuite[AHO+14]•  StressexistingRDFenginestorevealawiderrangeofquery

requirementsasestablishedbywebapplications•  Contributions– Definitionof2classesofqueryfeaturesusedtoevaluatethevariabilityofworkloadsanddatasets•  Structural(e.g.,numberoftriplepatterns)•  Data-driven(affectselectivityandresultcardinality)

–  In-depthanalysisofexistingSPARQLbenchmarksusingthestructuralanddata-drivenfeatures

– WatDivTestSuitetostressexistingRDFenginestorevealawiderrangeofqueryrequirements


WatDivStructuralFeatures(1)1.  TriplePatternCount

–  NumberoftriplepatternsinSPARQLGraphPatterns2.  JoinVertexCount

–  NumberofRDFterms(IRIs,literals,blanknodes)andvariablesthataresubjectsorobjectsofmultipletriplepatterns

3.  JoinVertexDegree–  Thedegreeofajoinvertexvisthenumberoftriplepatternswhose

subjectorobjectisv


SP2BenchQ5aSELECTDISTINCT?person?nameWHERE{?articlerdf:typebench:Article.?articledc:creator?person.?inprocrdf:typebench:Inproceedings.?inprocdc:creator?person2.?personfoaf:name?name.?person2foaf:name?name2FILTER(?name=?name2)}

TripleCount

JoinVertices

JoinVertexCount

JoinVertexDegree

6 ?article,?inproc?person,?person2

10 ?article:2,?inproc:2?person:2,?person2:2

WatDivStructuralFeatures(2)•  JoinVertexDegree&Countprovideagoodcharacterizationof

thestructuralcomplexityofaquery–  Numberoftriplepatternsdoesnotproperlycharacterizethequery:twoquerieswiththesamesetoftriplepatternscanhavedifferentstructures


?n

?m ?x

?l

C

E

?k

A

?y

?b

?z

?d ?o

Linearquery

?c

D D ?x

?b

B

?z

C

?w

D

?b

E

?w

Snowflakequery

?y

?b

?x

B

E

A D

?z

C

?c

Starquery

?m ?f

G

WatDivStructuralFeatures(3)•  JoinVertexType

–  PlayanimportantroleinthebehaviorofRDFenginestodetermineefficientqueryplans•  E.g.,starqueriespromoteefficientmergejoins

•  3(mutuallynon-exclusive)typesofjoinvertices–  Vertexx oftypeSS+ ifforalltriplepatterns(s,p,o)*, x isthesubject–  Vertexx oftypeOO+ ifforalltriplepatterns(s,p,o)*, x istheobject–  Vertexx oftypeSO+ ifforalltriplepatterns(s,p,o)*, (s’,p’,o’) x=s & x=o’


?n

?m ?x

?l

C

E

?k

?m type SS+

?x B

?z

C

?w

?x type OO+

?c

D D ?x

?b

B

?z

C

?w

?x type SO+

*Triplepa8erns(s,p,o) areincidentonx

WatDivData-drivenFeatures(1)•  Asystem’schoiceonthemostefficientqueryplandependson

–  (a)thecharacteristicsofthedatasetand–  (b)thequery

•  Ifthesystemreliesonselectivityestimationsandresultcardinality,thesamequerywillhaveadifferentqueryplanfordataset(s)ofdifferentsizes

•  Differentcases:– Querieshaveadiversemixofresultcardinalities

–  Sometriplepatternsareveryselective,othersarenot

– Alltriplepatternsareequallyselective


WatDivData-drivenFeatures(2)•  ResultCardinalityCARD(Ā,G)–  thenumberofsolutionsintheresultoftheevaluationofagraphpatternĀ = <A, F> overgraphG

•  FilterTriplePatternSelectivity(f-TPSelectivity)SELFG (tp)

–  theratioofdistinctsolutionmappingsofatriplepatterntptothesetoftriplesingraphG

•  Measures

1.  Resultcardinality

2.  Mean&standarddeviationoff-TPselectivitiesoftriplepatterns•  Importantfordistinguishingquerieswhosetriplepatternsarealmostequallyselectivefromquerieswithvaryingf-TPselectivities


WatDivData-drivenFeatures(3)•  ResultCardinality&f-TPselectivityarenotsufficient

–  Intermediatesolutionmappingswillnotmakeittothefinalresult(e.g.,duetofiltersormorerestrictivejoins)

–  Theoverallselectivityofagraphpatterncanbedeterminedbyasingleveryselectivetriplepattern

•  Run-timeoptimizationtechniques(e.g.,side-waysinformationpassing)toearlypruneintermediateresults

•  Introduce2featurestocaptureabovecases1.  BGP-Restrictedf-TPselectivity

2.  Join-Restrictedf-TPselectivity


WatDivData-DrivenFeatures(4)•  BGP-Restrictedf-TPselectivitySELF

G (tp|Ā)•  assesseshowmuchatriplepatterncontributestotheoverall

selectivenessofthequery

•  fractionofdistinctsolutionmappingsforatriplepatternthatarecompatiblewithsomesolutionmappinginthequeryresult.

•  Join-restrictedf-TPselectivitySELF G (tp|x)•  assesseshowmuchafilteredtriplepatterncontributestothe

overallselectivenessofthejoinsthatitparticipatesin

•  forxajoinvertexandtp atriplepatternincidentonx, thex-restrictedf-TPoftp overgraphGisthefractionofdistinctsolutionmappingscompatiblewithasolutionmappinginthequeryresultofthesub-querythatcontainsalltriplepatternsincidenttox


WatDivTestSuite(1)•  Components:DataGeneratorandQueryGenerator•  DataGenerator–  Allowsuserstodefinetheirowndatasetcontrolling

•  Entitiestoinclude•  TopologyofthegraphsallowingonetomimictherealtypesofdatadistributionsintheWeb– «well-structuredness»ofentities– probabilityofentityassociations– cardinalityofpropertyassociations

–  Important:Instancesofthesameentitydonothavethesamesetofattributes:breakingthe«relationalnature»ofpreviousRDFbenchmarks


WatDivTestSuite(2)•  QueryTemplateGenerator–  User-specifiednumberoftemplates–  Userspecifiedtemplatecharacteristics

•  Numberoftriplepatterns•  Typesofjoinsandfiltersinthetriplepatterns

–  TraversestheWatDivschemausingarandomwalkandgeneratesasetofquerytemplates

•  QueryGenerator–  Instantiatesthequerytemplateswithterms(IRIs,literalsetc.)fromtheRDFdataset

–  User-specifiednumberofqueriesproduced


WatDivTestSuite(3)•  QueryTemplateGenerator–  RandomWalkonaninternalrepresentationoftheschema

•  Entitytypesintheschemacorrespondtographvertices•  Relationships(i.e.,objecttypeproperties)aregraphedges•  Verticesareannotatedwithdatatypeproperties(i.e.,attributes)

–  ProducesasetofBasicGraphPatternswithamaximumntriplepatternswithunboundobjectsandsubjects

–  kuniformlyrandomlyselectedsubjects/objectsarereplacedwithplaceholders

–  PlaceholdersarereplacedwithactualRDFtermsrandomlyretrievedfromthedataset


ComparisonofWatDivwithotherRDFBenchmarks


Copy

right[A

HO+1

4]

•  QueryWorkload–  Largerangeofqueries

•  Meanjoinvertexdegreedistributedamong2and10–  JoinVertexTypes:

•  18%ofqueriesarestarjoins,4.4%inDBSB•  61.3%ofqueriesarepathqueries,5.4%inDBSB

ComparisonofWatDivwithotherRDFBenchmarks


Copy

right[A

HO+1

4]

•  Data-DrivenFeatures–  DBSBandBSBMcovertheendsofthespectrumofmeanJoin-Restrictedf-

TPselectivityvalues–  WatDivcoversthefullspectrumofRestrictedf-TPselectivityvalues–  WatDivcoversalowerrangeofvaluesformeanf-TPselectivitywhen

comparedtoDBSB

GeneralRemarks•  comparabletoDBSB•  morediversethanLUBM,SP2BenchandBSBM

FEASIBLE[SNM15]

•  Proposesafeature-basedbenchmarkgenerationapproachfromrealqueries–  Structure-based–  Data-drivenbased

•  ApproachissimilartoWatDivTestSuite•  Novelsamplingapproachforqueriesbasedonexemplarsand

medoids•  ProposeSELECT,ASK,CONSTRUCTandDESCRIBESPARQL

queries


FEASIBLEQueryFeatures•  NumberofTriplePatterns•  NumberofJoinVertices–  Distinguishingbetween«star»,«path»,«hybrid»and«sink»vertices

•  JoinVertexDegree–  Sumofincomingandoutgoingedgesofthevertex

•  TriplePatternSelectivity–  Ratiooftriplesthatmatchthetriplepatternoveralltriplesinthedataset


o1

x o2

p1

p2

x y p1 p2 z

Starvertex:x Pathvertex:x Hybridvertex:x

o1 x

o2

p1

p2 y

z

Sinkvertex:x

x

y

z

FEASIBLEBenchmarkGeneration•  3-stepbenchmarkgeneration•  Data-setCleaning–  Leadstopracticallyreliablebenchmarks

•  NormalizationofFeatureVectors–  Queryselectionprocessrequiresdistancesbetweenqueriestobecomputed

–  Normalizethequeryrepresentationssothatallqueriesareinaunithypercube

•  QuerySelection–  Basedontheideaofexemplars[NS11]


FEASIBLEBenchmarkGeneration

•  DatasetCleaning–  Removeerroneousandzero-resultqueriesfromthesetofrealqueriesusedtogeneratethebenchmark

–  Excludeallsyntacticallyincorrectqueries–  Attach9SPARQLoperators(UNION,DISTINCT,OPTIONAL,..)and7queryfeatures(joinvertices,joinvertexcountetc.)toeachofthequeries


FEASIBLEBenchmarkGeneration

•  NormalizationofFeatureVectors–  Queriesaremappedtoavectoroflength16whichstoresthequeryfeatures•  ForbinarySPARQLclauses(e.g.,UNIONiseitherusedornotused),storevalue1.Elsestorevalue0•  Allnon-binaryfeaturevectorsarenormalizedbydividingtheirvaluewiththeoverallmaximumvalueinthedataset•  Queryrepresentationsareassociatedwithvaluesbetween1and0


FEASIBLEBenchmarkGeneration•  QuerySelection

–  GivenanumberofNqueriestoselectasbenchmarkqueries–  AsetofcleanedandnormalizedqueriesL, |L| >>> |N| –  Computeanl-sizepartitionofqueriessuchthat

•  Theaveragedistancebetweentwopointsin2differentelementsinthepartitionishighand

•  Theaveragepointswithinapartitionissmall•  Selectthepointthatisclosetotheaverageofeachpartitionandincludeitinthebenchmark

–  Implementedby•  Selectingexemplars(pointsthatrepresentaportionofthespace)thatareasfaraspossiblefromeachother

•  PartitioningL bymappingeverypointofL tooneoftheseexemplarstocomputeapartitionofthespace

•  Selectingthemedoidofeachofthespacepartitionsasaqueryinthebenchmark






ConcludingWhathavewelearnedtoday?Whatshouldwedonext?


Whatdidwelearn?

•  Whyweneedbenchmarking•  Whataretheprinciplesandmethodsunderlying(orshouldbe

followed)duringbenchmarkdesign– ChokePointsdesignisthenewandinterestingtrend

•  Whataretheexistingcategoriesofbenchmarks–  Synthetic– Real– Benchmarkgenerationframeworks


Whatshouldwedonext

•  NostandardqueryingbenchmarksforRDFengines(àlaTPC)•  Nobenchmarksthatare100%RDF-oriented!– Benchmarkstendtobemorerelationallike

•  Benchmarkframeworksaregoodinprovidingmorerealisticdatasetsandworkloads– Beingschemaanddomainagnosticmakeseasierforpeopletousethosetoolsfortheirusecaseandfortheengineofinterest

–  Beinghighlyparameterizedsincetheyallowahighdegreeofvariabilityinthedataandworkloads


Whatshouldwedonext•  NobenchmarksthatconsidercomplexandexpressiveOWL

ontologyconstructs!–  36.4%oftheLODClouddatasourcesconsiderOWLontologies–  Timetodefinebenchmarksthatgobeyondtestingconformance

–  TimetodefineRDFbenchmarkswithadatabaseperspective•  Haveinmindqueryplans!

•  Needtolookatbenchmarksthatscale–  Needtodealwithbig,bigdata(orderoftrillionsoftriples)

•  NeedtolookatqueryworkloadsforSPARQL1.1–  aggregates,nestedsubqueries,…–  donotforgetupdates!


References•  [NPM+12]R.OthayothNambiar,M.Poess,A.Masland,H.R.Taheri,M.

Emmerton,F.Carman,andM.Majdalany.TPCBenchmarkRoadmap2012.InTPCTC,2012.

•  [L97]CharlesLevine.TPC-C:TheOLTPBenchmark.InSIGMOD-IndustrialSession,1997.

•  [Castro04]R.lGarcia-CastroandA.Gomez-Perez.AMethodforPerforminganExhaustiveEvaluationofRDF(S)Importers.InWISE2005Workshops,2005.

•  [W90]R.P.Weicker.Anoverviewofcommonbenchmarks.Computer,23(12):65–75,December1990.

•  [G93]J.Gray,editor.TheBenchmarkHandbookforDatabaseandTransactionSystems(2ndEdition).MorganKaufmann,1993.

•  [H09]K.Huppler.TheArtofBuildingaGoodBenchmark.InTPCTC,2009.•  [GPH05]Y.Guo,Z.Pan,andJ.Heflin.LUBM:ABenchmarkforOWLKnowledge

BaseSystems.JournalWebSemantics:Science,ServicesandAgentsontheWorldWideWebarchiveVolume3Issue2-3,October,2005,Pages158-182


References•  [BNE14]P.Boncz,T.Neumann,O.Erling.TPC-HAnalyzed:HiddenMessagesand

LessonsLearnedfromanInfluentialBenchmark.PerformanceCharacterizationandBenchmarking.InTPCTC2013,RevisedSelectedPapers.

•  [SHM+09]M.Schmidt,T.Hornung,M.Meier,C.Pinkel,G.Lausen.SP2Bench:ASPARQLPerformanceBenchmark.SemanticWebInformationManagement,2009.

•  [FOAF]FriendofAFriend.http://www.foaf-project.org/•  [SWRC]SWRCOntologyhttp://ontoware.org/swrc/•  [DC]DublinCoreMetadataInitiative.http://dublincore.org/•  [BS09]C.BizerandA.Schultz.TheBerlinSPARQLBenchmark.Int.J.Semantic

WebandInf.Sys.,5(2),2009.•  [BSBM]BerlinSPARQLBenchmark(BSBM)Specification-V3.1.http://

wifo5-03.informatik.unimannheim.de/bizer/berlinsparqlbenchmark/spec/index.html.

•  [RU09]N.RedaschiandUniProtConsortium.UniProtinRDF:TacklingDataIntegrationandDistributedAnnotationwiththeSemanticWeb.InBiocurationConference,2009.


References•  [UniProtKB]UniProtKBQueries.http://www.uniprot.org/help/query-fields.•  [NW10]T.NeumannandG.Weikum.TheRDF-3Xengineforscalable

managementofRDFdata.TheVLDBJournal,19(1),2010•  [NW09]T.NeumannandG.Weikum..ScalablejoinprocessingonverylargeRDF

graphs.InSIGMOD2009•  [SKW07]F.M.Suchanek,G.KasneciandG.Weikum.YAGO:ACoreofSemantic

KnowledgeUnifyingWordNetandWikipedia,InWWW2007.•  [Barton]TheMITBartonLibrarydataset.http://simile.mit.edu/rdf-test-data/•  [PHS10]H.Patni,C.Henson,andA.Sheth.Linkedsensordata.2010•  [TPC-H]TheTPC-HHomepage.http://www.tpc.org/tpch/•  [WordNet]WordNet:AlexicaldatabaseforEnglish.http://

wordnet.princeton.edu/•  [XMark]AnXMLBenchmarkProject.http://www.xml-benchmark.org/•  [MLA+14]M.Morsey,J.Lehmann,S.Auer,A-C.NgongaNgomo.DBpedia

SPARQLBenchmark-PerformanceAssessmentwithRealQueriesonRealData.ISWC,2011


References•  [NS11]A–C.NgongaNgomo,S.Auer.LIMES:atime-efficientapproachforlarge-scale

linkdiscoveryonthewebofdata.IJCAI'11.•  [NS09]A–C.NgongaNgomoandD.Schumacher.Borderflow:Alocalgraphclustering

algorithmfornaturallanguageprocessing.InCICLing,2009.•  [AHO+14]G.Aluc,O.Hartig,T.Ozsu,K.Daudjee.DiversifedStressTestingofRDFData

ManagementSystems.InISWC,2014.•  [SMN15]M.Saleem,Q.Mehmood,andA–C.NgongaNgomo.FEASIBLE:AFeature-

BasedSPARQLBenchmarkGenerationFramework.ISWC2015.•  [DKS+11]S.Duan,A.Kementsietsidis,KavithaSrinivasandOctavianUdrea.Applesand

oranges:acomparisonofRDFbenchmarksandrealRDFdatasets.InSIGMOD,2011.•  [Wilkinson06]K.Wilkinson.Jenapropertytableimplementation.InSSWS,2006.•  [AMM+07]DanielJ.Abadi,AdamMarcus,SamuelMadden,KatherineJ.Hollenbach:

ScalableSemanticWebDataManagementUsingVerticalPartitioning.VLDB2007:411-42

•  [BDK+13]MihaelaA.Bornea,JulianDolby,AnastasiosKementsietsidis,KavithaSrinivas,PatrickDantressangle,OctavianUdrea,BishwaranjanBhattacharjee:BuildinganefficientRDFstoreoverarelationaldatabase.SIGMODConference2013:121-132


Assessing the performance of RDF Engines: Discussing RDF Benchmarks

Engineering