Database/Data Warehouse Integration Architecture
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Database/Data Warehouse Integration Architecture
ContentsIntroduction.................................................................................................................................................2
Overview......................................................................................................................................................2
BusinessIntelligenceIntegration.............................................................................................................2
DatabaseIntegration................................................................................................................................2
BusinessIntelligenceIntegration.................................................................................................................2
ExtractionProcess................................................................................................................................2
ApplicationConfiguration....................................................................................................................4
Entities..................................................................................................................................................4
DatabaseIntegration....................................................................................................................................6
SynchronizationProcess.......................................................................................................................6
ApplicationConfiguration....................................................................................................................9
Entities..................................................................................................................................................9
LinkstoRelatedMaterial...........................................................................................................................12
IntroductionThisdocumentprovidesdetailedinformationregardingimplementationof:
• ExtractionarchitecturebetweenMarketoandanexternalBusinessIntelligencesystem(BI)• SynchronizationarchitecturebetweenMarketoandanexternalDatabase/DataWarehouse
system(DB)Entitiesaredescribed,andthespecificsofmaintainingsynchronizationofnewandupdatedrecords.
Overview
BusinessIntelligenceIntegrationThisusecaseanswersthequestion,"HowdoIgetMarketodataintomyenterpriseBIsolutionforcustomreporting?"BIsolutionspermitflexiblereportingonMarketomarketingdatacombinedwithotherdatasourcessuchasfromcustomerservice,operationsorfinance.Combineddataisoftenanalyzedandpresentedintheformofdashboardsthatareusedtotrackbusinessmetrics.
MarketorecommendsimplementationofBIintegrationapplicationsusingaone-waypollingmodel,retrievingnewrecordsfromMarketo,andpushingthoserecordsasmappedintheapplicationconfigurationtotheBIsystem.TheapplicationshouldqueryfornewrecordsperiodicallyandthenperformanynecessarytranslationandfiltrationbeforepushingtheupdatestoBIdatastore.
DatabaseIntegrationThisusecaseanswersthequestion,"HowdoIsynchronizeMarketowithanexternalDatabase/DataWarehouse?"Acommondatamanagementpatternistomaintaina"systemofrecord"(SOR),whichservesastheauthoritativedatasourceforagivenelementorpieceofinformation.TheSORistypicallyarepositorywheredataobjectsaremaintained.Keepinglead/contactdatainsynchbetweenmarketingcampaignsandbackendsystemprocessesensuresconsistency.ModelingMarketocustomobjectsusingdataobjectssuchassalesordersororderfulfillmentcanhelpimproveyourmarketingcampaignsbyincludingenricheddataincustomercommunications.
MarketorecommendsimplementationofDBsynchronizationapplicationsusingatwo-waypollingmodel,retrievingchangesandnewrecordsfromeithersystem,andpushingthosechangesasmappedintheapplicationconfigurationtotheothersystem.TheapplicationshouldqueryforchangesofselectedrecordtypesperiodicallyandthenperformanynecessarytranslationandfiltrationbeforepushingtheupdatestoDBorMarketorespectively.
BusinessIntelligenceIntegration
ExtractionProcessThesimplestandmostefficientwaytoextractdataistoimplementapollingprocessthatretrievesnewactivityrecordsfromMarketoandpushesdatatotheBIdatastore.Thecycleisthenrepeatedafterapredeterminedperiodeachtime.Thiscouldbeimplementedasadailyscheduledjobforexample.
RecommendedExtractionOrder
1. ExtractActivitiesfromMarketo2. ExtractLeaddetailforeachActivityfromMarketo(optional)3. UpdateBIdatastore
Activitiesareread-onlyaftercreation.TheyarereadfromMarketoviatheGetLeadActivitiesendpoint,ingroupsofupto10activitytypes.TheendpointacceptsanearliestcreationdateviaapagingtokenwhichisretrievedviatheGetPagingTokenendpoint.ThesetofresultsshouldbepagedthroughuntilthemoreResultparameterintheresponseisreturnedasfalse,andthecorrespondingactivitieswrittentoBIdatastore.BeginthenextextractioncyclebyadjustingtheearliestcreationdateandcreatinganewpagingToken.Movethedateforwardintimebythedesiredamounttoestablishanewearliestcreationdate.Theamountoftimeselectedbecomesthepollinginterval.
SinceMarketomaintainsonlyper-secondresolutionfordatetimes,itispossiblethatthesameactivityrecordcouldbereturnedintwodifferentextractcycles.Toavoiddataduplication,theapplicationshouldusetheactivityidwhichisauniqueidentifier.Duplicateactivityrecordscanbesafelyignored.Iftheactivityrecorddoesnotprovideallofthelead-relateddatarequiredtomeetyourbusinessneed,youmayperformanadditionallookupoftherelatedleadrecord.Tolookuptherelatedleadrecord,youpasstheleadIdfieldfromtheactivityrecordtoGetMultipleLeadsbyFilterType.ThisendpointallowsyoutosendabatchofleadIdsinasinglerequest.ThishelpsminimizetheoverallnumberofAPIcallsrequiredforthelookup.Youwillthencombineactivityrecorddatawithleadfielddata.Thelogicusedtocombinethetwodatasetsshouldreflectaninnerjoinoperation,usingleadIdasprimarykeyandasforeignkey.
ApplicationConfigurationTheapplicationshouldsupportasetofconfigurationoptionstocontrolprogrambehavior.Theseoptionscouldbestoredinapropertiesfileforexample.
FieldsNotallfieldsarenecessarytobemappedbetweenMarketoandanassociatedBIsystem.AnoptiontoenableordisableextractionofspecificfieldsfromLeadandActivityentitytypesisrecommended.OnlytheleadIdfieldismandatory,whileallothersshouldbeoptional.Reducingthenumberofextractedfieldswillimproveperformanceinallcases.
ActivityTypesAdminusersshouldbeabletoselectwhichactivitytypesareandarenotextractedtolimitAPIcallusageandimproveperformance.
PollingIntervalThetimebetweenextractionbatchjobsneedstostrikeabalancebetweenlowdatalatency,andhighutilizationofAPIcalls.
YourpollingintervalshouldbebasedonhowmanyAPIcallsatypicalextractioncyclewilltake.ThenumberofnewactivityrecordsthataclientexpectstobecreatedwillbethegreatestinfluenceonthecumulativenumberofAPIcallswhichwillbeusedinagivenday.ForaccountsprovisionedorrenewedafterMarch2016,thedefaultnumberofAPIcallsperdayis50,000.AdditionalAPIcallsmaybepurchasedingroupsof10,000/day.
TraditionalBIpresentshistoricaldataformanualanalysis,solatencyisnotaconcern.Inthiscasea24hourintervalisappropriate.Real-timeBIreliesoneventdrivenprocessing,solatencyisaconcern.Inthiscasea5minuteintervalisappropriate.
EntitiesThefollowingaretheMarketoentitiesthatapplyforBIintegrationusecase:Leads,Activities.
LeadsPrimaryKey:id
TheintegeridofaMarketoleadrecordandtheprimarykey.ThisissystemmanagedbyMarketo,andmayonlybeassignedbyMarketo.
InMarketo,leadsrepresentanyperson-recordwhichrepresentsasalesormarketingtarget.AllSmartCampaigns(commonlyreferredtoasa“workflow”innon-Marketosystems),filter,trigger,andoperateonleadrecords,basedontheircharacteristicsandactions.
Model
LeadsarehighlyextensibleinMarketoandmayincludealargenumberofcustomfields.Whenextractingdatafromanyparticularsubscription,theDescribeLeadendpointoftheRESTAPIshouldbeusedastheexclusivesourceoftruthtodeterminefieldavailabilityinaparticularsubscription.
Notethatthemodelforaleadisalsopotentiallydynamic,asfieldsmaybeaddedorhiddenbyend-usersatanytime.Theapplicationmustberesilienttosuchchanges,andnotbreakwhentheyoccur.
Relationships
LeadsarerelatedtonumerousaccessibleobjecttypesinMarketo.Forthisusecase,weareonlyconcernedwithActivities.
Object RelationshipType LinkType LinkFieldActivities 1:N ActivityField leadIdAccessibility
Inordertoreadleads,anAPIusermusthaveRead-OnlyLeadpermission.Theycanbereadthroughthefollowingendpoints:
• GetMultipleLeadsbyFilterType• GetMultipleLeadsbyListId• GetMultipleLeadsbyProgramId
LeadMetadataisreadthroughtheDescribeLeadendpoint.
ActivitiesPrimaryKey:id
ActivitieshaveauniqueintegeridthatissetuponcreationbyMarketo.
ActivitiesarerecordsofactivitiesassociatedtoleadrecordsinMarketo.Theymayrecordactivityofmanydifferenttypes,asindicatedbytheiractivityTypeId.Activitiesareread-onlyinMarketo.Certainactivitytypesareprunedafter90daysintheMarketosystem.
PrunedActivities• DataValueChange• AddtoList• RemovefromList• VisitWebPage• ClickLink• ChangeScore
Thetypesofactivitiesavailableinagivensubscriptionvarydependingonmanyfactors,includingthetypeofsubscription.TheavailabletypesandtheirmetadatashouldalwaysbedeterminedbycallingGetActivityTypesendpointfromthetargetsubscription.
Model
Activitieshaveasemi-strictschema.Thefollowingfieldsaredefined,butnotnecessarilyusedforallactivitytypes:
Name Datatype Descriptionid Integer Uniqueidforactivity.leadId Integer Idofthelinkedlead.Mapstoidonleadrecords.activityTypeId Integer Idofthetypeofactivity.CorrespondstoaresultofGet
ActivityTypes.activityDate Datetime Datethattheactivityoccurred.primaryAttributeValue String Valueoftheprimaryattribute.primaryAttributeValueId Integer Idoftheprimaryattribute.attributes Array Arrayofname/valuepairsrepresentingthesecondary
attributesoftheactivity.
Eachactivitytypehasaprimaryattributethatcorrespondstoavalueofsomekind.TheprimaryattributemayberelatedtoanytypeofassetorobjectinMarketo.Forexample,theprimaryAttributeValueoftheVisitsWebPagetypecorrespondstothenameorURLofthewebpagethatwasvisited.TheMarketonamewillbepresentedifthepagewasalandingpageandthepageURLifitisnotaMarketopage.Secondaryattributesconsistofanarrayofname/valuepairs,namingeachofthefieldsforanactivitytypeandthecorrespondingvalue.Continuingwiththeexample,someofthesecondaryattributeswouldbeClientIPAddress,QueryParameters,ReferrerURL,andUserAgent.
Relationships
ActivitiesinMarketoarealwaysrelatedtoleadrecordsthroughtheleadIdfield.SomeactivitytypesmayhavearelationshiptootherMarketoassetsthroughtheirprimaryAttributeValue.
Accessibility
Inordertoreadactivities,anAPIusermusthavetheRead-OnlyActivitypermission.
ActivitiescanbereadthroughtheGetLeadActivities,GetLeadChanges,andGetDeletedLeadsendpoints.
DatabaseIntegration
SynchronizationProcessThesimplestandmostefficientwaytomaintaincontinuoussynchronizationistoimplementapollingprocessthatretrieveschangestoleadrecordsinMarketoandpushesthemtoDBandthenfromLead/Contact,CustomObject,orCompanyrecordsinDBandpushesthemtoMarketo.Thecycleisthenrepeatedafterapredeterminedperiodeachtime.Thiscouldbeimplementedasascheduledjobforexample.
ToretrievechangesfromMarketo,ahighwatermarkmustbemaintainedforchangestolead/companyfieldswhichhaveoccurredsincethemostrecentlyretrievedchange.Thesearedatetimevalues.Whenretrievingchangesinasubsequentsynchronizationcycle,theexactdatetimeofthemostrecentlycreatedrecordshouldbeused.SinceMarketomaintainsonlyper-secondresolutionfordatetimes,itispossiblethatthesameleadchangerecordcouldbereturnedintwodifferentextractcycles.Toavoiddataduplication,theapplicationshouldusetheactivityidwhichuniquelyidentifiestheactivityrecord.Duplicateactivityrecordscanbesafelyignored.
RecommendedSynchronizationOrder
1. RetrieveLeadschangesfromMarketo*2. RetrieveLead/ContactchangesfromDB*3. RetrieveCompanychangesfromDB(optional)*4. RetrieveCustomObjectchangesfromDB(optional)*5. Resolveconflicts(optional)6. UpdateDBwithMarketoData7. UpdateMarketowithDBData
*Includesnew,changed,anddeletedrecords
LeadsandCompanies
SynchronizationforLeadsandCompaniesismaintainedprimarilyusingtheGetLeadChangesendpointwhichretrievesdatavaluechangerecordswhichoccurafteratimestampgivenbyapagingtokenwhichisretrievedviatheGetPagingTokenendpoint.ThisendpointwillreturnbothNewLeadactivities,whichindicatethecreationofanewknownleadinMarketo,anddatavaluechangeactivitiesforasetoffieldsgivenintheparametersofthecall.
ThechangeactivitiesshouldbeappliedintheorderofthecreatedDate,fromearliesttolatest,givenintheactivitytotherecordsinDBwhichcorrespondtotherecordgivenbyleadIdintheactivity.NewLeadactivitiesshouldbeaddedtoaqueueofnewleadrecordswhichneedtoberetrievedbyid,“leadId”intheactivity,usingtheGetLeadsbyFilterTypeendpoint,withidasthefilterType.Thesemayberetrievedupto300atatime.Itisrecommendedtowaituntilthereare300recordstoretrieve,andthentomakethecalltoretrievetheserecords,insteadofcallingwheneverthesebecomeavailable.If,uponreachingtheendofthesetofchanges,therearelessthan300records,thanthesetshouldberetrieved.
Inadditiontochangesandnewleads,inordertomaintainsynchronizationtheMergeLeadandDeleteLeadactivitymustberetrievedinorderaccountforleadrecordswhicharemergedtogether.Themergeactivityindicatesthattworecordshavebeenmergedintoasinglerecord.TheDBmayormaynotchoosetohonorthemerge/delete,deletethelosingrecordandretrievethechangesfromthewinninglead,oritmaybeignoredandhavea“DeletedinMarketo”flagsettoindicatethattheMarketoLeadIDforthatrecordisnolongervalid.
Aspartofastandardsynchronizationcycle,changesfromtheDBshouldalsoberetrieved.Ideallyonlyfieldswhichhavebeenupdatedsincethemostrecentsynchcycleshouldberetrieved,butthismaynotbepossiblegiventheconstraintsofthesystem.Ifthisisavailable,allthechangesforagivenrecordshouldbeaggregatedintoaleadrecordtobesubmittedtoMarketo.Ifachanges-onlyoptionisnotavailable,thenitisviabletoretrievethewholerecordwithalloftheMarketo-mappedfieldsforsubmissiontoMarketo.
Topushchangesforleadrecords,usetheCreate/UpdateLeadsendpoint.Create/UpdateLeadsallowsfortheinputofupto300leadrecordsasJSON.
ForincrementalsynchingofupdatesfromDB,thelookupFieldshouldbespecifiedastheprimarykeyselectedfromtheDBsystem,andthecreateOrUpdatemodeshouldbeused.ThisallowssharingofthesamequeuebynetnewleadsandleadupdateswhichneedtobepushedintoMarketo.
InthecasethataMarketoLeadislinkedtoacompanyrecordviaexternalCompanyId,commonlyconceptualizedasacontact,thecompanytypefieldsthatwerepartoftheleadrecordarenolongerwriteablethroughtheleadrecordandaredeferredtothelinkedcompanyrecord
Companies
ItisimportanttodeterminewhichfieldsareCompany-typefields,andwhichfieldsareLead-typefields.ThiscanbedonewiththeDescribeCompanyendpoint.AllfieldslistedthereareCompany-typefields,ofwhichmostaremirroredasleadfieldsforunlinkedleads.IfaCompany-typefieldisreflectedinaChangeDataValueoperation,thenthechangeshouldbereflectedagainstthecompanyrecordinDBifthechangewasmadeagainstaleadwhichislinkedtoacompanyrecordviaexternalCompanyId.Ifnot,thechangeshouldjustbereflectedagainsttheleadrecord.
CustomObjects
SincetheDBisthesystemofrecordforCustomObjects,synchronizationisone-wayfromDBtoMarketo.ThedefinitionofaCustomObjectshouldbederivedfromthetypeinDB.CustomObjectrecordsshouldbecreated,updated,ordeletedinMarketowheneveracorrespondingeventoccursintheDBsystem.Thischeckshouldbeperformedbytheintegrationsoftwareuponeverysynchronizationcycletoseeifchangesarerequired.
ApplicationConfigurationTheapplicationshouldsupportasetofconfigurationoptionstocontrolprogrambehavior.Theseoptionscouldbestoredinapropertiesfileforexample.
FieldsNotallfieldsarenecessarytobemappedandsynchronizedbetweenMarketoandanassociatedDB.AnoptiontoenableordisablesynchronizationofspecificfieldsfromtheLeadorCustomObjectentitiesisrecommended.OnlytheleadIdandDBforeignkeyfieldsshouldbemandatoryforsynchronization,whileallothersshouldbeoptional.Reducingthenumberofsynchronizedfieldswillimproveperformanceinallcases.
CustomObjectsAnoptiontoenableordisablecustomobjectsynchronizationisrecommended.
PollingIntervalMarketo’snativesynchronizationconnectorsqueueanewbatchofpushesandpulls5minutesafterthecompletionofthepreviousbatch.Thiscoversagreatdealofcasesandstrikesanacceptablecompromisebetweenlowsynchronizationlatency,andexcessiveutilizationofAPIcalls.Foryoursolution,youshouldbasethesynchronizationintervalbasedonhowmanyAPIcallsatypicalsynchronizationcyclewilltake.ForaccountsprovisionedorrenewedafterMarch2016,thedefaultnumberofAPIcallsperdayis50,000.AdditionalAPIcallspurchasedingroupsof10,000/day.
ThenumberofchangesthataclientexpectstooccurfortheirleadrecordswillbethegreatestinfluenceonthecumulativenumberofAPIcallswhichwillbeusedinagivenday,andthisshouldinfluenceyourdesign.
EntitiesThefollowingaretheMarketoentitiesthatapplyforDBintegrationusecase:Leads,Companies,CustomObjects.
LeadsPrimaryKey:id
TheintegeridofaMarketoleadrecordandtheprimarykey.ThisissystemmanagedbyMarketo,andmayonlybeassignedbyMarketo.Anyinsertoperationsattemptedbyaforeignsystemwhichincludeidwillberejected.
InMarketoleadsrepresentanyperson-recordwhichrepresentsasalesormarketingtarget.AllSmartCampaigns(commonlyreferredtoasaworkflowinnon-Marketosystems),filter,trigger,andoperateonleadrecords,basedontheircharacteristicsandactions.
Model
LeadsarehighlyextensibleinMarketoandmayincludealargenumberofCustomFields.Whensynchronizinganyparticularsubscription,asetofstandardfieldsshouldnotbereliedupon,andtheDescribeLeadfunctionoftheRESTAPIshouldbeusedastheexclusivesourceoftruthtodeterminefieldavailabilityandupdateabilityinaparticularsubscription.
Notethatthemodelforaleadisalsopotentiallydynamic,asfieldsmaybeaddedorhiddenbyend-usersatanytime.Theapplicationmustberesilienttosuchchanges,andnotbreakwhentheyoccur.
Relationships
LeadsarerelatedtonumerousaccessibleobjecttypesinMarketo.Forthisusecase,weareonlyconcernedwithCompaniesandCustomObjects.
Object RelationshipType LinkType LinkFieldCompany N:1 LeadField externalCompanyIdCustomObjects 1:N,N:1,orN:N SeeCustomObject
SectionSeeCustomObjectSection
Accessibility
LeadsmaybereadandwrittenfreelyinMarketogiventheRead-WriteLeadPermissionisgrantedtotheAPIuserbeingused.Theycanbereadthroughthefollowingendpoints:
• GetMultipleLeadsbyFilterType• GetMultipleLeadsbyListId• GetMultipleLeadsbyProgramId
Theycanbewrittenthroughthefollowingendpoints:
• Create/UpdateLeads• ImportLead
LeadsmaybedeletedthroughtheDeleteLeadendpoint.
LeadMetadataisreadthroughtheDescribeLeadendpoint.
CompaniesPrimaryKey:externalCompanyId,id
externalCompanyIdisanarbitrarystringfieldsetuponcreationbytheexternalsystem.TheprimarykeyofcompanyrecordsinDBshouldbemappedtoexternalCompanyId,whichisnotupdatableandmustbeunique.Idisauniquesystem-generatedintegerid.
Companyobjectsrepresenttheorganizationtowhichleadrecordsbelong.LeadsareaddedtoaCompanybypopulatingtheircorrespondingexternalCompanyId.Leadslinkedtoacompanyrecordwilldirectlyinheritthevaluesfromacompanyrecordasthoughthevaluesexistedonthelead’sownrecord.Attributesavailableonthecompanyrecordareavailablefortriggeringandfilteringonleadrecordsfromwithintheapplication.
Companyrecordsmayonlybecreatedbyexternalsystems,andtheDBshouldbetreatedasthesourceoftruth.
Model
• Companiesarefullyextensibleandmayhaveanynumberandtypeofcustomfields• DescribeCompanyshouldbeusedtoobtaintheschemaofcompany
Relationships
Object RelationshipType LinkType LinkFieldLead 1:N LeadField externalCompanyIdOpportunity 1:N OpportunityField externalCompanyId
Accessibility
Inordertoreadandwritetocompanyrecords,anAPIusermusthavebeengrantedtheRead-WriteCompanypermission.
Companiesmaybereadthroughasingleendpoint,GetCompaniesbyFilterType.Companiesmayonlybefilteredonalimitednumberoffields,whichareprovidedinthesearchableFieldsattributeoftheDescribeCompanyresult.
CompaniescanbewrittentousingtheCreate/UpdateCompaniesendpoint.
CompaniesmaybedeletedthroughtheDeleteCompaniesendpoint.
CustomObjectsPrimaryKey:marketoGUID,AdditionalUserDefinedkeys
CustomObjectsalwayshaveauniquesystem-generatedmarketoGUIDwhichissetuponcreation.Therewillbeatleastoneadditionalkey,andpossiblymore,whichareuser-definedintheCustomObjectdefinition.KeysmaybesinglefieldsfortypeswhicharelinkeddirectlytoLeadsorCompanies,andmayhavecompoundkeysfortypeswhicharelinkedtoLeadsandanotherCustomObjecttype.KeyscanbedeterminedbyusingDescribeCustomObjecttoretrievethelistofdedupeFields.
MarketoallowsthedefinitionofCustomObjecttypesbyuserstoextendtheMarketoschema.MarketoCustomObjectsmayberelatedtoLeadsorCompaniesineithera1:NorN:Nconfigurationthroughtheusageofintermediatecustomobjects.
Model
NotethattheschemaforcustomobjectswithinaDBispotentiallydynamic,asfieldsmaybeaddedorremovedatanytime.Theapplicationmustberesilienttosuchchanges,andnotbreakwhentheyoccur.
Relationships
MarketoCustomObjectsmayberelatedtoleadsorcompaniesineithera1:NorN:Nconfigurationthroughtheusageofintermediatecustomobjects.RelationshipscanbederivedfromtherelationshipsparameteroftheresultofDescribeCustomObject.
Accessibility
ToreadandwritetoCustomObjects,anAPIusermusthavetheRead-WriteCustomObjectpermission.CustomObjectsaremanipulatedusingthefollowingendpoints:GetCustomObjects,Create/Update/UpsertCustomObjects,andDeleteCustomObjects.AListCustomObjectsendpointisalsoprovidedtoagiveameansofdeterminingwhatCustomObjecttypesareavailableinagivensubscription.
LinkstoRelatedMaterial• RESTAPIreference:MarketoDevelopersSite• RESTAPIerrorcodes:MarketoDevelopersSiteErrorCodes• Bestpracticesandcodesamples:MarketoDevelopersBlog
Related Documents
