2013 Oracle Practice
Whodunnit? A Hyperion Murder MysteryCasey Ratliff, System ArchitectJon Harvey, EPM Practice Lead
HUGMN TechDay03/19/14Oracle/ Hyperion/ OBIEEStrategic assessments & technology roadmapsProject Managers with extensive implementation experienceBusiness consultants with planning, financial reporting & consolidation delivery experienceCertified Oracle Planning, Essbase, HFM, OBIEE ConsultantsCustomer enablement services (training, hosting, extended support and managed services)Employee lead engagements with minimal contractor utilizationBoard of Director Advisors to the MN Hyperion User Group
eCapital Advisors Overview
2
Founded in 2001 Profitable every year in businessPerformance management consulting firmOver 250 performance management customerseCapital Advisors employees: Dedicated to Enterprise Performance Management and Business Analytics, helping client better understand business driversProven record and high customer satisfactionExperience across a variety of industries and have worked with companies of all sizes
eCapital Advisors Overview
3330+ consultants, all W2 employeesSales or Service offices in Milwaukee, St Louis, DallasFounded by former Accenture & Arthur Andersen Audit or Technology employeesIncredibly deep set of business and technology contacts across multiple industries, products, and successesOne of the Fastest growing companies in the Twin Cities2013 Inc. MagazineInc. 5000 award: #1558
2013 MSP Business Journal Fast50 award winner, #17
eCapital Oracle BI & Hyperion Client Sampling
5Includes ALSACWell, agenda ruins the fun of it
Key points were going to coverWho changed the system?Who changed data?Who changed the application objects?New functionality in 11.1.2.3
Agenda66Why audit?Why are systems audits important?Mandated by data setPrevention of malicious actsUseful for troubleshooting
The perils of not auditing?
What do we mean by audit? For the purposes of this talk, we typically are talking about who changed what and when.Mandated by data set = regulatory, SEC, HIPPAAUBS = sys admin was pissed at his bonus, planted logic bomb, blew up the system. If they were able to identify what he had touched recently before walking out, they could have found it.7Our story begins
System went down at 3AM. IT responds the only way they know how (restart of services).8Our story beginsWe know that our nightly batch completed successfully at 10PM
Batch runs metadata updates, redeployment and calc scripts
We know the system was working fine at 10PM because our nightly batch (which has run without error for months) ran fine. This is a forecasting system and were not forecasting right now, so data shouldnt be changing.9Our first suspectWas it:
Who: Col. MustardWhere: Server consoleWeapon: The EPM Configurator
Put Caseys head on this pic10
Windows Event ViewerSecurity log failuresSystem rebootBackup conflictsEPM Deployment ReportArchitectureConfigurationsChanges to Registry
Casey walk through epmsys_registry reporting and sys changes11Digging inWhile the IT department continues to troubleshoot
The system is up and running fine now, but our tie outs are hosed
As is common, while the system architects are searching for root cause on the back end, the application guys are trying to figure out how to restore and an order of events on the front end12Who changed the data?Data audit options available to usSSAUDITPlanning Data Audit
Lets take a look at bothSince we know our batch ran without error, lets start by taking a look if someone changed the some key inputs to the forecast (for instance, a change in currency rates could have done this)13SSAUDITEssbase.cfg settingTracks Essbase writes only
Essbase tells us nothing changed14SSAUDITGenerates 2 filesGenerates an ALG and ATX file
15SSAUDITALG FileHistory records from every update transactionIncludes user name, time stamp, and number of updated rows
Planning apps = under app owner ID, writeback under user ID16SSAUDITATX FileTransaction records in a format that can be used as the input source for data load
Essbase tells us nothing changed17Planning Data Audit11.1.2.0, 11.1.2.1 (AdministrationApplicationReports)
11.1.2.2 (ToolsReports)
Planning Data Audit
If we had pulled up the Planning audit menu and had checked data, we could access the Planning data audit recordsNOTE: In 11.1.2.1 and prior, restart of Planning service required19Planning Data Audit
This audit data is ONLY accessible via SQL. Within the Planning repository, a table called HSP_AUDIT_RECORDS will get populated with all of the changes that are happening. Captures calc scripts, BRs (with prompt values), etc.20When we look into our audit table
Our next suspectWho: Nick ChihakWhere: Planning AppWeapon: Data Form
The Plot ThickensWho would try to frame Nick???
Go to cut his access to the system and notice that its not his AD account its a native account made to look like his AD accountLets hit the Shared Services and find out.23Shared Services Audit ReportsWhat information is available to us in Shared Services?
Shared Services has the capability of tracking every login to every service within Hyperion, user creation, group additions/deletions, provisioning changes, etc.Everything click you make in SS can be audited.24Shared Services Audit ReportsThe catch not enabled by default
BUT unfortunately the auditing is disabled by default. They even have all of the boxes checked like some sort of cruel joke25Looking back, the fake ID only changed one YearTotal dollar amount, but our variances are much bigger.
Could calcs or other objects have been changed?
Now what?We know the batch completed successfully, like it always does
The nightly batch does a bunch of stuff, including:Imports MetadataRedeploys the applicationRuns calc scriptsRuns report scripts
Lets take a look at what information is available to us about these objects change histories
Were Objects Modified? (Planning)
Weve talked through the capabilities of the Planning audit and how that works. That functionality extends to objects as well as data28Were Objects Modified? (Planning)
Doesnt look like anything was modified in Planning in the last month or so29Were Objects Modified? (Calc Manager)
No history of changes, but has user ID and dateCalc Manager just doesnt tell us much in most versions30Were Objects Modified? (Essbase)No history, no user ID, only update date
Essbase NEVER tells us much, could parse the log for who locked?31Where do we look next?One of the first places we should have looked, EAS:
Half of the Entity dimension is missing!
Were Objects Modified in EPMA?We found our culprit!
The CulpritIt was:
Who: Jim FarleyWhere: Dimension LibraryWeapon: Import Profile
It was Jim Farley in the dimension library with an import profile3411.1.2.3Fun exercise, but could have been a lot easier in 11.1.2.3
New in Shared Services 11.1.2.3: Artifact Change Reports
Artifact Change Reports
One thing worth noting if you havent seen 11.1.2.3 Shared Services is a tab in the workspace frame instead of a popout window36Artifact Change Reports
Q&A
