10/22/2019
1
Artificial Intelligence & Internal Control TestingSeries 1 – October 23, 2019
• Participate in entire webinar• Answer polls when they are provided• If you are viewing this webinar in a group
Complete group attendance form with• Title & date of live webinar• Your company name• Your printed name, signature & email address
All group attendance sheets must be submitted to within 24 hours of live webinar Answer polls when they are provided
• If all eligibility requirements are met, each participant will be emailed their CPE certificates within 15 business days of live webinar
To Receive CPE Credit
10/22/2019
2
Shauna Woody-Coussens, CFEManaging Director, Forensics & Valuation Services
Lanny Morrow, EnCE®, CTFIManaging Consultant, Forensics & Valuation Services
Our Goals Today• Discuss the benefits of AI-assisted internal control testing• Discuss general accounting system requirements to be able to
implement internal control testing using artificial intelligence (AI)• Understanding AI & complex systems • Discuss the technology used for AI-assisted internal control
testing
10/22/2019
3
Traditional Internal Control Testing• Very manual in nature
Submit electronic internal control questionnaires to various organizational personnel
Read through organization’s policies, procedures & internal control documentation
Perform interviews & walk-throughs of random transactions
Data Analytic-Assisted Transaction Testing• Use data analytics to identify potentially higher risk transactions
for testing Identify transactions that appear to possibly violated required internal
controls Identify transactions with common red flags for risk Identify transactions that are anomalies for the organization
10/22/2019
4
The Past• Very manual & time-consuming process• Subjective & random sampling means some risky transactions
may not be tested• Internal control gaps may not be highlighted
The Future• Less involvement of organizational personnel in the process• Population testing of transactions against organizational & best
practices for internal controls• Heighted assurance that financial operations are operating
within acceptable risk tolerances
10/22/2019
5
What Is AI?• AI systems typically demonstrate some of the following behaviors
associated with human intelligence Planning Learning Reasoning Problem-solving Knowledge representation Perception Motion Manipulation Potentially some social intelligence & creativity
… & What AI Is Not …
10/22/2019
6
Artificial General Intelligence• Intelligence of a machine that has the capacity to understand or
learn any intellectual task that a human being can • It is a primary goal of some AI research & a common topic in
science fiction & future studies• NOT the focus of what we’re talking about today
The Right Kind of AI• We will focus on machine learning, deep learning & similar AI
methods• Machine learning – computer programs that can learn from
experience using statistical methods• Deep learning – more sophisticated form of machine learning• Computer vision – useful for video & scanned PDF analysis
10/22/2019
7
Types of Machine LearningSupervised Unsupervised
Give examples & answers, machine finds more like it
Give data, machine finds patterns & applies its
own rulesExamples
• Fraud detection• Known controls weaknesses
Examples• Knowledge discovery• Grouping like things
Supervised Machine Learning
10/22/2019
8
Why AI?
AI IN INTERNAL CONTROLS TESTING?
YOU’RE KILLING ME SMALLS!
Organizations are complex systems
“A complex system included a large number of interacting components (agents, processes, etc.) whose aggregate activity is nonlinear (not derivable from the summations of the activity of individual components) & typically exhibits hierarchical self-organization …”
Source: https://www.informatics.indiana.edu
All These Organizations Had Internal Controls – They Even Tested Them• Equifax, Target, Home Depot, SWIFT• NASA• Three Mile Island• Enron• Wells Fargo• WorldCom• Encompass Health (formerly HealthSouth) • & one more … yours
10/22/2019
9
Complexity & Internal Controls• Complex systems behavior is intrinsically
difficult to model due to the dependencies, relationships, or interactions between their parts or between a given system & its environment
• Most internal controls treat the organization like an engine (component view)
Why AI? The Key Idea• AI is capable of finding patterns, relationships & threats that are
Too subtle for humans to recognize efficiently Too deeply buried in massive amounts of data Too disparate (different systems) In a bad format (scanned PDFs) Emergent – never seen before Cascading – only happens when something else happens Concealed – intentional acts to suppress the pattern or observation
10/22/2019
10
Using AI in Internal Controls Testing• Natural language processing
Natural language processing is AI that can read textual content AI can read, summarize & classify internal control policy documents AI will then identify transactions, communications & activities that
violate those policies Email analytics Invoice review Contract review Potential uses in interview speech & tone recognition for fraud Computer vision may help with handwritten/scanned content
Using AI in Internal Controls Testing• Predictive analytics
AI can be taught what controls departures have looked like in the past, then interrogate new, never-before-seen data to find those patterns
Example 1 – spending patterns related to vendors offering bribes or kickbacks
Example 2 – training AI to spot emails containing evasive, vague or other emotional tone that may indicate a controls violation
Example 3 – full coverage journal entry testing, not just sampling Systems learn from experience & improve with each iteration
10/22/2019
11
Logistic Regression• Concise “yes/no” answer for each
transaction, journal entry or activity• Associated “certainty” score• Near-instantaneous feedback
Using AI in Internal Controls Testing• Prescriptive analytics
Predictive tells us what could happen Prescriptive tells us what should happen AI system preprogrammed with “best practices” internal controls,
exposed to an enterprisewide system Using “best practices” template, identifies shortcomings in the system
being evaluated Automation is key
10/22/2019
12
Using AI in Internal Controls Testing• Agent-based modeling
Only type of analysis capable of modeling outcomes in a complex, nonlinear system
Great at identifying “unknown unknowns” & black swans For a given internal control or policy, potential outcomes can be modeled
& tested (in some cases, before it’s ever implemented) Difference between actual & best practices can be modeled Great approach for IT controls testing Demonstration
Using AI in Internal Controls TestingAgent-based modeling IT controls example
Outcome under current policies Outcome under best practices
10/22/2019
13
Using AI in Internal Controls Testing• Robotic process automation
Not AI, but will certainly support automation & efficiency efforts Robotic process automation (RPA) is the term used for software tools
that partially or fully automate human activities that are manual, rule-based & repetitive
RPA will be the glue that ties various AI internal controls testing components together
Any computerized process currently manual (contract review, extracting data, reporting, computing) can be automated; even mouse clicks
Potential to replace up to 48 percent of current manual functions
Using AI in Internal Controls Testing• Generative adversarial networks – warring AIs
AI playing “war games” simulations, over & over & over & … AI that detects internal control violations pitted against an AI that tries to
break the rules Systems on both sides learns & improves before next battle Result – new weaknesses reported to human internal auditors, system
becomes improved Potential to discover previously unknown weaknesses Further down the road – self-learning & self-correcting systems
10/22/2019
14
Using AI in Internal Controls Testing• The role of data visualization
Humans still need to understand AI output Data visualization tools (Microsoft Power BI, Amazon QuickSight,
Tableau) can connect to technical output for visual analysis Human eye can still spot patterns & put them into proper context in a
manner that rivals AI New trend – data visualization tools incorporating AI to provide “insight
generation” as the human analyst reviews content
Using AI in Internal Controls Testing• Summary of concepts
Predictive analytics – spotting IC weaknesses Prescriptive analytics – best practices versus current state Natural language processing – including text content in IC testing Deep learning & computer vision – better ML & accessibility Simulations – stress testing internal controls in varying situations Adversarial models – better controls through warring AIs Robotic process automation – “driverless” internal controls testing Data visualization – the human window into the AI mind
10/22/2019
15
QUESTIONS?
Continuing Professional Education (CPE) Credits
The information in BKD webinars is presented by BKD professionals, but applying specific information to your situation requires careful consideration of facts & circumstances. Consult your BKD advisor before acting on any matters covered in these webinars
BKD, LLP is registered with the National Association of State Boards of Accountancy (NASBA) as a sponsor of continuing professional education on the National Registry of CPE Sponsors. State boards of accountancy have final authority on the acceptance of individual courses for CPE credit. Complaints regarding registered sponsors may be submitted to the National Registry of CPE Sponsors through its website: www.nasbaregistry.org
10/22/2019
16
• CPE credit may be awarded upon verification of participant attendance
• For questions, concerns or comments regarding CPE credit, please email the BKD Learning & Development Department at [email protected]
CPE Credit
THANK YOU!Lanny Morrow, EnCE®, CTFIManaging Consultant Forensics & Valuation ServicesKansas City | 816.221.6300 | [email protected]
Shauna Woody-Coussens, CFEManaging DirectorForensics & Valuation ServicesKansas City | 816.221.6300 | [email protected]