Knowledge management for analytic teams jaime fitzgerald and alex hasha - presentation @info360 aiim conference 2011

Knowledge Management for

Analytic Professionals and Teams

Jaime Fitzgerald, President, Fitzgerald Analytics, Inc.

Alex Hasha, Chief Data Scientist, Bundle Corporation

March 2011

Architects of Fact-Based Decisions™

Introduction

Alex Hasha

Data Scientist @

Bundle Corp

Jaime Fitzgerald,

Founder @

Fitzgerald Analytics

Responsible

� Leading development of data products

� Designing statistical methods / algorithm � Transforming data into value for clients

2Knowledge Management for Analytics

Responsible

For…

At a

Company

That

Also

Working

On

� Designing statistical methods / algorithm

that transform data into insights for

consumers

� Helps consumers with financial mgt by

providing tools & spending behavior data

that are available nowhere else

� Is growing and hiring!

� Learning about Hadoop, Hive, Etc.

� Buying a home and renovating a bathroom

� Creating meaningful careers for employees

� Helps clients convert Data to Dollars™

� Brings a strategic perspective to improve

ROI on investments in technology, data,

people, and processes

� Writing a book about the role of “thought-

style” in the information-era

About Bundle Corporation and Bundle.Com

� Bridging the gap between personal financial management and the good

living that healthy finances makes possible.

� Helping people save well & spend well, combining free personal

financial mgt tools with data tools and recommendation engines built

from the anonymous spending behavior of over 20 million households.

A joint venture of Citi, Microsoft Money, & Morningstar, Bundle is:


from the anonymous spending behavior of over 20 million households.

� Cross-referencing this data with other public and private information,

we developed tools available nowhere else:

� "Everybody's Money”: learn how your peers spend and save

� Restaurant Recommender: based on card spending data

� Merchant Recommender: our newest recommendation tool

The restaurant recommender: predictive analytics meets dining!


1. Complex analysis is high stakes, risky, and hard to

manage informally

2. Success requires knowledge management

standards and tools, even within small teams

Executive Summary


standards and tools, even within small teams

3. “No Silver Bullet”: to empower analysts,

knowledge management methods must be tailored

to a team’s workflow

Table of Contents

1. Challenges of Analytic Thought-Work

2. The Role of Knowledge Management


3. Implications for all Thought-Workers

Challenges Analysts Face

In this section we will discuss…

1. A core set of universal challenges faced by analytic pros

2. Specific examples of these challenges at:

1. Challenges


1. Bundle Corporation

2. Other analytic teams that are working hard as we speak

It’s a Different World

While analytic teams are indispensible in today’s information economy, the nature of their work

makes teamwork, management, and coordination challenging.

1. Challenges


This creates a set of pitfalls…

source: www.xkcd.com

Problems Analysts Face

There are several pitfalls into which analysts fall.

Analysis is

often:The Embarrassing Facts

1. Hard to

understand� A top complaint regarding analysis is that it is confusing and unclear

2. Impossible to

verify, audit, � Few executives report they fully trust analysis they receive

1. Challenges


verify, audit,

or replicate

� Few executives report they fully trust analysis they receive

3. Flawed

� 90%+ of spreadsheets used in the field are estimated to have

material errors

� In 201- Aetna cancelled a 19% rate increase due to flawed analysis

4. Inefficient� Average analysts spend less than 10% of their time actually

performing core analysis (with most time going to data gathering,

troubleshooting, etc)

Analyst Pitfalls: Real Life Examples

Based on personal experience, but modified to protect the guilty…

Analysis is often: Examples (not @ Bundle) Impact

1. Hard to

understand

� An analysis involving both simple and

weighted averages of borrower credit

scores

� In various parts of the analysis, both

metrics were called “credit score”

� Misinterpretation

� Wrong inputs

� Unaware of risk

1. Challenges


2. Impossible to

verify, audit, or

replicate

� Documentation of key input variables was

not consolidated, hard to find, and

therefore not widely known

� Unaware of risk

� Hard to tell

whether correct

input was used

3. Flawed

� A chain of analysis mixed up simple vs.

weighted averages.

� Error persisted 5 years before it was

discovered.

� Consequences can

be significant

4. Inefficient� 2 PhDs took a week to solve a problem

that should never have happened.

� Expensive waste of

skilled time.

Data Science at Bundle: Lots of Prototyping

1. Goal: Solve a

New Problem

4. User Feedback 2. Design Solution

It is crucial for us to

document, share, and re-use

lessons learned from each

cycle of effort

1. Challenges


4. User Feedback 2. Design Solution

3. Testing +

Validation Data Product

on Website

Table of Contents





Challenges Analysts Face

In this section we will discuss…

1. How KM helps analysts in general

2. Case Examples from Bundle

2. The Role of Knowledge

Management


Knowledge Management to Avoid Analyst Pitfalls…

Knowledge Management makes a big difference in outcomes for analysts

Pitfall How KM Helps

Lack of detailed specifications often leads to

useless results, yet writing detailed

specifications for another teammate can take

longer than doing it yourself.

Slow on boarding due to “knowledge

Consistent coding and analysis standards

prevent make specifications easier to create

and execute

Learning curve remains, but progress is faster.

Senior teammates spend less time training new


Management


Slow on boarding due to “knowledge

diffusion” and lack of access

Reinventing the wheel due to lack of

awareness of previous work-products

Misinterpreting data definitions, or

misunderstanding provenance of data leads to

incorrect analyses.

Senior teammates spend less time training new

teammates.

Teammates post documentation of solutions to

common problems, and are encouraged to

search this documentation as a first step.

Enforcement of standardized, descriptive, field

names and centrally available data dictionaries

make these mistakes harder to miss.

Workflow at Bundle

Inputs Process Outputs

Credit Card

Transaction Data

Merchant

Listing Data

Natural Language

Processing/

Categorization

Customer/

Merchant

Spending Survey

Insights &

Recommendations for

Consumers

� Everybody’s Money™


Management


Anonymous Customer

Demographic Data

Census Data &

Government Consumer

Spending Data

Geographic Data

Statistical

Sample

Rescaling

Consumer

Demographic

Profiles

� Restaurant / Merchant

Recommender

• Loyalty/Popularity

Scores

• “Web of Offline

Merchants”

� Consumer Segment

Analysis

KM at Bundle

Wiki-Based

Knowledge

� Metric Definition

� Algorithms

� Dialog

� Meta-data


Management


Work Flow

“Persistent” Code & Scripts

(vs. ad hoc data processes)

In-System

Knowledge

Table of Contents





Key Implications for all Through-Workers3. Implications

Concept Basis

1. The more technical the work, the

more you need KM!

� Technical work draws upon more

dimensions of knowledge than teams can

manage informally

2. Good KM supports YOUR

workflow and thought-work

� Since the goal of KM is to improve Quality

and Efficiency of outcomes, it is essential


workflow and thought-workand Efficiency of outcomes, it is essential

to customize KM to worker processes

3. One size does not fit all

� Because workflow for technical workers is

so variable, they require flexible KM

solutions

4. At times technical KM must span

multiple platforms

� In contrast to less technical knowledge,

there are times when technical knowledge

is better managed in multiple places

Our Thought-Work Creates Value

Like a good chef, we need to have the ingredients in place when we need them…

“The Chef’s Shelf”

3. Implications


Individual Work Collaboration Communication

Let’s Stay in Touch

We look forward to learning from each other…contact us anytime.

Alex Hasha Jaime Fitzgerald


Twitter: www.twitter.com/alexhasha

LinkedIn: www.linkedin.com/pub/alexander-

hasha/8/26a/30a

Email: [email protected]

Twitter: www.twitter.com/jaimefitzgerald

LinkedIn: www.linkedin.com/in/jaimefitzgerald

Email: [email protected]

Knowledge management for analytic teams jaime fitzgerald and alex hasha - presentation @info360 aiim conference 2011

Documents

role of knowledge management

etcknowledge management

bundleknowledge management

speakknowledge management

knowledge management

knowledge management

data tools

challenges data science