Software Operation Knowledge

Devnology: Software Operation Knowledge in Practice

Dr. Slinger Jansen

Utrecht University

Introductie UU en Software~VOC

09-08-13 open dagen 3

Who am I?: Slinger Jansen

§  Scientist

§  Entrepreneur

§  Advisor

§  Dad

§  Driver of old cars

Software Intensive Systems Developers suffer from

§  Increasing quality requirements

§  Decreasing release times

§  Data overflow of feedback data

We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us. - Ray Ozzie

Solution: Generic Data Acquisition and Data Mining

§  Generic data acquisition through instrumentation

§  Data gathering without altering the source code (aspect-oriented development)

§  Microsoft found out through their crash reporting program that 50% of crashes are caused by 2% of their bugs

State of the practice

§  Release sub-process - Release at times that are convenient to customer: 57%

- Use of formal release planning: 45% - All stakeholders can access the planning: 85% - Use of formal release scenario: 56% - Use of versioned configuration management: 54% - Explicit management of in-house build tools: 70% - Use of COTS: 38%

§  Delivery sub-process - Informing the customer through a website: 84% - Other channels not used often - Delivering updates through the web: 85% - Respondents that use CD & DVD: 40% - Manual product pull: 92%

- Automatic product push: 12%

State of the practice

§  Deployment sub-process - Use of an update tool: 30% - Tool is able to update at runtime: 56%

- First time installation failure: 8% - Possibility of de-installing previous release: 89% - Undo an update: 59% - Check of customer configuration: 73%

- Data backup: 59% §  Usage sub-process

- Use of license agreement: 73% - Knowledge about the customer platform: 73% - Use of error reports: 35%

- Use of usage reports: 40%

Vision: Continuous Customer Configuration Updating

§  Continuously release software after every change to a release repository

§  Continuously (be able to) release to customers

§  Continuously (be able to) deploy an update (run-time?)

§  Continuously change your software

§  (You should at least strive for continuous release!)

Possibly part of a Software Supply Network

Software Developer

Cust. 1

Cust. m

. . .

Software Developer

Cust 1

Cust n

. . .

Operational Environment 1

Operational Environment n

Operational Environment m

. . .

. . .

Slinger Jansen and Wilfried Rijsemus: Balancing Total Cost of Ownership and Cost of Maintenance Within a Software Supply Network , proceedings of the IEEE International Conference on Software Maintenance (ICSM2006, Industrial track), Philadelphia, PA, USA, September, 2006.

Or even an ecosystem!

Do you “own” one?

SOK Propagation in Software Ecosystems

§ Relationship management in software ecosystems –  Keystones use various participation incentives to bind niche players to their

ecosystem(s)

–  SOK resulting from software compatibility monitoring is input for relationship management

–  Other incentives are used to strenghten relationship

•  Developer keys > certification programs > private SDK access

SOK Propagation in Software Ecosystems

§ Software quality management –  Vendors use SOK to manage various types of software quality

–  Particularly useful in:

•  Determining root cause of in-the-field software failures that do not occur at the software vendor site

•  Isolation of Heisenbugs

–  A computer bug that disappears or alters its characteristics when an attempt is made to study it

–  EFfective SOK utilization quality management success

Continuous feedback mechanisms §  Email

§  SMS

§  Ambient Orbs

§  Windows taskbar monitor

§  Sound

What does your office use?

Pragmatic Process Improvement through SOK

§  Introduction –  Many software vendors that acquire SOK, … just acquire it.

•  Limited data mining, limited ‘integration’ in software processes

•  Unclear who is responsible

–  No structural integration with processes and tools in-place

•  No protocols or workflows to involve acquired data

•  SOK is not utilized to support decisions and steering

–  Release management

–  Requirements prioritization

–  etc.

§  What do you collect, use and report on?

§  What kind of actions are connected to these data?

SOK Lifecycle

37

Who acquires data without using it? And yes, support calls count!

Einde Theoretisch Deel

SOK Integration Template Method

§  The method is designed to guide vendors in: –  Analysis of integration environment and target processes

–  Identification of relevant and valuable operation information

–  Integration of selected information in, and adaption of, target processes

–  Presentation of integrated operation information

§  We assume that, before using the method:

–  Software operation information

–  One or more target processes

§  are accessible and available for integration


§  What do you want to look at immediately when you enter your office tomorrow morning?

§  OK, but how should we use the method? –  For Stabiplan, we built Errata Garden

•  Crash report mining and -analysis tool

•  Allows access to in-the-field operation information based on acquired crash reports


§  OK, but how should we use the method? –  Stabiplan had questions like

•  How do we make people use this information?

•  How do we prevent productivity to be descreased by EG monitoring?

•  How frequently should Errata Garden reports be monitored?

•  Who is responsible?

§  How can information provided by Errata Garden be effectively integrated with our product software processes?


§  OK, but how should we use the method? –  Template method was instantiated with three processes

•  Software maintenance

•  Software product management

•  Customer support

–  Action research

•  Researchers present at the software vendor site

•  Observing phenomena while adhering to CAR principles

–  Researcher-Client Agreement, Learning through Reflection, etc.

Davison, R.M., Martinsons, M.G., Kock, N.: Principles of canonical action research. Information Systems Journal 14, 65–86 (January 2004)

Software maintenance

Software product management Customer support

Developers not used to involve reports during

maintenance

PMs rely on customer support and salesmen for release planning

Supporters rely on customer explanation for softw. operation

insight

Accelerated software maintenance

Directed software product (release)

management

Increased customer intimacy

Independent requests from developers Evaluation of EG tool Short talks

Error report meta information (status)

Aggregated error report properties, operation

trends

Operation information enabling particular

identification of customers

Activity and deliverable descriptions



Maintainers were selected by us and appointed by CEO

PMs were selected by us and appointed by

CEO

Customer supporters were selected by us

and appointed by CEO

Analysis of maintenance process: at least once a week

Analysis of SPM process: at least

every Scrum sprint

Analysis of customer support process:

possibly every call

EG list view with error report status

EG aggregated error report view

EG customer-spectific error report views

Creating and introducing error report labeling

Creating and introducing error

report aggregation; organizing meetings

Extending error report format and EG

analysis functionality




EG extension

EG extension, harvest meeting introduction for

frequent analysis of error reports

EG extension

Evaluative talks Evaluative talks Evaluative talks



§  Main result: Increased software quality –  Report submissions increased about linearly from February, 2009

until June, 2010

–  Trend was broken in September, 2010, while a major new version was released


§  Based on observations and interviews, lessons learned are identified –  What should a vendor do to effectively involve such information in its

product software processes?

–  What should it absolutely not do? What are considered pitfalls?

–  Observing integration in retrospect, what are your lessons learned?

SOK Integration Template Method: Lessons Learned

§  1. Integration Processes Should be Lean –  Handle additional (administrative) tasks pragmatically:

additional administration may negate the time gained by integration of operation information


§  2. Integration Responsibilities and Results Should Be Evangelized –  Shared responsibility is no responsibility

–  Increase awareness and acceptance of new way of working

•  Both among employees as at management level!

Who is your evangelist?


§  3. SOK Integration Opens Up Black Boxes –  Yes, informed development, software failure fix prioritization, etc.

–  But: improvement areas that were unnoticed before, are highlighted as such after SOK integration

•  Increased developer awareness (software quality, end-user situation)

•  More customer-central, pro-active development mentality

–  `Build what the customer will use, before the customer asked for it'

•  Clearer and quicker interdepartemental communication

•  No more HiPPOs!


§  4. Continuous Refinement of SOK Integration Objectives and Requirements Leads to Optimization of Integration Results –  Integration results depend on integration objectives and requirements

–  Both should be evaluated and refined continuously to correspond to your product strategy, as well as the needs of your customers

Challenges

§  Visualization

§  Alignment business vs technical metrics

§  Big data (or just real time data) & data mining

§  Errors at different levels

Data Mining and Visualization

Business Metrics vs…

65

What’s your company’s mission statement?

Technical metrics

66

Performance Metrics

Memory

Nonpaged System Memory SizePaged Memory SizePaged System Memory SizePeak Paged Memory SizePeak Virtual Memory SizePeak Working SetPrivate Memory SizeVirtual Memory SizeWorking Set

Processor

Privileged Processor TimeTotal Processor TimeUser Processor Time

Network Interface

Bytes Received/secBytes Sent/secPackets Outbound ErrorsPackets Received ErrorsTotal Bytes ReceivedTotal Bytes Sent

Disk

% Disk Time% Free SpaceAvg. Disk Queue LengthAvg. Disk sec/TransferDisk Reads/secDisk Transfers/secDisk Writes/secFree Megabytes

Web Tier

Page Type CountQuery String CountTotal Request DurationAuthenticate Request DurationAuthorize Request DurationResolve Request Cache DurationAcquire Request State DurationRequest Handler Execute DurationRelease Request State DurationUpdate Request Cache DurationModule Type CountMethod Type Count

Usage Metrics

User Details

Customer IDIP/Hostname

Application Details

Web site vs web applicationHosting

User Actions

Actions PerformedPage/function/method accessTimeframeEntry/Exit pointSession IDErrors

Tata Steel produceert, bewerkt en distribueert hoogwaardig staal voor producten die het leven vergemakkelijken.

Belastingdienst: Geld innen voor de BV NL

Aviva Solutions: In eerste plaats biedt Aviva Solutions u goede ICT-oplossingen door het bouwen van applicaties. Maar buiten zorgen voor continuïteit en vernieuwen van uw applicaties en systemen, helpt Aviva Solutions u graag bij andere ICT-vragen. Als dé Microsoft Platform Specialist zijn we immers expert op het gebied van: Integratie, Business Intelligence, Custom Solutions & Ontwikkelstraten, Information Worker en Internet & Commerce.

AFAS Software automatiseert bedrijven en organisaties uit alle branches met complete en moderne (online) bedrijfssoftware.

So… How to get started?

§  Assess most important KPIs

§  Collect them (tons of tools available)

§  Analyse them (tons of tools available)

§  Report them (…)

§  ACT!

?

Software Operation Knowledge

Technology

software processes

software vendors

data sok

software operation information

software operation knowledge

software ecosystems

software maintenance

field software failures