© Dennis Adams Associates Limited November 2006 Managing Production Systems: A Manager’s Guide to Prevent Firefighting. 15.

15 November 2006© Dennis Adams Associates Limited http://www.dennisadams.net

Managing Production Systems:

A Manager’s Guide to Prevent Firefighting.

15 November 2006

Dennis Adamsa s s o c i a t e s


Background

• Different Systems / Vendors– VMS, TRU64, HP-UX, Ingres, Sybase, Oracle, Monitoring Tools.

• Emphasis on IT Production– Support Administration, Sys Admin, DBA, Capacity Planning

• Assistance to IT Production Managers– Deployment of Tools and Processes

• IT Technical Architecture– Infrastructure Design and Architecture

• IT Production Strategy– Global Standards

• THERE ARE COMMON MANAGEMENT TECHNIQUES FOR RUNNING SUCCESSFUL IT PRODUCTION DEPARTMENTS

• A Consultancy Service Dedicated to the Challenges of Managing IT Production teams.


Objectives of this Seminar

• Identify some of the common challenges facing IT Production Teams today.

• Highlight some of the pitfalls.

• Introduce the MOPS Strategic Approach.• Explain how this can be applied in practice.

• Highlight some “Action Points” to improve the way IT Production can be managed in future.

• Some example brief Case Studies.

• But first, some Technology…


University of Kentucky HP Superdome cluster

• Four HP Superdomes• 256 total processors (64 procs per host/node)• Itanium-2 (Madison) processors rated at 1.25 TF sustained / 1.6 TF

peak.• HPUX 11i v2 (11.23)• 2 Gigabytes of memory per processor• 7 Terabytes of total disk space• High speed, low latency Infiniband internal interconnect• Gigabit connections to public network


Compaq “Laptop”

• The second Compaq luggable. • 20MB hard disk, 360K floppy drive, CGA screen. • A milestone in PC history.• It is the first of the clones.


HP 3000

• Advertised in • Computer Applications Pty • Australia


VAX 11/780

• CED Istituto di Radioastronomia 1981• 1 MIP• 512 KB Ram• 64 MB Mass Storage



VAXMate

• Intel 80286• 1 MB RAM• 5.25” floppy disk• 20 MB 8” hard drive• Ethernet


Common Theme: The two different sides of IT

• IT Development – Business Functionality

– Speed of Delivery

– Cost of Development

– Development Projects may take months

– Creating Competitive Advantage and ROI for the Business

• IT Production– Reliability, Resilience

– Stability, Scalability

– Cost of Support & Maintenance

– Production Support may be required over many years.

– Delivering Day-to-Day Competitive Advantage and ROI for the Business

IT Development and IT Production think and act differently.


Common Theme: Project Development and Deployment

Development Production


Inside IT PRODUCTION


The Challenges facing IT PRODUCTION

• Heterogeneous Data Centre• Lots of legacy applications to support.• More applications being thrown over the wall all the time.

• IT Production can focus on “quick fixes”, rather than long term resolution of the underlying problems.

• Lack of standardisation of production systems, leading to a broader set of skills needed to support it.

• IT Production can be a “reactive”, rather than “pro-active”.• Difficult to justify future investment.

• Threat of Out-Sourcing.


Dealing with the Challenge

• There is no SIMPLE solution to the challenges of Managing IT Production

• For IT production to succeed there should be Managers who are willing to “STEP BACK” and put in place a systematic STRATEGY.

• Four Key Criteria for making IT Production – more Pro-Active,

– more Client-Focused

– a Strategic Managed Environment

– in a better position to justify IT Infrastructure Investment

• METRICS• OPERATIONAL TOOLS• PROCESSES AND PROCEDURES• STANDARDS


The “MOPS” Approach

• The Basis for Pro-Active IT Production Management• Metrics

– Identify Key Information to be captured and maintained in order to make informed Management decisions and justify IT Investment.

• Operational Tools– Advise on the selection and integration of appropriate software

tools to manage the infrastructure and the people who look after it.

• Processes– Implement pragmatic processes based on ITIL and build linkages

with Business and Development team processes.

• Standards– Establish Technical Strategy and Architecture Standards to ensure

that future applications can be supported at optimal cost.


Creating an IT Production STRATEGY

• ANALYSE existing IT Production under the following headings:– Metrics– Operational Tools– Processes and Procedures– Standards

• IDENTIFY the gaps– under each of these headings

• PRIORITISE– from IT Production perspective, but also….

• ENGAGE with Sponsors and Business– Talk about what we are doing, and why

• CREATE the Strategic Roadmap– owned by all stakeholders

• INCREMENTALLY role out changes to the way the department works


METRICS

• “Some numbers which tell me what is happening” • You cannot control what you cannot measure.

• Metrics can help us in several ways:– Help us understand where our resources are being used.

– Identify which Applications are consuming our support costs.

– Explain / Justify what we do to the business.

• We must capture numbers that describe what IT Production does, in terms that make sense to the business.

• This is NOT the same as Technical Performance Monitoring– “CPU % used” means nothing to the business

– A SMALL no of basic Metrics ONLY.


METRICS: Actions

• Create a database of the infrastructure, what it is, and the business purpose it exists for.– E.g. SERVERS by APPLICATION.

• Capture the Time and effort your teams make. – Capture time by SUPPORTED APPLICATION, not by FUNCTION.

– Your biggest budget cost is probably manpower – where is it being used?

• For each APPLICATION, report on amount of time spent on support, no of support calls,

• Report on Key Indicators to business.– “CPU Units” (not Percentage used !)

– “Disk Units” (real allocation)

– Highlight the “costs” of support of business applications.


OPERATIONAL TOOLS

• Software Tools can assist in many ways:– Capture the METRICS on Time Used, Support Calls, Business

Transactions, Infrastructure Supported.

– AUTOMATE the day-to-day support role, Batch processing, Backup, Central Console, Alerting

• The biggest danger with software tools is that they are implemented “in isolation”

• Make sure that you have common naming conventions for all your applications etc. in different tools.– Reports can be cross-referenced.

– Use a “referential” approach.

• Remember that the more tools you have, the more complex the task of integrating them together.


OPERATIONAL TOOLS: Actions

• Review what Operational tools you have.

• Remove duplication of software

• Exploit the capabilities of the software you already have.

• Ensure that all software is integrated– Common naming conventions

– A “referential” approach.


PROCESSES & PROCEDURES

• Properly Implemented Processes can – Speed up the way support is managed

– Reduce costs

– Ensure that the business has a predictable service

• Badly Implemented Processes can– Slow down the ability to respond to business needs

– Increase Costs

– Lead to a loss of Service

• Introduce Processes that add Value, not Bureaucracy.• Concentrate on making it easier for external teams (developers,

business units) to ask you to do things.

• Be Pragmatic


PROCESSES & PROCEDURES: Actions

• Look at the current procedures you have in place, compared with ITIL Standards.

• Make sure that development teams, business units know how you work, and what they need to in order to get things done.

• Put in processes to get involved with development projects at the earliest possible time in the project life-cycle.


STANDARDS

• In some cases, the choice of Technology for a new Application can be driven by Developers’ Choice:– Useful Development Tools ?

– Design and Development Features ?

– Familiarity ?

– The desire to try out the latest technology ?

• Result: Applications whose Development costs may be Low, but the Support Costs may be high (even prohibitive).

• Defining IT Production Standards can redress this balance.• Standards can contribute to controlling Costs of Maintenance &

Support• Simplicity = Economies of Scale in Support


STANDARDS: Actions

• Establish a Production Architecture Role• Collect and publish your Standards Capability

• Make sure that the development teams get involved• Change the Development Project Management Methodology

– Include Production requirements when projects are Initiated.

• Have a process in place for looking at new technologies.

• Define what you mean by “Production Ready”• Validate new products or technologies for “production

readiness” and add them to the list (or not)


MetricsOperational ToolsProcesses & ProceduresStandards

How Does IT WORK

in Practice?

SOME CASE STUDIES


CASE STUDIES (1)

• Infrastructure Support Teams did not know where the costs were going, and there were pressures on budgets.

• METRICS: Introduce Timesheets into IT Production Team– Capture time which teams spent BY APPLICATION

– Enabled us to identify the “problem applications”

– Provided evidence for the Development teams.

– Resulting in application changes

• Reduced costs of IT Support.• Better visibility of costs.• Development Project teams became more “Support Aware”.


CASE STUDIES (2)

• 30 to 40 DBA overnight call-outs to fix database problems.

• OPERATIONAL TOOLS: Implement Alerts from Database Monitoring tool to central Operations Console.

• This was a product that was already licensed to the client, but was not being used !– DBAs advised of problems before they became incidents

– Arrange pro-active outages to fix problems ahead of time.

– Overnight work was not interrupted by incidents

– Better quality of service to the business.

• Number of call-outs was reduced - one or two per week.


CASE STUDIES (3)

• Number of Software changes was high (every day).– Often changes were replaced by “patches” within a few days.

– Significant support and implementation effort.

– Business was not getting the service it wanted.

• PROCESSES: Introduce Change Control Board and formal Change Control Processes, based on ITIL framework.– Number of changes reduced

– Development Testing Effort was signficantly increased.

– Post-implementation support effort reduced.

• Business saw software changes implemented “right first time”


CASE STUDIES (4)

• Business Development Proposal to introduce a new software package which was functionally rich, but would be impossible to support in production.

• STANDARDS: Created a Production Architecture Team.– Engaged with Developers who were looking at this package, to

emphasise the “Production Readiness” costs.– An alternative solution was chosen.

• The costs of the proposed package in PRODUCTION could have been many times more

• Although attractive to Developers, it could have resulted in massive financial loss to the company.


CASE STUDIES (5)

• Various teams had incomplete spreadsheets listing machines they were supporting and what applications they ran.

• METRICS: Initiated the creation of a centralised Configuration Management Database.– Servers: Location in Datacentre, Owner, Spec. – Applications: Server(s)

• A Major outage resulted in the loss of one of the 6 datacentres– A recovery strategy was needed.– What Applications were important?– What Servers to restore first?

• The information was available in minutes, instead of hours• Most of the services were restored within SLA.• Significant Financial Loss was reduced.


Conclusions: How to MANAGE IT Production

• Without a Strategic Approach, IT Production can become “Fire-Fighting”

• Step back - Create an IT Production Strategy• The Key Elements for a Strategic IT Production Approach:

– METRICS– OPERATIONAL TOOLS– PROCESSES AND PROCEDURES– STANDARDS

• Approach:– Analyse your current activity against the “MOPS”– Identify and Prioritise work.– Implement Incrementally

• Using appropriate management techniques, it is possible to run IT Production in a “Pro-Active” way, without “Fire-Fighting”


Managing Production Systems

http://www.dennisadams.net

email: [email protected]

Time for any Questions?


Managing Production Systems:

A Manager’s Guide to Prevent Firefighting.

15 November 2006

Dennis Adamsa s s o c i a t e s

© Dennis Adams Associates Limited November 2006 Managing Production Systems: A Manager’s Guide to Prevent Firefighting. 15.

Documents