#ISSS2015 Berlin - Gilbert et al - Understanding Systems Engineering Project Development

Dawn Gilbert – [email protected] Mertens – [email protected]

Mike Yearworth – [email protected]

Understanding Systems Engineering Project

Development – A Traditional and Complex Adaptive Systems

View

Question

How does complexity in the organization affect the ability of systems engineers to meet delivery expectations in terms of cost

and time?

The Research

• Critical realist exploratory empirical case study• Participant observation• Data supported reflection in real-time and analysis at later stage• Case study design informed by Yin (2014) and Palmberg (2009)

Novelty• ‘An Investigator has the opportunity to observe and analyse a

phenomenon previously inaccessible to scientific investigation’ (Yin, 2014)

• Empirical CAS research of Systems Engineering Development (The Evidence Centre, 2010)

• Logically coherent case study research in Systems Engineering (Martin and Davidz, 2007), (Valerdi et al., 2010)

Organizational PM

• ‘a field dominated by reductionist paradigms and perspectives’ (Chia, 2013)

Chia, R. (2013) Paradigms and perspectives in organizational project management research: implications for knowledge-creation. In: Drouin, N., Müller, R. and Shankar, S. (eds.) Novel Approaches to Organizational Project Management Research: Translational and Transformational. Series: Advances in organisation studies.

Copenhagen Business Press: Copenhagen, Denmark, pp. 33-55. ISBN 9788763002493

Complex Adaptive Systems

Holland, J. (2014) Complexity – A Very Short Introduction, Oxford University Press, Oxford, UK

• Comprised of elements called agents

• Agents learn or adapt in response to interactions with other agents

• Agents exhibit bounded rationality• Non-linearity • Self-organizing • Agents have 3 levels of activity:– Performance (moment-to-moment

capabilities)– Credit-Assignment (rating the usefulness

of available capabilities)– Rule-Discovery (generating new

capabilities)

Thales UK Website

Thales UK

Source: Hobday, M. (2000) The Project-Based Organisation: An

Ideal Form for Managing Complex Products and Systems? Research Policy, Vol 29, pp871-

893

SE Function of a Business Line

Analysis

• Key Stakeholder, plus two steps away in org chart• Three themes– Software sprints– Staff Circulation and Restructure– Management Tools and Databases

• Does a ‘traditional’ view explain what is observed– Are we surprised at what we find?

• Does a CAS view explain what is observed– Are we surprised at what we find?

Analysis – Software Sprints

Systems Engineering V

Scrum Framework

"Scrum Framework" by Source (WP:NFCC#4). Licensed under Fair use via Wikipedia - https://en.wikipedia.org/wiki/File:Scrum_Framework.png#/media/File:Scrum_Framework.png

"SampleBurndownChart" by Pablo Straub - Own work. Licensed under Public Domain via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:SampleBurndownChart.png#/media/File:SampleBurndownChart.png

Committed (x) vs Completed (y)

0 200 400 600 800 1000 1200 1400 16000

200

400

600

800

1000

1200

f(x) = 0.681261809060857 xR² = 0.948341699011878

All Data Points

Completed

Linear (Completed)

Committed (x) vs Completed (y)

0 50 100 150 200 2500

50

100

150

200

250

f(x) = 0.535438483008886 xR² = 0.383072959839063

Data Points Under 200

Completed

Linear (Completed)

What’s happening?

• Semi-structured interview – Scrum Master, Product_1

• Staff_2 (software head of function): • Programme Review Meetings review the health of the project, we want to underpin that with numbers….. 90% of

info of what’s going on on projects comes from walking the halls, the Programme Review presents the same info, and lands surprises, metrics aim to balance the Programme Review to 50/50 coherent data. We are doing agile across everything…We are using jira out of the box for tasking and defects…The burndown chart and velocities we get from jira…. We have a 3 month sprint look ahead, where we can compare the backlog versus the team size, but not all the work is in the backlog… I already look at software burndown daily.

• Staff_3 (programme management head of function):• The main risk is in software development. We look at the Programme Review pack and we don’t get good info on

the state of the project…. The business objective here, I want quantitative and predictable software performance, delivery against budget.

• Staff_7 (Product_2 programme manager): • I want to know, what is our capability? What we struggle to do is estimate and stick to it. Across a number of

projects it is pretty much the same thing year after year in the software domain. … I look at Jira almost daily. One cause of problems is other activities, like business winning that’s over and above, but knowing that, we said we do the planned work.

• Staff_5 (Product_1 Scrum Master): • If issues are not authorised (by the CCB – Configuration Control Board) they don’t get on the backlog…. .stories

as soon as you create them appear on the backlog Backlog isn’t a bucket of all the work left to do – it a ‘no worse than’ view... Reports are frozen with history, containing planned and actuals…The asterisk means added to the sprint after the start, if it goes in, its urgent, it can come out if its not needed, something else changes, or there’s no time… Velocity doesn’t work for us, it assumes we have a constant team, and the team doesn’t work on the sprint work 100% of the time… It would be interesting to know how much of the Primavera and Business Planning backlog is in the jira.

Staff_5: (Product_1 Software Scrum Master)‘The programme review packs still don’t get looked at, they don’t look at my supplementary pack [which reports sprint performance, lessons learned, engineering metrics, context, achievements, and upcoming milestones] it’s all the SOFT, finance, QA stuff, it makes it soul destroying…I strive to give full transparency and highlighted where there have been issues elsewhere’.

Staff_1: (Acting Operations Director, and Head of Systems Engineering Function) I think from a business perspective we need, what we needed was some relatively simple metrics that could be collected easilyResearcher: I think on Product_1 you’ve been seeing some software metrics the last couple of sessions in there [referring to Programme Review Meetings]Staff_1: (Acting Operations Director, and Head of Systems Engineering Function) It might do if I’d been in the last session with Product_1, I don’t think I was in the last session with Product_1 so, no if I’m honest, no it doesn’t.

Analysis 2 – Staff Circulation and Restructure

• Business Line Performance Plan– Staff rotation target 10%/yr.– This underpins a business challenge to develop a high performance team

culture by optimizing and enhancing capability.– Feedback would be gathered from managers and staff, quarter by quarter

to continuously improve the approach

Analysis 2 – Staff Circulation and Restructure Staff_5 (Product_1 Software Scrum Master)We have 1000-1100 hours of work per sprint. What it is doesn’t really matter, we know we need a good range. When we change people, we change velocity. The team productivity is the velocity, velocity changes from sprint to sprint. There is an administrative overload related to juggling tasks so people who have fewer skills are occupied, so there are pick-ups and drop-offs of work. If you look at capacity in advance, it reduces the capacity! We have varying utilization a rule of thumb is that we have about 10 percent waste, someone will be off, and chew up several days. A 2-day task for handholding is non-specific, you have to add in the time, it looks like the sprint got bigger but it was expected to increase anyway. I looked at work pick ups and drop offs versus focussed depth and asked why? We got support interrupts (ie tech support from users in the field), we hit a blocker like we were waiting for SE/PDA who needed something, or we are juggling skills of task to the skills of the people…. We’ve got 2 new staff, they are struggling to get their machines up to date….. People moved off the project…’

Staff_6: (Product_2 Software Scrum Master)‘We are rotating staff left right and centre so we have programme mobility we need a consistent approach … A restructure is afoot…. The last 2 sprints had to pay significantly for new people. You don’t get it and you won’t get it for several months, this is the rotation plan, One or two people are so heavily relied upon, like the tech lead, its like fighting a losing battle….’

https://forio.com/simulate/k.e.v.oorschot/get-fat-fast-hiring-chains-in-projects/simulation/#p=page3

Staff_1: (Acting Operations Director, and Head of Systems Engineering Function)‘ah, it might be that we’ve had a worse impact that we thought, that’s enlightening, we want to cope with the perturbation…maybe we are a more competent engineering organisation than we thought’

After conceptualizing discrete event modelling• Staff_1‘I like that, I like that…you know what, I’d be quite keen to try and do something like this on something like Product_4’

Two weeks later I heard Staff_1 had submitted his resignation

Analysis 3 – Management Databases

• Staff_1: (Acting Operations Director, and Head of Systems Engineering Function)• We have an operations meeting, the outputs of project review meetings don’t go in to it, it’s a supply and

capability review for Business Planning…..the demand on the business human resources now and out to 2 years looking at potentials and set demand…. We have a heatmap for 12 months, Business Planning hasn’t changed the tactical planning over the next 3 months, we’re not very good at 3-12 months, 12-24 month Business Planning is better, it used to be a 12 month rolling wave, now we have reliable data for 15 months, 18-24 months is not so reliable. I think we make decisions on up to 15 months, this is what we can reliably believe, we keep watch on 5-18 months. Staff_18 [Thales UK Corporate Business Planning] was hard on prescriptive direction, now it’s ‘you tune the process for what works for you’, we’ve gone full circle back with everyone being in the room at the same time…..You’ve got to have human beings in the loop.

• Staff_2: (software head of function• There is general support for jira, we’ve transitioned to tool up for agile. We want to integrate to primavera for

work package management and earned value management. We have a 3 month sprint look ahead, where we can compare the backlog versus the team size, but not all the work is in the backlog… Business Planning deals with resourcing… I already look at software burndown daily.

• Staff_3: (programme management head of function):• We’ve been trying to sort out how we do agile, using Primavera for schedules and reviewing in project

reviews...If we have our bow-wave of known work and forecast wave beyond that, we can match it to resources…Primavera has always been a bit of a dark art. Last year we had a workshop on linking primavera to jira...Primareva tells you how many people you need in the sprint to do it. The actuals should be in Jira and Primavera. The Primavera month is a financial month, we export from Jira back to Primavera as target EVM.

The Functional Manager View

• Staff_5: (Product_1 Software Scrum Master)• It would be interesting to know how much of the % primavera Business Planning backlog is in the Jira backlog.

[When looking in Jira] make sure what you are looking at is what you are thinking you are looking at.• Staff_6: (Product_2 Software Scrum Master)• Jira is the backlog manager…. We don’t have backlog reviews yet, which will be weekly with programme

managers and the integrated product team (we have Jira drive this). We have only got so much capacity, what you do is what you do…. Software estimating is crap… we are underperforming, we are incapable, and you can’t ascertain that from a sprint burndown. Jira will help us in time track how much time we spend, but we haven’t met one sprint yet, we are on sprint 5 at the moment….. in Jira we know day by day we are ‘not performing’, but even when we know we don’t use open language in stand ups

• Staff_7: (Product_2 programme manager):• I look at Jira almost daily…. Demand is now run from Primavera, which has problems, its driven by Business

Planning. We use the same tool for different purposes.• Staff_9:Systems Engineering Manager – Product_3 • Jira is used to monitor, which is the best tool seen so far…. I monitor the burndown in jira, it gives KPIs which

are tweaked and added to… • Staff_10: ‘Front-End’ Systems Engineer – Product_3• The project has three pulls, the PM reports to the PM line…the SEM reports up through to OPS about cost,

money, time, the PDA reports to the design criteria – there is a question of whether this reflects the customer… we don’t have a sound approach to estimating. I’ve made an estimate, what happens to that info? When it comes to lessons learned, we cut and paste the last lot. In actuals, is overtime visible or not? – it’s not visible on actuals. We never see positive methods of capturing, we are mixing apples and pears, we are mixing the contents….

The Project Staff View

• Staff_14: Project Controller – multiple projects• The risk management pack is in the project management pack and is referred to in the Organisational

Management System. There is a monthly update to the risk register during each project, there is an independent assessor at the meeting along with the PM and senior technical team people – the PDA… You enter the risk and its weight in line with Organisational Management System criteria….Risk happens, it is built in to the project schedule, then budget and resource allocated. We should then maintain a log of what happened. In the perfect world we wouldn’t be surprised by late work and cost overruns, they will be in the PMR, the top 5 risks or maybe 10. I wouldn’t be surprised, but Staff_16 (Business Line General Manager) might be. We are not running risk through Primavera. Horse-trading of scope, time, and budget is down to commercial, Staff_15 (Business Line Technical Director) is an independent assessor. The [Organisational Management System] instruction is clear, it works, project managers should be aware of the instruction, I can only assume project managers get it…. I don’t know who makes the decision on what the risks are, I’d say the risk is our estimates are bad. The pressure is coming from above, not from projects….Over a 3 year project we forecast finishing to one day, but we don’t know if that’s a 95% chance of getting there, and we don’t know how long it will be until we get to 100% done – is it 2 months past the end date? If we show a SPI or CPI of 50% we get questions from above, so here we massage it so we show no variance and don’t get questions…. The Work Breakdown Structures are defined prior to Primavera, the project managers say to software, “I don’t care how you do it, just do it”. The cost collection is actually at different levels, for example, from subcontractors and from Jira. The lowest level we go should be the level you get actuals against. Primavera breakdown was mapped against delivery structure routed through work packages by customer, but we estimate and collect cost through product and function. I’m restructuring Primavera to match against what we measure…. I’m resorting and simplifying Primavera to reduce dependencies that are likely to cause errors in updating that are also unnecessary levels of detail for one-person staffing

The Project Staff View

Discussion

• Analysis 1 – Software Sprints– A traditional view would not expect to see this sprint performance

(expected vs completed), and could only explain it as poor performance. Continuous improvement-type approaches within that paradigm could explore why this is, but they weren’t undertaken • (‘its pretty much the same thing year after year’)

– A CAS perspective shows bounded rationality, and agents (staff) adapting in response to others

– The traditional view can’t explain why the scrum metrics data isn’t reviewed by Staff_1 in the Programme Review Meetings that his presence is ‘mandatory’ at (according to the organisational management system). A CAS view understands that a staff member in 2 roles will personally prioritise, and that staff in 2-day-long review meetings may also choose not to review everything’

Discussion• Analysis 2 – Staff Circulation– The traditional view produced the Business Line Performance Plan which

called for staff rotation, outlined it’s aims and described it’s monitoring and improvement approach.

– Staff Circulation was implemented (25% staff rotation during the 6 months field work, a further 25% in the subsequent 6 months was observed)

– Prior to this research, monitoring had not taken place. The traditional view would not expect this, and can’t explain this.

– A CAS perspective explains staff circulation and restructuring inherently.– A CAS perspective also explains a lack of monitoring, as Staff_1 was acting

in two roles, overburdened with work, and made personal decisions about how to prioritize work (agent performance).

– In presenting the system dynamics model to Staff_1 the researcher supported Staff_1 in rule discovery, generating new capabilities, in seeing the impacts of staff rotations differently (i.e. supported by evidence from the business line). Since his resignation followed shortly after, we can’t say whether this new capability would have been assigned sufficient credit to impact the performance (chosen actions) of Staff_1.

Discussion

• Analysis 3 – Management Databases Contd.– The Software and Project Management functional managers show strong

reductionist preferences, believing the data in the databases are explicit and objective, and can be, or are linked in a sensible, valuable way.• This can drive a lack of questioning of the data.

– The staff see value, and use the databases with caution, or interpretation. In the risk example, the databases are explicitly manipulated to avoid questions from ‘above’. The prevalent reductionist approach of senior management understands deviation from plan (or expected) only in terms that means something is at fault.

Question

How does complexity in the organization affect the ability of systems engineers to meet delivery expectations in terms of cost

and time?

Reflection

• The ‘traditional’ approaches don’t explain or expect what is evident

• Staff see the effects of complexity – They don’t necessarily have frameworks to understand them– May instinctively navigate them well• Do they have targets on their backs?

• An unwillingness to be open to ‘other ways’ of viewing the organization can be explained, but not overcome by the CAS view– Performance (moment-by-moment capabilities)– Credit-assignment (rating the usefulness of capabilities)– Rule discovery (generating new capabilities)

Reflection

• Thales UK View– The data is probably familiar to other Tier 1 Systems Engineering

Contractors• The case study is expected to resonate with others, and provide an

illustrative example of the value of novel research approaches in systems engineering management

– The methodology was richly revealing about the state and nature of the Business Line.

– The BL was so dynamic, more standard methods are immediately obsolete.

Thank you