Designing a maturity model people will trust

A maturity assessment is only as good as the model behind it. If people don't recognise their work in the questions, they answer carelessly. If the levels don't match what everyone already knows about a team, the results get dismissed. And if the model changes between rounds, the trend line means nothing.

This guide covers the decisions that make a model credible: its structure, the wording of its questions, its answer scales, its levels, and how to change it safely.

Borrow, then adapt

You rarely need to start from a blank page. Established models give you vocabulary that people already know:

• Generic process maturity, in the tradition of CMMI's five levels, works for most business processes.
• The DORA metrics cover software delivery: deployment frequency, lead time for changes, change failure rate and time to restore service. They come with years of published research behind them.
• Your own organisation's standards, such as an operating model or a target process design, are often the most relevant source of all.

Borrow the structure, then cut it down to the capabilities your transformation is actually trying to change. A model that covers everything gets answered as if it matters to nobody.

Structure: domains, capabilities, questions

A three-level structure keeps a model readable:

• Domains group related capabilities, such as Design, Execution, Measurement and Improvement. They make the heatmap easier to scan.
• Capabilities are what you score and act on, such as Process ownership or Hand-offs. Each should be something a unit could improve with a few specific changes.
• Questions are the evidence behind each capability score. Two or three per capability is usually enough.

Aim for a questionnaire someone can finish in ten to fifteen minutes. In practice that means twenty to thirty questions. If it's longer than that, cut capabilities rather than questions; a capability with one weak question is worse than no capability.

Write statements about observable practice

The wording of individual questions does more for honest answers than anything else.

Describe what happens, not how good it is. "The process is documented end to end, including inputs, outputs and hand-offs" can be checked. "Our process documentation is high quality" invites everyone to say yes.

One idea per statement. "Plans are reviewed weekly and shared with stakeholders" is two questions. Someone whose plans are reviewed but never shared can't answer it honestly.

Avoid absolutes. "Always" and "never" push people to the extremes. "Releases go out without a manual approval step" is better than "We never need manual approval".

Use the unit's language. If the finance team says "month-end close", don't write "period-end reporting cycle".

Ask about the present. Respondents drift towards how things should work, or how they will work after the transformation. A short instruction at the top helps: answer for how this works today, not how it should work.

Choose answer scales deliberately

Different questions need different scales. Mixing them is fine, as long as each one is consistent wherever it is used.

Agree/disagree scales suit statements about practice. Five points with a neutral middle is the common choice (the Likert scale, after the psychologist Rensis Likert). Keep the labels the same throughout the questionnaire.

Frequency wording often fits transformation work better than pure agreement, because practices are usually partly in place. "Agree", "Agree sometimes", "Neutral", "Disagree sometimes", "Disagree" lets people say it happens, but not reliably. That is often the most useful answer you will get.

Yes/no suits facts: "Does the process have a named owner?" Don't use it for anything that is really a matter of degree.

Single choice suits questions with natural steps. "How often does the team deploy to production?" with options from less than monthly to on demand gives a cleaner answer than any agreement scale.

Numbers suit things that are already measured, such as change failure rate or days to close. Decide which end is good. For many measures lower is better, so the scale should run from worst to best rather than from smallest to largest.

Whatever the scale, decide the score for each answer up front and apply it the same way everywhere. Even spacing (0, 25, 50, 75, 100%) is the sensible default. Only deviate when you have a reason you could explain to a sceptical respondent.

Make the levels match reality

Levels are where a model wins or loses trust. If the assessment says a team people regard as excellent sits at level 2, nobody will believe the rest of the results.

Describe each level in observable terms. "Documented and followed across the unit" is a level; "Good" is not.

Set thresholds, then calibrate. If capability scores come from questions, each level needs a threshold, such as level 3 from 40%. Before the first real round, score one or two units you know well. If the result doesn't match what experienced people would say about them, adjust the thresholds or the questions, not the people.

Keep the bottom level at zero and the top reachable. A level 5 nobody could ever reach demotivates. A level 5 that half the units hit in the first round tells you nothing.

Use weights sparingly

Weighting lets one question or capability count for more than another. It is tempting to weight everything you consider important. Resist.

Start with equal weights. Introduce a weight only when two conditions hold: the capability genuinely matters more to the programme's outcomes, and you can explain the weighting in one sentence to the people being assessed. Weights that nobody can explain make a model feel rigged.

Pilot before you roll out

Run the model with one or two friendly units before the full baseline. Look for:

• Questions everyone answers the same way. They carry no information; replace or cut them.
• Questions with lots of comments or skipped answers. These are usually unclear.
• Capabilities whose score surprises people who know the unit. Check the questions and the thresholds behind them.
• The time taken. If people are spending more than fifteen minutes, cut it down.

A pilot costs a week. A baseline with a broken question costs you the whole trend line.

Change the model without breaking history

Models improve as you learn, but every change risks making rounds incomparable. Three rules keep history intact:

1. Version the model. When you change it, publish a new version and keep the old one. Each round should record which version it was asked against.
2. Change between phases, not mid-round. Make changes when the programme moves to a new phase, not while units are answering.
3. Compare like with like. When comparing rounds on different versions, compare only the capabilities whose questions didn't change, and say so when you present the trend.

In short

• Borrow a recognised structure, then cut it to the capabilities your transformation is changing.
• Keep to twenty or thirty questions, two or three per capability.
• Write statements about observable practice, one idea each, in the unit's own language.
• Pick the scale that suits the question, including frequency wording for partly-adopted practice.
• Describe levels in observable terms and calibrate thresholds against units you know.
• Weight rarely, pilot first, and version every change.