Patterns

A field guide for buyers of engineering consultancies, with the research behind it.

Patterns that keep engagements running long after the work has stopped justifying them. None of these are rare. Most are invisible to the buyer until it is too late, because the consultancy controls the information, the vocabulary, and the narrative.

Where a claim can be grounded in a study, the study is named, so you can check it yourself rather than taking our word for it. That is the whole philosophy of an audit: evidence over assertion.

If you recognise three or more of these, the engagement is not going well. You are being managed, not served.

A note on the numbers

Two findings frame everything below.

First, most large software projects underdeliver, and this is measured. A University of Oxford study reviewing more than 5,400 IT projects each costing over fifteen million dollars found that on average large IT projects run 45 percent over budget and 7 percent over time while delivering 56 percent less value than predicted (Bloch, Blumberg and Laartz, 2012).¹ The Standish Group's CHAOS research, tracking project outcomes since 1994, has consistently found that only around a third of projects fully succeed, and that large projects succeed far less often than small ones.

Second, the practices that predict success are measurable and well established. A decade of DORA (DevOps Research and Assessment) research, the largest ongoing study of software delivery, identifies four key metrics that distinguish high performers: deployment frequency, lead time for changes, change failure rate, and failed deployment recovery time. (A fifth, reliability, was added in a later iteration.) The 2024 State of DevOps Report, drawing on over 39,000 respondents, found only 18.7 percent of teams qualify as elite, and that the industry is getting worse. The high-performing tier shrank from 31 percent to 22 percent year over year. The low tier grew from 17 percent to 25 percent. Nearly a quarter of respondents deploy less than once a month.

The gap between those two facts — projects fail at scale, yet the practices that prevent failure are known and rare — is the space consultancies operate in. The patterns below are how that space gets exploited.

The account manager shield

The work is poor but the relationship is excellent. The account manager is warm, responsive, available, and deeply concerned about your satisfaction. Meanwhile the delivery team is underperforming, and you never speak to them directly. The account manager exists to keep you comfortable while the meter runs. Their job is retention, not delivery.

How to spot it. If your primary contact is not the person writing the code or leading the technical work, ask what the account manager actually produces. If the answer is meetings and status reports, you are paying for a buffer between you and the truth.

Moving the goalposts

The original scope was clear. Six months in, what was promised has quietly shifted. This rarely happens in a single dramatic moment. It happens across dozens of small conversations, each reasonable in isolation: "we discovered some complexity," "the requirements evolved," "we re-prioritised based on learnings." By the time you compare the current state to the original proposal, the two bear no resemblance to each other.

This is a documented failure mode. Scope and requirements instability is one of the most consistently cited contributors to project failure in the Standish CHAOS research, alongside lack of user involvement.

How to spot it. Keep the original proposal and SOW somewhere you can find them. Every quarter, compare what was promised to what is live and working in production. Anything described as "in progress" or "almost done" does not count. The gap is the finding.

Vocabulary inflation

The consultancy uses precise-sounding language that makes simple things sound complex and complex things sound handled. "We're implementing a microservices architecture with event-driven choreography" might mean "we split the app into pieces and now nothing works reliably." The vocabulary serves two purposes: it makes the consultancy sound expert, and it makes the client feel unqualified to ask questions.

How to spot it. When a consultancy uses a technical term, ask them to explain it in plain language and show you where it is. "Show me a deployment that happened this week" is better than any definition. If the explanation is more jargon, or they cannot show you the thing they named, the term is decorative.

Claiming expertise that does not exist

The proposal said the team had deep experience in your domain or stack. The people who show up have little or none. The experienced architect who appeared at the pitch is not on the project, or is "available for consultation" but never actually present. The pitch team and the delivery team are different people, and the handoff happens after the contract is signed.

How to spot it. Ask for the CVs and experience of the people who will actually be on the project: the delivery team, not the pitch team. Compare to the expertise claims in the proposal. Revisit when team members rotate, which they will.

The "AI exception"

"AI changes everything. Traditional testing approaches don't apply. We need a different kind of quality assurance for AI-generated code. Our team has developed a proprietary methodology."

This is the current favourite. AI is new enough that buyers cannot easily challenge the claim, and it provides cover for abandoning the engineering discipline that was already absent. The truth: AI-generated code has to integrate, be tested, deploy, and survive production, exactly like code written by a person. A consultancy claiming AI requires different standards is using novelty as cover for lowering the bar.

How to spot it. Ask what, specifically, is different about their AI methodology. Ask to see it. Ask how many deployments to production happened this month. If AI is accelerating delivery, the deployment log should show it. If it does not, the acceleration is theoretical.

Reframing failure as learning

The sprint failed. Features were not delivered. But the retrospective says "we learned a lot this sprint" and "the team is building velocity." Learning matters, but it cannot be the only deliverable, sprint after sprint, in an engagement you pay for by the week.

How to spot it. Count what shipped to production, not what was discussed. If the same "learnings" appear in multiple retrospectives, nothing was learned; the same problems are repeating and being narratively repackaged. Deployment frequency cuts through: a team that is "learning" but not deploying is a team that is failing, regardless of how the retro reads.

The complexity excuse

Everything takes longer because the consultancy "discovered complexity." Sometimes this is true. Often it is cover for poor estimation, inadequate discovery, or a team not skilled enough for the work. The consultancy framed an aggressive timeline to win the deal and is now managing the gap by attributing the shortfall to forces beyond their control.

A team that estimates from empirical lead-time data — measuring how long similar work actually took in the past — catches this before the estimate leaves the room. A team that estimates in story points or gut feel does not. The estimation model determines whether the excuse has anywhere to hide.

How to spot it. Ask whether the complexity was knowable when the proposal was written. If they did a discovery phase and still missed it, the discovery was inadequate. If they did not, they estimated blind. Ask what specific complexity was found, when, and what the revised estimate is with a rationale. Vague complexity means nothing. Specific complexity with a documented impact is worth discussing.

The staffing shell game

Senior people rotate off. Junior people rotate in. The consultancy calls it "team evolution." What is happening: the expensive senior staff won the deal and established the relationship, and now that the contract is signed they are moved to the next sales engagement while delivery is handed to less experienced, less expensive staff, billed at the same rate.

How to spot it. Track who is actually on the team, month by month, against what was proposed. If the team that pitched is not the team that delivers, and the replacements are more junior, the margin is being optimised at your expense.

The never-ending discovery

The engagement starts with a discovery phase — legitimate and often valuable. The trick is when discovery never ends. Phase one recommends phase two. Phase two recommends a pilot. The pilot recommends a proof of concept. At each stage the consultancy produces a document that justifies the next document. Working software is always one phase away.

How to spot it. Ask when working software will reach production, where real users can use it. If the answer is always "after the next phase," the engagement is structured to produce documents, not software.

Deliberate dependency creation

The consultancy builds the system so only they can maintain it. Key decisions are undocumented. Knowledge lives in the heads of the consulting team and leaves when they do. This is what happens when an engagement is structured around delivery with no plan for handoff, and the consultancy is incentivised to stay rather than make itself unnecessary.

The research is clear. Analysis of consultant dependency identifies it as a structural outcome of contract design: most consulting contracts are time-and-materials or phased deliverables, and neither model rewards finishing faster or transferring capability. Organisations that do not explicitly plan for the capability to live in-house end up in a dependency state where the work was done but the knowledge did not transfer.

How to spot it. Ask: could our internal team operate this system tomorrow if the consultancy left? If the answer is no, ask what the plan is to change that, and when. If there is no plan, the dependency is the business model.

The dashboard of distraction

The consultancy produces impressive dashboards. Velocity trends upward. Story points completed per sprint increase. Burndown charts look healthy. None of these measure whether working software is reaching production. Velocity measures effort. Story points measure estimates. A burndown chart can reach zero in a sprint where nothing ships.

The DORA metrics exist specifically because these proxy measures fail. Deployment frequency, lead time, change failure rate, and recovery time were validated because they measure outcomes that cannot be gamed by working harder at the wrong thing. Velocity and story points appear in none of the DORA key metrics, because they measure the inside of the process rather than its result.

How to spot it. Ask the one question the dashboard does not answer: how many times did we deploy to production this month? If the dashboard tracks velocity and burndown but not deployment frequency or lead time, it measures the theatre.

The transformation that never transforms

The consultancy was engaged to run an "agile transformation" or "digital transformation." Months in, there is a new org chart, new role titles, new ceremonies, squads and tribes, perhaps a Transformation Lead and a Head of Ways of Working. Nothing about how the software is built or shipped has changed. The same code sits in the same long-lived branches. Deployments happen at the same frequency as before.

This gap is measurable. Industry analysis finds that while a large majority of companies claim successful agile adoption, only about half actually operate in an agile way. The CHAOS research is clear that genuine agile practice roughly triples success rates over waterfall. The cosmetic version forfeits a real, measured benefit.

How to spot it. Compare how the team was described before the transformation with how software reaches production now. If the vocabulary changed but deployment frequency, integration cadence, and test automation did not, the transformation is cosmetic. Ask to see the deployment log from before and after. That comparison is the finding.

The demo that is not production

The demo looks excellent: features work, the design is polished, stakeholders are impressed. What is not mentioned is that it runs on a local machine or a purpose-built demo instance that has never been through the deployment pipeline and has no real data behind it. The gap between "works in a demo" and "works in production" is where most delivery failures live.

A team practising continuous delivery treats deploy and release as separate events. Code is deployed to production continuously; features are released to users when ready. In this model, there is no demo-production gap, because the demo is production. Any team that can only show you a demo and not production is revealing, by that fact alone, that continuous delivery is not happening.

How to spot it. When shown a demo, ask: is this running in production right now? Can I use it with real data? A team shipping continuously should be showing you production, because there should be no meaningful difference.

Metric gaming

The metrics look good because they have been optimised at the expense of the work. Velocity rises because estimates are inflated. Tickets are closed as done when merged to a branch but not deployed. Test coverage is high because trivial tests were written to hit the target. This is the predictable result of measuring a team by proxy numbers: Goodhart's law, where a measure that becomes a target stops being a good measure.

Story-point estimation is particularly susceptible. There is no external standard for what a "point" means; the unit is defined by the team using it, which means the team can redefine it at will. A velocity chart that trends upward may reflect faster delivery, or it may reflect nothing more than inflated estimates. You cannot tell from the chart alone.

How to spot it. Look for divergence between metrics and reality. If velocity rises but deployment frequency is flat, work is being counted but not shipped. If coverage is 80 percent but bugs keep reaching production, the tests are not catching what matters. The hardest metric to game is deployment frequency: either code reached production or it did not.

The "best practice" shield

The consultancy justifies decisions by invoking "industry best practice." The phrase sounds authoritative and is almost impossible to challenge. In reality it usually means "what we already know how to do." There is no governing body that defines best practice in software, no standard, no register. The phrase is a rhetorical device.

How to spot it. When a consultancy says "best practice," ask: whose? Where is it documented? What evidence supports it? A genuine evidence-based practice (continuous integration, trunk-based development, the DORA metrics) can be cited and verified. A vague "best practice" that conveniently matches the consultancy's existing playbook is opinion wearing a lab coat.

The technical sections

The microservices sales pitch

The consultancy recommends a microservices architecture. It sounds modern and scalable. What is not mentioned: microservices are a tool for large organisations with many teams that genuinely need to deploy independently. For most teams, and nearly every greenfield project, they add operational complexity, slow delivery, and create failure modes the team is not equipped to handle. The consultancy often recommends them because microservices are what they sell: they have the architects, the platform engineers, and the Kubernetes expertise on the bench. The recommendation follows the bench, not the need.

The academic literature treats this as a known hazard. A 2023 systematic review in IEEE Transactions on Software Engineering covering the decomposition of monoliths into microservices (Abgaz et al., 2023) catalogues how difficult correct decomposition is and how often it goes wrong. Industry migration studies (Fritzsch et al., 2019) document that teams routinely underestimate the practices required.

How to spot it. Ask why microservices and not a monolith. If the answer is "scalability," ask how many users the system has and what load is anticipated. If "team independence," ask how many teams work on the system. One team does not need twelve independent services. If the consultancy cannot articulate the specific problem microservices solve that a monolith does not, the architecture is a solution in search of a problem.

Microservices without the prerequisites

The architecture has been adopted, but the practices that make it survivable have not. This is where the real disasters happen, and it has a name in the literature: the distributed monolith — a system with all the operational complexity of distributed services and all the tight coupling of a monolith, delivering the benefits of neither.

The prerequisites that are almost always missing:

No contract tests. Each service exposes an API others depend on. When a service changes its API, dependents break — silently, in production, sometimes days later. Contract tests verify each service's API still matches what its consumers expect, on every change. Without them every deployment is a gamble. Ask: do you have contract tests between services? "We have integration tests" is not the same answer.

A shared QA gate. The entire point of microservices is independent deployability. A shared QA stage where all services must be tested together before any can release destroys this: the services are coupled at the testing layer. You have a distributed monolith with a shared bottleneck. Ask: can each service deploy to production independently, without waiting for any other service? If the answer involves a shared environment or a release train, the independence was never real.

End-to-end test dependency. The team relies on a full end-to-end suite running all services together to feel safe releasing. No service can release until every service passes every test. The suite is slow, flaky, and grows into the single biggest impediment to delivery. Ask: how long does the full E2E suite take, and how often does it fail for reasons unrelated to the change being tested?

No service ownership. In a functioning architecture each service is owned by a team that builds, deploys, and operates it. In a failing one, services are collectively owned by everyone, meaning no one. When something breaks, nobody knows who is responsible. Ask: for each service, who is on call, who deploys it, who decides what changes?

Shared databases. Two or more services read and write the same database. This is the most common way to build a distributed monolith. Services that share a schema cannot be deployed or scaled independently, and a change in one service's data model can silently break another. Ask: does each service have its own data store? If services share a database, they are a monolith accessed via network calls — strictly worse than a monolith accessed via function calls.

No observability. A monolith fails in one place. Microservices fail across a network, partially and confusingly. Without distributed logging, tracing, and monitoring, the team cannot tell what is happening when something breaks. Ask: when a request fails, can you trace it across every service it touched?

The cumulative effect. These prerequisites are almost always missing together, because the team that lacks contract tests is the same team that relies on E2E testing, shares a QA gate, and has no service ownership. The practices were never established before the architecture was adopted. The result is a system harder to build, test, deploy, and debug than the monolith it replaced, delivering none of the benefits that justified the change.

How to spot it. Ask the six questions above. If the answers are: no contract tests, shared QA gate, slow E2E suite, collective ownership, shared databases, no tracing — the architecture is a liability. That is a finding on its own, regardless of how the code looks.

The AI sections

The research here is recent, consistent, and largely unflattering to the marketing. It is worth reading before accepting any AI pitch.

AI as a substitute for engineering discipline

The consultancy claims AI has changed how software is built — code is generated faster, velocity is up, the team is "AI-native" — with the implication that traditional discipline matters less. This is the most dangerous version of the AI pitch, because the evidence points the other way: AI-assisted development demands more discipline, not less.

Three findings matter.

First, the Stanford user study (Perry, Srivastava, Kumar and Boneh, "Do Users Write More Insecure Code with AI Assistants?", CCS 2023) found that participants with access to an AI assistant wrote significantly less secure code than those without, and were simultaneously more likely to believe their code was secure. The authors named this a false sense of security. The participants who trusted the AI least and engaged most critically with their prompts produced the most secure code.

Second, this compounds at the delivery level. For two years running, DORA has found that AI tooling correlates with worsened software delivery performance: faster individual code production, but worse throughput and stability for the team as a whole. More code arriving faster, without the practices to absorb it, degrades delivery.

Third, a 2025 Sonar study analysing output from five leading models across 4,442 coding assignments (Sabra, Schmitt and Tyler, 2025) found that all evaluated models introduced bugs, security vulnerabilities, and code smells including hard-coded credentials and path-traversal flaws. A model's functional benchmark performance did not predict the quality or security of its code. Passing the tests does not mean the code is sound.

How to spot it. Ask: since adopting AI tooling, has deployment frequency increased? Has the test suite grown proportionally to the code produced? Is every AI-generated change integrated to trunk daily and tested automatically? If "AI generates code faster" but deployments have not increased and test coverage has not grown, the AI is producing inventory, not delivery.

The AI strategy that is just a deck

The consultancy was engaged to deliver an "AI strategy." What was delivered is a presentation: a maturity model, a roadmap with phases, a recommendation to "establish an AI Centre of Excellence," and a list of technologies named without specifying what problem each solves for your business. No working software. No model trained or evaluated. The strategy recommends more consultancy work to implement the strategy.

How to spot it. Ask what working software the engagement will put into production. If the answer is "the strategy informs the next phase," the engagement is a document that sells more engagement.

The AI demo problem, amplified

AI makes the demo-to-production gap dramatically worse. A consultant can build an impressive AI demo in an afternoon: a chatbot over your documents, a classifier, an "autonomous" agent. The demo works. What it does not show: how the system handles wrong answers (it will produce them), behaviour on real data at scale, how outputs are evaluated for accuracy, the failure mode when the model hallucinates confidently, production monitoring, what happens when the underlying model is updated or deprecated, and the real cost at production volume.

The false-confidence effect Perry et al. found in developers is present in the systems themselves: AI output is often superficially plausible and syntactically correct while containing subtle logical errors. A demo is precisely the setting where superficial plausibility is most convincing and least tested.

How to spot it. Ask: what is the accuracy on real data, measured how? What happens when the model is wrong? What is the monthly cost at full production volume? What monitoring exists for output quality? Vague or deferred answers mean the demo is a sales tool, not a prototype.

Selling "agentic AI" without the foundations

The current pitch: autonomous agents that perform workflows, make decisions, and take actions without human involvement. What is not mentioned: agentic AI is the most failure-prone, hardest to test, hardest to monitor, and least predictable form of AI deployment. An agent that autonomously takes actions can autonomously take wrong ones, at speed, at scale, with no human check.

This is the microservices sales pitch repeated with higher stakes: the advanced architecture sold to organisations that often lack the basics — clean data, reliable pipelines, even a working CI/CD practice.

How to spot it. Ask:

• What actions can the agent take? What is the blast radius if it takes the wrong one?
• How do you test an autonomous agent? What does the test suite look like?
• How do you monitor agent behaviour in production? How do you know when it is wrong?
• What is the rollback mechanism when an agent makes a bad decision at scale?
• What evaluation framework measures whether the agent produces correct outcomes?
• Has the team built and operated a simpler, non-agentic AI system in production first?

If the answers are vague, or the team has never shipped a simpler AI system, they are proposing to skip the fundamentals.

The prompt engineer as expert

The project is staffed with an "AI engineer" whose primary skill is writing prompts. They may be good at eliciting useful outputs from a model. What they may lack: software engineering skills, production experience, understanding of testing and deployment, the ability to build the infrastructure that makes a model production-grade. Perry et al. is again relevant: the participants who did best with AI were those with the engineering judgement to engage critically with its output, not those most fluent at prompting.

How to spot it. Ask the AI team member to describe the production infrastructure — not the prompts, the infrastructure. How is the system deployed? How are outputs evaluated? How are costs monitored? How is the model updated? Answers about prompts rather than systems mean the role is miscast.

Patterns that cross the line

Everything above is bad practice. What follows is worse: grey areas that thrive on ambiguity, and in some cases, fraud. A buyer who recognises these should consider not just an audit but legal advice.

Budget burning

Common in government. The department was allocated a budget for this financial year. If it does not spend it, next year's allocation is cut. The consultancy knows this. Work is commissioned not because it is needed but because the budget must be consumed. The deliverable is the invoice.

How to spot it. Engagements starting in the last quarter of the financial year with aggressive timelines and vague scope. Work difficult to connect to any strategic objective. A general atmosphere of "we just need to get this done before June."

Why it matters. This is public money. The pattern is rarely illegal, but it is waste, and it creates a cycle where budgets are justified by spending rather than outcomes.

Writing the rules, then winning the game

The consultancy helps write the RFP, the technical specification, or the evaluation criteria — then bids on the work, or a closely affiliated firm does. The specification has been written, consciously or not, to favour the consultancy's own capabilities. Competing bidders face requirements that happen to describe the incumbent's exact offering.

How to spot it. Check whether the firm that helped define the requirements is also bidding, or whether a closely affiliated firm is. Look for specification language unusually specific to one vendor's approach.

The revolving door

The person who awards or manages the contract leaves the department and joins the consultancy. Or the consultancy's engagement lead joins the client and continues to engage their former employer. Not always corrupt — people change jobs — but when systematic, the conflicts are structural and the incentives to challenge poor delivery are undermined.

How to spot it. Look at the career histories on both sides. LinkedIn makes this trivially easy. If the client-side sponsor previously worked at the consultancy, or the account lead previously worked at the client, the relationship may not be arm's length.

Why it matters. This is a governance observation, not a legal finding. But it explains a great deal about why poor work goes unchallenged.

Ghost resources and rate inflation

The consultancy bills for twelve. Eight are active. Two attend occasional meetings. Two have never appeared. Or the proposal named senior developers at senior rates and the work is done by juniors at the senior rate. Billing for people who do not exist is straightforward fraud. Billing senior rates for junior staff is a grey area that depends on the contract.

How to spot it. Track who is actually working, by name, each week, against the invoice. If the invoice says twelve and the standups have eight, the other four are the finding.

The "independent" review that is not independent

The client suspects failure and engages a second consultancy to assess the first. The second has a commercial relationship with the first: a partnership, a referral arrangement, a shared parent company, or simply a mutual interest in not setting the precedent that consultancies audit each other harshly. The review returns mild findings and recommendations to "improve communication." The client paid for assurance and received confirmation.

How to spot it. Before engaging anyone to review a consultancy's work, check for any commercial relationship. Ask directly. Check their website for partnership logos. If the reviewer lists the firm under review as a "partner," the review is compromised before it starts.

Why this matters here. This is the specific problem vendor-blind auditing exists to solve. Withholding the consultancy's name removes the conflict at the root: an auditor cannot be biased toward a firm they do not know they are assessing.

Manufactured urgency

The deadline was set by the consultancy during the proposal, or accepted without challenge. It is now at risk — because it was always unrealistic, or deliberately underestimated to win the deal. The response: a war room, overtime, weekend work, additional consultants at surge rates. The crisis feels real, and the team genuinely works hard. But the crisis was manufactured by the estimate that created the engagement. The consultancy is now paid more — for the surge — to fix a problem they created — with the estimate.

How to spot it. When a project enters crisis mode, return to the original estimate. Was the timeline realistic when proposed? Did anyone challenge it? If the consultancy set the timeline, sold the deal on it, and is now billing surge rates to meet it, the crisis is a feature of the business model.

Undisclosed vendor incentives

The consultancy recommends a specific tool, platform, or vendor. The recommendation sounds technical and objective. Undisclosed: the consultancy has a referral arrangement, partnership deal, or revenue share with the vendor. The advice is not independent; the recommendation is a sales channel.

How to spot it. Ask: do you have a commercial relationship with them? Are you a partner, reseller, or referral partner? Check the consultancy's website for partner logos and certification badges. If they recommend a platform and their site says "Advanced Consulting Partner" for that platform, the recommendation is not independent.

What a healthy consultancy looks like

Not every engagement fails. These are the signs that the work is real:

They show you production, not demos. The team can show working software in production at any time, because it is deployed continuously. There is no "demo environment" that diverges from reality.

They introduce you to the engineers. Your primary contact is the person doing the technical work, not an account manager. You talk to the team directly and regularly.

They document decisions as they go. Architectural decisions, trade-offs, and context are recorded continuously, not reconstructed into a handover document at the end.

They make themselves progressively less necessary. Internal staff work alongside the consulting team from the start, paired on real work, building the capability to operate without the consultancy.

They can explain the architecture in plain language. No jargon shields. If they cannot explain what they built and why in terms a non-technical stakeholder can follow, the complexity is a warning sign, not a virtue.

They measure outcomes, not activity. The team tracks deployment frequency, lead time, and change failure rate — not velocity, story points, or hours logged. They can show you the deployment history and it tells a clear story.

They estimate from data, not from hope. Forecasts are based on how long similar work actually took (lead-time distributions), not on planning poker, t-shirt sizes, or whatever the most persuasive person in the room asserts. The data is visible to you, not just to them.

They limit work in progress. The team finishes work before starting more. One active thing per person, or close to it. If everyone is working on five things, nothing is shipping.

They treat deploy and release as separate events. Code reaches production continuously. Features are released to users when ready, via feature flags or similar mechanisms. There is no high-drama "release day."

They welcome scrutiny. They are comfortable with an independent audit and do not resist it. A consultancy that objects to its work being independently examined is telling you something.

References

These patterns range from common bad practice to potential fraud. If you recognise them in your engagement, an independent audit can document the evidence, with each finding tied to the actual artefacts. What to do with that evidence is a decision for you and your advisors. mainline.systems