How to choose an AI agent implementation partner: a scorecard

Choose an AI agent implementation partner on evidence, not on labels: give every candidate the same description of one workflow and score their answers on ten criteria. These run from whether the partner will tell you an agent isn't needed, through evaluation before build and an audit trail, to the cost of a single task and whose repository the work lives in.

Before you look for a partner

This guide assumes you have decided to work with an outside partner (if not, start with build vs buy) and have one concrete workflow for agentic AI. The scorecard only works if every candidate answers about the same workflow; otherwise you are comparing pitches, not answers. If you're not sure the task needs an agent at all, first check whether an assistant or a workflow is enough and run through the agent or script checklist.

The partner scorecard: ten criteria

The scorecard has ten rows, and each one ends with a request for evidence. Zero points for a claim, one for a description, two for an artefact you have seen. Before you add anything up, mark the must-have criteria: a zero on any of them rules the candidate out.

Why pilots stall is covered in why AI pilots don't scale. Here, each row says what to require from a partner and what evidence to collect before you sign.

Scoring scale: 0 = no answer or only a claim · 1 = described but no artefact · 2 = backed by an artefact you have seen (a sample, a redacted document, a demo on your data).

The guide doesn't prescribe weights. Before scoring, write "M" next to the rows that are must-haves for you.

Criterion	What good looks like	Evidence to ask for	Score 0-2
1. Tests whether the task needs an agent	Says in writing when a workflow or a script is enough, and why	A redacted example where they recommended the simpler pattern	0 / 1 / 2
2. Real data path	The pilot uses the real, permissioned access path, not data pasted in once	The planned access method, the permission list and how access is revoked	0 / 1 / 2
3. Evaluation before build	A test set with expected outputs and a quality threshold agreed before build, plus a stop rule (how to tell whether an agent works)	A redacted evaluation set or report from previous work	0 / 1 / 2
4. Human in the loop and guardrails	Named approval points where a human in the loop decides with context rather than clicking "OK", plus rules that block misuse	The approval map for your workflow (agentic architecture that passes audit)	0 / 1 / 2
5. Audit trail and observability	Every agent action logged: who asked, what the model returned, who approved. Observability and monitoring continue after go-live	A redacted log or trace, and who reads it after launch (monitoring AI agents)	0 / 1 / 2
6. Integrations and permissions	Access through a controlled interface, such as MCP or a gateway, with least privilege. Credentials never sit in prompts or spreadsheets (MCP and agent integrations)	The integration diagram and the agent's permission list	0 / 1 / 2
7. Model and vendor independence	The model can be swapped without breaking controls. Commercial ties to model, cloud or platform vendors are disclosed (build vs buy)	A written disclosure and a description of how a model swap would work	0 / 1 / 2
8. Cost per unit of work	Running cost estimated per task before start, including usage-based lines, not only the build cost	A cost model for your workflow, with its assumptions (what AI in production really costs)	0 / 1 / 2
9. Ownership and handover	A named process owner on your side. Code, prompts, evaluations and runbooks in your repository. A handover plan exists	The contract clause on IP and repository location, and the handover checklist	0 / 1 / 2
10. Governance and data protection	Data classification, permission limits and GDPR and AI Act questions handled from day one, not bolted on later (governing AI agents)	The data-classification approach for your workflow and who signs it off (AI and GDPR, EU AI Act for businesses, data privacy)	0 / 1 / 2

Operator's rule: a partner who can't tell you when an agent is unnecessary won't tell you when to stop building either.

Partner types, in brief

A software house, a large consultancy (Big 4), an independent architect, a platform vendor's implementation partner and an in-house team can all deliver an agent. The label doesn't predict the result: score each one on the same ten criteria.

Partner type	What they usually bring	Scorecard rows to check first
Software house	An engineering team that builds and ships code	1, 3, 8
Large consultancy (Big 4) or integrator	Coordination across many areas and organisation-wide programmes	2, 8, 9
Independent architect or boutique	Architecture and control design, a small team	5, 6, 9
A platform vendor's implementation partner	Depth in one platform and its tooling	6, 7, 8
In-house team plus freelancers	Inside knowledge of the workflow and the data	3, 5, 10

The middle column describes a role, not quality.

Questions for the first meeting

Each question maps to one scorecard row, so you can score the answer straight away.

1. "When did you last advise a client against an agent, and what did you suggest instead?" (row 1)
2. "What data will the pilot run on, and how do we revoke your access?" (row 2)
3. "What quality threshold would stop the rollout, and who sets it?" (row 3)
4. "Where exactly does a person approve what the agent does?" (row 4)
5. "Who watches the logs after launch, and what do they do when something looks wrong?" (row 5)
6. "What permissions will the agent get, and where will the credentials live?" (row 6)
7. "What happens when we change the model?" (row 7)
8. "What does one completed task cost?" (row 8)
9. "Whose repository will the code and evaluations live in?" (row 9)
10. "Which data in this workflow do you treat as sensitive, and who on our side signs that off?" (row 10)

Red flags

The most serious red flag is a partner who proposes an agent for everything and can't show an evaluation before build. The second is running costs that only appear after go-live.

• Everything is an agent. Every problem ends in an agent proposal, even when a rule or a script would do.
• The demo never runs on your data. Questions about your data get "after signing".
• Evaluation only after build. Quality is to be judged "in use", with no test set and no threshold.
• "Human in the loop" means an OK button. A person approves without context and with no real way to change the outcome.
• A closed platform with no exit. Prompts, evaluations and logs can't be exported.
• Running costs "we'll see". The proposal covers the build; the cost of one task comes after launch.
• The IP stays with the partner. Code and evaluations are licensed to you, not handed over.
• One team in the pitch, another in the build. Nobody can say who will actually build and run the agent.

How to run the selection, step by step

Choosing a partner takes five steps: one workflow, a short list, the same material for everyone, scoring with must-haves first, and a narrow first engagement.

1. Describe one workflow on one page. Inputs, outputs, who does it today, which systems and data are involved, and what an error costs you.
2. Shortlist a few candidates, ideally of different types, so you compare approaches, not just proposals.
3. Send all of them the same description and scorecard. Ask for answers row by row, with evidence or a note that there is none.
4. Score the answers and the evidence. Apply the must-haves first, then add up the remaining rows.
5. Start with a narrow, time-boxed first engagement with stop conditions. What to buy first is in AI audit vs proof of concept, and how to set stop conditions and a single metric in small pilot, hard proof.

If you'd like to run this scorecard against one of your own workflows, tell us about it.