AIChoosing a vendorMethodGDPR
How to choose an AI development agency: criteria, questions to ask and red flags
11 min read
To choose an AI development agency, judge it on what happens after the demo: systems already running in production, a written method for measuring answer quality on your real cases, control over personal data, security and running costs, and deliverables you can take over. Ask precise questions in the first meeting, be wary of promised accuracy figures, and test the agency on a short pilot with your own data before committing a large budget.
This guide does not rank agencies and names none. It gives you a scorecard for the ones you meet, ours included.
First: does your need really call for AI?
Many projects pitched as "an AI project" are better solved another way. Before you contact anyone, classify the need:
- A stable, explicit rule (if the amount exceeds a threshold, ask for approval) belongs to classic automation. It is more reliable, easier to test and cheaper to run than a language model.
- An off-the-shelf tool already does it: writing assistants, meeting transcription, translation. A subscription costs less than a build.
- Free text to understand, summarise, classify or extract, documents to query, requests to route: this is where generative AI earns its keep.
Our article on adding AI to a business application covers the use cases that work, and the ones that disappoint.
What "AI agency" actually means
The label covers very different businesses. Knowing who you are talking to stops you from comparing proposals that aim at different outcomes.
| Type of provider | What it does well | What to check |
|---|---|---|
| Strategy consultancy | Strategy, use case prioritisation, change management | Who builds and runs the system afterwards? |
| IT services firm, integrator | Rolling out vendor tools, connecting them to your systems | Can it build when the packaged tool falls short? |
| Software development studio | A full application: data, permissions, interface, AI, ops | Depth on model evaluation |
| Data science specialist | Predictive models, training on your own data | Can it ship software non-specialists can use? |
| Marketing agency with AI tools | AI-assisted content production | This is not a software project |
No type is better in absolute terms. When AI is one feature of a business application, most of the work is still ordinary software engineering, and that is often where projects fail.
The criteria that really separate agencies
1. It understands your business before it talks about models
A good partner starts with your processes, your documents, your users and what currently costs you time or money. If the first meeting is about the latest model or a fashionable framework, you are being sold a technology, not an outcome.
2. It has systems in production, not just demos
An AI prototype can impress in a few days. Keeping it working in production, with users asking unexpected questions, badly scanned documents and models that change version, is a different job. Ask to see a system that has been live for several months, and ask for a reference you can call.
3. It measures quality, and shows you how
An AI answer is not right or wrong by default: it has to be measured. The agency should offer to build, with you, an evaluation set of real cases whose correct answer is known, then score the system on that set after every change. Without that measurement, nobody knows whether a new version is better or worse.
4. It handles "I don't know"
A serious system grounds its answers in your data, cites its sources when it has them, and can say that it does not know. Ask what the system does when the information is not in your documents. The answer tells you a lot.
5. It takes security seriously
Applications built on language models have their own weaknesses. The OWASP Top 10 for LLM Applications (2025 edition) ranks prompt injection first: text read by the system carries instructions that hijack it. It also lists excessive agency, where an agent holds more permissions than it needs. A competent agency can explain how it limits both risks.
6. It knows where your data goes
Which data goes to which model provider, in which country, kept for how long, and is it used to train models? The answers must be in writing. In the EU and the UK, the GDPR applies to any AI system that processes personal data. France's data protection authority, the CNIL, publishes recommendations for developing AI systems in line with the GDPR, available in English. Our GDPR checklist for a business application covers the basics.
7. It knows the EU AI Act, without using it as a scare tactic
The EU AI Act assigns obligations by role, provider or deployer, and by the risk level of the system. A company that has an AI system built and puts it into service under its own name may count as the provider. The Act also reaches companies outside the EU whose systems are used in it. Our article on the EU AI Act for business software sets out the timeline and the roles.
8. It prices the running costs, not just the build
An AI system costs money every time it is used: model providers bill by the volume of text processed, according to their public price lists. A serious proposal estimates that cost at your real volume, explains how it will change, and how to cap it. Our article on the cost of a custom AI agent breaks those lines down.
9. It does not lock you into one model
Models change quickly, and so do their prices. A sound architecture lets you switch provider without rewriting everything, and the evaluation set lets you check that the replacement does at least as well.
10. It plans for adoption
The best system is worthless if nobody uses it. Ask how future users will be involved, trained, and how their feedback will be collected after launch.
Questions to ask in the first meeting
| Question | Reassuring answer | Answer that should worry you |
|---|---|---|
| Can you show me a system that has been in production for several months? | A walkthrough with its usage and cost metrics | Only prototypes or mockups |
| How will you measure quality on our cases? | A set of real cases built with you, scored every release | "The model is very accurate" |
| What does the system do when it does not know? | It says so, and hands over to a person | It always answers something |
| What will it cost to run at our volume? | A reasoned estimate, with a cap | "It's negligible" |
| Where does our data go, and what does the model provider keep? | A written answer, flow by flow | A vague or delayed answer |
| What if the model provider changes its prices or terms? | A switch plan, tested against the evaluation set | "That won't happen" |
| Who will build it, and who maintains it after delivery? | Named engineers and a maintenance agreement | You meet the developers after signing |
Red flags
- Guaranteed accuracy ("almost no errors") with no word on which cases or how it was measured.
- A demo only on the agency's own data, and reluctance to try your documents.
- A quote with no questions asked about your data, volumes or users.
- Training or fine-tuning a model proposed up front, before simpler approaches have been tried.
- No line for running costs, quality measurement or maintenance in the proposal.
- A sales-only contact, with nobody able to answer technical questions.
- Promises to replace jobs rather than to help teams.
Analysts share that caution. In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. It is a forecast, not a measurement, but it points at exactly the three things to check in an agency.
Test before you commit
Rather than choosing on paper, have the agency work on a short pilot:
- One use case, chosen for its value and its simplicity.
- Your real data, anonymised if needed.
- Success criteria fixed before starting: share of correct answers on the evaluation set, time saved per task, cost per use.
- A handful of real users to try it.
- No commitment to what comes next: the pilot must be able to stop without penalty.
Within a few weeks, the pilot shows what no sales meeting can: how the agency asks questions, handles the unexpected and reports back. Our guide to prototype vs MVP vs proof of concept helps you pick the right format.
Comparing proposals
AI proposals are even harder to compare than those for classic software: different scopes, running costs included or not, quality measurement present or absent. Bring them back to one grid: deliverables, assumptions, exclusions, build cost, estimated running cost, maintenance and ownership. Rights over the code, the model instructions and the evaluation set are defined case by case in the contract: read the clause before signing. Our method for comparing software development quotes applies as is.
When not to hire an AI agency
- Your need is a fixed rule: classic automation will do.
- An off-the-shelf tool does the job: try it for a few weeks first.
- Your data does not exist yet, or is scattered across inboxes: structure the process first, AI can come later.
- Nobody in-house will own the project: without a business owner who has time, no vendor can measure quality on your behalf.
Where to start
- Describe one precise use case: who, which task, which documents, what volume.
- Collect about twenty real examples with the expected answer.
- Define what success looks like in simple numbers.
- Meet two or three providers with the scorecard above.
- See the result before you pay for development.
For that last step, we build a free visual prototype, presented on a video call within 5 working days: you see how AI fits into your screens and workflows before any commitment. Our article see your software before you pay for it explains what the prototype demonstrates.