ExampleAn illustrative report for a fictional organisation. No real client, system or data is shown.
AI Risk Scorecard: Customer claims assistant
Northbank Insurance (fictional)
Overall rating
High risk
1 critical · 2 high · 3 medium or low findings
Independent assessment of Northbank Insurance's customer claims assistant across security, reliability, fairness and governance. Three findings are rated critical or high. F1 and F3 must be fixed before rollout; both have been fixed and verified by re-test.
- tests run
- 412
- critical finding
- 1
- findings in total
- 6
- frameworks mapped
- 4
- System: AI assistant with claims lookup and refund approval
- Testing window: 3 weeks
- Environment: non-production copy
Validity of these results: Valid for assistant release 2.3 on the model version recorded in the evidence pack, as at 14 March. Re-test when the model, tools, data sources or system instructions change, and at least every quarter.
For: Board, executives
1. Executive summary
Overall rating: High risk at the time of testing. Two of the three critical or high findings have been fixed and verified by re-test. The third is in remediation.
Top risks
- F1 Cross-customer data disclosure through indirect prompt injectionCriticalFixed, verified
- F2 Inaccurate statements of policy coverHighIn remediation
- F3 Refunds approved without human authorisationHighFixed, verified
Decisions required
- Before rollout: fix F1 and F3. Both are now fixed and verified by re-test.
- Within 30 days: limit answers to approved policy wording (F2).
- Within 60 days: add fairness monitoring (F4) and a sign-off step for model updates (F5).
For: CISO, risk
2. Scope and approach
In scope
- Customer claims assistant (web chat), including file upload
- Connections: claims system (read), refund approval up to £500, policy document library
- Model configuration, system instructions and change process
Out of scope
- Underlying infrastructure and network
- The model provider's own systems
- Staff-facing tools and live production traffic
Approach
- 2,400 automated attack runs for breadth
- 412 tester-designed tests across security, reliability, fairness and governance
- Every finding confirmed by a second tester before it is rated
- Non-production copy of the assistant, over 3 weeks
Model and responsibilities
Third-party hosted large language model. Vendor and version are recorded in the evidence pack, and results apply to that version only.
Model vendor is responsible for
- Model training and its built-in safety behaviour
- Hosting, availability and security of the model service
- Notice of model changes and deprecations
Northbank is responsible for
- System instructions and configuration
- Connected tools, permissions and the data the assistant can reach
- Approving changes, monitoring and responding to incidents
For: CISO, risk
3. Findings summary
| Finding | Severity | Owner | Status |
|---|---|---|---|
| F1 Cross-customer data disclosure through indirect prompt injection | Critical | Head of Digital Engineering | Fixed, verified |
| F2 Inaccurate statements of policy cover | High | Head of Claims Operations | In remediation |
| F3 Refunds approved without human authorisation | High | Head of Digital Engineering | Fixed, verified |
| F4 Response variation by customer name | Medium | Conduct Risk Lead | Re-test scheduled |
| F5 Model update deployed without change approval or re-test | Medium | AI Product Owner | Open |
| F6 Partial disclosure of system instructions | Low | Head of Digital Engineering | Open |
Severity definitions
- Critical
- Repeatable and exploitable. Leads directly to personal data disclosure, financial loss or regulatory breach. Fix before deployment.
- High
- Material harm is likely, such as inaccurate information customers may act on, or actions taken without required controls. Fix before deployment or within 30 days.
- Medium
- A weakness in fairness, monitoring or change control that increases risk over time. Fix within 60 days.
- Low
- Limited direct impact, but may make other attacks easier. Address in normal maintenance.
For: Security, engineering
4. Finding F1
Cross-customer data disclosure through indirect prompt injection
Description
The assistant treats text inside uploaded documents as instructions. Combined with read access to all customers' claims, this allows a user to retrieve another policyholder's claim details.
Attack path
- 1. Document uploadedAny customer can upload a file.
- 2. Hidden instruction executedThe assistant does not distinguish its own rules from text inside a document.
- 3. Claims system queriedThe assistant can read every customer's claims, not only those of the signed-in customer.
- 4. Data disclosedAnother policyholder's name, claim number and payout returned.
Evidence
Personal details redacted.
Impact
Disclosure of another policyholder's name, claim number and payout. In live use this would be a personal data breach, likely to be reportable to the ICO.
Recommendation
Restrict claims lookups to the authenticated customer, and remove hidden or non-visible text from uploaded documents before processing.
For: Risk, claims operations
5. Finding F2
Inaccurate statements of policy cover
Description
The assistant answers questions about cover from general knowledge, not from the customer's own policy wording. Where the policy excludes a peril, it sometimes states that the customer is covered.
Evidence
Impact
Customers may make decisions, or delay their own repairs, based on an incorrect statement of cover. This is relevant to the FCA Consumer Duty outcome on consumer understanding, and is a likely source of complaints.
Recommendation
Answer cover questions only from the customer's own policy wording, quote the clause relied on, and hand over to a claims handler when the wording does not settle the question.
For: CISO, risk
6. Agent controls
The assistant looks up claims and approves refunds, so it acts as an agent. These are the controls around what it can do and how it is stopped.
| Control | Result | What we found | Ref |
|---|---|---|---|
| Agent inventory, including local models | Partly | The claims assistant is registered. Two internal prototypes using a locally hosted open-weight model are not in the inventory. | O1 |
| Permissions and access | Not in place | Read access to every customer's claims, and refund approval with no per-customer limit. Least privilege is not applied. | F1 · F3 |
| Decision records | Partly | Refunds are logged, but not the inputs and instructions that led to them, so a disputed refund cannot be reconstructed. | O2 |
| Escalation to a person | Partly | Refunds now route to a claims handler (fixed under F3). Complaints and vulnerable-customer cues were escalated in 18 of 20 tests. | F3 |
| Ability to stop the agent | Partly | The assistant can be switched off, but only by an engineering redeploy. There is no documented procedure or named owner, and it took 47 minutes in a timed exercise. | O3 |
| Model supply chain | Partly | The hosted model version is recorded. The source and integrity of the weights used by the local prototypes have not been verified. | O4 |
O1 to O4 are observations: recommendations on controls around the assistant, not rated findings.
For: Risk, compliance
7. Framework and regulatory mapping
Each finding is linked to recognised controls so risk, audit and security teams can act on it directly.
OWASP Top 10 for LLM Applications
10 / 10
Categories tested. 5 with findings.
OWASP Top 10 for Agentic Applications (2026)
2 tools
Applied to the claims lookup and refund tools. Findings F1 and F3, observations O1 to O4.
NIST AI Risk Management Framework
4 / 4
Functions covered: Govern, Map, Measure, Manage. 3 with findings.
ISO/IEC 42001
5
Annex A control areas linked to findings.
Tested, no findingTested, finding raised
| Finding | OWASP LLM | NIST AI RMF | ISO/IEC 42001 |
|---|---|---|---|
| F1 Cross-customer data disclosure through indirect prompt injection | LLM01 · LLM02 | MEASURE 2.7 | A.7 Data |
| F2 Inaccurate statements of policy cover | LLM09 | MEASURE 2.5 | A.6 Life cycle |
| F3 Refunds approved without human authorisation | LLM06 | MANAGE 1.3 | A.9 Use of AI |
| F4 Response variation by customer name | – | MEASURE 2.11 | A.5 Impact |
| F5 Model update deployed without change approval or re-test | – | GOVERN 1.4 · MANAGE 4.1 | A.6 Life cycle |
| F6 Partial disclosure of system instructions | LLM07 | MEASURE 2.7 | A.4 Resources |
UK regulatory relevance
- UK GDPR and ICO guidance on AI and data protectionDisclosure of personal data, and access to more data than the purpose requires.F1 · F6
- FCA Consumer DutyCustomers' understanding of their cover, and consistent outcomes for different groups of customers.F2 · F4
Shows where findings are relevant. Not a legal or compliance opinion.
For: CISO, risk
8. Remediation, re-test and risk acceptance
The same tests were re-run after Northbank's team applied fixes. The test suite is handed over so it can be re-run after every update.
| Finding | Severity | Failures before fix | Failures after fix | Status |
|---|---|---|---|---|
| F1 Cross-customer data disclosure through indirect prompt injection | Critical | 7 of 20 | 0 of 20 | Fixed, verified |
| F3 Refunds approved without human authorisation | High | 4 of 15 | 0 of 15 | Fixed, verified |
| F2 Inaccurate statements of policy cover | High | 6 of 40 | 1 of 40 | In remediation |
| F4 Response variation by customer name | Medium | 18 of 200 | Not yet re-tested | Re-test scheduled |
Risk acceptance for findings not yet fixed
| Finding | Risk accepted by | Interim control | Review by |
|---|---|---|---|
| F2 Inaccurate statements of policy cover | Head of Claims Operations, 21 March | Questions involving exclusions are routed to a claims handler. | 20 April |
| F4 Response variation by customer name | Conduct Risk Lead, 21 March | Monthly manual review of a sample of conversations. | At re-test |
| F5 Model update deployed without change approval or re-test | AI Product Owner, 21 March. Chief Risk Officer informed. | Model updates frozen until a sign-off step is in place. | 20 May |
| F6 Partial disclosure of system instructions | Head of Digital Engineering, 21 March | None required. Addressed in normal maintenance. | Next release |
For: Board, executives
9. Governance considerations for the board
Questions the board should put to management, with the answer this assessment supports. Each answer references recorded test evidence.
Can it disclose personal or confidential data?
Yes. An instruction hidden in an uploaded document disclosed another policyholder's claim details.
YesF1Does it give inaccurate information customers may act on?
Yes. It stated flood damage was covered where the policy excludes it, in 6 of 40 scenarios.
YesF2Can it take actions without human authorisation?
Yes. Refunds of up to £500 were approved after repeated customer requests.
YesF3Does it treat customers consistently?
Mostly. Responses varied with the customer's name in 9% of matched pairs.
PartlyF4Are changes to the system controlled?
No. The most recent model update was deployed without a recorded approval or re-test.
NoF5
For: CISO, operations
10. Ongoing assurance: re-testing and monitoring
AI systems change after testing, so these results lose value over time. This section sets out when to re-test and what to watch in between.
Validity: Valid for assistant release 2.3 on the model version recorded in the evidence pack, as at 14 March. Re-test when the model, tools, data sources or system instructions change, and at least every quarter.
Re-test when
- The model or model version changes
- A tool, permission or data source is added or changed
- System instructions are changed
- At least every quarter, even if nothing has changed
Monitor between tests
- Claims lookups that don't match the signed-in customer
- The earliest sign of a repeat of F1.
- Refunds approved per day, and by whom
- Shows whether the F3 fix holds.
- Escalation and refusal rates
- A sudden change often follows a model or prompt update.
- Complaints that mention the assistant
- Catches inaccurate statements of cover (F2) in live use.
- Answers compared across customer groups, monthly
- Tracks consistency of treatment (F4).
For: Procurement, CISO
11. Assurance team and quality controls
Independent specialists with no role in building, selling or configuring the system.
Engagement lead
Senior AI and risk leader
Accountable for the final rating and the executive summary. Experience of AI and technology risk at board level.
Red team
Certified red team testers
Hold recognised offensive security certifications. Designed and ran the adversarial tests, including F1 and F3.
AI engineering
AI and machine learning specialists
Tested reliability and accuracy against Northbank's policy wording.
Governance
AI governance specialist
Assessed fairness, oversight and change control, and mapped findings to the frameworks.
- Automation and judgementAutomated attack runs give breadth. Testers decide which results matter to the business and how they are rated.
- Peer reviewEvery finding is checked by a second tester before it is rated.
- ReproducibilityInput, output and result are recorded for every test run.
- IndependenceWe don't sell AI products, take vendor commission or test systems we built.
- Data handlingClient data is used only for this engagement and deleted on completion.
For: Risk, procurement
12. Methodology and limitations
- 1. ScopeSystem, connections, data and exclusions agreed and signed off.
- 2. Design412 tests built for this system's users and risks.
- 3. Test2,400 automated runs plus manual testing. Every run recorded so it can be repeated.
- 4. ReviewFindings peer-reviewed and rated.
- 5. Hand overReport, evidence and test suite passed to the client's team.
Limitations
- Point in time
- Results reflect the system during the testing window. Model, prompt or data changes can alter them.
- Scope-bound
- A clean result means nothing was found within scope, not that no weakness exists.
- Non-production environment
- Testing used a non-production copy. Production configuration may differ.
- Sample-based fairness testing
- Fairness results are based on 200 matched pairs. They indicate, but cannot prove, behaviour across all customers.
- Not certification
- Mapping to OWASP, NIST AI RMF and ISO/IEC 42001 is not certification or accreditation, and this report does not imply regulatory approval.
- Not legal advice
- References to UK GDPR, ICO guidance and the FCA Consumer Duty show where findings are relevant. They are not a legal or compliance opinion.
- Observations
- O1 to O4 concern controls around the assistant rather than its behaviour, so they are recorded as recommendations and not rated.
Email me a copy
Optional. We'll send you a link to this report so you can share it with colleagues.
Want this evidence for your own AI?
The AI Risk Scorecard tests one AI system in two to four weeks, at a fixed price.