The Financial Conduct Authority (FCA) has found financial services firms overstating the accuracy of their artificial intelligence (AI) models while some automatically exclude vulnerable customers from AI journeys.
The findings coincide with a warning from the regulator for consumer AI services such as ChatGPT, Gemini and Claude to include disclaimers or warnings to help consumers understand the limits of financial information and advice generated by their models.
In a speech this morning, FCA chief data, information and intelligence officer Jessica Rusu, warned that increasingly, AI was changing what happens before a consumer ever reaches a regulated firm.
“Consumers are turning to general-purpose large language models (LLMs), like ChatGPT, Gemini and Claude, for financial information and advice, while regulated firms are beginning to integrate their services into AI platforms,” Rusu said.
“The good news – this can significantly widen access to financial tools and information. But it can also blur the boundary between regulated and unregulated activity, and the protections consumers receive.
“Four in five less-experienced investors aged 18-40 have used AI to help with investing, but only 40% of consumers correctly recognise that there is no recourse if something goes wrong after acting on advice from a general-purpose AI tool.
“Our answer is not simply to expand our remit, but to make the existing boundary clearer.“
To enable this, Rusu wants clear warnings and disclaimers to be published by the services.
“Through InvestSmart and the FCA Firm Checker, we already help consumers understand the protections they have. But warnings may be more effective at the point where AI-generated financial information is delivered,“ she said.
“So we are calling on AI providers to help consumers understand the limits of financial information and advice generated by their models through clear, proportionate and consistent disclaimers.”
Review findings and responsible use
As part of its review the regulator told financial services firms to be upfront with customers about the technology’s limitations, monitor live outcomes to prevent issues from escalating and understand it’s impact on vulnerable customers.
The FCA also warned firms they must define why AI was needed before designing and deploying its use, properly test consumer outcomes and manage bias and accuracy, .
Its AI in financial services: good practice and areas for improvement report shared examples from the regulator’s industry engagement, supervisory work and its AI Lab initiatives to support the safe and responsible adoption of AI in financial services.
The publication, promised earlier this month, revealed the regulator wants firms deploying AI to do so safely and responsibly so consumers see the benefit and firms continue to meet their Consumer Duty obligations.
Alongside areas of concern and improvement the FCA published examples of how and where AI had been appropriately installed.
“Where firms are deploying AI-supported tools, they remain responsible for consumer outcomes,” the FCA said.
“This includes outcomes they deliver through any products, services, communications and support.
“When deployed safely and responsibly, we saw that AI can deliver significant benefits to consumers by improving efficiency, reducing costs, and enabling more accessible and tailored services,” it added.
The FCA said firms using AI safely and responsibly successfully defined the consumer need and target market before designing and deploying AI-supported products and services, and tested consumer outcomes across different consumer characteristics and scenarios.
They also used proportionate controls to manage AI bias and accuracy, and to ensure that decisions could be explained.
These firms also told consumers how they had used AI and its limits, and tested their understanding where appropriate, and monitored live outcomes so they could spot and deal with issues quickly.
Products and services
Looking specifically at products and services, the FCA said firms using AI should design them to meet consumers’ needs, characteristics and objectives.
In particular, they should take particular care where AI could affect outcomes, including people’s ability to access services, how firms prioritise issues, and support and guidance for consumers.
It warned that some firms were building tools without establishing the consumer problem, target market or predicting foreseeable harms which raised concerns the product would not meet the needs of consumers likely to use it.
They were also found to be introducing guardrails that addressed one risk but created another, with some firms focused on technical model guardrails with limited consideration of how to support good consumer outcomes.
Furthermore, some vulnerable consumers were found to be already excluded by default.
To reduce risk, some firms automatically excluded consumers with vulnerability indicators from AI journeys.
While the FCA said this may be appropriate in some scenarios, the regulator warned this could unfairly exclude those consumers if the firm did not provide another way for them to get support or benefits that met their needs.
Examples of good practice included firms choosing where AI should support, automate or abstain.
Firms considered which tasks AI can support or automate, which should remain staff-led, and where the AI tool should abstain, redirect or escalate to a person.
This meant building proportionate technical checks and continuous monitoring into the AI system, rather than only relying on a person to review, approve, challenge or override its outputs or decisions.
The FCA added firms should be able to show that the controls they choose work in practice.
Other good practices highlighted included defining the consumer need before developing or deploying a product. Firms started with a clear consumer problem, target market, and intended outcome before deciding whether AI was the right tool and in what form, e.g. agentic and generative.
Firms were also found to be testing outcomes across different consumer characteristics, including vulnerability and digital or financial capability, to support inclusive design.
Consumer understanding
To aid better consumer understanding, the FCA called on firms to explain how they had used AI in their products, services and journeys in a way consumers could understand, so they could make effective and properly informed decisions.
Firms should also consider whether their records, explanations and monitoring were sufficient to support challenges and help them deal with complaints.
Where firms could up their game, the FCA found some firms assumed their AI outputs were clear and helpful without testing whether different consumer groups understood them.
This included consumers with lower financial capability, accessibility needs or vulnerability characteristics.
They also gave consumers technical information without explaining what it meant for them, providing technical information about an AI model without explaining what it meant for the consumer, why it produced a certain outcome, what the consumer could do next, or whether the information was right for the consumer.
Firms were found to be overstating the accuracy, reliability or role of AI systems, or were unclear about whether AI or a person performed a particular function which could mislead consumers and undermine informed decision-making and trust.
And some firms were found to be focusing only on how a model works, rather than whether they could show good outcomes, spot poor outcomes, understand the likely causes and support challenge or redress.
In terms of good practice, the regulator found some firms were explaining the role and limits of AI at the right level for the consumer, making it clear to consumers that they were using an AI-enabled tool, and explaining what it could and could not do and when the consumer should look for help from another source.
They were also found to be using AI to make consumer communications clearer, with human review, using AI to assess consumer communications and propose clearer wording, while staff reviewed all the changes before approving the final wording.
They were found to be using approved source material and guardrails to reduce hallucinations or inappropriate responses, limiting chatbots and generative AI assistants to approved source material, with checks and escalation routes.
This meant issues could be identified and passed to a person where needed.
Consumers in vulnerable circumstances
Turning to support for customers in vulnerable circumstances, the FCA found firms left long gaps between reviews of live outcomes, or defined their escalation triggers too vaguely.
Although there is no set timescale for firms’ reviews and they will need to determine what is most appropriate for their use case, running them more frequently can help spot harm faster. This can be particularly effective when the firm has clear responsibilities and processes to address issues.
The regulator also found some firms placed too much confidence in a large language model’s (LLM) ability to identify text-only indicators of vulnerability, particularly temporary ones.
It said firms should test their systems thoroughly to make sure they can identify gaps in approaches and make changes, for example adding vulnerability indicators or questions the LLM could ask.
Some automated interfaces were found to make it harder for consumers to get support.
Problems included unclear responses, not addressing the consumer’s issue and not allowing them to move to another channel.
In more positive actions, firms were found to be using AI tools to spot potential vulnerability indicators, such as key words in consumer responses, spending patterns, or signs a consumer was struggling with a digital journey.
Where appropriate, they allowed consumers to choose between AI-enabled or non-AI support for straightforward tasks, so consumers could switch channels without having to repeat information.
This helped maintain continuity and consumer control, and allowed firms to answer questions more efficiently.
