Data governance and data protection as the foundation
Governance of AI is underpinned by how effectively an organisation manages inherent risks associated to AI and ML. Given the sheer volume of data required to train models there are heightened risks around data. The advent of generative AI only heightens these risks further. Financial services firms in particular need to pay close attention, given the significant amount of data that they both produce and consume, particular on their customers. As with any form of ML and AI, both 'traditional' and generative, organisations are expected to be able to answer questions around transparency, traceability, quality and potential bias of the data that feeds a model.
For many financial services organisations that have come out the other side of BCBS239, there will be existing data governance frameworks and arrangements that can be leveraged to help solve for many of these challenges. It is vital that these frameworks not only cater for the risks around AI, but have also been implemented over the AI development lifecycle.
As part of these arrangements, data protection risks need to be front and centre when developing AI use cases as organisations, and particularly technical teams within those organisations cannot forget they are accountable for complying with data protection laws.
Continued relevance of Data Protection Impact Assessments (DPIAs)
There is an acknowledgement that not all uses of AI in the financial service industry will involve types of data processing that are likely to potentially impact an individual's rights and freedoms. However, organisations must still demonstrate the existence and use of governance processes in place that assess and mitigate any data protection risks. In the vast majority of cases, the use of AI will involve a type of processing likely to result in a high risk to individuals’ rights and freedoms. It will therefore trigger the legal requirement to undertake a DPIA. You need to make this assessment on a case-by-case basis. The ICO’s AI toolkit is designed to provide further practical support to organisations to reduce the risks to individuals’ rights and freedoms caused by their own AI systems.
The iterative and fast paced nature of AI technologies, including generative AI, within organisations mean that DPIA's should be reviewed regularly and reassessed where appropriate where the nature, scope, context or purpose of the processing, and the risks posed to individuals, alter for any reason. Put simply, data protection needs to be front of mind, and data protection stakeholders need to be brought on the AI development journey from start to finish, from idea generation to implementation, given the way in which AI use cases and solutions tend to morph as they are developed.
Transparency and trust
Generative AI only serves to magnify the issues that financial services organisations will face in deploying the technology. The complexity and lack of transparency of generative AI models makes it much harder for organisations to manage the data risks that AI presents. This stems from the challenges in truly understanding how the model is built and where its training data is sourced from.
Risks are compounded compared to traditional use of AI techniques, as organisations are forced to rely on third party large language models (LLM) given the costs and amount of training data required to build their own. This means that many organisations are either augmenting foundational models trained by third party providers, or are leveraging these models out of the box. This raises concerns over the lack of transparency from vendors about models and training data which may either lead to unfair, discriminatory, or bias outcomes, or may have been trained using the data of individuals without their consent. For organisations not building their own in-house generative AI models, that have enabled users to interface with technologies such as ChatGPT and BARD, a different type of risk emerges given that many do not have clear line of sight in the type of information being entered into these tools, and are instead relying on policies to govern their use.
Multi-faceted teams are required to govern AI technologies
The management and governance of this new technology cannot also be simply the responsibilities of those that are involved in the development and deployment of AI capabilities, typically sitting in data scientists or engineering teams. A comprehensive governance council should be established at both senior leadership levels and at the operational level where the work is being executed. Senior management leading Financial services organisations such as in the Chief Data Office and other stakeholders such as Data Protection Officers, are also accountable for understanding and addressing privacy risks appropriately and promptly. This ensures that the organisation is equipped with the knowledge on governing any new uses cases of AI or managing existing projects.
For now, financial services organisations need to ensure that foundational data governance frameworks and data protection protocols have been implemented effectively and are operating to manage the data risks presented during the AI development lifecycle.
Whilst many organisations are treading carefully when it comes to generative AI, over time we will no doubt see the evolution of customer-focused use cases. Getting the foundations around data governance and data protection right today means that organisations will be well-placed to face into the challenges that generative AI will pose in the years to come.
