Governing agentic AI
It seems inevitable that the near-term horizon for the future of work will involve the integration of AI agents into our workflows, but how many organisations could say that they are ready for this shift? How many organisations even know the issues they should be thinking about, writes Jared Browne, Fexco’s Head of Privacy and AI Governance.
As a financial services, business services, and technology services company, operating both a suite of Fexco-owned and operated products and services, but also offering products and services on an outsourced basis to clients and partners, these are among the questions we grapple with daily, especially in my role as Fexco’s Head of Privacy and AI Governance.
It is clear that AI agents, if harnessed correctly, could lead to innumerable benefits across all sectors, but as with all powerful technologies, realising that potential rests on deploying sound governance structures to manage the associated challenges. In this article, we will discuss some of these key challenges and analyse suitable solutions for their mitigation.
First, however, it is important to understand what AI agents are and why it is that they pose novel challenges. The concept of an AI agent concerns an AI system that can work with significant levels of autonomy by using memory, reasoning, and planning abilities.
Compared to passive chatbots, that simply respond to user queries, the major difference with AI agents is that they act on the environment around them.
In practice, this means that through memory they are able to recall previous steps in a workflow, with planning they are able to foresee the next steps, and with reasoning capacities they are able to navigate their way through the workflow with greater independence.
This, then, creates the possibility that AI could complete complex, longer-term workflows, freeing up considerable time, and reducing cost for organisations.
Unlike traditional software agents, although they are goal-directed, AI agents are, crucially, ‘non-deterministic’ in nature. Non-deterministic means that when given any goal they are capable, to some degree, of independent problem-solving to achieve that goal. This allows them to handle novel problems and, for example, analyse issues, weigh up options, and potentially correct their own mistakes along the way to completing any task.
The major hoped-for efficiency gain is that agents, subject to appropriate oversight, will be able to reliably work on individuals’ behalf and free up substantial amounts of time for higher-value activities.
The challenges
Given the semi-autonomous nature of AI agents, the ability to effectively oversee their activity is the core challenge for any governance function. Article 14 of the EU AI Act requires human-in-the-loop controls to be in place for the use of high-risk AI systems.
Additionally, from the data protection perspective, Article 22 of the GDPR, subject to exceptions, requires human oversight where personal data is subject to fully automated decision-making.
However, if meaningfully overseeing AI was challenging, doing so in the case of agentic AI is still more so. This is primarily owing to the dynamic nature of agentic AI, its in-built autonomy-by-design structure and its often opaque way of making decisions.
With respect to its dynamic nature, most governance controls, which tend to be only periodic in nature, will likely struggle to adjust to the fluid nature of AI agents which are built to adapt and modify their activities using chain-of-thought reasoning.
This makes it difficult for a human reviewer to understand what normal behaviour is for any AI agent. In other words, how exactly will they know if they should intervene to overrule an agent’s activities, if the agent starts to function in new ways?
An AI agent may, for example, decide to call on a completely new tool because it has determined that it is more efficient to do so. Should the human allow this or not, and what safeguards need to be put in place at design phase to set the appropriate boundaries?
Additionally, the operational reality of AI agents is that they are designed to potentially make numerous micro-decisions per minute, something which any human will find very challenging to keep pace with.
“AI could complete complex, longer-term workflows, freeing up considerable time, and reducing cost for organisations.”
Although this productivity may be desirable on a commercial level, if an AI agent begins to complete many times more work than a typical human would in the same time period, how could any human overseer hope to keep track of such a volume of output and also be satisfied that the agent is acting ethically and lawfully?
Even if the whole point of AI agents is that they can work autonomously and realise greater gains, there is no world in which they can be permitted to operate outside real human control and oversight.
AI agents are developing rapidly but the governance structures for providing real human oversight in the context of dynamic, autonomous, and opaque process flows do not yet exist in a consistent and mature form, even if the legislative obligations to do so are clear.
This is an area we have placed particular focus in Fexco, to ensure the AI products and services we offer align with the governance requirements and legislative obligations for both ourselves and our clients.
A salient example of this oversight challenge is the difficulty many companies are now facing in reviewing AI-generated code. AI can generate code much quicker than traditional QA teams can assess it. This means that defective code and bugs can go unnoticed simply because the volume of code challenges the capacity for human review.
With respect to oversight of agentic AI, it is advisable to start small with simple workflows, learn about agents’ abilities and limitations, and then design the workflow, with appropriate human intervention built in including trigger points and hand-offs.
The key to effective deployment is optimising the role of the human in any agentic workflow, and planning for their involvement in advance. We have seen the success of this approach in our own products and services, where starting small has allowed us to make large improvements in a more controlled, governed, and outcome-focused manner.
Shadow agentic AI
Organisations may be accustomed to the risk of shadow IT, namely the introduction of IT systems and technologies without the express knowledge or permission of an organisation’s management, leading to the risk of unknown, and unmonitored systems processing company information.
The long-term risk with shadow IT is that once systems are deployed in this ambiguous way, they are often forgotten, yet can remain active within an organisation, continuing to pose an ungoverned, hidden risk.
Although shadow IT continues to be a risk for all organisations, regardless of the technology involved, the introduction of AI agents into organisational workflows clearly exacerbates this risk.
This is because of the increased autonomy and ability of AI agents to act upon the external world, such as by accessing websites and other software tools, retrieving information from databases and potentially even interacting and sharing information with other AI agents.
While this should not stifle innovation and curiosity, if allowed to operate unchecked, they may potentially not only continue to carry out their goals without oversight but may actually learn and adapt over time and begin to conduct actions entirely contrary to their original intent.
Also, research shows that the adoption rates of agentic AI indicate that the risk of ungoverned AI agents leaking personal data or gaining access to sensitive information are already here. This is evidenced by Microsoft’s Cyber-Pulse Report which shows that over 80 per cent of Fortune 500 companies have active AI agents that were built by low code/no code tools.
By way of example, if we compare this to the traditional risks associated with shadow robotic process automation, the new risk environment becomes clearer.
When robotic process automation (RPA) became commonplace, it was not unusual for RPA use cases to be set up without proper oversight, and for the underlying bots to continue processing information and performing tasks without any explicit knowledge.
The risk is somewhat mitigated by the fact that the bot is not generally capable of autonomous behaviour and will simply continue to operate as designed, executing the same defined process, in the same defined way.
This means that RPA, from the governance perspective, still behaves like traditional software: “predictable, bounded, and under human command”, as the ACM Europe Technology Policy Committee has put it.
However, with agentic AI, as mentioned, its potential ability to adapt means that the hidden risks of their deployment may increase over time. They present, in other words, dynamic, evolving risks in line with their dynamic, interactive nature.
Key considerations for deploying AI agents
As always, a lot will come down to preparation as we have proven with the AI products and services we have delivered. If we rush to integrate AI agents into our work, it risks failing commercially and leading to unnecessary friction and unfairness.
The new art that is required is to calibrate agent-human interaction, and, even more so, how to optimally design workflows so that multiple agents and humans can more efficiently achieve the desired outcome.
“If we rush to integrate AI agents into our work, it risks failing commercially and leading to unnecessary friction and unfairness.”
Within these workflows the critical junctures for the success of future work will be the hand-offs. These are the points, designed by humans, where the agent can proceed no further and needs to pass the task to a human worker to move forward with the workflow. These trigger points need to be placed very carefully to make sure that humans are able to intervene before the agent oversteps its brief.
Acceptable use policies
On a fundamental policy level, each organisation should develop an AI ‘acceptable use policy’ to determine how they want to deploy agentic AI within their networks. Without a clear policy, the alternative is that, owing to the proliferation of agent-building tools, staff will simply go ahead and build and deploy AI agents without any meaningful oversight.
With respect to AI agents, at a minimum, an AI acceptable use policy should identify:
- Areas where the use of agentic AI is appropriate. Organisations should determine that they work well for certain tasks but are perhaps not suited to other more critical use cases. These should be clearly outlined in the policy for all staff to understand what is and is not permissible;
- Authorised individuals that are permitted to generate and use AI agents in their work area and who will also co-ordinate and oversee the use of AI agents; and
- Human-in-the-loop oversight and approval processes appropriate to the risk involved.
General considerations
At a minimum, organisations should ask themselves the following questions before deploying agentic AI:
- Is the process/task suited to automation?
- As currently designed, is it a good process? If not, you may be only automating and turbo-charging a bad process.
- How will the AI agent(s) fit into the existing workflow?
- Do your workflows need to be redesigned?
- How much autonomy is too much autonomy? An agent’s level of independence should be calibrated to the level of risk involved, and it should never be fully autonomous, unless the use case is trivial and effectively risk-free.
- Do you understand the hand-offs between AI agents and humans, and what can go wrong at these junctures? These hand offs are critical.
- Is it a critical use-case? If so, it is probably best not to deploy AI agents at all, or if you do, it will need constant human oversight, but this will obviously impact the efficiency gain.
- If something went wrong in the workflow, would you be able to establish where the fault lies and know how to remedy it?
- Do you have a plan in place to govern AI agents? How they are built, deployed, and tracked, and who gets to use them?
|
Jared Browne
Jared Browne is Fexco’s Head of Privacy and AI Governance. He provides external training and consultancy services informed by the latest regulatory developments and global standards. |
