Human in the Loop Isn't Enough. Someone Must Be Accountable for the Loop.

A senior risk officer I worked with once described how her role had changed over her career. Early on, the controls she oversaw assumed that decisions were made slowly enough for a second person to review them and catch errors. Later, many of the same controls applied to systems that scored thousands of cases an hour, and errors were often found only at audit, well after the decisions had taken effect. The principles of risk management had not changed, but the speed of the processes they governed had.

Most AI in financial institutions still makes recommendations: a model flags a transaction and an investigator decides what to do, or a tool drafts a reply and an employee reviews and sends it. Accountability stays with the person who acts, so existing controls largely continue to apply. AI agents work differently. They can update records, send messages, initiate payments, open or close cases and decline requests, and they can carry out several steps in sequence without waiting for a person. Unlike a recommendation, which can be ignored or corrected before it has any effect, an action has already changed something for a customer or for the business.

The limits of "human in the loop"

The usual response to this concern is that a human will remain in the loop. The phrase covers very different arrangements. A person might approve every action, review only exceptions, examine patterns once a week, or have the right to override the system but rarely use it. At the volume and speed at which agents operate, a person who is formally accountable but cannot realistically observe what the system is doing provides little real control. A more useful question is who is accountable for the loop itself: who decided what the system is allowed to do, who monitors it, who can stop it, and who answers for the outcome when it is wrong.

Regulators are beginning to ask

Supervisors have started to raise these questions directly. In June 2026, the Reserve Bank of India published a draft model risk management framework stating that no AI model should operate without the ability to be shut down immediately if it produces harmful or erroneous outputs. In October 2026, Reuters reported that US banking regulators had begun asking banks how they would shut down autonomous systems and whether they could track and audit the decisions their AI agents make.

Building a technical "kill switch" is relatively straightforward. The harder issues are organizational: who is authorized to use it, at what level of the system, how quickly, and with what consequences for the person who uses it.

A model from another regulated industry

Offshore oil and gas offers a useful precedent. In the oilfield, where I worked earlier in my career, any worker on site is expected to stop a job if they see something unsafe, and doing so is treated as part of the job. US offshore safety rules that took effect in 2013 require operators to give anyone working offshore the authority to stop work when they see an imminent risk, and to define clearly who holds ultimate work authority on a facility at any given time. For financial institutions, the relevant lesson is that the ability to stop a system is effective only when people know they are permitted to use it, when it is clear who has final authority, and when stopping has been practiced often enough to be routine.

Questions to answer before an agent goes live

For any AI agent that acts on customers, money or records, leaders should have written answers to five questions before it is scaled.

1. What can the system technically do? Connecting an agent to several systems often gives it far more reach than its intended use requires.

2. What is it allowed to do? The permitted actions should be deliberately narrower than the possible ones, and the difference should be documented.

3. Who can stop it, at what level and how quickly? Stopping may apply to a single transaction, a customer segment, a workflow or the whole system, and the procedure should be tested.

4. Who is accountable when it makes errors at scale? This should be a named executive at a level that matches the potential consequences, rather than a vendor or a committee.

5. How will errors be detected? Errors made by agents often look acceptable one at a time and become visible only as patterns across many actions, so monitoring needs to cover the overall population of decisions as well as individual cases.

Defining the accountable roles

In practice, four roles need to be filled for each agent: the team that designs what it does, the team that operates it day to day, the executive accountable for its outcomes, and the board committee that oversees whether the overall arrangement is adequate for the risk. In the organizations I have seen, the first two are usually in place, while the third and fourth are often missing until a significant incident occurs. Customers affected by an agent's decisions also need a way to understand those decisions and to challenge them, which in regulated industries is part of the license to operate.

Conclusion

Organizations that define accountability before they scale autonomous systems will be able to move faster, because their agents will operate within limits that are clear to everyone involved, with named people responsible for them. Organizations that add these arrangements after an incident are more likely to see their programs paused. AI can take on a growing share of actions, but accountability for those actions remains with the institution that deploys it.