A regulator’s sandbox is a good place to test a proposition. It is not a substitute for the environment where the work should already have started. Most firms are experimenting in production without realising it.
The choice most firms are actually making
A regulated firm that wants to test an AI use case before deploying it faces a narrow set of options. Internal tooling is often inadequate for the task. External sandbox provision is available, but many firms are unaware of it or unprepared to use it well. The result is not that experimentation stops. It is that experimentation happens in production, without guardrails and without baselines, because that is where the tools already are.
This is the argument for treating an internal AI testing environment, an AI workbench, as governance infrastructure rather than an engineering convenience. Not because engineers want one, though they do, but because without it a firm cannot generate the evidence its own accountability framework requires.
Product governance comes before the build
The most common mistake is to approach this as a technical procurement. An AI workbench is a product governance instrument first. Before anything is built, there should be clear documentation of what the AI has been instructed to do, by whom, for what purpose, and with what guardrails. That documentation is not overhead. It is the artefact a senior manager will rely on when asked to stand over the system’s behaviour.
Six parameters worth specifying
A benchmark defined before deployment. What good looks like, how performance will be measured, and what constitutes a successful or failed outcome. Without this, drift is undetectable and return on investment is unprovable.
Explicit workflow mapping. Which workflows the AI operates within, who owns those workflows, and precisely where human oversight is required.
An accountability layer around the model. A documented framework of rules, guardrails and escalation protocols defining acceptable and unacceptable outputs. This is the governance wrapper, not an attempt to open the model itself.
A synthetic-to-live data pathway. The conditions that must be met before live data is introduced, and the evidence that must be generated during the transition.
Model selection evidence. What alternatives were tested, what the results showed, and why the chosen model is the appropriate tool for this specific use case. This is one of the first things an institutional buyer or a regulator will ask for, and one of the last things most suppliers can produce.
Isolation from production. The point of the environment is that experiments cannot touch live customer data or systems. If that boundary is soft, the environment is not doing its job.
On governing what you cannot see inside
The objection raised most often is that large language models are black boxes, and that governing them is therefore impossible. The more useful framing is that you do not govern the black box. You govern the inputs, the rules, the guardrails around the output, and the outcomes. What matters is whether the system produces the right results consistently, whether edge cases are identified and routed appropriately, and whether there is a human in the loop who can be held accountable when it matters. A person who is there only to click approve is not a control. A person who understands what the system is doing and can exercise judgment is.
You govern the inputs, the rules, the guardrails around the output, and the outcomes. You do not govern the black box itself, because you cannot.
The commercial argument
There is a sales case here as well as a compliance one. Institutions across the market know they should be using AI and feel the pressure not to be left behind, but many have not defined a use case, established a baseline, or decided how success would be measured. A supplier that arrives with a clearly defined problem statement, tested evidence of performance and a coherent answer on return on investment is in a categorically stronger position than one arriving with a demonstration. Internal sandboxes are where that evidence gets made.
Author: Stephen Walsh, CEO, Keeper Solutions
READ THE REPORT
This article draws on the AI Capability Building working session, convened by FinTech-Tables with Keeper Solutions and Higgs LLP. The full report sets out the keynotes, the three working group findings, the five key challenges and the ten takeaways in full.
