An LLM in a financial workflow is usually not the decision-maker. It drafts the customer message, summarizes the KYC file, or ranks the alert queue. Because a person or a rules engine sits downstream, teams often skip model-level monitoring and rely on the existing controls around it. That works until a summary starts omitting adverse media for one group of applicants, or a provider updates a model and the triage flag rate shifts by a third without anyone changing a line of code.
This article describes a monitoring architecture for LLM governance for financial services: what to measure, where to put the measurement, what to record, and when to take a model out of the path. It treats drift and bias as separate problems with separate detectors that write to one shared evidence trail.
Why an LLM in a financial workflow fails differently from a credit model
A credit model takes a fixed set of features and returns a score. You can validate it against outcomes, freeze it, and monitor input distributions for each feature. Its failure modes are well catalogued, and most model risk teams already have templates for them.
An LLM breaks several of those assumptions:
- The input is free text plus context. There is no fixed feature vector, so “input drift” has to be defined on things like topic mix, language, document length, and retrieved content.
- The output is also free text. There is no single score to compare with an outcome. You have to extract measurable properties from the output.
- The model can change without your involvement. A hosted model alias can point to a new version, and the behavior changes while your code does not.
- The prompt is part of the model. Editing a template changes behavior as much as retraining would, yet it often goes through a lighter review than a model change.
- Identical inputs may not give identical outputs. Sampling settings, batching, and provider-side changes all affect this.
The consequence is that a monitoring design borrowed directly from credit scoring will watch the wrong things. You need to monitor the whole behavioral unit: model version, prompt template, retrieved context, and post-processing, together.
Drift you can actually detect: prompts, model versions, and output distributions
Drift in an LLM system has three sources, and each has a different detector.
Configuration drift
This is the cheapest to catch and the most often missed. Record the exact model identifier, the sampling parameters, and a hash of the prompt template on every call. Pin model versions explicitly instead of using a floating alias, and alert whenever the recorded identifier or hash differs from the approved value. A change you did not approve should be an alert. A change you did approve should be a change ticket with a re-evaluation attached.
Input drift
Track properties of what goes in, per workflow and per day:
- Document length and token counts
- Language mix and the share of non-English or mixed-language inputs
- Topic or category distribution, using a lightweight classifier or embedding clusters
- Source mix, such as the share of KYC files from each onboarding channel
- Retrieval statistics, such as which knowledge-base documents are cited and how often retrieval returns nothing
Compare each window against a baseline window using a distribution distance. The population stability index is a common choice for categorical and binned measures, and the conventional rules of thumb treat values around 0.1 as worth watching and around 0.25 as a significant shift. Those cutoffs are conventions, not standards, so tune them to your volumes.
Output drift
Extract measurable properties from the output and track them the same way:
- Refusal and escalation rates
- Length and readability of customer-facing text
- For triage: the flag rate, the distribution of assigned categories, and the distribution of priority levels
- For summarization: the share of summaries that contain required fields, such as source of funds, beneficial owners, or sanctions-screen results
- Rate of guardrail triggers, such as prohibited phrases or missing disclosures
Add a golden set: a fixed collection of representative and difficult cases with reviewed expected behavior. Run it on a schedule and on every configuration change. Scores on the golden set catch regressions that aggregate statistics smooth over, and they give you a before-and-after record for each change.
Finally, keep a sampled human review stream. Distribution metrics tell you something changed. Only a reviewer can tell you whether the change made the output worse.
Bias checks that fit a text-generating system
Bias monitoring for an LLM cannot be a single fairness metric on a score. There are three practical techniques that fit text systems, and they work best together.
Counterfactual pair testing
Take real or synthetic cases and create variants that differ only in a attribute that should not matter: a name associated with a different ethnic background, a different postcode, a different first language reflected in phrasing, a different stated age. Run both through the workflow and compare the outputs. Look at whether the triage flag changes, whether the summary includes the same risk-relevant facts, whether the tone of the customer message differs, and whether the escalation decision changes.
This works before launch, on every configuration change, and on a schedule in production. It isolates the model’s behavior from differences in the underlying population.
Outcome rates by segment
Where you hold legitimate data on segments, whether collected for fair-lending monitoring or inferred through approved proxies, compare rates across groups in production: flag rates in transaction triage, escalation rates for KYC reviews, and complaint or reversal rates after customer communications. Differences do not prove bias, because the underlying populations differ, but an unexplained change in a gap over time is a signal worth a review. Some jurisdictions restrict the collection or inference of protected attributes, so confirm with counsel what you may use.
Content-level audits of generated text
For summaries and messages, score properties that can differ by group: omission of required facts, hedging language, negative framing, and reading level. A KYC summary that consistently drops mitigating context for applicants with certain name patterns is a bias failure that no flag-rate metric would show.
Summarizing the difference from drift: drift detection asks whether behavior changed compared with last month. Bias detection asks whether behavior differs across groups right now. A model can be perfectly stable and still biased, and a drifting model can be unbiased. That is why they need separate detectors. Both detectors should write to the same store so a reviewer can see a case from both angles.
Where the monitoring layer sits: gateway, evaluation store, and review queue
Three components carry the architecture.
The gateway. Every model call goes through one internal service. It attaches the request ID and workflow ID, enforces pinned model versions and approved prompt templates, applies redaction rules, records the call, and runs inline guardrails. It is also the place where a kill switch takes effect, because routing decisions live there and nowhere else. If teams can call a provider directly from application code, the monitoring has gaps by design.
The evaluation store. An append-only store that holds the logged calls, the metrics computed from them, the golden-set results, the counterfactual test results, and the later outcomes joined back by request ID. Computed metrics should be reproducible from the raw records so that you can show an auditor how a number was derived. This is where the drift and bias detectors read and write. For the storage guarantees such a store needs, the approach in building audit trails that survive a regulator’s scrutiny applies directly: write-once records, ordering you can prove, and access controls on who can alter anything.
The review queue. Human reviewers receive two kinds of items: a random sample from each workflow, and cases the detectors flagged (a guardrail hit, a counterfactual mismatch, an output far from the baseline). Reviewers record a structured disposition: correct, incorrect, incorrect and material, and the reason. Those dispositions flow back to the evaluation store, where they become the ground truth for tuning thresholds and for estimating error rates.
The random sample matters as much as the flagged cases. Flagged cases only tell you about failures your detectors already know how to see.
What to log so an auditor can replay a decision
“Replay” needs a careful definition for an LLM. Because outputs can vary between runs, you generally cannot regenerate the same text later. What you can do is reconstruct exactly what the system saw, what it was configured to do, what it produced, and what happened next. That requires logging at the time of the call.
For each call, record:
- Request ID, workflow ID, timestamp, and the case or customer reference the output belongs to
- Model provider, exact model version, and sampling parameters
- Prompt template ID and hash, plus the rendered prompt or a pointer to retained inputs
- Identifiers and versions of any retrieved documents or policy text included in the context
- The raw model output, before any post-processing
- Post-processing steps applied and the final output used downstream
- Guardrail results, with the rule that fired and the action taken
- The downstream action: message sent, alert flagged, summary attached to the case
- Human involvement: who reviewed it, what they changed, and when
- Latency, errors, and any fallback path taken
- Evaluation scores attached afterward, and the configuration snapshot in force
Two practical points. First, personal data in prompts and outputs falls under your retention and residency rules, so decide per field whether you store the content, a reference to it, or a hash, and where it is stored. The considerations in mapping data sovereignty laws to cloud architecture decisions apply to the evaluation store as much as to production data. Second, retention periods for these records should follow the longest period that applies to the underlying record. A KYC summary that informed an onboarding decision should be retained as long as the onboarding file.
A useful test: pick a decision at random from six months ago and ask someone outside the team to explain, from the logs alone, what the model was told, what it said, and what a person did with it. If they cannot, the log is missing something.
Thresholds, alerts, and the kill switch: deciding when a model leaves the path
Monitoring without pre-agreed action levels turns into dashboards that nobody acts on. Define the levels before launch, with an owner for each.
Three response levels
- Alert. A metric crosses a watch threshold. An owner investigates within a set time and records the conclusion in the evaluation store.
- Degrade. A metric crosses an action threshold. The gateway routes affected traffic to the fallback: a human queue, a deterministic template, or the previously approved pinned model version. Work continues at lower automation.
- Remove. A critical threshold is crossed, or a defined event occurs. The model is taken out of the path, and the workflow runs on its non-LLM process until a named person approves reinstatement.
Example triggers
These are starting points to adjust to your volumes and risk appetite, not recommendations of specific numbers:
- Any unapproved change in model identifier or prompt hash: degrade immediately.
- Golden-set score below the approved baseline on a required behavior: block the release or roll back.
- Counterfactual pair mismatch rate above the tolerance you set: remove from the path for the affected workflow until investigated.
- Reviewer-confirmed material error rate above tolerance in a sampling window: degrade, then remove if it persists.
- Guardrail failure on required disclosures in customer communications: remove from the path for that message type.
- Provider incident or deprecation notice affecting a pinned version: planned switch with re-evaluation.
Set tolerances using the baseline period and reviewer data, and write down the reasoning. An auditor will care less about the exact number than about whether you set it deliberately and followed it.
Make the fallback real
A kill switch that nobody has pulled is an untested assumption. The fallback process needs staffing, tooling, and capacity for the volume it would receive. Include it in your game days; the approach in automated chaos engineering for financial systems covers running failure experiments safely, and a forced model removal is a good one to schedule. Measure how long the switch takes and how the downstream queue behaves.
Mapping the monitoring signals to the controls your regulator will ask about
Regulators rarely ask for “LLM monitoring” by name. They ask about the controls they already expect around models and automated processes, and your monitoring signals are the evidence. The table below maps them. Which obligations actually apply depends on the jurisdiction and the use case, so treat it as a starting point for a conversation with your compliance and legal teams.
Control question Evidence from this architecture Do you have an inventory of models and their uses? Gateway registry of workflows, pinned model versions, and approved prompt templates How is the model validated before use and after changes? Golden-set results and counterfactual test results tied to each configuration change Is there ongoing monitoring and outcomes analysis? Drift metrics, reviewer sample dispositions, and outcomes joined to requests How do you manage change? Hash and version alerts, change tickets linked to re-evaluation records How do you detect unfair treatment? Counterfactual pair tests, segment outcome comparisons, content-level audits Is there meaningful human oversight? Review queue dispositions, reviewer overrides, escalation rules Can you reconstruct a specific decision? Replay-ready logs with retention aligned to the underlying record What happens when the model fails? Documented thresholds, kill switch drills, fallback process records How do you manage third-party providers? Version pinning, provider change monitoring, exit and fallback path
Supervisory guidance on model risk management, such as the US SR 11-7 letter, emphasizes validation, ongoing monitoring, and outcomes analysis, and a monitoring layer like this is a direct answer to each. Where a workflow falls under the EU AI Act’s high-risk category, which includes systems that evaluate the creditworthiness of natural persons, the requirements for record-keeping and human oversight map to the logging and review queue components. A KYC summarizer or a customer-message drafter may sit outside those categories, but the same evidence still shows the control environment around it. For the broader infrastructure side of this mapping, the CTO’s framework for automated financial infrastructure governance covers how to turn controls into automated checks.
A first-90-days rollout for a team starting from nothing
Sequence the work so each stage produces evidence and reduces the largest risk first.
Days 1 to 15: inventory and gateway
List every place an LLM is called, by whom, with which model and prompt. Rank the workflows by customer impact and regulatory exposure. Route the highest-ranked workflow through a gateway, pin its model version, and put its prompt templates under version control with hashes recorded.
Days 16 to 30: logging and the evaluation store
Implement the log fields above for that workflow. Decide on retention and residency per field. Stand up the append-only store and confirm that someone outside the team can reconstruct a sample of decisions from it.
Days 31 to 50: baselines and the golden set
Collect a baseline window of inputs and outputs. Build a golden set of representative and edge cases with reviewed expected behavior, and run it against the current configuration. Compute the first drift metrics and set watch thresholds from the baseline.
Days 51 to 70: bias testing
Build the counterfactual pair suite for the workflow, choosing attributes with your compliance team. Run it on the current configuration and record the results as the starting position. If the results show gaps, fix or constrain the workflow before adding more automation.
Days 71 to 85: review queue and thresholds
Launch the sampled review queue with structured dispositions. Write the three response levels, name the owners, and set the action and critical thresholds using the baseline and the first reviewer data.
Days 86 to 90: drill and evidence pack
Run a kill switch drill: force a model removal in a controlled window, route traffic to the fallback, and record the time taken and the effects. Assemble the evidence pack: the inventory, the configuration history, golden-set and counterfactual results, drift reports, reviewer dispositions, the threshold document, and the drill record. Then extend the same pattern to the next workflow on the ranked list.
By the end of this period, one workflow has a full evidence trail, and the template for extending it exists. That is a much stronger position to be in when a regulator asks than a plan to build monitoring after the question arrives.