Principle 5:

Reliability and Operational Integrity 13

AI financial products or services produce consistent, accurate, and predictable outputs across varied inputs and conditions.

Technical Accuracy and Consistency of Information

  • The product’s outputs are factually correct, current, and reliable—including in financial calculations, data sourcing, and replicability across equivalent prompts.14
    • The outputs the product generates are up-to-date and factually correct.
    • Outputs are consistent, reliable, and reasonably replicable when similar prompts are used across users, including when demographic information or user context varies.
    • The product accurately represents the currency of its information and proactively flags when its outputs may be out of date.
    • The product provides limited, topical information relevant to the financial services role it is designed to fulfill and immediate tasks requested.
    • Predictive accuracy metrics are cross-referenced and scrutinized using appropriate measures.
    • The product handles edge cases in financial calculations accurately, including rounding, currency conversion, and compound interest.
    • The product does not generate hallucinated financial figures or fabricated account information, and provides appropriate disclaimers when performing calculations that carry inherent uncertainties.
  • The product maintains appropriately calibrated uncertainty under sustained multiturn questioning.
    • The product shows limited or no performance degradation when transitioning from single-turn to multiturn interactions.
    • The product does not escalate its own confidence beyond what the underlying evidence supports.
    • The product does not make assumptions when information is unavailable.
    • The product does not build on incorrect or inaccurate information once an issue is identified.
    • When incorrect or inaccurate information is identified during multiturn interactions, the product identifies the source of the information and explains how the conclusion was reached.

Resilience and Fail-Over

  • Consumers retain proper access to services through backup mechanisms in the event an AI system fails or experiences disruptions.
    • In the event of an AI system failure or degradation, users retain access to all core account functions through a non-AI-dependent channel without being required to interact with the failed system to access this channel.
    • Users are notified promptly through a channel independent of the affected AI system when an outage or degraded service condition occurs, with clear information about which functions are affected, which backup pathways are available, and, when possible, a realistic estimate of when full service will be restored.
  • The underlying infrastructure is built to anticipate and avoid fatal errors or service degradation.
    • Version controls for models, code, and parameters are created and maintained, with a log of all changes, including which developers were involved.
    • Data pipelines have clear stages for development and implementation, with automated metric reporting between stages.
    • Every model deployed or trained is documented with key parameters and relevant quality control metrics.
    • Model performance is monitored regularly before and after deployment, with automated monitoring and developer notification for anomalies in usage, error rates, or latency.
    • The model’s infrastructure is scalable, with sufficient capacity for project usage and opportunities for growth.
  • Fail-Over procedures are in place to handle service incidents, with human oversight for critical consumer-facing functions.
    • Fail-Over procedures are in place to handle service incidents; any AI agent doing the work for a human on behalf of a consumer has a fallback human.
    • Critical services conducted with AI (e.g., account lockout, fraud reporting, closing accounts) have direct human supervision that is immediately responsive to consumer needs.
  • Product failure, user issues, and customer service are efficiently and resiliently handled.
    • The product is able to determine when a task, a prompt, or an issue requires an urgent resolution and/or should be escalated to a human representative.
    • The product escalates in a tiered manner in response to increasing levels of urgency or distress expressed by the user and potentially other estimations of harm.
    • All issue resolutions or customer service representatives, whether human or automated, are helpful, respond in a timely manner, and facilitate the complete resolution of the issue.5
Back to all principles Read next Principle
Back to Paragraph
13

This principle governs technical accuracy, the factual currency of information, the correctness of financial calculations, and the system-level consistency of outputs across interactions. Whether the product maintains accurate responses under social or emotional pressure from users is addressed in Principle 4: Honesty and Non-Manipulation.
14

This standard addresses the reliability of the product’s information and computational outputs, not the product’s behavioral disposition toward honesty under user pressure, which is governed by Principle 4.