Executive Summary

This study compared the performance of Amplifi (a specialist reg-tech tool) and Microsoft Copilot (in MS Word) in simplifying complex, regulated communications, specifically within the UK financial services sector.

Main Findings

The study concludes that the primary difference between the two tools is systemic: Amplifi functions as an operational risk control, while MS Copilot acts as a generative drafting assistant.

• Measurable Intelligibility: Amplifi produced significantly higher gains in document intelligibility. In paragraph-by-paragraph testing, Amplifi improved a credit card agreement's score by 66%, compared to MS Copilot’s 31%.
• Diagnostic vs. Generative: Amplifi identifies specific complexity drivers prior to editing. MS Copilot lacks native diagnostics, forcing users to rely on subjective judgment to find risks.
• The Scoring Inflation Risk: A critical finding revealed that MS Copilot significantly over-scored its own work. In one instance, MS Copilot gave a simplified press release a score of 82/100, while Amplifi’s objective assessment using its validated method was only 50/100. This creates a material compliance risk by providing false evidence of clarity.

 Impact on Compliance and Governance

The study highlights several potentially high stakes impacts for firms managing Consumer Duty and other regulatory obligations:

1. Semantic Drift: MS Copilot frequently weakened precise legal qualifiers or altered the sequencing of terms. This introduces hidden risk where a document becomes easier to read but loses its legal integrity.
2. Auditability: Amplifi provides a structured journey from initial assessment, to editing, and then reporting. It generates a traceable audit trail suitable for senior management and regulators. MS Copilot lacks an integrated governance architecture, shifting the burden of validation entirely back to the human user.
3. Efficiency vs. Effectiveness: While MS Copilot accelerates the speed of drafting, it does not reduce the diagnostic burden. Amplifi reduces the total effort by directing the user to high-risk clauses, ensuring interventions are high-impact and evidence-led.

Conclusion

For regulated environments, the study suggests that Amplifi is a risk-managed system that provides defensible proof of improvement. Conversely, MS Copilot is a more unstable environment for sensitive text, where performance is highly dependent on individual prompt engineering and manual human oversight to prevent semantic drift.

MS Copilot’s limitations in text simplification and assessment

High dependence on prompt wording

Simplification is contingent on how the task is phrased, which introduces user variability into the process.

Poor control over editing scope

MS Copilot does not always act only on the selected or intended text, increasing the risk of unintended edits and weakening trust in the editing workflow.

MS Copilot’s interaction model is inconsistent

Sometimes allowing the user to review alternative text before applying it, at other times it writes directly into the document. This reduces transparency and control. 

Inconsistency across repeated runs

Similar prompts can produce materially different outputs, both in content and in how the system applies the task. 

Weak reversibility and unstable editing recovery

Restoring or reviewing the original wording is not always straightforward. This shifts the burden back onto manual document management and version control.

MS Copilot takes initiative beyond the user’s explicit instruction

The system not only generates language but sometimes redefines the task itself. That increases the likelihood of unintended changes.

MS Copilot’s assessment of intelligibility is not reliable

It tends to produce scores that are materially more favourable.  It cannot be relied upon to validate the success of its own simplification.

Risk of semantic and structural drift

Legally significant wording was frequently softened, qualifiers weakened, and sequencing was altered.  This required manual checking, reducing efficiency and increasing risk.

Governance and auditability limitations

Prompt history, output history, and revision logic are not always captured in a way that supports robust audit trails. Governance controls must be recreated externally.

Download the full report here

‍