How we test

An audit runs in two halves. A testing harness we built captures your product and records what a keyboard user, a screen reader and an AI agent meet at every step, and then a named person confirms every finding by hand before it is reported. This page says what each half does, so a buyer, an engineer or a journalist can judge the method rather than take the signature on trust.

What is in scope

A web audit covers the building blocks your product is made from (a component is one distinct interactive pattern, counted once however many pages use it) and the customer journeys that matter to revenue (a flow is one task, up to twelve steps). An app audit covers the screens and flows of an iOS or Android build. Third-party embeds, PDFs and email templates are out unless priced. The proposal lists the exact counts.

The automated sweep

The harness drives a real browser through every flow at a desktop and a mobile viewport and stores, for each step, the page as it was (its markup, its accessibility tree, a screenshot, a video of the step), where keyboard focus went after each action, and what any live region announced. On the stored pages it runs axe-core and Lighthouse's accessibility audits, and a set of our own checks for the things those two do not judge: whether focus is visible, whether a dialog takes and returns focus, whether a status message is announced, whether targets are large enough, how the page behaves at 400 percent zoom, with wider text spacing, in forced colours and with reduced motion requested, and whether a keyboard user can reach every control and leave again.

The vision pass

A multimodal model reviews every screenshot beside its accessibility tree and flags what a scanner cannot see: a state shown by colour alone, text that exists only in an image, low contrast over a photograph or gradient, a visible label that does not match the name a screen reader gets, text that is clipped or overlapping at that viewport, and anything on screen with nothing behind it in the tree. These are candidates for the human pass, never conclusions.

The agent benchmark

Two AI agents, one from Anthropic and one that is not, are given each flow as a plain-language task, the way a customer would phrase it, and left to complete it. Agents are not deterministic, so one run is an anecdote: each flow runs five times per agent. We report, per flow and per agent, how often the task was completed, the median number of steps and the median time, the element the agent stalled on, and the model name, version and run date, because the models change monthly and you will want to rerun it. This is a benchmark on a date. It is not a certification; there is no standard to certify against.

Agent runs happen on a staging or test environment with test accounts, or on production with a flow that stops before payment. Never a real order, never a real card, never a real customer account.

The public benchmark

Twice a year we run the same harness across fifty consumer sites in one market, the UK first and then Spain: Lighthouse's accessibility and agentic-browsing scores, and one agent asked to reach the site's main call to action. We publish the distribution, the common failures in plain words and the names of the top performers only; nobody is named for failing. Public pages only, robots respected, one visit per site, no login. The agent stops at the first form: it reports whether it could reach and identify the main call to action and its fields, and never submits, signs up, adds to a basket or creates an account. Every score is dated with the Lighthouse and model versions, the dataset is available to journalists and the accessibility community on request, anonymised, and any site can ask to be left out of future runs.

The screen reader runs

Real screen readers, driven from the browser rather than imitated: NVDA with Chrome on Windows and VoiceOver with Safari on the Mac walk each flow the way a screen reader user would (headings, landmarks, forms, then the task by keyboard), and every phrase spoken is logged against the step and the element, five runs per flow per reader, with the audio recorded. Where what the browser's tree implies and what the reader actually said differ, that difference is itself a finding. iPhone VoiceOver and Android TalkBack cannot be driven this way and are tested by hand.

The manual pass: always, and pointed

The harness evidences focus, announcements and spoken output; a named human confirmed every reported finding on a real screen reader on the current build. The manual pass has two parts and they are stated plainly because the split is where the method lives.

Always, on every audit: a person goes through each flow end to end with NVDA on Windows and VoiceOver on an iPhone, listening for what the logs cannot judge (reading order that makes sense, verbose or misleading announcements, custom widgets where the tree and the reader disagree); a visual pass at 400 percent zoom, in forced colours and with reduced motion; dragging movements, which cannot be judged from the markup; your own AI chat widget, if you have one; and anything the agents or the readers did that looked like a design fault rather than a markup fault.

Pointed: every candidate the harness raised is shown to the person with its annotated screenshot and a plain-text description, ranked by severity. They confirm or reject each one, recording the platform and the date, and the worst findings are confirmed on a real screen reader before they are marked. Nothing reaches the report, the tracker or a statement until a named person has done this on the current build. Candidates the person rejected are kept for the next run and never reported.

What you receive

Each finding carries what a customer experiences, the technical cause and the fix, its component, its scope (fix it once in the library, fix it on this page, or fix the page itself), a guess at the owning team, the WCAG 2.2 success criterion and the EN 301 549 clause from a checked lookup table, and the evidence: an annotated screenshot with a plain-text equivalent, and for the failures that need motion to be understood, a short clip with captions and a transcript. The report's headline is two numbers: what the design-system team can close, and what stays with product teams regardless.

Findings arrive in your Jira as issues with the evidence attached, imported by your own engineer with their own login; no access is given to us and your admin is not involved. The import is a small, public, client-run tool with no dependencies and no connection to us, and the bundle it reads is written so that a coding agent can work from it directly. The import bundle is built to be handed to a coding agent, which imports the findings into your tracker or starts fixing from them with the same evidence attached. Another tracker: say which, and it goes there.

What we do not test

• Third-party embeds (payment widgets, chat, maps, cookie banners from a vendor) beyond noting how they behave when met, unless they are priced in.
• The interface of an AI assistant itself. If you deliver your service through an assistant, we audit what your side returns and requires; the assistant's own streaming, focus and confirmation dialogs belong to its provider.
• Physical devices for agent runs on iOS, and mobile screen reader runs more than once by hand. Simulators and emulators differ from devices, and every app report says so.
• Your conformance. We advise; you remain responsible for the conformance of your product. The report says what was tested, what was found and what we recommend.

Apps

An app audit mirrors the web one: Appium drives the iOS simulator and the Android emulator through each flow and stores the screen, the accessibility tree and a recording; Apple's own accessibility audit and Google's Accessibility Test Framework run over them with our own checks for unnamed controls, small targets, missing roles and contrast; a mobile agent runs the same five-run benchmark; and a person tests on a real iPhone and a real Android phone with VoiceOver and TalkBack, the largest text size, Switch Control basics and reduced motion.

The site you are reading

This site is built to WCAG 2.2 AAA and run through the same harness. How it is checked, and how to tell us when it fails, is in the accessibility statement. Questions about the method: [email protected].