Lenna Price Zitter - Digital Strategist

It is easy to assume that a simple automated accessibility test tool captures everything. However, it was never intended to be a single measure of the accessibility of a site.

Digital accessibility tests are more than automated accessibility tests alone.
Digital accessibility tests are designed to measure how well your website, service or app is compatible with people with different abilities. This requires a balance between automated and manual testing methods in order to achieve an appropriate measurement.

Automated tools were designed only to accelerate QA tests. They work by comparing the code of a website with some success criteria in WCAG. This leads to several gaps, including errors that can only be detected through manual testing and the lack of error analysis.

These tools tend to generate long lists of problems. Despite catching every instance of an error they are programmed to catch and identify as an error, they will not identify the root cause. For example, if there are many color contrast errors, they will not tell you that they come from a line in the CSS. This requires additional time and effort for your team to examine.

As a result, despite the number of problems it raises, automated tests will not provide an exhaustive or refined list.

Why does it not test all criteria?
Automated tools rely exclusively on code and heavily on code quality. If the code is clean, they may not show any problems, despite obvious problems. As a result, the output can be incomplete. These missing errors can then be pushed undetected to the live site. Here are a few examples.

The contrast of color is not met with text and images.

For example, it is common practice to have text overlaid on hero banners on corporate websites.

There are multiple ways to code this, such as:

• Overlaying a text box on top of the image.
• Adding the image using CSS as a background property of the text box.
• Embedding text as part of the image.

All of these scenarios pass automated tests, but may not meet the minimum color contrast ratio with manual tests. The background color is not shown in the code here.

Manually test with color contrast tools:

To check this in design software, identify the background image color with a color picker. If the image is busy, try to blur the image slightly. This forces the pixel colors to converge on average. If you have large color blocks as your background, choose the closest tone to the font color. Simply enter this value together with the font color in a tool like the WebAIM color contrast checker to assess the contrast ratio.

Many modern automated testing tools are already warning testers to use manual tools to check for this.

• Screen readers and alternative text
• Alternative text quantity
• Automated tests only reveal whether images have alt tags. It will not recommend whether these images should be marked as decorative.

Adding alt tags to every image for access is certainly noble. However, there is a risk of sacrificing the quality of experience for people who use screen readers by providing too much distracting information. This can lead to a very repetitive and overwhelmingly verbose experience, especially for those new to this technology.

For a simple example, imagine a pencil icon with a label. If the alt text and label both say ‘Edit’, the screen reader can sound like this: “Edit image, Edit. Edit button.” This message can obscure the fact that it is simply an edit button.

Alternative text quality
Automated tools cannot check whether the alt text serves as an appropriate or accurate descriptor for the image. If there is an image of an apple and the alt text says “orange,” it will not flag this. It is not a substitute for manual tests with screen readers.

Test screen readers: Try this by using one of the most common screen readers, such as NVDA. Check the whole website and access the content.

Font is too small or unreadable
There is no official WCAG directive that specifies the minimum font size. Automated tests do not identify poor design decisions. If parts of your website have an extremely small font size, it must be reassessed. If it’s too small, it may not be there at all.

Tests with user research: These types of problems are often revealed by careful observation in user research with all people, not just people with disabilities. If users with clear vision need to lean in or reach the zoom tool, rethink this design decision.

As you can see in these examples, automated tools should be paired with other testing methods to form an accurate accessibility assessment.

So where does automated testing shine?
Despite their shortcomings, automatic tests are not all bad. Your site could be a complicated mess with hundreds of pages of code under the hood. Considering the length a QA tester has to go through to perform regression tests. Automated tests can serve as a canary in the coal mine to draw attention to specific problem areas or identify site-wide error patterns.

What else can my team do?
Access to people with different skills is much more than just meeting WCAG and legal requirements. Next time you think about an accessibility test strategy for your website, consider automated testing as a complement and not a substitute for user research and manual testing.

Of course, prevention is the best medicine here. All teams can prevent most of these issues by adopting an accessibility-first mindset, in all things they do.