Automated vs. Manual Testing

If you’ve ever done an online search to look for resources to begin auditing the accessibility of your website it’s likely you’ve seen, and possibly have used, an automated accessibility testing tool. While manual testing involves a real person going through a website using assistive technology to evaluate its accessibility, automated testing uses software to scan a website’s code and flag automatically detectable issues.

What a lot of automated tools fail to convey is that using them alone will not catch off all website’s accessibility issues.

What is automated testing

Automated testing software—often a plugin or browser integration—runs a script to analyze the underlying code of a webpage or website against known accessibility rules. These tools evaluate rules that have a clear and detectable pass/fail condition that will either be met, or not met, within the code.

These tools are typically free to low-cost with results that are fast, repeatable, and require little to no accessibility experience to interpret. Problematically, a lot of accessibility tools fail clarify that the rules used to analyze a website is not the complete WCAG standard. Due to the subjective and experiential nature of a lot of digital accessibility requirements, most of the guidelines cannot be evaluated without human-review. If this is not understood, many accessibility tools—especially those that provide an “accessibility rating”—may create a false sense of confidence to the businesses using them.

What automated testing can reliably do

Automated testing should absolutely be used as a first line of defense against accessibility issues—but it’s important to know what the strengths and limitations of this testing method are.

1. Identify missing alt-text on images

Automated tests can quickly scan images across an entire website, check for alt-text, and flag instances where this attribute is missing. Adding alt-text to your website’s images is both an SEO and an accessibility requirement.

2. Identify low color contrast ratios

Poor color contrast (the difference between foreground text colors and background colors) can make it difficult for users to read content and discern actions on your website. While all users can be affected by this, it is especially impactful for those with vision issues. Accessibility tools are a powerful way to identify and evaluate the contrast of all color combinations in use across your website.

Having your website or brand color palette evaluated for accessibility can make the process of remediating these issues easier. By identifying what accessible color combinations are available to use before you run automated testing will allow to fix any flagged issues quickly and more confidently.

3. Identify missing form labels:

Ensuring form inputs are properly labeled is critical for screen reader users to be able to know what exactly an input is, and what information is required of them. Automated testing tools can identify and flag inputs across your website that are missing properly defined labels.

4. Identify missing page language declaration:

While the language of a website may seem visually apparent, it needs to be explicitly defined within a website’s HTML in order for screen readers to know what language page content should be announced in. This is especially important for contexts in which a the primary language of a user’s screen reader or device is different than that of a website. Without a proper language declaration the screen reader will may be unable to recognize characters and create comprehension issues by incorrectly synthesizing the content into an incorrect language’s accent and speech.

Through automated testing tools you may be able to check that your website’s page language has been declared, and aligns to the correct language of your content.

5. Identify empty links and buttons with no name:

Empty links and buttons lack a clear destination or description of action. Without this information, screen reader users who are missing any visual clues that identify this information may become lost or confused. Automated testing tools can scan the links and buttons across your website to flag instances where this information may be completely missing.

What automated testing cannot reliably do

While there are differences between automated testing tools, the results in their analyses largely overlap. It’s best to pick a tool you feel comfortable and confident using to focus automated testing efforts with; running tests through multiple tools will take additional time and won’t do much to expand your sites coverage.

Most sources indicate that automated testing tools can only catch about 30%-50% of a website’s accessibility issues. Even Deque’s own research—the company behind industry leading AxeDevTools—shows their software identified just 57% of problems. While this data, according to Deque’s own reporting, show’s higher results than other tools, it still fails to detect almost half of a website’s accessibility issues. These undetected issues are ones that are simply not able to be effectively evaluated through automated methods, not a signal of these tools reliability, and it’s important to understand where these limitations begin.

1. Evaluate image alt-text

Automated tools can only detect whether or not alt-text is present, they cannot make a judgement as to the accuracy or meaningfulness of alt-text that exists. Due to the subjective and highly contextual nature of this determination a human-reviewer must be responsible for the creation and evaluation of an image’s alt-text.

2. Evaluate error messages

Another area that is highly subjective and contextual is the clarity and effectiveness of an error message. While automated tools may be able to flag when inputs do not have error messages defined, a human-reviewer needs to write and evaluate the usability of the message itself.

3. Evaluate heading structures

While automated testing tools may flag certain explicit conditions—like not having multiple H1 headers on a page (although even this rule can have an exception)—automated tools cannot determine if the heading structure is defined in an appropriate and logical manner, or if it is properly presented through a screen reader. Additionally, automated tools cannot determine if the structural relationship of headers aligns to the content, or make recommend improvements.

Heading structure has significant impact on both accessibility and SEO. As such, its critical that the content and structure of these elements goes through manual testing and human-review.

4. Catch interaction issues

Some issues may only present themselves during on page interactions like completing a form or opening a menu. These interactions can trigger dynamic content or introduce new conditions within a page. Even if these elements are structurally correct—according to the criteria of automated testing tools—the dynamic method by which they appear may cause screen readers to announce them incorrectly, or not at all. These types of issues in an experience are subjective and a result of unique scenarios within your website. Often-times only manual testing with a human-reviewer is able to identify these issues.

5. Test screen-reader or keyboard navigation experiences

One of the biggest reasons a website will never reach full compliance through automated testing tools alone is because while automated tools can evaluate a website based on best practices and WCAG guidelines with definitive conditions, they cannot evaluate the actual experience of a user who rely’s on a screen reader or keyboard to navigate. Only when website is viewed through these user’s experiences can it be evaluated for factors like arduous navigation, blocked actions, and unclear or ineffective content.

Even following the development specifications and requirements of WCAG, the uniqe behavior and structure of every website may cause unique scenarios. Ultimately, the only way to catch these experience issues is to conduct human-led manual testing.

Commonly used automated testing tools

WAVE (Web Accessibility Evaluation Tools)

WAVE is available as a browser extension for Chrome, Firefox, and Internet Explorer or as a stand-alone API that can be integrated into your development frameworks. It is made and maintained by WebAIM and Utah State University; WAVE has been an industry leading tool for automated accessibility testing since it’s creation in 2001.

When testing with the browser extension, all analysis is ran entirely within your browser, on your device. No information is transferred to WAVE’s servers, making the browser extension a reliable and 100% private method for evaluating internal or password restricted content.

Cost: Free for the browser extension.

Axe DevTools

Axe DevTools is available as a browser plugin for Chrome, Firefox, and Internet Explorer; developed and maintained by Deque. It’s one of numerous tools and resources the company offers to accessibility professionals.

Cost: Free for limited functionality—premium tiers unlock additional tools, AI supported automations, and additional functionality and integrations.

Google Lighthouse

Lighthouse is only available within Google Chrome, built directly into the browser’s development tools. Lighthouse measures a website’s accessibility as well as other aspects such as overall performance and SEO.

Cost: Free

What is manual testing

While there are numerous aspects of a website’s content and semantic structure that can and should be human-reviewed, manual testing typically refers to the process in which an reviewer navigates the website using assistive technology—like screen readers and keyboard-only navigation inputs—to deliberately evaluate it for ease-of-use, clarity, and alignment with WCAG criteria. Manual testing requries familiarity and use of accessibility tools, and knowledge of accessibility standards, in order to interpret what is or isn’t a point of failure.

What manual testing reliably catches

While manual testing can take significantly more time to complete than automated testing it is the only reliable way to capture the accessibility issues that automated tools miss or are unable to effectively evaluate. It’s important to understand some of the aspects of accessibility that rely more heavily on manual testing methods.

1. The usefulness of alt-text

While automated testing tools can flag where a site is missing alt-text, manual testing can help evaluate the effectiveness of alt-text, as it is announced by a screen-reader, to ensure it is providing meaningful and useful context. Manual testing can also aid in evaluating where alt-text becomes unnecessary or distracting from the important content.

Automated tools will typically flag alt-text as a requirement of all visual assts. Adding alt-text to some visuals, like graphic elements or purely decorative background images, that aren’t providing important context can actually be distracting and disrupting to the experience of screen reader user. Manual testing can help identify instances in which alt-text descriptions aren’t providing the context they need to, and where they should be removed from non-essential decorative assets.

2. Usability of forms

once again, while automated testing tools will flag form elements that are missing required formatting, like labels, they cannot evaluate the actual experience of completing the form itself. Manual testing will help determine if a form is genuinely operable and and understandable from start to completion—and can help detect any failures or weaknesses regarding the way form requirements are communicated, error validation, and success confirmations.

3. Usability of focus order

Automated testing tools may flag when interactive elements—such as links, buttons, and inputs—are missing focus states but they cannot evaluate the effectiveness of the focus interactions. When using the tab key to navigate by keyboard, all focusable elements should receive focus in a logical and expected order.

Commonly a bright, high-contrast outline; all focusable elements should display a clear and prominent indicator when they receive the focus. Without this, alternate navigation users may be unable to discern where—if anywhere—they are within a page. Manual testing can help determine if a website’s focus states are visually effective, and if focus states move in a logical and expected order.

4. Effectiveness of error messages

Automated testing tools may be able to detect if inputs or forms are missing defined error states—but the evaluation ends at that binary condition. Manual testing should be used to determine if the error messages are clear and effective, and evaluate the ease with which users can identify and remediate errors that appear.

5. Coherency of the website

Many websites are visually coherent; if the content’s hierarchy and context relies on visual structure and styling to create that coherency, the experience is likely to degrade when presented audibly through a screen reader. Manual testing should be used to review the navigation, presentation of content, and overall coherency of a website when it is heard instead of seen.

Who conducts manual testing

The great thing about accessibility tools is the free availability and wide access to them; in large part due to how much native support is integrated within most common devices. While many individuals with disabilities and accessibility experts may choose to use stand alone software or external accessibility devices, the barriers for anyone to begin learning and using the accessibility tools within their device are incredibly low.

Because of this, it’s entirely possible to begin conducting your own manual testing to identify issues within your website—there will likely be a learning curve as you adjust to using these tools and the new interaction patterns. If you want to save time, and have the available budget to bring on additional support, you can explore the following options:

• Dedicated accessibility specialists auditors: These individuals will be able to confidently and efficiently conduct manual testing and provide a report of their findings and recommendations for remediation. Depending on resource and the type of remediation required, you may be responsible for implementing the corrections yourself.
• Designers and developers trained in assistive technology: Not every designer or developer will have experience in this area, but if you find a resource with accessibility knowledge they may be able to help perform automated and manual testing, provide recommendations, and assist with implementing corrections.
• Users with disabilities: While this is different than traditional manual testing, incorporating users with disabilities into usability testing can provide valuable and immediate insights as to the accessibility and experience of your website. We recommend making an effort to identify and remediate most of the clear cut compliance issues (like those identified through automated testing) before conducting usability tests with disabled users. This way, feedback can be more focused on improving the true quality of your website’s accessible experience.

A practical approach to accessibility testing

Many small businesses are aware of the need for accessibility testing but may feel that they lack the budget or resources to fully commit to it. Delaying accessibility work is only putting your business at greater risk. Remember, reducing your business’s legal risk is about being able to demonstrate the considerations and consistent efforts you make—starting slow and small is better than delaying the project entirely.

1. Start with automated testing

Install a free, industry endorsed plugin like WAVE or Axe DevTools on your website. If your website is large and scanning all of it feels overwhelming, start by testing your most important pages first—these are likely the homepage, contact page, product or service pages, and checkout. Work on fixing any issues that have been flagged within these pages, and then move onto another group of pages within your website.

2. Layer in basic manual checks

After you clear the accessibility issues on some pages, you can try to run your own manual tests. These tests might not be as effective or comprehensive as a professional manual audit would be—but you will catch issues and begin to get more comfortable using these tools and identifying issues.

1. Using your keyboard’s tab key try to see if it allows you to reach ever interactive element. Make sure the selected element is always obvious and that selection progresses in a logical and expected order; try typing when an input is selected to check you can input text without additional action; and check that the enter/return key is able to open menus, make selections, and trigger links or actions. Check that any action able to be performed by a standard mouse user is able to be triggered using these keys.
2. Turn on your device’s screen reader—VoiceOver (Mac or iPhone), NVDA or Jaws (free downloads for Windows), or TalkBack (Android)—and try to navigate entirely by ear. Listen for any issues in the order that content is announced; check that links, actions, and labels are clear and descriptive; and make sure that any important visual context, or dynamic content is announced out loud.
3. Increase your browser’s default text size to 24px; make sure that your website layout responded appropriately. Next, increase your browser zoom to 200% and review the website layout again. In both instances, page content should adapt responsively, interactions should still work, and content should not fall off the viewport or overlap.

3. Recognize when to bring in a professional

It’s possible, and encouraged, to take ownership of evaluating and remediating as much of your website’s accessibility issues as you can. Doing so can deepen your awareness of website accessibility issues and improve your businesses approach to accessibility in future updates for new projects. There are certain points however when it becomes necessary to bring in a professional expert:

• Upon receipt of a complaint or demand letter: If this occurs you should consider bringing on a professional resource to address concerns or perform deeper evaluations before complaints escalate into lawsuits.
• Before a site launch or major redesign: Accessibility should be a consideration throughout your website’s design and development process, but if you have been self-managing that process it might be a good idea to bring on a professional to perform a final, deeper audit—before major site changes are finalized, or new features a launched. Doing so can help identify any complex issues that had they to be identified and help ensure that the most accessible baseline for your website goes live.
• When internal efforts or the best path to remediation is unclear: While some accessibility requirements are relatively straightforward to identify and correct, there is a subjective aspect to the process that can make it difficult to always self-identify the appropriate solution. A professional expert can help in these moments to clarify what are legitimate accessibility concerns, and provide guidance on the best ways to remediate issues you’re not confident on.

About accessibility audits

A formal accessibility audit—when performed by a credible professional—should identify specific failures on your website, map them to the related WCAG success criteria, and prioritize them by severity. While professionals will and should use automated testing tools as a first step in this process, full-service reviews should always include manual testing methods as well. Even automated testing should include a level of professional human evaluation to determine the validity and impact of flagged issues.

When evaluating accessibility professionals or auditing services watch out for the following red flags:

• Services that claim to be “full accessibility audits” but promote or provide reports that that are fully generated from automated testing tools. Unless an audit includes human evaluation and manual testing, it cannot be considered comprehensive.
• Vendors who price and promise website audits and remediation as a bundled service. These are distinctly different services; the scope of work required to remediate a website’s accessibility issues cannot be determined until the scale and breadth of those issues is identified through a comprehensive audit.

Why knowing the difference between automated vs. manual testing matters for small businesses

Getting the “all-clear” on an automated accessibility test means only that most obvious, machine-detectable errors have been identified and, ostensibly, resolved—it does not mean that website is actually accessible. In fact, a website can pass every automated test performed and still fail to support screen readers and alternate navigation.

Knowing the difference between automated and manual accessibility testing is the best way to protect yourself, and your business, from developing a false-confidence and over-reliance on the results of automated accessibility tests. There is a fundamental gap between the point at which a website passes automated testing, and the point at which it becomes legitimately accessible; this gap is where most business owners under-estimate their website’s level of risk. Manual testing; the only method that can reliably evaluate a website’s actual accessibility; is how to close said gap.

Being more knowledge in the different methods of accessibility testing can also reduce the decision paralysis many business owners fell upon becoming aware of accessibility requirements. It’s important to know that you don’t need a full investment or a perfect accessibility audit to reduce your legal risk. Starting with a free, automated scan is a legitimate and valuable first step to take—as long as you are aware that it is a step to build off of, not to stop at.