PDF accessibility checker comparison: Tingtun and Acrobat Pro

Provided that you understand their limitations and differences, PDF accessibility checkers can be useful tools. But which one to use? This article compares the web based Tingtun PDF accessibility checker with the built-in Acrobat Pro checker. (I rarely use PAC 3 for a number of reasons, but that is perhaps for another day.)

For the first part of this test I used a PDF that has been a key part of my training documentation for years. It has a number of problems built into it specifically for the purpose of demonstrating how to test a PDF for accessibility. I then tested a further nine documents created by different authors and organisations.

Test results for Tingtun

Firstly, I uploaded my test document to the Tingtun site and then ran the checker. The results panel can be seen below.

Analysis

Tingtun reported four categories of “errors” in this document, namely headers and footers, bookmarks, alt text and tables.

Headers and footers

The first reported “error” was the absence of running headers and footers. Tingtun reported this as a fail against WCAG’s PDF14: Providing running headers and footers in PDF documents, despite the fact that the document has footers throughout containing page numbers. (Incidentally, another document I tested had headers throughout except on its cover page. This also failed the Tingtun test.)

Bookmarks

Tingtun correctly reported that the document has no bookmarks. In this respect it performed better than Acrobat which overlooked this problem.

Alt text

Tingtun also correctly reported that the document has three images with no alt text.

Tables

Lastly, Tingtun reported eight tables with tagging problems, although unfortunately, unlike the Acrobat checker, it doesn’t specify what kind of problems and in which tables. There are in fact nine tables in the document, five of which are tagged correctly and four which aren’t.

Comment

Tingtun correctly identified the absence of alt text and bookmarks. However, its results regarding headers and footers should be taken with a pinch of salt. Requiring headers and footers for every document would be unnecessarily restrictive to say the least. What about, for example, a single page document? Also, its results regarding tables are, I’m afraid, unreliable and unhelpful. And lastly, as we will see, it failed to identify some significant problems that the Acrobat checker does pick up.

Test results for Acrobat Pro

For the same document, in contrast to Tingtun’s four categories of “errors”, Acrobat identified nine: title, tagged content, tagged annotations, tab order, forms, alt text, tables, lists and headings.

There were only two categories in which Tingtun and Acrobat told the same story:

• Both correctly reported three images with no alt text.
• Both failed to pick up an overall document language setting of “Arabic” on an English language PDF.

There were two categories in which Tingtun performed better than Acrobat:

• Unlike Tingtun, Acrobat failed to notice the absence of bookmarks in this document.
• There is a skipped heading in the test document. It was put there deliberately. Changing it would make the document less accessible not more so. Acrobat incorrectly flagged this perfectly legitimate skipped heading as a fail. Tingtun, correctly, didn’t.

In all other respects the Acrobat checker performed better than Tingtun. Acrobat correctly reported:

• that the document title was not being displayed
• a logo that wasn’t tagged, and a page number that should have been an artifact
• a link tagging problem that would cause a dead (or silent) item in the tab order
• that the tab order had not been set
• an untagged form field
• that one table had no headers. However, it failed to pick up the absence of ColSpan and RowSpan attributes in three others
• that a list was missing its parent <L> tag

“Page could not be downloaded” error

Next, I tried to upload a further nine documents to the Tingtun site. Four of them could not be uploaded at all, but instead produced the error message “string indices must be integers” (see below). This message will be unintelligible to virtually all potential users of this service, including myself.

All four of these documents, by the way, are fully accessible. They work fine with a range of assistive technologies and all pass the Acrobat checker.

None of the remaining five documents provided any further significant insights that were not already highlighted by my regular test document.

Conclusion

Having run a selection of documents through Tingtun, I would conclude that the Acrobat checker gives a much truer picture and is less buggy. For that reason, Acrobat is still by some margin my preferred checker, caveat, of course, the ever-present danger of over-reliance on automated PDF accessibility checkers.