At a glance
No single public crawl can prove it found every PDF on a website. Combine reachable links and sitemaps with known download URLs and a CMS or file-storage export from the website owner. Reconcile those lists using source URLs and retrieved file identity, then report the scope you actually checked.
A CMS export can reveal documents a public crawler never had an opportunity to see.
| Source | Example evidence | Follow-up |
|---|---|---|
| Public links | 90 PDF download addresses | Check retrieval and revisions |
| Sitemap | 12 additional addresses | Verify that they serve PDFs |
| CMS export | 5 unlinked assets | Ask whether they should remain public |
| JavaScript portal | Documents behind a search form | Obtain an authorized source list |
Scroll sideways to see all columns. Use arrow keys when the table is focused.
Start with four sources of evidence
A linked PDF is discoverable only if the crawler can reach a page that exposes its address. Navigation depth, host restrictions and dynamic page generation affect that path. A search-engine query can give useful clues, but its results are not a complete inventory or proof of current availability.
Sitemaps can expose URLs absent from navigation. They can also be outdated, incomplete or omit asset downloads entirely. The owner’s CMS export gives another perspective: it may include drafts, duplicates and private assets that should never enter a public scan.
- Crawl the intended public host and record its boundary.
- Read its published sitemaps and sitemap indexes.
- Add known public download URLs for important services.
- Ask the owner to reconcile with a CMS or storage export.
A PDF does not always end in .pdf
Some download endpoints use an identifier in a query parameter or a path such as /download/123. Keep document-specific parameters when they distinguish actual files. A filename-only search can miss these downloads; a link that ends in .pdf can instead return an HTML login or error page.
Separate candidate links from verified PDF retrievals. Record redirect destinations and failed outcomes without keeping credentials or temporary access tokens in a shared log. If a redirect goes to another host, confirm that host belongs in the agreed scope.
- Do not strip every query string as if it were tracking.
- Check what anonymous retrieval returns, not just the link label.
- Keep failures in the reconciliation; removing them makes the list look falsely complete.
Publish a count with a coverage statement
Compare the lists by URL first, then group confirmed identical bytes. Ask owners about unmatched CMS items: are they current public documents, drafts or obsolete assets? Record the decision rather than assuming every stored PDF is a public requirement.
A useful final statement names the date, hosts, discovery sources, confirmed totals and unresolved areas. For example: 102 public PDF links verified; five owner-listed assets not publicly linked; a search portal excluded pending an export. That is more informative than claiming all PDFs were found from one successful crawl.
Sources and scope
Methods, worksheets and illustrative examples are DocAccessible editorial contributions. These references support the technical or product details described above. W3C techniques are informative guidance; they are not a certification scheme.
- Sitemaps protocol
URL and sitemap-index formats used in public discovery.
- RFC 9309: Robots Exclusion Protocol
How crawl rules work; robots rules are not access authorization.
- DocAccessible PDF inventory
The counting workflow, report options and purchase boundary.
Written by the DocAccessible team under our editorial policy. Suggest a correction if a source or product detail needs updating.
See the evidence in a working example
Explore the public sample inventory report to try its table, full-screen view and missing-count reasons. The downloadable counting fixture check records what the local counter returned for seven self-authored inputs, with hashes and explicit limits.