Crawlability of regulatory sources
Each tracked source is requested twice — once with a desktop browser user-agent and once with an AI-agent user-agent — and the responses are recorded. 10 of 23 return the document to a standard HTTP client.
This records observed HTTP responses. It expresses no view about any publisher's choices and draws no conclusion about why a source responds as it does.
Answer a browser, refuse an AI agent (2)
- FDA AI guidance — browser 200, agent 404
- FDA ETP — browser 200, agent 404
Answer an AI agent, refuse a browser (2)
- GAMP AI Guide — browser 403, agent 200
- GAMP 5 2nd Ed — browser 403, agent 200
Full dataset
| Instrument | Authority | Browser | AI agent | Crawlable | Render | robots.txt |
|---|---|---|---|---|---|---|
| EMA QIG | European Medicines Agency | 200 | 200 | yes | server | no AI-specific disallow |
| EU AI Act | European Parliament and Council | 202 | 202 | partial | client | no AI-specific disallow |
| EU GMP Annex 11 | European Commission | 200 | 200 | yes | server | no AI-specific disallow |
| EU GMP Annex 22 | European Commission | 200 | 200 | yes | server | no AI-specific disallow |
| FDA AI guidance | US Food and Drug Administration | 200 | 404 | yes | server | no AI-specific disallow |
| FDA CSA | US Food and Drug Administration | 404 | 404 | no | unknown | no AI-specific disallow |
| FDA ETP | US Food and Drug Administration | 200 | 404 | yes | server | no AI-specific disallow |
| FDA guidance agenda | US Food and Drug Administration | 404 | 404 | no | unknown | no AI-specific disallow |
| ICH Q12 | International Council for Harmonisation | 200 | 200 | partial | client | no AI-specific disallow |
| ICH Q13 | International Council for Harmonisation | 200 | 200 | partial | client | no AI-specific disallow |
| ICH Q14 | International Council for Harmonisation | 200 | 200 | partial | client | no AI-specific disallow |
| ICH Q2(R2) | International Council for Harmonisation | 200 | 200 | partial | client | no AI-specific disallow |
| ICH Q9(R1) | International Council for Harmonisation | 200 | 200 | partial | client | no AI-specific disallow |
| ISO/IEC 42001 | ISO/IEC | 403 | 403 | no | unknown | disallows GPTBot |
| GAMP AI Guide | ISPE | 403 | 200 | yes | server | robots.txt unreachable |
| GAMP 5 2nd Ed | ISPE | 403 | 200 | yes | server | robots.txt unreachable |
| NIST AI RMF | US National Institute of Standards and Technology | 200 | 200 | yes | server | no AI-specific disallow |
| PIC/S GMP Guide | Pharmaceutical Inspection Co-operation Scheme | 200 | 200 | yes | server | no AI-specific disallow |
| prEN 18229-1 | CEN/CENELEC JTC 21 | 500 | 500 | no | unknown | no AI-specific disallow |
| prEN 18229-2 | CEN/CENELEC JTC 21 | 500 | 500 | no | unknown | no AI-specific disallow |
| prEN 18281 | CEN/CENELEC JTC 21 | 500 | 500 | no | unknown | no AI-specific disallow |
| prEN 18282 | CEN/CENELEC JTC 21 | 500 | 500 | no | unknown | no AI-specific disallow |
| 21 CFR Part 11 | US Food and Drug Administration | 200 | 200 | yes | server | no AI-specific disallow |
Machine-readable at /api/crawlability.json.
Method: a source counts as yes when at least one of the two probes returned
HTTP 200 with 600 or more characters of extractable text.
Last updated: