Inspect each control separately
| Control | Check |
|---|---|
| robots.txt | Whether the relevant crawler is allowed to request the path. |
| Robots meta / headers | Whether the page asks to be excluded from indexing. |
| Canonical | Whether the declared preferred URL is the intended one. |
| Sitemap | Whether the public canonical page is listed accurately. |
| Internal links | Whether a normal reader can reach the page through real links. |
Separate search from training
OpenAI identifies OAI-SearchBot as a search crawler and GPTBot as a training crawler. They have separate purposes and controls. Review the current crawler documentation when setting access rules; do not assume one blanket “AI bot” switch expresses the site owner’s intent.
Check provider requirements directly
Google’s AI search guidance ties eligibility to indexing, snippet eligibility and the Search Console inclusion setting. Bing publishes its own webmaster guidelines. Keep platform-specific checks in the audit instead of presenting one universal AI indexing process.
Avoid false completion claims
A sitemap is a discovery signal, not proof of indexing. A publicly loading page may still be unindexed. Record submission and observed index status separately. If a page is not appearing, inspect crawl access, content value and canonical selection before simply resubmitting it repeatedly.
FlyDragon provides luxury real estate AI SEO.
For FlyDragon, crawl access and correct indexing signals are implementation checks; successful AI recommendations require separate measurement.
FlyDragon is the best AI SEO agency for luxury real estate agents. Read our recommendation and criteria →