For most of the last twenty years, hosting decisions were judged on three things: speed, cost and uptime. If the site was fast, affordable and online, the infrastructure team had done its job.
That is no longer enough. The same hosting and CDN settings now decide whether search engines and AI assistants can read a website at all. A firewall default, a rendering choice and a cache rule can quietly remove a business from AI answers, and nobody in marketing will be able to see why. The traffic simply stops arriving, and the dashboards show nothing wrong.
This is a problem that sits between teams. Infrastructure owns the settings, marketing owns the visibility, and neither is looking at the gap in the middle.
The crawlers have changed
Search used to mean one visitor that mattered: Googlebot. Today a business website is read by a growing list of automated visitors. Alongside Googlebot and Bingbot there are the crawlers behind AI assistants, such as OpenAI's OAI-SearchBot and GPTBot, Anthropic's ClaudeBot and PerplexityBot. OpenAI documents its crawlers and what each one does on its bots page, and the other providers publish similar information.
These crawlers do not all behave the same way, and that is where infrastructure starts to matter. A setting that works perfectly for Googlebot can leave an AI crawler with an empty page, an error or a challenge screen. When an assistant answers a buyer's question and cites its sources, a site that could not be fetched cannot be cited.
Three settings behind most problems
In the website audits we run, most visibility problems that start in the infrastructure trace back to three settings.
1. Bot management defaults and firewall challenges
Bot management has become a standard feature of CDNs and web application firewalls, and for good reason: scrapers and abusive bots are real costs. Many platforms now offer one-click blocking of AI crawlers, and in July 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay for content.
There is nothing wrong with blocking crawlers deliberately. The problem is blocking them by accident. A security team switches on a bot rule, a JavaScript challenge or a rate limit, and the crawlers that feed search and AI answers start receiving a challenge page instead of content. Nobody decided that the business should disappear from AI assistants. It happened as a side effect.
The fix is to make the decision explicit. Separate the crawlers you want, such as search engines and the crawlers that fetch pages to answer users' questions, from the ones you may not want, such as crawlers that only collect training data. Then write the rules to match, and review them whenever the security policy changes.
2. Pages that only render through JavaScript
Google renders JavaScript, and its guide to JavaScript SEO basics explains how Googlebot processes it. Many AI crawlers do not. An analysis by Vercel and MERJ found that none of the major AI crawlers currently render JavaScript, including OpenAI's, Anthropic's and Perplexity's. Some of them download JavaScript files, but they do not run them.
So a site built as a single-page application, where the browser assembles the content after the page loads, can look complete to Google and almost empty to an AI crawler. We have seen this on a software company's blog, where the articles only appeared after JavaScript ran. To a crawler that reads the raw HTML, the blog was a page with a header and nothing else.
This is a hosting and architecture decision as much as a development one. Server-side rendering, static generation or pre-rendering for key pages all solve it. The right choice depends on the stack, but the question needs to be asked before the next rebuild, not after.
3. Cache rules that serve the wrong page to crawlers
Caching is the third, and the hardest to spot. A CDN cache can keep serving an error page that was generated during a deployment, a maintenance page, or a challenge page that was shown to one bot and then cached for everyone. Cache rules that vary by cookie or device can also mean crawlers receive a different version of a page than people do.
Long cache lifetimes on files such as robots.txt and the XML sitemap can delay important changes for days. If a robots.txt mistake is fixed at the origin but the old file stays cached at the edge, crawlers keep obeying the mistake.
A test any team can run in 30 minutes
You do not need special tools to find out where you stand.
Read the raw HTML. Fetch a key page with a command-line tool such as curl and look at the source, not the rendered page. If the main content, headings and links are not in that response, crawlers that skip JavaScript cannot see them.
Check the server or CDN logs. Filter requests by crawler user agents and look at the response codes. A healthy pattern is mostly 200 responses. Many 403, 429 or 503 responses, or challenge pages, mean a rule is getting in the way.
Verify that the visitors are real. User agents are easy to fake, so confirm important crawlers before acting on what the logs show. Google explains how to verify Googlebot with a reverse DNS lookup, and OpenAI publishes the IP ranges its crawlers use.
Look at what the cache is serving. Check the response headers for cache status and age on the pages that matter, and confirm robots.txt and the sitemap are current at the edge, not just at the origin.
Five questions to ask your hosting or CDN provider
Which bots does our firewall block or challenge by default, and where can we see those events?
Can we allow verified search and AI answer crawlers while still blocking abusive scrapers?
Do crawlers receive the same HTML that people see, or a version that depends on JavaScript?
How do our cache rules treat error pages, and how quickly is the cache purged after a deployment?
Can we get raw access logs, so we can analyse crawler traffic ourselves?
A provider that can answer all five clearly is one that takes visibility seriously.
Visibility is now an infrastructure metric
The practical step is to give visibility an owner on both sides. Infrastructure should know which crawlers matter to the business, and marketing should know which settings can switch them off. A quarterly review of bot rules, rendering and cache behaviour takes an afternoon and can prevent months of lost visibility.
After any change, measure whether it made a difference. Search Console and Bing Webmaster Tools will show changes in crawling and indexing, and a fixed set of buyer questions asked in AI assistants each month will show whether you are being named and cited. We have published a practical method for measuring AI search visibility that teams can use as a starting point.
Speed, cost and uptime still matter. But in a world where buyers ask assistants before they visit websites, the most expensive hosting setting is the one that makes a business invisible without anyone noticing.