Search Traffic Fell After a Redesign: Check Robots, Redirects and Indexing Signals

Establish whether search visibility fell or measurement changed

A traffic graph can drop after a redesign even when visitors still reach the site. Before changing robots.txt or requesting reindexing, compare analytics sessions with Search Console clicks and impressions for the same dates. A broken analytics tag, changed consent behavior or a different reporting filter can change the graph without the same change in search visibility.

If Search Console also shows a decline, separate affected pages, queries and devices. Compare equivalent weekdays and account for normal reporting delays. A redesign's date is a useful clue, not proof that every movement in search traffic came from its code.

Save a list of previously valuable URLs before investigating. If those URLs changed, you need a mapping from each important old page to its intended replacement. Without it, a technically valid redirect can still send the visitor somewhere irrelevant.

Check the exact page Google should index

Inspect a representative page using a GET request, then inspect its rendered HTML. A header-only request can follow a different application path and cannot show an HTML robots tag.

curl --silent --show-error --connect-timeout 3 --max-time 10 \
  --dump-header page-headers.txt --output page.html \
  https://example.com/products/widget

Run this from a diagnostic machine and substitute your real public URL. Review the HTTP status and any X-Robots-Tag header. In the saved document, check robots metadata and the canonical link; also check the rendered page if JavaScript modifies them. These files are diagnostic evidence, not an HTML parser or a search-engine emulator.

For an indexable page, an accidental noindex directive is a direct problem. It can come from the template, middleware, a web-server header or CDN configuration. Removing the template tag will not fix a noindex header still added elsewhere. Google's noindex documentation ↗ (opens in a new tab) explains both delivery forms.

Do not use robots.txt as an indexing switch

robots.txt controls crawling for cooperating crawlers. It does not provide authentication, and disallowing a URL is not a reliable way to remove it from search. A crawler blocked from fetching a page may also be unable to see the noindex directive you placed on it.

Read the deployed robots.txt at the production hostname, not just the repository template. Compare relevant rules before and after the release. A broad staging restriction carried into production can hide important routes from crawling. Equally, do not remove legitimate restrictions on private or unbounded application paths just to make a warning disappear.

Google documents the distinction in its robots.txt introduction ↗ (opens in a new tab). Keep staging protected by access control; robots.txt alone does not make its content private.

Verify redirects against a URL map

For a permanently moved page, inspect the initial response without automatically following it:

curl --silent --show-error --connect-timeout 3 --max-time 10 \
  --dump-header - --output /dev/null \
  https://example.com/old-product

Compare its status and Location with the planned destination. Then request the destination and check its content, indexability and canonical. When checking a chain, bound the number of redirects and preserve every hop rather than reading only the final 200.

Avoid sending every removed product to the homepage. The destination needs to satisfy the old URL's intent; when no appropriate replacement exists, a genuine not-found response may be more truthful. For URL changes, review Google's site migration guidance ↗ (opens in a new tab).

A local fixture verified a narrower case: a page exposed noindex in both a response header and HTML, while its corrected replacement exposed neither. A separate old route returned 301 to the intended replacement. This demonstrated inspecting those signals, not crawling by Google or a measured recovery in ranking.

Treat canonical and sitemap as consistency checks

A redesign can accidentally make every detail page canonicalize to the homepage or retain a staging hostname. Inspect several page types rather than assuming one correct homepage proves all templates are correct. Check alternate hostnames, trailing-slash behavior and parameterized URLs according to your intended canonical policy.

Review the generated sitemap for the canonical public URLs you want discovered. Look for staging hosts, redirecting URLs, removed pages or a suddenly tiny output after a query changed. Validate the actual XML and any sitemap-index children that matter to your site.

A sitemap helps discovery; its presence does not guarantee that every submitted page will be indexed. Google's sitemap overview ↗ (opens in a new tab) describes that role. Do not turn a successful XML fetch into a claim that Google has accepted every URL.

Use Search Console to confirm the crawler's view

After fixing a technical issue, inspect important URLs with Search Console's URL Inspection tools. Distinguish a live fetch from the stored indexing state: fixing today's response does not rewrite yesterday's crawl history.

Compare the user-declared canonical with Google's selected canonical where the tool provides it. Check whether important content renders, whether access is blocked and whether the reported crawl predates your fix. Request indexing where appropriate, but do not promise a fixed recovery date.

If technical signals look correct, investigate changed content, internal navigation and page intent. A redesign may remove useful copy or weaken links to valuable pages while every HTTP and XML check still passes. Technical monitoring narrows the investigation; it cannot diagnose every ranking change.

Turn the release into a repeatable SEO check

Maintain a small URL set spanning the homepage, category, detail and editorial templates. Before and after a release, compare status, redirect destination, robots directives, canonical and the important internal links. Include mobile rendering when navigation or content visibility changed.

DeployWatchers can monitor robots.txt, sitemap and redirect changes as part of that picture. Review whether a detected change is intentional, and verify Search Console after the fix. A monitoring alert can identify when a technical signal changed; it cannot guarantee that search traffic will immediately return.