A Toowoomba accounting firm once rang around asking why their brand-new website had disappeared from Google two weeks after launch — not a ranking drop, a total vanishing act. The culprit was one line left over from the build: Disallow: / in their robots.txt file, quietly telling every search engine to stay out entirely. Nobody had touched it since launch day. Robots.txt is one of the smallest files on your website and one of the easiest to get wrong, so it's worth understanding properly.
What Is a Robots.txt File, Really?
Robots.txt is a small text file that sits at the root of your website and gives instructions to search engine crawlers — the automated programs (like Googlebot) that visit your pages to work out what's there. It's part of the Robots Exclusion Protocol, a long-standing web standard that Google, Bing and other search engines respect.
Think of it as a note pinned to your front gate for delivery drivers: "use this entrance, skip that one." It doesn't lock any doors — it's a set of polite instructions that well-behaved crawlers follow. Google Search Central, Google's own documentation hub for how search works, is the authoritative source on how its crawlers interpret the file.
What Robots.txt Can (and Can't) Do for Your Website
This is where most confusion — and most damage — happens. Robots.txt controls crawling (whether a bot fetches a page), not indexing (whether that page can appear in search results). Those are two different things, and mixing them up is the single biggest misunderstanding small business owners have about it.
| What it can do | What it can't do |
|---|---|
| Tell crawlers which folders or pages not to bother fetching | Guarantee a page stays out of Google's index — a blocked page can still be indexed if other sites link to it |
| Reduce unnecessary crawling of admin areas, search results pages or duplicate content | Remove a page that's already indexed (use a noindex tag or Google Search Console's removal tool for that) |
| Point crawlers to your XML sitemap | Stop a human from typing in the URL and viewing the page directly |
| Save "crawl budget" on very large sites | Act as a security measure — it's a public file anyone can read |
If you genuinely need a page kept private, robots.txt is the wrong tool. Use a noindex meta tag, password protection, or removal through Google Search Console instead. And because the file is publicly viewable, never list sensitive folder names in it — you'd be handing out a map of exactly what you're trying to hide.
Where to Find Your Site's Robots.txt File
Every robots.txt file lives in the same predictable spot: yoursite.com.au/robots.txt. Type that into a browser right now and you'll see your live file (or a message saying one doesn't exist, which is also worth knowing).
There's only ever one robots.txt file per domain, and it must sit at the root — a copy tucked inside a subfolder won't be read. If your business runs on a subdomain (like shop.yoursite.com.au), that subdomain needs its own robots.txt too.
The Handful of Directives You Need to Know
The good news is robots.txt only uses a handful of commands, and you don't need to be technical to read one.
| Directive | Plain-English meaning |
|---|---|
User-agent | Which crawler these rules apply to (* means "all crawlers") |
Disallow | A path you're asking crawlers not to fetch |
Allow | An exception that overrides a broader Disallow rule |
Sitemap | The web address of your XML sitemap, so crawlers can find your important pages easily |
Rules are read top to bottom for whichever User-agent block applies. A single stray slash after Disallow: is the difference between blocking one folder and blocking your entire website — which is exactly what happened to that Toowoomba firm.
Common Robots.txt Mistakes That Cost Australian Businesses Traffic
Blocking the whole site by accident
Disallow: / blocks everything from the domain down. It's standard practice on a staging or development site so search engines don't index unfinished work — the trouble starts when that same file gets copied straight to the live site at launch and nobody remembers to remove it. This is exactly what happened in Toowoomba: the developer's staging robots.txt made the trip to production, and the firm's rankings, enquiries and Google Business Profile visibility all quietly cratered before anyone noticed. If your traffic drops off a cliff right after a redesign, this file is the first thing to check.
Blocking CSS and JavaScript
Older advice suggested blocking scripts and stylesheets to "save crawl budget." Google now needs to render your pages properly to understand them, and blocking the CSS or JS that makes the page look and function correctly can make it appear broken or low quality in Google's eyes — directly undermining your technical SEO checklist.
Forgetting the sitemap line entirely
A missing Sitemap: line isn't fatal, but it's an easy win to leave on the table. Pairing your robots.txt with a well-structured XML sitemap makes it far easier for crawlers to find and prioritise your key pages.
Assuming it protects private information
Because the file is public, listing /staff-portal/ or /invoices/ in a Disallow rule is effectively advertising the folder exists. Genuinely sensitive areas need proper authentication, not a polite request in a text file.
A Simple, Safe Robots.txt Example
Most small business websites need something close to this — allow everything important, keep crawlers out of the WordPress admin area (while still letting the file that makes forms and dynamic features work through), and point to the sitemap:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com.au/sitemap.xml
That's genuinely all most sites need. Resist the urge to add extra rules "just in case" — every unnecessary Disallow line is a chance to accidentally block something you actually want found.
Editing Robots.txt in WordPress With Yoast or Rank Math
You don't need FTP access or a developer to change your robots.txt file if you're running WordPress. Both major SEO plugins give you a built-in editor:
- Yoast SEO: under Yoast SEO → Tools → File Editor, which will show your current robots.txt and let you edit it directly in the dashboard.
- Rank Math: under Rank Math → General Settings → Edit robots.txt, with the same kind of direct text editor.
If you don't already have one of these installed, it's usually bundled into a broader website audit to check what else might need attention. After any change, use Google Search Console's URL Inspection tool to confirm Google can still fetch the pages you care about, and keep an eye on the Pages report for anything newly marked "blocked by robots.txt."
Key Takeaways
- Robots.txt controls crawling, not indexing — it's not a privacy or security tool.
- It always lives at
yoursite.com.au/robots.txt, one file per domain. Disallow: /blocks your entire site — check for it immediately after any launch or redesign.- Blocking CSS or JavaScript can hurt how Google renders and understands your pages.
- Yoast and Rank Math both let you edit robots.txt directly from the WordPress dashboard.
- Always include a
Sitemap:line pointing to your XML sitemap.
Frequently Asked Questions
Does robots.txt keep my site out of Google search results?
Not reliably. Robots.txt asks crawlers not to fetch a page, but if other sites link to that page, Google can still index the URL without ever visiting it. To reliably keep a page out of search results, use a noindex meta tag or request removal through Google Search Console.
What happens if I don't have a robots.txt file at all?
Nothing dramatic — search engines will simply crawl your site as normal, as if every page were allowed. A robots.txt file is a helpful tool for guiding crawlers efficiently, not a requirement for your site to function or rank.
Can robots.txt speed up my site?
Not directly. It can make crawling more efficient on very large sites by steering bots away from low-value pages like internal search results, but it has no effect on how fast your pages actually load for visitors. For real speed gains, look at hosting, images and code — see our guide on how fast a website should load.
Should I block WordPress admin pages in robots.txt?
Blocking /wp-admin/ is common and sensible, since there's nothing there for search engines to index. Just remember to leave admin-ajax.php allowed, since some WordPress themes and plugins rely on it to load content correctly on the front end.
How do I check if my robots.txt is blocking my whole site?
Visit yoursite.com.au/robots.txt in a browser and read it — if you see Disallow: / under User-agent: *, your entire site is blocked. Google Search Console will also flag a sudden spike in pages "blocked by robots.txt," which is often the first warning sign business owners notice.
Do I need a different robots.txt for staging and live sites?
Yes. A staging or development copy of your site should block all crawling so it never gets indexed, but that file must never be copied across when you go live. Making a robots.txt check part of your launch process avoids the exact mistake that cost the Toowoomba firm two weeks of invisibility.
Does robots.txt affect Google Ads or paid traffic?
No. Robots.txt only governs organic crawling by search engines; it has no bearing on Google Ads campaigns, which send visitors directly to your landing pages regardless of crawl rules. It's worth understanding both — our SEO vs Google Ads comparison explains how the two channels fit together.
Get Your Technical Foundations Sorted
Robots.txt is a small file, but it's exactly the kind of detail that quietly makes or breaks your visibility on Google — alongside sitemaps, canonical tags and the rest of your site's technical health. If you're not sure what your current file is doing, or you want a proper check across your common SEO mistakes and technical setup, have a chat with Pixel and Pine. We'll make sure the things Google can and can't see on your site are exactly what you intended.
