A robots.txt file provides crawling instructions for a website. Google explains that it manages crawler access but does not reliably prevent an address appearing in search. A disallowed address can still be discovered through links elsewhere. Source: Google: introduction to robots.txt

Use the right control for the purpose

Before requesting a change, say whether the aim is to manage crawling, prevent indexing or protect private information. These are different requirements. Google points to controls such as noindex for indexing and password protection for restricted access. Source: Google: introduction to robots.txt

Check representative travel pages

Ask the website team to test an important guide, a glossary page, a commercial landing page and a filtered search address against the proposed rules. This gives the review a useful spread of page types instead of assuming one rule affects only the intended section.

Keep the previous rules and the approved change together. After publication, recheck the selected addresses and record the result. If an important page becomes inaccessible to the intended crawler, the team then has a clear change record to investigate.

Written and updated by Travelgenix. Sources checked on 16 October 2026. Worked examples and checklists are editorial illustrations and recommendations, not customer results.