Robots.txt: Legal Web Crawling for Stores

More and more online stores decide to auto-populate their catalog or monitor competitor prices through web crawling, yet few teams check, at the very first step, a small file that can directly influence the legality of the project: robots.txt.

Robots.txt is a public text file placed at the root of a website (e.g. example.com/robots.txt), through which the site owner communicates to software robots which pages can be accessed and which pages should not be visited; respecting it does not automatically guarantee that a crawling project is legal, but deliberately ignoring it significantly increases the risk that data collection will be considered unauthorized, especially if it also conflicts with the target site's Terms of Service.

This guide explains what a robots.txt file actually contains, what the User-agent, Disallow, Allow and Sitemap directives mean, why respecting it does not equal full legal compliance on its own, how it relates to GDPR when personal data is involved, and what a practical verification checklist looks like before launching a crawling project for an online store.

What Robots.txt Is and Where to Find It on a Website

Robots.txt is part of the Robots Exclusion Protocol, a technical convention first proposed in 1994 and formally standardized by the IETF as RFC 9309 in 2022. In practice, any website can publish this text file at the root of its domain (for example, https://example.com/robots.txt), publicly visible to anyone, including search engine robots and any crawling script.

The file's role is to tell crawlers visiting the site which sections can be freely browsed and which sections the owner would rather robots avoid — for example admin pages, internal search results, or duplicate content sections.

According to Google's official documentation for webmasters, robots.txt is not a security mechanism and cannot guarantee that a page stays completely invisible in search engines; for genuinely sensitive data, the correct protection is authentication or access restriction, not robots.txt.

Which Directives a Robots.txt File Contains and What Each One Means

A robots.txt file is made up of simple text blocks, each targeting one or more robots identified by their technical name (user-agent). The most common directives are:

User-agent — identifies the robot the following rules apply to (e.g. Googlebot, or * for all robots).
Disallow — indicates a path (URL or directory) that the given robot is asked not to access.
Allow — explicitly permits access to a path, usually used to make an exception inside an otherwise blocked directory.
Sitemap — points to the location of the site's sitemap.xml file, helping robots discover available pages more efficiently.

One important detail, confirmed by Google's own documentation: different crawlers may interpret the syntax of a robots.txt file slightly differently, and the instructions cannot technically force a robot's behavior — it ultimately remains up to whoever wrote the crawler whether the published rules are respected or not.

Does Respecting Robots.txt Automatically Mean Legal Crawling?

Not automatically. Robots.txt is a technical convention, not a law, and respecting it strengthens the good faith of a crawling project, but it does not replace checking the other applicable rules — the Terms of Service published on the site, copyright over the extracted content, and, where relevant, GDPR.

In practice, a crawling project is better protected legally the more of these signals it satisfies at the same time: robots.txt allows access to the targeted pages, the Terms of Service do not explicitly forbid automated data extraction, and the collected data does not include personal information without a clear legal basis.

Conversely, deliberately ignoring a restrictive robots.txt, combined with massively extracting copyright-protected content or explicitly violating the Terms of Service, increases the risk that the target site's owner will consider the access unauthorized and demand that the project stop or seek damages.

What Risks Appear If You Ignore Robots.txt or a Site's Terms

These risks are not just theoretical; they have direct consequences for a business project built on data collected from external sources.

Risk	What can happen	How to reduce it
IP or crawler blocking	The target site detects abnormal traffic and blocks access	Respect Disallow rules, use a reasonable crawl-delay
Cease-and-desist request from the site owner	Notice or complaint about violating the Terms of Service	Check the Terms of Service before launching the project
Exposure related to personal data	Unintentional collection of names, addresses or other personal data	Explicitly exclude personal-data fields from the crawling script
Copyright-related dispute	Reproducing original descriptions, images or text in full	Process and adapt the data, don't just copy it verbatim

These risks are indicative and depend on each project's context; for high-volume projects or sensitive sources, the recommendation is to seek specialized legal advice before starting collection.

How GDPR Relates to Data Collected Through a Crawling Project

The General Data Protection Regulation (GDPR) applies to any organization processing data of individuals in the European Union, regardless of where the company is based. The regulation defines personal data broadly, as any information that can identify a natural person, directly or indirectly — including names, email addresses or other identifiers.

For an online store, the practical implication is simple: the fact that a piece of information is public on a website does not automatically remove it from GDPR's scope, if that information qualifies as personal data. A crawling project focused strictly on product data (title, price, code, description, images) carries reduced GDPR risk; a project that would extract, even incidentally, names, email addresses or reviews signed with identifiable details raises a separate issue that needs to be addressed explicitly before collection.

The practical recommendation is that any crawling script built for an online store should explicitly exclude fields that could contain personal data and should collect strictly the product information the project needs.

Practical Checklist: How to Check Robots.txt Before a Crawling Project

Before building a crawling script for any source site, the checklist below should be completed in full:

Open site-name.com/robots.txt directly and read the rules applicable to the * user-agent and, if relevant, to the specific user-agent your script uses.
Identify all Disallow directives relevant to the pages you want to access and exclude them from the crawling plan.
Check whether a Sitemap directive exists; a valid sitemap helps with more efficient, less aggressive crawling of the target server.
Read the target site's Terms of Service, especially sections about automated use, data extraction or reproducing content.
Set a reasonable access frequency (crawl-delay), so the source server is not overloaded.
Exclude any field that could contain personal data from the collection, if it is not needed for the project's purpose.
Document the decision internally (which site, which data, which rules were checked), for traceability in case of a later request.
For high-volume projects or sensitive sources, request a legal review before launching collection at scale.

In-House Crawling vs. an External Service: Who Owns the Legal Check

Many in-house development teams can technically build a crawling script, but they do not always have the time or the expertise to systematically check robots.txt, the Terms of Service and the GDPR implications for every new source. A custom web crawling service treats this verification as a standard part of the project, not an optional step added later.

The practical difference is that a specialized provider builds the crawling script with robots.txt and the risks discussed above already accounted for during the planning phase, while an in-house solution built quickly, without prior checks, risks being blocked or challenged after the project is already in production.

Frequently Asked Questions About Robots.txt and Legal Web Crawling

Is it illegal to ignore robots.txt?

Ignoring robots.txt is not, by itself, automatically illegal, because the file has no legal standing — it is a technical convention. The real risk appears when ignoring robots.txt is combined with violating the Terms of Service, massively extracting copyright-protected content, or collecting personal data without a legal basis.

What happens if a crawler doesn't respect robots.txt?

Technically, nothing automatically blocks a crawler that ignores robots.txt — the instructions cannot be enforced. The target site, however, can detect unusual traffic and block the relevant IP or user-agent, and the owner can later request that the collection stop.

Is robots.txt enough for GDPR compliance?

No. Robots.txt only controls which pages robots can access, not what happens to the data collected. GDPR compliance depends on the type of data extracted, the legal basis for processing it, and how that data is subsequently stored and used.

How do I quickly check a site's robots.txt?

Add /robots.txt directly to the address bar after the target site's domain (for example example.com/robots.txt). The file is public and opens directly in the browser, as a plain text document.

Who bears the legal risk in a crawling project: the client or the provider?

It depends on the agreed service contract; that is why it matters that, from the proposal stage, checking robots.txt and the target site's Terms of Service is clearly stated as a project step, with each party's responsibilities made explicit.

Conclusion: Robots.txt Is the First Signal, Not the Only Legal Check

Robots.txt quickly tells you which pages a site wants robots to access, but a truly legally safe crawling project also checks the target site's Terms of Service and the GDPR implications of the collected data. Ignoring these signals does not technically stop a crawler, but it increases the risk of blocking, disputes or, in sensitive cases, litigation.

See the full web crawling service, built with legal verification included in every project, or discuss your project directly with the HappyWeb team, for an evaluation that also considers robots.txt and the legal risks specific to your data source.

Sources

Google Search Central — Introduction to robots.txt: developers.google.com/search/docs/crawling-indexing/robots/intro.
IETF RFC 9309 — Robots Exclusion Protocol: rfc-editor.org/rfc/rfc9309.
GDPR.eu — What is GDPR: gdpr.eu/what-is-gdpr.
HappyWeb web crawling service: happyweb.ro/en/services/services-web-crawling.

Note: the legal information in this article is for general guidance only and does not constitute legal advice. For crawling projects involving large data volumes or sensitive sources, we recommend checking the target site's Terms of Service directly and, where relevant, consulting a legal specialist.

Image generated with AI, used for illustrative purposes.

What Is Robots.txt and How It Affects Legal Web Crawling for an Online Store