English
Romana
Picture this: you want to auto-populate your catalog with product data pulled from other websites, but among the results you also find reviews signed with a full name and email address. At that point, your crawling project is no longer just about titles and prices — it has stepped into GDPR territory.
GDPR applies to any collection of personal data, regardless of whether that data is publicly visible on the internet or not; the fact that a piece of information can be seen by anyone on a website does not automatically strip it of its status as personal data, and it does not, by itself, give you the right to extract and store it.
This guide explains what personal data means in the context of a crawling project, what legal basis you need to process it, which data categories are usually safe to collect, how to reduce risk through anonymization and filtering, and when a large-scale collection project requires a formal impact assessment (DPIA) before it starts.
What counts as personal data under GDPR
The General Data Protection Regulation defines personal data as any information relating to an identified or identifiable natural person — directly (name, national ID number, personal email address) or indirectly, through a combination of elements that together allow a person to be identified (for example, a combination of city, occupation, and date of birth).
In practice, a crawling project can encounter personal data in unexpected places: customer reviews signed with a real name, blog comments, public user profiles, personal email addresses listed in contact sections, or photos that include identifiable faces.
Company names, product codes, prices, technical descriptions, or catalog images are generally not personal data, because they don't relate to an identifiable natural person — this is why most eCommerce-oriented crawling projects carry a low GDPR risk, as long as they stay strictly within this type of data.
Public data online vs data you can legally collect: not the same thing
A common misconception is that "if it's public online, I can copy it." GDPR doesn't work that way: a personal data item being published by a person or a third party does not automatically constitute consent for someone else to collect, store, and process it for a different purpose.
The practical difference is between viewing information and processing it systematically. A human visitor who reads a review on a site is not "processing" that data in the GDPR sense; a crawling script that automatically extracts thousands of reviews with names and stores them in its own database, for a purpose different from the original one, does exactly that.
That's why the right question for a crawling project isn't "is it public?", but "do I have a legal basis to collect and store this data for my specific purpose?"
What legal basis you can rely on for a crawling project
GDPR requires an explicit legal basis for any processing of personal data. For business-oriented crawling projects (price monitoring, catalog population, market research), the relevant legal bases are usually:
- Legitimate interest — the most commonly invoked basis for collecting product or market data, but it requires a balancing test between your business interest and the impact on the data subject; this basis isn't automatically valid if the collected data is sensitive or the volume is large.
- Consent — rarely applicable in crawling, because the data subject usually doesn't know their data is being collected by a third party through an automated script.
- Contract performance — relevant only if the data subject already has a direct contractual relationship with the company collecting the data, which is rarely the case in an external crawling project.
For most crawling projects, the practical recommendation is to avoid collecting personal data altogether, unless there's a clear business reason and a documented legal basis before the project starts — it's simpler and safer to filter out personal data at the source than to build a solid legal basis for every category of data collected.
Data categories: what's usually safe vs what needs attention
The table below summarizes, as a general guide, the GDPR risk level for data categories commonly encountered in eCommerce and market research crawling projects.
| Data category | GDPR risk | Practical recommendation |
|---|---|---|
| Title, price, product code, technical description | Low | Standard collection, no additional measures |
| Catalog images (product, packaging) | Low | Standard collection, just check copyright |
| Reviews with full name or identifiable username | Medium-high | Exclude the author field or anonymize before storage |
| Publicly listed email addresses or phone numbers | High | Explicitly exclude from the crawling script |
| Photos with identifiable people | High | Exclude or anonymize (blur); needs separate analysis |
This classification is a general guide; the actual risk depends on the specific purpose of the project, the data volume, and how the data is used after collection.
How to reduce GDPR risk directly in the crawling script
The most effective protection against GDPR issues is to not collect personal data at all, rather than collecting it and protecting it afterward. A few practical steps, applied directly at the script-building stage:
- Define explicitly, before writing the script, exactly which data fields are necessary for the project's purpose (for example, only title, price, code, image).
- Exclude from the extraction rules any field that could contain names, emails, phone numbers, or other personal identifiers, even if they appear accidentally on the source page.
- If you extract reviews or comments for sentiment analysis, remove or anonymize the author before storage, not after.
- Don't store IP addresses, cookies, or other technical identifiers of the source site's visitors — these are usually not relevant to the purpose of a product data project.
- Document internally the project's purpose, the categories of data collected, and the legal basis used, before launching collection at scale.
- Set a clear retention period for the collected data and a deletion process once the purpose has expired.
When you need a Data Protection Impact Assessment (DPIA) for a crawling project
For most crawling projects focused on product data, a formal Data Protection Impact Assessment (DPIA) isn't required, because the risk to data subjects is low. GDPR recommends a DPIA when processing involves a high risk to the rights and freedoms of data subjects — for example, large-scale systematic monitoring of individuals or processing special categories of data.
Signals that should trigger a discussion about a DPIA, not necessarily a firm conclusion that it's mandatory:
- The project collects, even incidentally, a large volume of identifiable person profiles (not just products).
- The collected data could be combined with other data sets to build detailed profiles of individuals.
- The source includes special categories of data (health, political opinions, biometric data).
For these scenarios, the recommendation is to consult a Data Protection Officer (DPO) or a legal specialist before designing the collection script, not after the data has already been extracted.
GDPR vs robots.txt: two different checks, both necessary
Robots.txt and GDPR answer different questions and can't substitute for each other. Robots.txt tells you which pages a site technically allows robots to access; GDPR tells you what you can legally do with personal data found on those pages, regardless of whether technical access was permitted or not.
A properly built crawling project checks both: it respects the directives in robots.txt (covered in detail in HappyWeb's guide on robots.txt and legal crawling) and, separately, filters or documents the legal basis for any personal data encountered during collection.
Frequently asked questions about GDPR and crawling
Is it legal to collect product data (price, title, code) through crawling?
Yes, generally. Product data usually isn't personal data, so it doesn't directly fall under GDPR. Other checks still matter, such as robots.txt and the target site's Terms and Conditions.
Can I collect customer reviews if they're public on the site?
You can collect the review content, but you need to handle the author's name or other personal identifiers carefully — the safest approach is to remove or anonymize the author before storage, even if the review is publicly visible.
What happens if my crawling script accidentally collects personal data?
The recommendation is to delete that data from the collected set immediately and adjust the script's rules to exclude those fields going forward; document the incident internally, especially for large-volume projects.
Does GDPR apply even if my company isn't based in the EU?
Yes, if the processing involves data of individuals located in the European Union, GDPR can apply regardless of where the company collecting the data is established.
Do I need consent for any personal data collected through crawling?
Not necessarily explicit consent, but you need a valid legal basis, documented before collection. For most business projects, the safest path is to avoid collecting personal data altogether, rather than relying on a legal basis that's hard to justify later.
Conclusion: filter at the source, don't fix it after collection
A GDPR-compliant crawling project doesn't mean avoiding all data collection — it means explicitly excluding personal data that isn't necessary for your business purpose. Clearly defining the required fields, filtering them directly in the collection script, and documenting the legal basis turns a risky project into one that can be defended, if it's ever checked.
Want a crawling project built with GDPR filtering included from the planning stage? Discuss your project with the HappyWeb team or see the full web crawling service.
Sources
- GDPR.eu — What is GDPR: gdpr.eu/what-is-gdpr.
- GDPR — Article 4, definition of personal data: gdpr-info.eu/art-4-gdpr.
- GDPR — Article 6, legal bases for processing: gdpr-info.eu/art-6-gdpr.
- HappyWeb's web crawling service: happyweb.ro/en/services/services-web-crawling.
Note: the legal information in this article is for general guidance only and does not constitute legal advice. For large-volume or sensitive crawling projects, we recommend consulting a Data Protection Officer (DPO) or a legal specialist before launching collection.
Image generated with AI, used for illustrative purposes.
Write a comment