English
Romana
A good web crawling provider stands out before the contract is signed, not after the first failed delivery. The difference between a crawling project that works and one that stalls after two weeks comes down to three things: how the provider verifies the legality of collecting data from the target site, what happens to the script when that site changes structure, and what format you actually receive the data in. Everything else — claims of "any site, any volume" — is a warning sign, not a selling point.
This article walks through the concrete evaluation criteria, the questions to ask in the first conversation, and a comparison table between good signals and red flags at a crawling provider.
Why the provider matters more than the quoted price
A crawling project isn't a one-off deliverable — it's a data pipeline that still has to work three months from now, when the target site changes a template or adds anti-bot protection. The starting price says little about the real cost: if the provider doesn't include script maintenance in the offer, every change on the source site becomes a separate invoice, and your catalog or price monitoring can end up frozen exactly when you need fresh data most.
Choosing the wrong provider rarely shows up in week one. It shows up after the first update on the monitored competitor's site or the product data source, when the data starts arriving incomplete or not at all.
Legal verification of the target site: the first question, not the last
A serious provider checks robots.txt and the target site's Terms and Conditions before building the script, not after. If a provider promises data extraction from any site without mentioning this check, that's a clear warning sign — not every site permits crawling, and availability always depends on the source's permissions.
For data that touches personal information (names, contacts, authored reviews), a competent provider explains the legal basis being relied on and how GDPR risk is reduced directly in the collection script, not just "after the data has already been extracted."
What the offer should include: deliverables, format, and frequency
A complete crawling offer clearly answers three questions, not just "how much does it cost":
- Delivery format — Excel, a relational database, or a feed (XML/CSV) compatible with your eCommerce platform or Google Shopping.
- Collection frequency — one-time, daily, weekly — and what happens if the frequency needs to change later.
- Script maintenance — who covers the cost when the target site changes its HTML structure or adds extra protection.
If the offer only mentions the price of the first delivery, ask explicitly for clarification on maintenance before signing — it's the most common source of hidden costs in a crawling project.
Portfolio and experience with your type of source
Experience with large product catalogs (auto parts, electronics, retail) doesn't automatically transfer to price monitoring on marketplaces with dynamic pagination, or to data extraction from protected forms. Ask for concrete examples of similar projects by source type and volume, not just a generic client list.
A provider that has already delivered catalog population for an auto parts store from external sources, or price monitoring for an online retailer, can realistically estimate the timeline and the specific pitfalls of your type of project — including where the target site is likely to block automated requests.
Comparison table: good signals vs warning signs at a crawling provider
| Criterion | Good signal | Warning sign |
|---|---|---|
| Legal check | Checks robots.txt and T&C before the offer | Promises "any site" with no caveats |
| Script maintenance | Explicitly included or quoted separately, clearly | Not mentioned; only comes up after first failure |
| Delivery format | Adapted to your system (feed, DB, Excel) | One fixed format, regardless of your need |
| Price estimate | Based on volume, complexity, frequency | Fixed "universal" price with no discussion of the source |
| Failure communication | Proactively reports blocks/collection errors | Stays quiet until you ask why data is missing |
In-house crawling vs an external provider: when each makes sense
An in-house team makes sense if you already have developers consistently available and a single, stable target site with rare changes. For variable volume, multiple simultaneous sources, or no dedicated technical team, an external provider absorbs the maintenance and adaptation cost — a cost that, in-house, translates into unplanned developer hours.
Practical decision rule: if script maintenance would consume more than a few hours a month of your internal team's time, an external provider is usually more cost-effective overall, not just cheaper upfront.
Practical 4-step plan for evaluating a crawling provider
- Ask for a list of similar sites already processed, matched by source type and structure, not just by industry.
- Ask explicitly about robots.txt and GDPR for your target source — the answer should be specific, not generic.
- Clarify the delivery format and frequency before discussing the final price.
- Confirm who covers maintenance costs when the target site changes, in writing, in the offer or contract.
Frequently asked questions about choosing a crawling provider
How much does a crawling project cost?
It depends on data complexity, volume, and collection frequency; the first collection is usually more expensive, since it requires a dedicated script built specifically for the target site.
Can a provider guarantee data extraction from any site?
No. Availability depends on the target site's permissions; a serious provider verifies this before starting, rather than promising universal access.
What happens if the target site changes structure after delivery?
The collection script needs to be adapted; clarify in the offer who covers this cost and how quickly the fix happens.
Do you need legal advice for a crawling project?
For sensitive data or large-scale projects, we recommend additional review with a legal specialist; the provider can guide you, but the final decision on legal risk stays with the client.
What sets a crawling provider apart from a scraping tool you buy directly?
A provider delivers a dedicated script tailored to your source and required format, plus ongoing maintenance — a generic tool requires in-house setup and upkeep.
Conclusion: choose the provider who explains the risks, not just the price
A trustworthy crawling provider talks openly about the legal limits of data collection, script maintenance, and delivery format from the very first conversation. Offers that skip these points tend to generate hidden costs and incomplete data in the following months, right when you need a steady data flow.
Want to automate data collection for your store? Contact us for a custom quote, or see the full service: Web crawling services.
Write a comment