Your server is under a scraper attack (load average 37): how to block it with .htaccess without blocking real customers

The symptom

The server's load average, the number that summarizes how many requests are queued to be processed, climbs within minutes to something off the scale: from a couple of points, normal for a healthy store, to over 30. The site responds in fits and starts, pages that used to load instantly now take seconds, and at some points it stops responding altogether. There was no deploy, no catalog change, no ad campaign launched at that moment: the "normal" traffic from real people hasn't changed, something else has piled on top of it.

Watching the access log live (tail -f on the Apache or Nginx log, filtered to the last few minutes) makes the cause obvious within seconds: hundreds of requests per minute against category pages and product pages, coming from a handful of IP addresses that belong to a cloud computing provider, not residential or mobile connections. This is a scraper, a program systematically downloading the catalog, not a traffic spike from people.

What it looked like, and wasn't

The first idea, the most obvious one, is to block the IP addresses showing up in the logs: it looks like the most direct fix, and for the first few minutes it even seems to work, the load drops as soon as those IPs stop getting a response. The problem comes back shortly after: the scraper is coordinated across several addresses from the same cloud provider, rotating, and every blocked IP gets replaced by a new one within minutes. Blocking IPs one at a time, in this scenario, is a chase you never win: you spend your time adding lines to a list the scraper simply routes around by switching IP.

There is a detail, though, that IP rotation does not hide: every request from the scraper, regardless of which IP is making it at a given moment, declares the same narrow set of browser identities (the user agent), and specifically Chrome versions that are several years old, exactly the kind of string a real browser updates automatically and that you would not expect to see in the wild anymore. Rotation covers the address, not the signature.

The real cause and the blocking decision

An effective block, in this scenario, combines both criteria in the same rule, at the top of the store's .htaccess file, right after RewriteEngine on, so the request is rejected by Apache before it ever reaches PrestaShop and the database: that is the difference between responding with an error at close to zero cost, and still running the entire application stack on a request that is going to be discarded anyway.

# Known IPs of the cloud provider involved
RewriteCond %{REMOTE_ADDR} ^203\.0\.113\.10$ [OR]
RewriteCond %{REMOTE_ADDR} ^203\.0\.113\.11$
RewriteRule ^ - [F,L]

# Scraper's User-Agent signature (dated Chrome builds, used across rotating IPs)
RewriteCond %{HTTP_USER_AGENT} "Chrome/87\.0\.4280\.141" [OR]
RewriteCond %{HTTP_USER_AGENT} "Chrome/61\.0\.3163\.100" [OR]
RewriteCond %{HTTP_USER_AGENT} "Chrome/74\.0\.3729\.169"
RewriteRule ^ - [F,L]

(The addresses above are examples: in the real case there were four IPs from a single cloud provider, identified from the traffic at the time. They should always be pulled from actual live traffic, never copied from a list found somewhere else.)

Blocking by signature is what actually survives rotation: even when the scraper switches IPs, the user agent string stays the same until they change that too, and at that point the rule needs updating, not another round of chasing individual addresses.

The real risk of this kind of block

Blocking by signature, not just by address, is more effective, but it also raises the risk of a side effect nobody checks for until it happens to them: catching a real customer whose device happens to declare something that resembles the blocked signature. This is not a theoretical risk: on the same store, with a different filter but the same underlying logic (recognizing automated programs from a string), the first version of that rule also caught a legitimate phone brand's name inside it, and PrestaShop, whenever it decides a request comes from a robot, silently skips adding it to the cart: the customer saw "done," the cart stayed empty, no error in any log. The full story, with the numbers from that case, is in this article, it is worth reading before writing a signature-based block, because it is the real danger of this technique, and almost nobody checks for it before turning one on.

Because of that, before turning on a signature-based block, you need to explicitly verify it does not catch legitimate traffic. In the case that led to this rule, three categories of traffic were checked one by one before the block was considered safe:

• back-office access from an already authenticated admin session (must stay out of the block: an .htaccess-level firewall cannot tell a logged-in administrator from an anonymous visitor, other than by the URL they're hitting);
• a real smartphone visit, with the mobile browser's genuine user agent (must stay out: a block that is too broad on "old Chrome versions" can also catch legitimate mobile browsers with unexpected version numbers);
• legitimate automated requests from a third-party service integrated with the store, for example a carrier or a payment system querying the store's API with its own declared user agent (must stay out: it is automation, but it is wanted automation).

Only after confirming none of these three categories fell inside the block was the rule considered safe to leave active.

How to verify it holds

• From the command line, with curl, send a request declaring one of the blocked user agents: it should return 403. Send one with a normal, up-to-date user agent: it should return 200, on the homepage, a category, and a product page.
• Watch the server load for the minutes right after the block goes live: it should drop steadily, not in fits and starts, as long as the scraper hasn't switched IP or signature yet. Load creeping back up after an initial drop is the signal that the signature has changed and the rule needs updating.
• Check the logs for a few days after activation that the three legitimate traffic categories listed above keep working normally: this is the check that actually prevents the wrongly-blocked-customer problem, not something to do once and forget.
• Keep a backup of the .htaccess file from before the block, with a rollback ready (an actual copy, not a command you'll remember later): if something goes wrong, the fastest way back to a working store is restoring the file, not fixing the rule under pressure with the site down.

The store where this block was applied stays anonymous on purpose. An .htaccess-level block, by IP and by signature, is the right move when a server is under attack right now and needs immediate relief: it costs almost nothing, because it responds before PrestaShop has to do anything at all. But it is a surgical move, not a permanent one: if both the addresses and the signatures change completely, a different approach is needed (for example, a block based on request rate rather than identity). And above all, before turning it on, always check what it must NOT catch, not just what it must stop. For urgent work of this kind on a PrestaShop store, there is a summary of services on this page.