How Much of Your Store's Traffic Isn't People

The number that didn't add up

PrestaShop's statistics log every request that reaches the store as a "visit", unless an internal filter recognises it as an automated program and discards it first. The first day we actually checked our own store's numbers, the statistics table said 43,478 visits. Real people, that same day, were about 300. A hundred and forty-five fake visitors for every real one.

This is not a rare edge case: it is what happens when the bot filter built into PrestaShop is never updated. And the table that fills up with those numbers is not free: on our own store it had grown past 1.3 GB, one useless row at a time. After the cleanup it was down to 24 MB.

Why the built-in filter is not enough any more

PrestaShop already ships a method meant to recognise bots, Connection::isBot(), and it works by matching the request's user agent against a list written into the code. The problem is that list has never been refreshed: it contains names like Scooter (AltaVista's crawler) and other search engines that closed down years ago, but none of the crawlers built more recently to train AI models, and no updated versions of the crawlers belonging to search engines that are still around today. A filter built for the web of fifteen years ago lets through, unrecognised, exactly the programs generating the most automated traffic today.

The practical consequence is that "number of visits" on an unmaintained PrestaShop store does not measure people: it measures people plus a share, often the majority, of programs the filter cannot recognise. Decisions made by looking at that number, from ad budget to judging whether a page "works" or "doesn't convert", start from a wrong figure.

The mistake we made ourselves, first

Updating that list sounds harmless enough: add a generic rule that recognises the word "bot" in the user agent, plus the proper names of the crawlers that actually visit the store (search engines, Meta, the various AI crawlers). The first version we wrote said, roughly, "if the user agent contains bot, it's a robot." It worked perfectly, except for one detail: it also matched the tail end of CUBOT, an Android phone brand.

PrestaShop uses the same isBot() not only for statistics, but also inside the controller that adds products to the cart: if a request looks like it came from a robot, the add-to-cart action is silently skipped, and the store still answers "done", with no visible error. The result: a real customer with a CUBOT phone would click "add to cart", see the confirmation message, and the cart would stay empty. No error in the logs, no warning: just a customer who couldn't buy anything and probably had no idea why.

We found it by running the new rule against 153,113 real log lines from five days of store traffic, comparing how many robots the old rule caught against the new one: the difference was exactly the 31 requests from a CUBOT phone. We fixed it with an explicit exception for that brand, and re-ran the same comparison before shipping the rule to production. This is the kind of bug that only shows up if you go looking for it: a filter that is too strict does not throw an error, it produces a customer who leaves without telling you why.

A working filter is not the whole story

Recognising incoming bots is not enough if something else then explicitly tells them to come in. Around the same time, our Apache log showed 38,144 requests in a single day from one crawler, the one Meta uses to train its models: 78% of that day's entire traffic. These were not scattered hits: the crawler had fallen into a loop, following a login return parameter that kept nesting inside itself at every pass, and every single request was a fully rendered PrestaShop page, not a cheap response.

The cause was not a bug in the crawler: it was a robots.txt file that, as part of a configuration meant to welcome the "good" AI crawlers, had ended up welcoming that specific one too, the one that should stay out because it trains models rather than sending traffic back. Once the configuration was corrected, that crawler's traffic dropped 95% within fifteen hours. The lesson here isn't about the crawler either: a "good" filter written in one place can be overridden by a rule written somewhere else, and the only way to know is to actually read the logs, not to trust the configuration on paper.

What to check before you trust your own statistics

• Don't look at the "visits" number in isolation: compare it against an independent source (a browser-side analytics tool, for instance) for the same day. If the gap is an order of magnitude, the bot filter is not doing its job.
• If you maintain your own list of recognised bots, test it against a real sample of logs before shipping it, not just a handful of hand-typed user agents: a phone brand, a library, a legitimate HTTP client can accidentally contain the same word that identifies a robot.
• Check whether the filter feeding your statistics is the same one deciding whether a customer can buy. If it is, an overly strict filter doesn't just pollute a number: it silently blocks an order.
• Read the server logs, not just the configuration: a rule meant to open the door for one crawler can end up opening it for another, and only real traffic proves it.
• Remember the list of bots worth recognising is never finished: new crawlers keep appearing, especially for AI training, and a list written once and never touched again goes stale within months, not years.

In short

The "visits" number an online store shows in its back office is not a neutral fact: it depends on a filter that has to be maintained, and a badly maintained filter can fail in two opposite directions, letting too many bots into the statistics or blocking too many real customers from the cart. On our own store, the day we first looked, the ratio was 145 fake visits for every real person. Cleaning it up did not take a complicated project, but it did take actually reading the logs, fixing a mistake we made ourselves first, and re-checking everything against real data before trusting it.