Bot Traffic: What You Need to Know
Bot Traffic: What You Need to Know
Over the last few years, bot traffic has become a massive headache for organizations across all industries. While bots themselves are nothing new, bot traffic volume and sophistication have risen drastically as hackers sought ways to scale their malicious activities. In this article, we’ll explain the difference between good and bad bots, explore the risks they pose, and cover the steps you can take to protect your business website and web applications.
What is Bot Traffic?
Bot traffic consists of any non-human interaction with your website and online applications. Bots — short for robots — are computer programs written to complete repetitive tasks automatically. Bots complete tasks much faster and more accurately than humans and can also scale their activities easily. After overtaking human traffic in 2016, bots account for more than half of all Internet traffic. While bots were initially confined to simplistic tasks, modern bots are often highly sophisticated, and some even include self-learning algorithms.
Search Engine Bots or DDoS attacks
So, what exactly do bots do? Among the traffic your website experiences, you’ll find a mixture of bots programmed to carry out benign and malicious tasks.
The most common example of ‘good’ bots are the search engine ‘spiders’ used by Google and Bing to discover web pages and catalog their content. Without these bots, search engines wouldn’t be able to serve users with relevant search results, and it would be hard for website owners to attract attention. However, while some bots are essential, bad bot traffic can be highly detrimental. The most extreme example comes in the form of Distributed Denial of Service (DDoS) attacks, where a large number of compromised devices are used to flood a website or application with more traffic or requests than it can handle. DDoS attacks can be extremely damaging, particularly for organizations that rely on their web assets for financial or operational success.
Good vs. Bad Bot Traffic
However big your company website is, you can expect a significant amount of bot traffic. Good bot traffic consists mainly of bots managed by online service providers like search engines and SaaS companies. On the other hand, bad bots are created by malicious actors to conduct fraud, spread spam, and carry out cyber attacks.
Some of the most common bots include:
|Good Bots||Bad Bots|
|Search engine crawlers (e.g., Google, Bing)||Spambots|
|SEO and backlink checkers (e.g., SEMrush, Moz)||Website information scrapers|
|Aggregator feed bots (e.g., Feedly)||Fake account creators|
|Digital assistants (e.g., Siri, Alexa)||Checkout and business logic abuse bots|
Maintaining oversight of the bots that visit and interact with your website and applications is essential. While good bots are often valuable, too much bot traffic can hurt website or application performance, damaging legitimate customers’ experience. To make matters worse, sophisticated bad bots can cripple a website, either by disrupting or damaging its operations or by abusing business logic.
Hoarding bots are common on websites that sell limited stock that’s highly in-demand, such as ticketing websites for music and sporting events. These bots create fake accounts and repeatedly add stock to their baskets, preventing legitimate customers from purchasing it. This is usually done to ‘hold’ stock while it is fraudulently sold elsewhere for a higher price.
This automated ad clickers target websites that include paid advertisements and are used to ‘click’ ads hundreds or even thousands of times. This may be done to damage the success of a company’s advertisements, exhaust its advertising budget, or prompt advertising providers to ban the website from their network.
How To Detect Bot Traffic
To ensure your website and applications are properly served by good bots and protected from bad bots, you first need a mechanism to detect bot traffic. Unfortunately, this is far from simple. A major complaint from webmasters is that it’s difficult to distinguish between human and bot traffic in web consoles like Google Analytics. This makes it harder to understand user behavior and track the success of pages, features, and advertisements.
However, this is merely an unfortunate byproduct of good bots. Bad bots are much harder to detect and have far more significant repercussions. Some of the indicators you can look out for to identify bad bot traffic include:
- Unusual or very high website or application activity.
- Sudden spikes in page views, clicks, or account creation.
- High levels of spam comments or form submissions.
- Unusual checkout activity, e.g., checkout ‘hoarding’ of in-demand items.
Unfortunately, while these indicators can be helpful, it’s challenging to reliably identify bad bot traffic through monitoring. Even if the indicators above can be flagged for automatic identification, only the most basic bots are likely to be detected.
To make matters worse, modern bots are extremely good at avoiding detection. Malicious actors are increasingly designing bots to mimic human behavior, and some advanced bots even include self-learning algorithms to help them go under the radar.
How To Stop Bot Traffic from Damaging Your Business
While there are several things you can do to limit the damage caused by bot traffic, most are only partially effective. For instance, you can set rules for bots using a simple file called robots.txt, a standard component of any website. While this is an important step to take, it will only affect good bots like search engine crawlers. Bad bots will simply ignore your instructions.
Other standard precautions include:
- Rate limiters cap the number of requests permitted from a single IP address within a specified period. This approach has some value but is mostly ineffective against bad bot traffic because many bots use paid anonymous proxy services to rotate through millions of unique IPs. This enables bots to imitate traffic from residential IPs, making them practically indistinguishable from legitimate users. Equally, many destructive bots only require a handful of requests to complete their objectives.
- Human observation and analysis can be effective for small websites with low traffic but is completely unscalable. To make matters worse, even the most experienced and knowledgeable analysts can’t keep track of the latest bots (good and bad).
- Web Application Firewalls (WAFs) can identify some bots, but only if they are a known entity (i.e., they are included in a blacklist) or they attempt to attack a website or application in an obvious way (e.g., using SQL Injection). As a study by Cequence Security and Osterman Research found, WAFs are rarely able to detect bots that conduct more sophisticated, business logic-related attacks.
To ensure websites and online applications are adequately protected against bots, organizations need the means to distinguish between good and bad bots in real-time — even when those bots have never been seen before. More specifically, organizations need a bot management solution that benefits from the same advantage as bots themselves — the ability to complete actions quickly and automatically while ‘learning’ as they go.
Managing Bot Traffic On Your Business Website
To manage bots effectively, you need a bot management solution that provides full control over the wide range of bots that access your website and web applications each day.
Link11’s advanced bot management solution uses proprietary AI and Machine Learning algorithms to distinguish between good and bad bots in real-time — with zero human intervention — and block bot traffic only if it poses a threat.
Bots that are known to be malicious are blocked instantly, while new, unknown bots are identified and mitigated in under ten seconds on average. This is essential for full protection, as new bots are under continual development to bypass lower-quality controls.
As a result, your organization gets:
- Better website performance and improved user experience for real customers.
- Real-time defense against all malicious bot traffic, including volume-based attacks.
- The power to categorize, manage, and block bots individually.
- A drastic reduction in cyber risk caused by bots — one of the top threats to websites, web applications, and online services.
To find out how you can take control of bot traffic on your business website, visit our bot management page.