The Bots Are At The Gates – What You Need to Know to Protect Your Website

Bots. Whether it is the movies, or on the internet, bots have a unique place that sit somewhere between good and bad in the story. On one hand they can automate tasks, but on the other – they can take too much effort to manage and spin wildly out of control as a result. Worse yet, they can be used by malicious actors to do substantial harm. In the story of your organization's website, you should know where bots fit into your strategy and how Spindustry is helping to keep bots useful to your business, while also protecting your site from malicious actors that would use the cleverest bots to do your site, business and even you harm.

What is a bot?

Not as cool as Star Wars' R2D2 or Disney’s Baymax, bots on the internet are automated programs that access a site as a "user" -- a fake user. This allows a computer to view a site and all of its public information. Bots can even be smart enough to use login dialogs and buttons, simulating a user to the point it becomes hard look at activity alone and discern a real human user from a bot.

So why let bots hit the site at all if they are fake users?

Bots are used by a variety of companies to perform regular processes and testing that would take humans far too long to do. A great example of this is Google Bot – this little guy goes out to your site, catalogs all the pages and then indexes the data so when someone searches for your products or services, your website can be searched on Google. Google Bot will even reindex the site every now and then, to keep the search results up to date. There are many other types of bots out there used for user testing, vulnerability scanning, uptime monitoring, etc. This functionality can be further extended by leveraging AI to combine and update bot-driven data consistently and pull valuable insights from the collected information. With the advent of AI, bots can really become intelligent agents for enhancing marketing, security, and technology use in a business. Spindustry uses a variety of tools that leverage bot operations to maximiize the effectiveness of your marketing spend and to find ways to improve your site's SEO. The good bots help us keep your site visible to the world and allows us to strategize the next step in your marketing journey.

Of course, where there is good, there is bad.

The good things aside, bots are often used by malicious actors to commit automated, brute force hack attempts, as well as executing large, distributed denial of service (DDoS) attacks that drive an enormous amount of bad or damaging traffic to your site, bringing your site down and preventing legitimate users from accessing your site. Once again, AI can enhance these attacking bots to adapt to mitigation and prevention attempts, making them dangerous and difficult to deal with for any security team. In fact, websites are not the only ones at risk: these bots can be turned on other equipment, services, or anything that can be reached from the internet. There are even concerns that AI-powered bots could be used in the future to attack individual people with internet-connected vehicles, medical devices, nearby internet-connect industrial equipment.

Remember, bots are tools. It's how they are used that determines their value to the business and your customers. Leveraging bots effectively, can put you at the top of the search results, ineffective or malicious use of bots can result in lower results or complete paralysis of the site.

Can good bots cause bad problems? 

Unfortunately, bad guys aren’t the only ones that can cause havoc with bots though. Legitimate businesses can use bots with bugs, or misconfigurations to attempt legitimate use cases, only to result in DDoS style attacks, completely unintentionally bringing your site to its knees.

Case in point, Spindustry saw a similar situation recently. For the last 3 months Spindustry Support has been tracking a rising amount of traffic to our systems, as well as client hosted systems, and have been working to track down where it is coming from, why it is increasing and how to mitigate any potential issues. Luckily, the Spindustry hosting systems, as well as our hosting datacenter, are capable of handling increased traffic to a large degree – so no immediate effect was felt by anyone. However, the trend was concerning, quickening, and could eventually result in a full-fledged saturation or DDoS scenario.

Over the course of those 3 months, we identified more than a 150% increase over expected traffic values to our entire hosting environment – but on a site level, things were even more staggering. Some sites were seeing more than a 300% increase in traffic – and nearly all of it was coming from a single legitimate bot. This bot has been known to have saturation problems in the past, however, Spindustry has never seen the behavior over the last 15 years, until recently. Still – despite the increase, and continued trend, Spindustry Support didn’t want to outright block this traffic. The bot owner does its own legitimate indexing, like Google, but also pulls information about links when you share them on their popular social media platform. So outright blocking isn’t an option without potentially hurting SEO.

Then how do we stop the bad bots?

Though you can’t block them without sacrificing SEO, you can limit them. By default, every site should have a file called robots.txt – this cleverly named file is there for one reason: give bots a set of rules when browsing the site. At Spindustry, we set up these files for you and make sure to keep bots out of areas they don’t belong. We also place the needed request limits on indexing bots like Google, so they should not result in a DDoS scenario. Combined with CAPTCHAs, behavioral analysis, continuous application monitoring, and device identification techniques, sites hosted at Spindustry see bot traffic managed effectively 98% of the time. As you can imagine, malicious actors don’t follow these rules and actively look for ways around the other mitigation factors. Unfortunately, bots that are misconfigured or buggy from legitimate sources can ignore rules and limiters as well.

In this instance, despite our standard procedures and protections, the traffic from this offending bot continued to increase and hit our clients’ sites across the all of our hosting environments. Spindustry Support took the next steps to limit this traffic while maintaining functionality. This resulted in extended DDoS protections being put in place at the edge of the hosting environments, firewall limiters, and even server level limiters – all being pointed at this offending bot. The effect was noticed immediately with a sizable chunk of bot traffic dropping off on the whole environment. For many sites, this was all that was needed, and Spindustry Support implemented the changes transparently, as a part of existing hosting standards.

Some sites still needed some help though. Despite best efforts, some sites continued to see targeted increases of traffic from the offending bot and our developers were called in to devise a solution that would allow the site to automatically identify blacklisted bot traffic and implement limits at the site or even outright block the bot if configured to do so. Following implementation on some of the highest load sites, the solution was enormously successful in limiting and continuously protecting the site from the offending bot. The code solution brought stability to the site and increased a site's capabilities, giving admins the agility to respond to future threats quicker when another bad or misconfigured bot appears to wreak havoc again.

How do you keep your website protected?

This is a strong example of why you need close monitoring of your website. These risks must be identified, before they can be mitigated -- otherwise your website and your organization could see negative impacts without any idea as to why and without understanding the context of the impact.

Spindustry has set up some excellent web monitoring packages that are available right now. With these packages, we can help monitor these issues and respond for you. These plans allow us to take the necessary steps to keep things working as expected, even in light of an unexpected incident.

I'm Ready to Protect my Website

Return to All News