Today, bots are a hot topic; one that affects all web applications. As a result, many bot mitigation vendors are trying to latch onto this trend by claiming to have the ability to detect and mitigate bots. It’s only natural that you’ll want evaluate the claims of these vendors. Use these 10 questions to help differentiate between solutions to make an informed decision.

1) How does your solution deal with advanced persistent bots?

Today’s bots are more sophisticated than ever. In fact, these bots are so different from the primitive bots of yore, we’ve created a new term to describe them: APBs, or Advanced Persistent Bots.

An Advanced Persistent Bot has the ability to dynamically rotate IP addresses and to distribute their attacks over hundreds or thousands of IP addresses. Rotating IP addresses helps a bot avoid detection by circumventing rate limiting and blacklisting controls. Likewise, distributing attacks across many IP addresses reduces requests per IP allowing these attacks to “fly under the radar.” After all, 1,000 IP addresses making one request each achieves the same goal as 1 IP address making 1,000 requests.

To combat these tactics, your solution should include device fingerprinting to track bad bots across IP addresses. When evaluating bot mitigation vendors, ask if their solution includes device fingerprinting. Solid fingerprinting technology will stick to the bot even if it attempts to cycle through random IP addresses or hide behind an anonymous proxy. In contrast IP-centric bot detection will suffer from high false positives and false negatives.

2) Does your solution make use of modern bot detection techniques?

Legacy bot detection solutions lean heavily on IP-based rate limiting, IP blocking, user agent validation and JavaScript checks. Ask your bot mitigation vendor what modern, proactive security mechanisms they utilize. Does their solution inject active challenges and honeypots into HTTP traffic to trap bad bots? Are they making use of behavioral modeling and machine learning to understand what is normal behavior for your website, then look for deviations from the norm? Besides being more accurate, modern bot detection techniques are self-optimizing, minimizing their administrative burden.

3) Does your product make use of machine learning? If so, how does it work?

Machine learning works by establishing and constantly updating a baseline of behavior for your website, then identifying anomalies to that baseline in real time. This type of automated analysis greatly bolsters the power of the bot detection and mitigation solution.

Keep in mind that running machine learning algorithms on a big data platform is great, but intelligence requires a feedback mechanism. The system needs a way to understand the impact of its decisions. Also, machine learning shouldn’t be a black box. You should have some degree of control over its settings. Can you dial up the settings on critical websites or URLs, or turn it off for specific applications? Make sure you fully understand the product’s machine learning capabilities so you can select a solution that’s more than just a marketing buzzword.

4) Does your solution incorporate external and community sourced intelligence feeds?

Since bots can rotate IPs, mimic human behavior and distribute their attacks, simply getting a list of “malicious” IP addresses and piping them into your defense strategy may not be effective against this type of adversary. If the source of your intelligence feeds doesn’t make use of techniques proven to detect bots, then the data they provide may add little value.

One type of threat feed which is extremely valuable in your bot mitigation pursuits is an external and community-sourced threat intelligence feed derived of bad bot device fingerprints from other companies using the same anti-bot solution you have implemented.

5) Are you able to secure my APIs from automated threats?

When you start blocking bots from accessing your website, you’re cutting off a revenue stream for a would-be hacker. In response to this, many determined attackers will shift their focus to your APIs in hopes of regaining access to their revenue stream. For this reason, it is also important to secure your APIs against these threats, as well as developer errors, integration bugs, and general abuse. Talk with your potential vendors and ask what their capabilities are for handling automated threats launched at your application programming interfaces.

6) What deployment options do you offer?

Generally, a bot mitigation vendor offers solutions in a few deployment types: cloud services, appliances, and API calls. Each of these deployment types has its pros and cons, so make sure to find one that fits your web environment and operational style.

  • Cloud Based Network –Typically this means that the vendor is operating a content delivery network (CDN), which has bot mitigation as one of its features. To use this type of solution, you’ll route traffic to the vendor’s network, they will provide their service, and route the clean traffic back to you. This deployment type is easy to implement, and may provide other useful services such as content caching.
  • Appliances – Appliances are a great option for many customers. They work well behind CDNs and can seamlessly integrate with your existing infrastructure. Additionally, since they are in your datacenter they are subject to your security protocols. When evaluating a vendor with an appliance you should determine if they offer both physical and virtual appliances and how long it takes to get the solution up and running.
  • API call based solutions – Finally there are solutions on the market which function via API calls. These solutions may add latency to websites as requests must be sent to the vendor over their API for inspection before the traffic is allowed to reach the protected website.

7) Can your solution be implemented and configured on a per domain basis?

Depending on how an anti-bot solution works, it may be possible to provision and configure the service on a per website basis, or even in bulk by managing hundreds or thousands of domains at once. A solution architectured in this manner will greatly reduce the management overhead your team will experience with the tool.

Once you’ve established your initial deployment, you may have specific pages which are more sensitive and for which you’d like to ratchet up protection. Some examples of these types of pages might be account login pages or account registration pages. Having a solution that can be managed in bulk is important, but also look for one that offers per URL customization and security controls to help you fine tune your protection.

8) What parameters does your access control list include and does it perform self-maintenance?

Ask your bot mitigation vendor if your solution is able to block by country or organization. Instead of blacklisting specific IP addresses, can your solution blacklist or whitelist entire countries or organizations and dynamically obtain IPs for these entities?

Instead of blocking an IP into perpetuity, a timed approach where specific violations are blacklisted for specific lengths of time can be used. Repeat offenders are blocked for longer periods of time. Using this approach, your access control list (ACL) is more likely to be accurate and to have current data.

9) What kind of visibility do you provide in terms of bot traffic, trends, and motives?

We’ve all used tools that generate great data but in an unabsorbable way such as massive tables or log files. Perhaps they include some filters to help you make sense of it all, but there are problems with this approach. Notably, it doesn’t provide you any context. You might be able to identify the fact that you blocked 100,000 bots but what were they doing, and why?

Ask your potential bot mitigation vendors what pre-made dashboards and reports are available to help you understand your bot problem. Look for solutions that give you more insight than the standard traffic distribution (good bots vs. bad bots vs. humans) or that a certain number of bots were blocked.

10) What options do I have for enforcement?

Let’s assume that whatever solution you implement finds some bad bots, because it will. Once you’ve identified these bots, you’ll need to do something about them. Talk with your vendor and figure out what mitigation options you have. Can the solution be run in a “monitor or alert only” mode? If you want to block traffic, what enforcement mechanisms do they offer? Can you customize the blocking pages to include your own messaging and branding?

Making informed decisions begins with asking the right questions. Don’t forget to be diligent in your pursuit of the perfection solution. It’s out there; you just need to know how to look for it.