This blog explains bot attacks in detail, how to distinguish good from bad bots, and explains how to let good bots do their useful work while preventing bad bots from doing any damage. It also explains why bot attacks on and by mobile apps are particularly problematic and require special defenses to be put in place.
A bot attack is an automated cyber attack using scripts or modified apps to engage with a service, endpoint or API with the intent of disrupting service, stealing data, making fraudulent purchases, or carrying out other malicious actions.
Bad bots are computer programs that mimic human behavior and are a growing threat to businesses, costing them billions of dollars yearly in fraud and data theft.
A recent study found that companies are losing about 3.6% of their revenue to bots. According to the study, 60% of sampled organizations detected attacks on APIs and 39% detected attacks on mobile apps.
Source: netacea.com
Identifying and eliminating all automated traffic is not an option since some bots are good:
Good bots provide valid and useful services. Search engine web crawling is the best known example of a good bot. Other useful tasks that good bots may be performing on websites include checking for changes for archiving purposes, website health checks and usability and security tests commissioned by the website owners.
These activities introduce a small and fairly uniform overhead to most sites, but do not constitute a problem. They are in fact beneficial, making websites available for search and ensuring they are working correctly. Without google crawling the entire internet looking for search results, it would be much harder for any e-business to connect with their customers.
Bad bots are malicious and very targeted, focusing on individual sites and either degrading performance, spamming them, searching for weaknesses in their security or stealing information. This can be costly in a number of ways. DDoS attacks are the most straightforward, directly increasing your server costs and worsening the experience for your customers.
It can also lead to extortion as companies are forced to pay to stop the attack. However, perhaps most insidious of all are scraping bots which do serious damage to the business and reputation of online retailers, travel sites, aggregators and price comparison sites.
Automated web scrapers target websites looking for valuable information which they will attempt to extract via automated tools. They do this for various reasons. On a site which is built on the quality of its own content, a web scraper can potentially steal that data in order to resell it to a competitor. A flight aggregation site may have built a reputation based on the quality of the search results it returns, after investing time and money in making those as good as possible in order to attract and retain customers. If someone can scrape the pricing information and use it to populate their own competitive site then they can attempt to undercut the original site, thus stealing traffic and revenue.
The airline Ryanair even took the Wegolo website to court to prevent them from scraping airfares from the Ryanair site.
The problem of competitive web scraping is certainly something that all commerce sites are very concerned about.
Bot protection is a measure taken by developers to protect their apps from illegitimate traffic or activity that can skew analytics and impact business decisions. It filters out bad actors, like scrapers and spammers, and allows developers better to understand their app's true usage and engagement.
There are a few reasons why bot protection is so important:
By protecting your app from bots, you can avoid all of these issues.
A bot manager is any software product that manages bots. Bot managers should be able to block some bots and allow others through, instead of simply blocking all non-human traffic.
In a nutshell, Bot Managers help to keep the internet safe and running smoothly by controlling the bots that crawl across it. By identifying which bots are good and which are bad, they can block their activities accordingly. For example, in a web context good bots are allowed access to certain content and resources, while bad bots are denied access and may even be served alternative content.
Like search engine crawlers, good bots help us access information quickly and efficiently. Bad bots, on the other hand, can be used for everything from data scraping to DDoS attacks.
A good bot manager should be able to identify bots vs. human visitors, identify bot reputation, and add good bots to allow lists. They should also be able to challenge prospective bots with a CAPTCHA test or other techniques, limit any potential bot's excessive use of a service, and deny access to specific information or resources for malicious bots.
It is important to note that protecting mobile first or mobile-centric businesses - where the mobile app is the main touchpoint for end users - presents a difficult scenario because mobile apps can be downloaded and reverse engineered by anyone and can actually be turned into very effective bots by bad actors if you don't take steps to prevent this. Techniques used by web-based “traditional” bot managers are no longer effective in the mobile world.
Your mobile apps, once published, can be downloaded by anyone and once downloaded they may reverse engineer, study, modify or manipulate your app to suit their purposes. Since important business logic is contained within your mobile code, alongside required credentials such as API keys, your mobile app truly is a treasure trove of data for attackers. As a result, you must deploy specialist mobile app and API security techniques.
With the rise of mobile apps the vendors that provide the current tools to identify and counter bots have a new challenge. Browser signature based approaches in particular do not work. In addition, browser identity analysis extraction code in Javascript cannot be injected through mobile APIs.
Bots go wherever attackers send them, which tends to be the most popular endpoints across your mobile app, website, and APIs. In a world where more than 50% of global online traffic comes from mobile devices, a lot of traffic hits mobile apps. In addition mobile apps and the devices they run on can be modified and used to create bots themselves. It’s important to add bot detection to mobile apps and APIs.
You can take steps to protect your mobile app from bad bot attacks. There are six more traditional steps to keep bad bots away from your backend systems and a new one focused on mobile apps in particular:
1. Block or CAPTCHA outdated user agents/browsers
2. Block known hosting providers and proxy services
3. Protect all bad bot access points
4. Evaluate traffic sources
5. Investigate traffic spikes
6. Monitor failed login attempts
7. Invest in mobile bot protection
Let’s look at each one in more detail. Clearly it is preferable to identify and block bad bot traffic at the edge, before it enters your backend infrastructure and causes damage and increases your operating costs.
1. Block or CAPTCHA outdated user agents/browsers
Bad bots often masquerade as old browsers or devices when they make requests to your server. By blocking requests from outdated user agents/browsers, you can reduce the number of bad bot requests your server receives. Thankfully, most modern browsers force auto-updates on users.
2. Block known hosting providers and proxy services
Many bad bot requests come from compromised computers that are part of a botnet. These computers are usually located on shared hosting providers or proxy services. You can prevent many bad bot requests from reaching your server by blocking known hosting providers and proxy services, although you should be aware that masking where communications come from is not too complex.
3. Protect all bad bot access points
Bad bots will try to access your website or app through any open door, including any unprotected API endpoints. Protecting all your access points with authentication and authorization can make it more difficult for bad bots to reach your server, although it should be noted that credential stuffing attacks do use valid usernames and passwords.
4. Evaluate traffic sources
If you see sudden spikes in traffic from unknown sources, it could be a sign that someone is using a bot to DDoS your website or app. By monitoring traffic sources, you can quickly identify and block malicious traffic before it does any damage.
5. Investigate traffic spikes
Not all traffic spikes are caused by bad bots. But if you see a sudden spike in traffic that doesn't match up with any changes you've made to your platform, it's worth investigating further. It could signify that someone is trying to DDoS your site or steal data from your app.
6. Monitor failed login attempts
A high number of failed login attempts could signify that someone is using a brute force attack to gain access to your app, e.g. a credential stuffing attack. By monitoring failed login attempts, you can quickly identify when you are under siege and minimize the damage.
7. Invest in a bot protection solution for mobile
The traffic actually coming from your mobile app is particularly difficult to differentiate from traffic claiming to come from your mobile app. The best way to protect your mobile business from bot attacks is to invest in a bot protection solution like Approov. Approov uses a runtime attestation system that verifies the integrity and origin of every request made to your mobile backend APIs using cryptographically signed tokens bound to the requesting device/application couplet - protecting against automated bot attacks, malicious replay recordings, and fraudsters as well as stopping man-in-the-middle (MitM) hijacks and other active breaches.
Bot management solutions are designed to protect mobile apps from various automated attacks. Perhaps the most well-known type of bot attack is the distributed denial-of-service (DDoS) attack, in which a backend platform is overwhelmed with traffic from many bots to bring it down.
However, cybercriminals can also use bots to launch simpler denial-of-service (DoS) attacks, in which a single bot pings an app repeatedly to slow it down or crash it. This is particularly relevant for mobile apps that use 3rd party services which are accessed directly from the app code; a single bot can overwhelm a 3rd party service, potentially causing that 3rd party service not to respond which in turn might cause the mobile app to become unresponsive.
Credential stuffing is another type of bot attack that involves using stolen usernames and passwords to gain unauthorized access to accounts. Credential stuffing may also be used to steal payment card numbers or other personal information.
These attacks are particularly difficult to defend against because they operate at low frequencies, usually passing under the radar of traditional security mechanisms.
The challenge of mobile requires a new solution. Instead of examining the behavior of a device and trying to infer whether it is a bot, Approov uses a positive authentication model. Our custom SDK integrates seamlessly with the genuine app, allowing it to present an authorized app identity to the server. Real customers can then confidently be given full access to the backend server assets while suspicious activity can be blocked or rate limited.
Our technology incorporates sophisticated anti-tamper mechanisms and helps secure mobile APIs against the new bot threats developing in the mobile app channel. One of our penetration testing activities revealed a clear example of this problem. The API in question did not require a user login to perform searches and because the search is done inside the app, there is little to identify the agent requesting the information. This is very common in travel apps. If the app was enhanced with Approov technology, it could present a token proving that the software used to perform the request was the genuine mobile app. The server could then only respond to requests it knows are from a valid client.
The world of anti-bot technology is evolving rapidly. The days of very simple, easy to detect bots are long gone. As bots become harder to detect and switch their attack vectors to the mobile app channel, more sophisticated approaches are required to effectively detect them. Approov is an anti-bot solution for mobile APIs and adopts a positive model to allow you to authenticate the software being used to communicate with your servers prior to granting access, hence removing an important vector for scraping backend data.
Schedule a demo with us to find out how Approov Mobile Security will keep your mobile business safe from bot attacks.
- VP Marketing, Approov
George is based in the Bay Area and has an extensive background in cyber-security, cloud services and communications software. Before joining Approov he held leadership positions in Imperva, Citrix, Juniper Networks and HP.
