Bots are everywhere, crawling all over the internet. Some are good, cataloguing websites and enabling you to search for pictures of cats with ease. Others are all about information gathering, theft and fraud, are known as API abuse, and are bad news. More and more time is being spent accessing the Internet from mobile devices, and apps are becoming increasingly important as the software performing this access. Apps are a new and challenging arena for existing bot mitigation techniques and attackers are starting to shift their focus from the mobile web channel to mobile apps to try and circumvent current protection mechanisms.
There are many valid uses for bots, search engine web crawling probably being the most common. Other tasks that the good bots may be performing on websites include checking for changes for archiving purposes, website health checks and usability and security tests commissioned by the website owners. These activities introduce a small and fairly uniform overhead to most sites, but do not constitute a problem. They are in fact beneficial, making websites available for search and ensuring they are working correctly. Without google crawling the entire internet looking for search results, it would be much harder for any e-business to connect with their customers.
The malicious bots on the other hand tend to be much more targeted, focusing on individual sites and either degrading performance, spamming them, searching for weaknesses in their security or stealing information. This can be costly in a number of ways. DDoS attacks are the most straightforward, directly increasing your server costs and worsening the experience for your customers. It can also lead to extortion as companies are forced to pay to stop the attack. However, perhaps most insidious of all are scraping bots which do serious damage to the business and reputation of online retailers, travel sites, aggregators and price comparison sites.
Web scrapers target websites looking for valuable information which they will attempt to extract via automated tools. They do this for various reasons. On a site which is built on the quality of its own content, a web scraper can potentially steal that data in order to resell it to a competitor. For example a flight aggregation site may have a reputation based on the quality of the search results it returns and invests time and money in making those as good as possible to attract and retain customers. If someone can scrape the pricing information and use it to populate their own competitive site then they can attempt to undercut the original site, thus stealing traffic and revenue. The problem of web scraping is certainly something that budget airlines like Ryanair are very concerned about. In fact the airline actually took the operators of the Wegolo website to court to prevent them from scraping airfares from Ryanair’s website. They have also used terms and conditions on the website as well as litigation to try and prevent companies from accessing their data but this can be a slow and expensive approach.
This new set of challenges requires a new solution. Instead of examining the behavior of a device and trying to infer whether it is a bot, Approov from CriticalBlue uses a positive authentication model. Our custom SDK integrates seamlessly with the genuine app, allowing it to present an authorized app identity to the server. Real customers can then confidently be given full access to the backend server assets while suspicious activity can be blocked or rate limited. Our technology incorporates sophisticated anti-tamper mechanisms and helps secure mobile APIs against the new bot threats developing in the mobile app channel. One of our penetration testing activities revealed a clear example of this problem. The API in question did not require a user login to perform searches and because the search is done inside the app, there is little to identify the agent requesting the information. This is very common in travel apps. If the app was enhanced with Approov technology, it could present a token proving that the software used to perform the request was the genuine mobile app. The server could then only respond to requests it knows are from a valid client.
The world of anti-bot technology is evolving rapidly. The days of very simple, easy to detect bots are long gone. As bots become harder to detect and switch their attack vectors to the mobile app channel, more sophisticated approaches are required to effectively detect them. Approov from CriticalBlue is an anti-bot solution for mobile APIs and adopts a positive model to allow you to authenticate the software being used to communicate with your servers prior to granting access, hence removing an important vector for scraping backend data.