How Python Coders Killed my Supposedly Secure JavaScript API Service

One Developer's Bad Dream

Disclaimer: No Python or JavaScript coders were harmed during the writing of this article. All actions involving coders were safely simulated and purely fictional.

Have you ever tried to launch an app only to see your digital assets lost to a tiny hole in your API security? Do you imagine being as wildly popular as Pokémon GO, but like Pokémon GO, worry that you’ll lose control of your application’s features and anger your user base?

In Nov’16, the guys at Fallible created an online tool to reverse engineer any android app to look for secrets. Of over 16,000 apps tested, they found roughly 2500 apps contained API keys and/or secrets. That’s bad for security.

Code Surfing

Recently, to improve my JavaScript skills, I didn’t want to do an intense multi-day hack-a-thon, just get together for a few evenings occasionally with other coders. From this, the JavaScript Code Surfing app was born. The concept is simple; a host coder with an interesting project and decent WiFi advertises a meetup which other coders can sign up to attend. The host gets some attention on his project, and the attendees get to build up their JavaScript chops. The business model was purely altruistic, though I could see ways to monetize the service enough to make it self-supporting.

I built the backend with Node.js/Express, went mobile first, launching an Android app initially with iOS right behind, and I tied things together with a simple RESTful API. Both front and back end implementations were completely open-sourced.

I used several 3rd-party API services. For example, I used the GitHub API to estimate a coder’s JavaScript reputation based on their public repository activity and contributions. From the start, I implemented OAuth2 user authorization with two-factor authorization. With location disclosure involved, I wanted to ensure that all information access was properly compartmentalized and protected.

A Sputtering Launch

After launch and a bit of social promotion, things began to pick up, and that’s when the trouble began.

The first symptom was complete failure of the Android app to gather JavaScript reputation scores. With no reputation, no one could sign up for meetups.

I traced the problem back to GitHub, all API calls were being completely rate limited! I was calling the GitHub API directly from Android, and the API id/secret keys were stored in my app, safely I thought. However, someone had stolen my GitHub credentials and was flooding the API, completely locking out my real JavaScript user base. Turns out my app was being djangoed by jealous Python programmers.

No API keys were stored in the open source repositories. The keys were being stolen directly from the Android APKs- my app was failing the Fallible test. I tried code obfuscation and encoding the secret, but that didn’t slow them down much at all. Every time I changed the secret meant a full upgrade of the entire installed base. This was a losing game.

A Pitched Battle

My next step was to get the 3rd party GitHub API keys out of the Android app. Instead of calling the GitHub API directly from Android, I extended my own API to proxy calls to GitHub through my own back-end servers. At least I wasn’t exposing the GitHub API keys any more.

Though a step in the right direction, that didn’t slow those pesky pythonistas down very much. My app’s API key was the one key still hidden in my app, and they easily found that. Their response was to reverse engineer my own API and use it to cause my back end server to repeatedly calculate a user’s GitHub reputation, shutting my app down just as dramatically as before.

I countered with rate limiting and by caching user reputation scores on my server. They countered by creating a bunch of new users and flooding my service with fake meetup requests.

I was stuck in a never ending game of hide and seek, and all the fake traffic was costing me real money at Amazon Web Services. This was not sustainable.

No More Secrets

Why did I have an API key? It was supposed to be a secret known only to the app so that only the app could successfully call my API, but keeping that secret on the app was problematic.

My user authentication was solid. In fact, I delegated it to an OAuth2 authorization service which returned a user authorization token. My app never saw any user credentials.

So I used a similar approach to remove my last API secret. This time I delegated authentication of the app to an app authentication service. The service challenges the app and based on its responses can attest that the app is authentic and untampered. Like the user authentication service, the app authentication returns an authorization token. The special secret which generated the token is known only to my back end server and the authentication service; it’s never exposed to the app, so there’s nothing left there to steal.

Attesting app authenticity is fast and requires no human inputs, so the lifetime of any authorization token can be quite short. If any token is somehow stolen, its value is limited.

Once I got the last secret out of the app, my service stabilized. Attempts to repackage my Android app or replay API calls all failed to attest, and my costs came back down.

That would be the end of the story except for one more mistake I made. During development, I generated a long lifetime app token for testing, and I left it in the open-source repository. That put the pythonistas right back in business. Fortunately, since the secret was not in the app anymore, I simply switched to a new secret. As soon as the secret was changed, that dangerous test token was impotent, and no updates to the installed application base were needed.

Lessons Learned

When I finally woke up from my bad dream, I reflected on what might have happened.

Rather than adding features to my app and growing my user community, I had to spend a lot of time and money fighting off malicious attacks. I couldn’t afford to stay in that fight for long.

You must consider any mobile platform to be a hostile environment. It’s not just about user authenticity, which is what most developers focus on. it’s also about the authenticity of the app. Using a 3rd party service to attest your app provides a best practice approach to keeping your mobile apps secret-free and secure.

And never ever underestimate the tenacity of programmers- JavaScript and Python coders alike.

Full Disclosure: I love to code and occasionally dream in javascript, python, and other languages; they all have their delights and their challenges. I work on software performance and API security for CriticalBlue, and safely handling secrets is a part of our business.