** No data or access was compromised. **

On the 20th of May at 6:45 AM, I got a phone call alert from Spike.sh from our Twilio integration about a surge in calls.

Scammers abused our phone verification by otp generating thousands of phone calls. This qualifies as Toll fraud

Alt Text

Timeline of how it went down


7 AM 👨‍💻

Within the next 15 minutes, I was awake and glaring into my screen trying to find out what the hell is happening.

I learn about these keys facts -

40 new registrations in the last one hour using disposable email addresses, mainly Mailinator (.com and .net)

Our Twilio account had been recharged which was unusual

So far, I was knee-deep with $107 bill on Twilio because scammers had made about hundreds of phone calls to a number in Moldova

Also, Twilio, being awesome, detected suspicious activity and blocked several phone calls by that time. I had emails from their Fraud detection team. Thank you Twilio


7:15 AM 😧

After digging into logs, I learn that the scammers were generating phone calls from our Phone verification section. We make phone calls to verify your phone number after signing up on Spike. I knew this was not the best idea going in too... 🙁 We now send SMS to verify instead of making a phone call.

There were two such sections. Both of them now look like this -

Alt Text

7:30 AM 💸

I got a call from our credit card company who inform us we have been charged about $870 in total so far 😭

We have barred making calls to Moldova and some other regions scammers had been targeting using Twilio's GeoPermissions

By default, Twilio blocks voice call to some high-risk areas.

Twilio blocking high risk areas

And this is us blocking Moldova.

Alt Text

All in all, about 2400+ phone calls were made so far.


7:45 AM 🤖

Integrate Google reCaptcha on the dashboard. Block spam user's accounts. I was personally very sceptical if things would go back to normal.

When you do this, make sure to authorise the request on the backend too.


8:00 AM 🚫

So far, I have made about 20 deployments to production, thanks to GitHub Actions for automating that.

Replace phone call verification with SMS verification. This took some time along with testing.


9:00 AM 📖

Revoke all Twilio tokens and read their guide again on toll frauds. Are we missing anything?

Facepalm - Twilio has explicitly written in their docs to not use a phone call for verification via OTP and that's exactly what we did 🤦‍♀️

We released some of our old phone numbers and bought new ones. This didn't take long.


10:00 AM 🧟‍♂️

Deploy to production but we still see a number of signups coming in via bots using Mailinator email addresses.


11:00 AM 🕵️‍♀️

Our search for a service API to tell disposable email addresses apart has begun.

Shortlisted:

  1. https://block-temporary-email.com
  2. Kickbox.com
  3. NameAPI
  4. SO question on blocking disposable email addresses

The problem is creating newer disposable email addresses is easy. It's perhaps too much to expect that one service would cover all of them.

Interesting stat - A ton of our users were using Google OAuth, so we made the bold move.

Allow only Google OAuth. Remove login and signup using Email/Password combinations.


1:00 PM 👮‍♂️

Time to take extra measures -

Cloudflare is pretty awesome at this. When you visit Spike.sh, Cloudflare will try to determine if you are human or a bot. Sometimes, a test also appears for verification using hcaptcha.

Alt Text
  • We installed Needle.sh, which provides a security layer with their NPM module. In their dashboard, we noticed that security scanners have been probing our web properties for vulnerabilities multiple times a day. This made us realise the importance of using security tools for our web products from an early stage, obviously.
Alt Text

Rate limiting was super helpful and easy to setup. I have seen tons of 429 response status after we activated this.

Setup usage triggers on twilio


3:00 PM 🤞

All of our measures are now live. Now - it's all about monitoring carefully.

Scammers were unable to create new accounts. Yipee!
Some literally created accounts using Google OAuth and tried to make more calls but over 99% of them ended up in either 401 or 429 response codes.

What did we learn?

  • Setup critical alerts, had I not gotten the phone call from Spike.sh, I easily would have gotten a bill of more than $5000 by the time I woke up.
  • Security is super important. Never too early to start.
  • Use the right tools for the right job, we are using Cloudflare and Needle.sh
  • Better safe than sorry

Resources

These are some good reads I came across during the entire ordeal. Hope this helps you -

This doesn't cover all the steps we have taken to stop abuse or DDoS.

Cover photo by Hugo Jehanne on Unsplash