Hey there! 👋 Thank you for your interest in my article.

Today I will talk about how I optimized an application to save thousands of $$$/month in operational costs. 🔧💸

Let me give you a bit of a back story 📜.

A few months ago, when the COVID-19 epidemic spread around the world, I was tasked by Adeva, to create a data gathering portal for one of the worlds largest medical research organizations. The data will help scientists to understand and better prepare for the virus.

But, there was an issue.

The application must lunch as soon as possible. The virus won't wait for anyone.

So, we managed to develop a full application in under 2 weeks (crazy, right? 🤯) and lunched helpstopcovid19.com.

Once the marketing campaign kicked in, thousands of users started using our application. 🚀

The problem

The application is scheduled to send email and sms notifications to users based on actions they take in the system.

For example (there is a new task to complete, or a reminder for incomplete tasks, and so on...)

We fetched all users and checked if they need to receive a notification.

$users = User::all();

foreach ($users as $user) {
    if ($user->hasNewTask()) {
        $user->notify();
    }
    if ($user->shouldReceiveReminder()) {
        $user->notify();
    }
    ...
}

This job runs every 5 minutes and with a database of thousands of users, you can imagine how much CPU was used in the process. 🔥

Luckily, we had an expensive database server that could handle the requests, but it costs thousands of USD/month to operate. And obviously, this won't scale to millions of users.

The solution

My idea to solve the problem is simple.

Instead of fetching all users and checking each one, we need to query only the users that we need.

$users = User::all();

foreach ($users as $user) {
    if ($user->hasNewTask()) {
        $user->notify();
    }
    if ($user->shouldReceiveReminder()) {
        $user->notify();
    }
    ...
}

$users = User::where('has_new_task', true)->get();
foreach ($users as $user) {
    $user->notify();
}
  
$users = User::where('last_reminder_at', '<', $dateToSend)->get();
foreach ($users as $user) {
    $user->notify();
}
...

This will dramatically reduce the memory consumption and prevent "lazy loading" data for each user.

However, this was easier said than done, because we had a lot of different notifications, with a lot of variables that trigger the notification.

I had to extract all the variables that we need, and add columns on the users table than can be easily queried. I call these table columns, flags 🏁.

Now, those flags need to be properly set, so I added a migration that will initially loop all users and set them.

Finally, I need to implement adjustments of the flags at various points in the application. In other words, the entire system is going to be affected.

Deployment

This change has the potential to break the system. If there are any bugs or errors 🐛, the system can start sending notifications out of control.

So, I decided to release the solution in dark mode. The system will migrate the flags and start adjusting them, without changing the notifications job.

Also, I implemented a command that will check the flags and log any errors. I wanted to make sure if the system finds any inconsistency in the production data I can see a log of the errors and fix them before we release the new notifications job.

After completely testing everything and fixing all bugs, I deployed a new release and the new notifications job took over.

The results were extraordinary. 🤩

The database CPU utilization was reduced from 40-50% to 1-2%.

Final thoughts

Thank you for taking the time to read my article. I appreciate it.

P.S. If you are a developer looking to work on exciting projects, you can use my referral link to register at Adeva - an exclusive developers network.

Cheers 🍺