Unveiling Cloud Cost Anomaly Detection from Xi Beam

By Harold Bell
| min

Cloud cost anomaly detection is a vital tool in keeping your cloud costs under control. To put it in perspective, balancing your own budget isn’t always easy – even if you’re the most frugal person you know. Every now and then you may treat yourself to a new shirt or a nice meal from somewhere pricey. Occasionally you’ll find yourself hanging out a few more nights than you planned to. Unfortunately, there’s usually one thing that’s never on your mind when you splurge – how that will impact paying for things you actually need. That awareness hits when you’re staring at your credit card statement trying to remember how many drinks you had Thursday night. By that point my friend, the train has already left the station.

Oddly enough, balancing your cloud spend is very similar. Instead of treating yourself to material goods or libations, your organization periodically treats itself to new cloud applications. The development, testing, and configuration of these applications can lead to some very real financial consequences if not executed correctly. If you are an IT manager or cloud architect, it’s likely you’ve encountered one of the following scenarios:

  1. Your Dev team forgot to turn off compute instances that they are no longer using
  2. You’re using pay-as-you-go instances and not effectively utilizing the option to reserve instances that will be used for a long period of time
  3. You have no idea who spun up what, but you are still paying at the end of the month
  4. Your pricing plans were changed by the cloud provider and you missed the announcement
  5. You budgeted a certain amount for a new project but turns out you need more cloud resources for it. Your engineering team decides to go ahead with over-provisioning without adequate checks and balances in place

Industry leading analysts predict that organizations without a robust cost optimization strategy could end up overpaying by as much as 40%.

So, what do you do?

You could strengthen your IT staff. I mean, who wouldn’t want to hire more public cloud architects and app developers to guarantee each setting is compliant? The only problem there is, nobody has the money to do that – nor is it a full proof response. Let’s say onboarding your cloud dream team was feasible, not all spikes in cloud spend are due to internal human error. There are also security related reasons that may explain a sudden increase in your bill. These situations involve bad actors who infiltrate your infrastructure and utilize your resources for malicious activities. So unfortunately, hiring 100 people with AWS Certified Solutions Architect – Professional certifications wouldn’t help in the slightest.

Early detection is the answer

Before we get into the details of how to solve this, lets first identify what “this” is. The expensive mistake I just described is what’s known as a cloud cost anomaly. And just as unpredictable as when they occur, there is the unlimited financial impact if left uncorrected too long. Your protection from spend anomalies lies in the ability to continuously monitor how your budget is being utilized in the cloud. Your management platform must be able to detect spend activity that doesn’t align to normal spending patterns. Once an anomaly is identified, you will also need visibility into what happened so you can prevent it going forward. Not surprisingly at this point, your answer to a balanced budget is real-time cloud cost anomaly detection.

Cloud Cost Anomaly Detection 101

Cloud cost anomaly detection is vital for environments that are dynamic in nature, especially as your cloud environment scales. It works by analyzing the trend of spending on the resources in your environment. Essentially activity that doesn’t align to your anticipated spending behavior is identified for its deviation from the established patterns. You can also get insight into which resources are causing the anomalies to take corrective actions. For instance, optimizing or shutting down resources that are not legitimate. The secret sauce behind detection in modern times leverages an emerging discipline within data science – machine learning.
To get a little more technical, deviations are officially known as outliers. Outliers…anomalies…. same difference. There are three fundamental categories to describe the unusual behavior:

  • Point anomalies: these are single instances of deviations that heavily contrast from the normal patterns in the larger data set.
  • Contextual anomalies: these are generally seasonal deviations that occur during specific times that call for modified spending – like the Christmas holiday or tax season.
  • Collective anomalies: these occur when a single outlier doesn’t possess a stark deviation from the established patterns in the data set, but two or more of these “moderate” outlier’s alert cause for concern.

Now that you know what anomalies are, let’s talk about considerations in evaluating how well a platform can detect anomalies.

  • The data itself: This is a no-brainer but still discussion worthy. Aside from the ability to collect data, does the platform collect the right data? From every appropriate source? And is the available collected in real-time or is there a delay in reporting? These are all questions you must ask to gauge how accurate and effective a platform will be monitoring cloud spend.
  • An appropriate model: When considering contextual and collective anomalies, the model being used must address seasonal and trend patterns. Otherwise, if only utilizing normal behavior data, the tool is essentially useless. Contextual and collective anomalies can only be detected when the model accounts for changes in season and trend.
  • Competency in causal relationships: Considering the breadth of contextual anomalies, you must possess the ability to identify the relationships between data sets. Models must include algorithms that learn these relationships for detection to be effective.

If you find a solution that checks those boxes, then it’s a highly favorable investment in my eyes. What makes a solid solution will be its ability to handle those three variables in real-time. Cloud cost anomaly detection can be executed through manual processes – which is not a bad thing. However, in order to stay abreast of your environment, you need a solution that can alert you when cost anomalies occur. Unexpected increases in cloud spend can heavily influence spend for other critical business functions. It may even be the reason you forgo making any additional investments into your cloud infrastructure. So, a solution that can keep you from either one of those scenarios, has unquantifiable value. Receiving timely alerts can help you take corrective actions to remain within budget and arguably still employed. Now only if you could’ve had something like this last week at the bar!

Announcing Automated Cloud Cost Anomaly Detection for Xi Beam

To start your year off right, Nutanix has launched an automated cloud cost anomaly detection feature for Beam. With this feature, Beam will be able to:

  • Automatically analyze historical data
  • Establish a baseline of cloud spend
  • Track current and future spending
  • Alert the user when a cost anomaly is detected

Previously, Beam was able to provide cost anomaly detection, but it was a manual process that required users to set up several components. Now, by leveraging machine-learning algorithms, we are automating the process and enabling real-time detection. Beam will also outline the reasons for the cost anomaly so you can resolve the root cause(s). The automated cloud cost anomaly detection feature is available to all Beam customers as of today.

Finally, what should you be doing?

With all the utility that cloud computing provides, it’s hard to fathom that managing public cloud spend would be so hard. But considering the multiple regions, accounts and cloud services, it adds up to be quite a feat. If you’re already a Beam customer, then I’m happy you stuck with me to the and I hope you enjoy the new feature. But to those of you reading who aren’t using a cloud cost optimization platform, or you’re dissatisfied with your provider, now is the time to act. I’d like to give you an opportunity to experience how Beam can directly impact your bottom line. If you’re interested, enjoy a free 14-day trial of the platform to test it out. What do you have to lose?

© 2019 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and the other Nutanix products and features mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).