The serviceability of any product relies on four fundamental actions: (i) observe, (ii) inform, (iii) fix, and (iv) learn. While most products attempt to implement these in some form or another, very few get it right. Today I would like to share with you the secret sauce required to build the most serviceable system in the world.

1. Reliable observation. This is toughest problem and also the easiest to overlook. The component that observes failures, should by definition, be the most reliable component in the system. It needs to work even when all other things are failing. In today’s complex distributed systems, it needs to be truly highly available, it needs to be resistant to network partitions, and it needs to record data reliably. For high availability and partition tolerance, Nutanix has had great success with Apache Zookeeper. And for data storage, any reliable database is fine. We use a NoSQL database with a replication factor of 3. Go this extra mile and you will never miss a failure.

2. No-noise Information. While it is fairly easy to get systems to “call home” when things go wrong, it is both science and art to make the call homes precise and meaningful. First of all, never send multiple emails for the same underlying problem, even if it means comparing stacktraces in core dumps, or doing automated root cause analysis on the box itself. In fact, we designed a 2-step call home on any error. In the first step the cluster contacts Nutanix with the error code and consults a database of current issues. If there is a match, then the real call home is sent with an analysis of the problem. Our L1 support could see a call home that reads “Hit bug # 3212. Version 2.5.0-stable has a fix for this”. The primary and relentless goal should be to make the support organization highly efficient by improving the quality of the information flowing in from the customer machines. For cross-customer repeat problems, this reduces resolution time from hours to just minutes.

3. Live analysis. Never (or rarely) ship logs out from customer machines for troubleshooting. In my opinion, that is just a lame excuse for an on-call developer to have an extended lunch break. All debugging and analysis tools need to be available in real-time on the live box. The goal should be to resolve the issue before the customer finishes his lunch. To make this happen, we use (i) “reverse tunnels” that can create an on-demand VPN from the system to the Nutanix Service Center with the click of a button, (ii) distributed log viewing and analysis right on the box, (iii) on-demand cluster-wide live and historical stats visualization, (iv) click-of-a-button problem analysis tools on the box, (v) live, on-demand, debugging tool chest upgrade (vi) and adherence to “be nice” philosophy which means that troubleshooting should only consume unused cpu and memory resources in the system. None of these are trivial to implement, but the founders of Nutanix decided to build a product for the long haul, and the early investments are bearing fruit today.

4. Adaptation — the anti-virus model. Think of bugs like viruses. Your job is to limit the number of customers who get affected by the same bug. A serviceability system that only improves when you upgrade the entire system software, is quite useless. It is like an anti-virus software which does not update its virus definition files everyday, but only when you buy the next version of the software. You need to design the serviceability component of your system on the design principles of anti-virus software. What is monitored, how it is monitored and what actions are taken, should be constantly updated from as the community learns about the system. At Nutanix, we designed such a component called Aegis that enables serviceability to constantly learn (with permission from the admin) how to monitor the system better. Careful design keeps it completely isolated from the I/O path. If a problem occurs that has happened at other customer site, Aegis might already know how to fix it through pre-canned fix procedures leading to truly zero service times. If the error is in the I/O path, Aegis will inform the customer with a very meaningful message and the course of action.

Nutanix strives to adhere to each of the serviceability principles I’ve outlined, to become a leader in next-generation serviceability, one of the key concerns of datacenters both large and small.

This will be the first post in our new series, “Meet Nutanix,” where various employees at Nutanix will be featured. Read on and learn more about life at Nutanix and what various departments here are working on!

Name
Laura Jordana

Title/Position
System Reliability Engineer

What’s your background?

I started my career as a QA engineer at a small public company, and then moved on to a product support role at a software startup before coming to Nutanix. I enjoy the support role because it allows me to interact with customers and see how our product is being used in the field, and allows me to work with the different technologies that our customers use.

Why did you choose Nutanix?

I wanted to be a part of a great product that is changing the way the industry uses virtualization, and wanted to be challenged and inspired in my work. So far, working at Nutanix has met and exceeded my expectations (in a good way!).

What is your favorite part about working here?

Everyone is talented, friendly, and extremely helpful. When any issue arises, internally or otherwise, engineering does not hesitate to jump in and help solve the problem in any way they can. Also, ping pong and beer tastings on Fridays are always fun!

What does your job entail?

My job is to ensure the customer has a great experience while using the Nutanix product. It also involves improving and automating support processes to ensure our customers are getting the best service, and relaying product feedback from the field to our product team to improve the product.

Walk through of a day on the job

On a day to day basis there may be e-mails and calls that come from customers. These could be either actual product issues, configuration issues, or simple questions on how to do a certain thing in the product. I also help out with writing internal documentation for our team, and sometimes work on various projects in collaboration with the documentation or QA team.

What are some interesting challenges that have come up?

I have constantly been challenged and inspired while working here. I had only a little experience with virtualization when I first joined. So it was challenging to learn both the Nutanix product and how virtualization works, and how they work together. It has been a fantastic learning experience!

What has been your favorite project?

One of my favorite projects was when we recently worked on an upgrade of a 32 node cluster, as I had not done an upgrade of that size before and I learned a lot from the experience.

What are your future plans?

To continue growing along with the company, and to continue to help make our product even more amazing while ensuring that our users have a smooth experience.

Something interesting about you?

I love traveling! When I was a teenager I was part of a dance company and every summer we would travel to different places and do shows. I got to visit UK, France & Italy before I was 16. I have family in Spain, Australia and the Philippines and have been to the latter two (and hoping to visit Spain soon!)

What do you enjoy doing after work?

Anything active (taking classes at the gym, biking, jogging, snowboarding), hanging out with my friends and family, trying new restaurants, traveling

The Network is the Bottleneck

Cloud Computing is being considered a nirvana for enterprise computing and SMB computing alike. Service-oriented IT — both Software-as-a-Service (SaaS) and Infrastructure-as-a-Service (IaaS) — are a seminal innovation of the last decade. Broad adoption of Salesforce.com, Amazon AWS, and Google Apps, etc. underscores how important the notion of a “service” will become in this new decade. At the core of a service is “user experience,” and public cloud vendors are scrambling to improve that each passing day. SaaS vendors continue to emphasize usability and website performance. IaaS vendors are scratching the surface of user experience by deploying data centers worldwide, replicating data geographically and thus closer to the user, leasing transcontinental network cables, and renting CDNs (think Akamai) for mostly static content. The name of the game is to push content and computing as close to the user as possible. WAN is public cloud’s biggest enemy.

Private clouds are an equally important piece in the cloud computing puzzle. Enterprises and government agencies that want better control over data and computing — security, compliance, and overall quality of service — will continue to implement their own clouds. User experience — or the ability to push data and computing closer to the user — is as critical to the distributed (geographically dispersed) enterprise as it is to public clouds. But unlike the big service providers, most enterprises do not have the economies of scale to lease private network links or dictate good quality of service from CDNs. Notwithstanding all the WAN optimization innovation of the last decade, latency is a user experience killer. Satellite and branch office survivability is an important service goal, but extremely expensive to achieve because of the oligopoly of network service providers who charge exorbitant rates for the failover link to the corporate hub. The private WAN is an even bigger enemy of the private cloud.

SANs at the edge are overbearing

The leased line to the corporate hub is a bane, given the latency-sensitive and bandwidth-hogging workloads of this decade, e.g., rich media, virtual desktops, etc. Distributed enterprises have had a bittersweet experience with server rooms in satellite offices. They were clunky, ever-growing, and extremely hard to manage remotely from headquarters. Server room equipment was no different than that in large central data centers — unwieldy in form factor, power-guzzling, HVAC-hungry, noisy, and cluttered with haywire cables implementing complex networks. A big reason for this expansive clutter was the overbearing compute-network-storage architecture in which different pieces of the computing and storage puzzle fit in separate dedicated hardware. Quite reminiscent of the personal computing era when we used a cellphone for calling, a PDA for personal organization, a media player for music, a camera for photography, and a PC for emails, word processing, and web browsing. Then came the Apple iPhone. The rest is history.

An “iPhone” for Enterprise Computing

The private cloud at the edge needs an “iPhone-like” revolution that converges compute, network, and storage into one server-room-friendly equipment that is 5x smaller in form factor, extremely portable, green, content-rich for unparalleled user experience, and backed up to and easily managed from the “iCloud”. There is no separate SAN or NAS. There is no separate blade chassis. There is no need for hundreds of spindles of storage. There is survivability, in case the “3G” network is misbehaving. There is fault isolation because two “iPhones” don’t fail together. There is an “iOS” experience when managing individual devices at the edge. There is “Bonjour” to detect neighboring machines. There is impeccable serviceability with “auto downloads”. Most importantly, there is an “iCloud-like” corporate hub from where these “iPhones” can be managed, secured, and disaster-protected.

We need an Apple revolution in enterprise computing, period.

And that would be true homage to the genius who changed our lives forever.

Last month, a group of business students from Meiji University visited Nutanix HQ to learn about startup culture in Silicon Valley. These students are attending an International Gateways program. Before I joined my previous employer as a training developer, I taught in the TOEFL preparation program at International Gateways. Since I left there, I have kept in touch with my former colleagues and have been privileged to speak to a number of classes.

As their teacher and I planned their visit, the students were instructed to visit the Nutanix web site. When they arrived, the first activity was a quiz about the company; answers could be found on the web site. Students who correctly answered a question chose a Nutanix water bottle or T-shirt as a prize.

Once the quiz was finished, I gave the students an overview of traditional computer architecture and how virtualization differs from that model. Then on to the ugly part: while virtualization solves a lot of business and technical problems, the complexity and cost of managing storage and servers and hypervisors and the network to tie them all together cause a host of new problems. This is the problem for which Nutanix has the solution. At this point I opened up a Nutanix block I happened to have and showed the students the hardware.

With the basics out of the way, I took the floor and talked about the market opportunity and the kind of successes we have been experiencing. While virtualization architecture is outside the specialty of the students, the potential for a 40-60% CAPEX reduction in a smaller box relative to other solutions in the market is not. I finished by explaining the four domains where Nutanix has been found to be most appropriate: VDI, private cloud, disaster recovery, and big data. Naturally they were curious if any of our systems are in Japan, and we were happy to be able to say that there are, at a university and a telecommunications company.

After I went, Member of Technical Staff Chern Yih Cheah spoke to the students about working at a startup and how it differs from a large company like Oracle, where he worked before. Chern likes knowing everyone in the company and having better awareness of how the product is being used in the field, both of which are not possible at larger companies. Although the risk is larger at a startup, the benefits and rewards may be as well.

The students concluded their visit by joining the company for lunch, where they were able to talk with the employees and even the CEO Dheeraj Pandey. They were impressed by the casual dress and interaction. I think that drove home for them what working at a startup is really about: get the work done and don’t let peripheral concerns interfere.

US Federal agencies enter calendar 2012 with decreasing budgets and an increasing push to adopt cloud technologies.  I was honored to spend the last several years helping to implement cost-saving technologies, such as virtualization, at various Federal agencies, including both CONUS and OCONUS.  When I was first introduced to the Nutanix converged virtualization appliance (CVA), I felt it would be revolutionary for Federal datacenters, and wrote about my views in this blog post.

For those familiar with virtualization, Nutanix’s 2U CVA has four ESX hosts, each with their own FusionIO card, SATA SSD, and SATA spinning disks.  The Nutanix CVA has the horsepower to support hundreds of virtual machine instances, whether server or virtual desktop based.  For those whose support Tactical installations where a reduced footprint is of paramount importance, the Nutanix 2U appliance is a strong candidate.  When considering mobile solutions, solutions on tactical vehicles, solutions dropped from planes, or fly-away kits, the reduction in weight, power, and size that Nutanix offers is compelling.

Not every agency’s mission is to support these type of deployments, however, but the lessons and requirements from the Tactical space are still relevant in the largest of Federal datacenters.  For those with a large Network Enterprise Center (NEC), space, power, and cooling are still important.  While the parameters may be less stringent than deployed stacks, every square foot that is occupied has a cost associated with it.  Datacenters, no matter how large, have a finite amount of power capacity.  I recall being engaged at an agency several years ago where physical servers had to be powered off before new ones could be powered on.

How do you avoid or prolong such a scenario?  High-density virtualization.  High density virtualization is likely impossible in the legacy server virtualization designs, where rackmount servers or large blade chassis are connected to multi-U switch solutions which then connect to half or full rack storage arrays.  While these legacy building blocks do indeed offer the technical requirements for virtualization, they do not offer an efficient model of scale.  Scaling linearly in a 2U appliance is far more efficient than disproportional scale in 12-42U building blocks from disparate manufacturers.

On-premise cloud computing is another big driver this year, as Federal datacenters consolidate and work to deliver increased levels of self-service and automation.  With multi-tenant licensing considerations to be had, it’s extremely important that the underlying building blocks for Federal on-prem clouds are as small and cost effective as possible.  I believe this is another area in which Nutanix’s 2U converged virtualization appliance will be a strong fit, as it’s architecture is modeled after some of the largest names in cloud computing, like Google and Facebook.

I’m excited to work with the Federal community this year as they further their adoption of virtualization, VDI, on-prem cloud computing, hadoop, and mobile application management.  All of these initiatives will require a cost effective model of scale for the underlying compute and storage; all of these initiatives will also help further the mission of their respective agencies.  The budgetary concerns of this Federal fiscal year means that technologies must provide high value and a low cost of entry; legacy low-density and high-cost infrastructure will be a thing of the past.

Big Data and Virtualization are the two biggest revolutions underway in enterprise data centers. In the coming 3 to 5 years, they would have unquestionably changed the face of datacenters as we know them today. Interestingly, both these phenomena have been going on in parallel, with little to no intersection or impact on one other. That is about to change, as the waves of the hitherto independent “ripples” intersect, overlap, and create a massive cadence to shake enterprise IT even further.

This essay explores how the frontiers of these two ripples are about to meet, irreversibly changing each other’s fate along the way. Over the course of the next few years, they would become one massive ripple that couldn’t be traced back to two disparate epicenters of change. That is how interesting yet predictable this saga would be.

The Great Public-Private Divide: Amazon chargebacks keep them honest 

The biggest player in Big Data, Hadoop, is 100% virtualized when run in public clouds. Amazon’s Elastic Map Reduce (EMR) and every EC2/S3 customer that runs Hadoop jobs is running in a virtual machine container. True elasticity of Big Data jobs, in which machines can be added or removed programmatically, can only be attained by leveraging OS virtualization. True economies of scale — where multiple jobs can be run simultaneously and yet sandboxed from each other and from errant runaway jobs — can only be attained by leveraging the sandboxed “isolation” of a virtual machine.

Private cloud Hadoop, on the other hand, is running amuck with very little regard to hardware utilization and overall capital spend. Developers and scientists, who’ve never had to budget capex and opex before, are ordering rack-mounted servers like there is no tomorrow. These servers are running at 10-20% utilization because of lack of virtualization. IT is a mute spectator because they don’t really understand the Big Data phenomenon well enough to bring best practices to massive hardware clusters. And now, DevOps wants to pass the messy maintenance and monitoring of Hadoop clusters to IT, as they get impatient carrying pagers, replacing failed disks and network cards, and monitoring long-running data-crunching jobs.

So why this divide? Why are data scientists so careful optimizing hardware and emphasizing elasticity when running in public clouds than when they are running internally. Well, the answer is simple — Amazon chargebacks are for real, while IT’s are artificial. When real money changes hands, people automate, they share, they optimize, and they focus on system utilization. That time has come for private cloud environments. Private clouds will only thrive if they bring the same efficiencies as their public counterparts. Otherwise, CIO’s will see their teams shrink, as the efficiency divide between the two clouds grows.

Revenge of The Statistician: Can VDI keep them relevant?

The world has come to (synonymously) associate big data with Hadoop and data scientists with a new breed of Java-centric developers. In the enterprise though, data analytics will continue to be driven by a potent combination of this new Hadoop developer and the traditional statistician who uses SAS, R, OLAP cubes, and other tools that were built for the “desktop-centric modeler”. While statistical laws of sampling make it possible for these data scientists to develop models using their favorite desktop tool, Big Data is pushing the envelope on this conventional paradigm of data science and computing. The statistician’s laptop is neither secure nor elastic to handle demands of larger samples, iterative/agile modeling, and capturing outliers for fraud analytics.

The erstwhile data scientist’s role is not diminishing in any way, Hadoop notwithstanding. Their business domain experience are here to last; their Windows-based tools are here to coexist with Hadoop. The skill sets will blend over time with subject matter experts on either side. Interestingly, the biggest enemy of the statistician is not Hadoop but their physical desktop with its loose security and meager hardware resources. A desktop in the cloud, leveraging the strengths of cloud computing and virtualization, could be their biggest friend in the coming years. With VDI, IT plugs numerous security holes, as data never leaves the datacenter. With VDI, IT provides a truly elastic computing model for scientists, as their data processing demands grow and shrink.

Big data, bigger data: Whither Storage Virtualization?

While OS virtualization is beginning to come up quite often in the context of big data, storage virtualization is dangerously missing from the conversations. Amazon EC2 users continue to scratch the surface of storage tiering by shuttling massive amounts of data back-n-forth between the inexpensive (yet reliable!) S3 and the more expensive (and yet unreliable!) EC2 and EBS storage. But that manual process is ugly and kludgy, to say the least. In private clouds, the situation is even worse. There is literally no talk of virtualizing Hadoop’s storage onto various tiers of the enterprise storage stack. Non-Hadoop MPP platforms have an even more telling story. None of EMC Greenplum or HP Vertica or Teradata Aster have a distributed file system — unlike Hadoop HDFS — such that individual nodes can spill over or fail over into each other; or if disks fail or get full, the user queries continue to run.

There is so much simplicity to be had in the big data world by separating the storage logic from the data analytics logic. There is so much value to be had by transparently leveraging fast SSDs for NoSQL or random IO workloads; or transparently moving data between the more expensive server-attached tiers and the less expensive network-attached storage (archive) tiers. The big data stack is sorely missing storage virtualization, and things are beginning to hurt already. Ask any Amazon EC2-based company that shuttles data between S3 and their EC2 tier every morning and night when they spin up and down their Hadoop clusters!

Orchestration and Cloud Directors: Whither Big Data?

You’ve come this far reading this essay. I hope you and I are beginning to agree on how virtualization and cloud computing help big data in the enterprise. But what about the other way around? Is virtualization missing a big Big Data component? If you look at the embedded SQL databases underneath the enterprise management servers for virtualization, you will know what I mean. Most control plane products had SQL developers who understood how to write monitoring and system management applications with SQL as the cornerstone for persistence and querying. That is hurting us real bad today, what with fine-grained statistics for performance monitoring and chargebacks, and some very agile provisioning decisions that businesses have to make in today’s dynamic datacenter environment. The embedded SQL database in a management server has become the veritable tail that is wagging the orchestration dog. Virtualization administrators are having to painfully learn how to manage and debug unscalable SQL database environments, and make suboptimal decisions on stats gathering and chargebacks. Most of them don’t even know that their virtualization environments are slow or unreliable because of that pesky transaction log in that hidden 800-pound gorilla embedded in their vCenter server!

Orchestration, fine-grained chargeback, and performance debugging in multi-tenant cloud environments will continue to suffer as long as management products take a narrow SQL (and strictly transactional/ACID) view of persistence. Big data and NoSQL, with their minds free of the SQL and ACID-clutter, are the real need of the hour for large-scale cloud environments.

Big Virtualization’s Biggest Nemesis: Storage

We all know by now that virtualization over-promised and under-delivered vis-à-vis costs, complexity, and performance. And storage was the culprit. Network storage was built for physical environments, and is inarguably out-of-place and out-of-time in today’s highly dynamic virtualization environments. Solid-state-drives make storage networking and spindle-based storage controllers look archaic. Fetching data that is sitting multiple hops away in a separate storage aisle is making virtual machines crawl. VDI projects are running into rough weather because the legacy storage underneath cannot predictably scale between the pilot proof-of-concept and the real production scale.

So are there any lessons from the big data world for “big virtualization”? Turns out, scaling and simplifying large virtual environments share a solution that is at the heart of big data: collapsing the wall between compute and storage, and bringing storage as close to compute as possible. Now that is an epiphany moment! Storage networking and its massive complexity is the bane of virtualization and multi-tenant cloud environments. We need dramatically simpler architectures that beautifully scale out with time. We need a virtualized data center to resemble a Hadoop cluster, or a Google cluster, or a Facebook shared-nothing cluster powered completely by a FusionIO tier. We need enterprise data centers to look no different than those of the big cloud companies. We need convergence,period.

The Tale of The Two Ripples

While big data and virtualization originated in isolation, they share more in common than it appears on the surface. The two ripples will seamlessly meld into one…

We’ll be discussing two UI elements in this blog: Alerts and Statistics.

Alerts

Our design philosophy is UI alert by exception. We don’t display a “normal” status label because in reality, we don’t expect the admin to look at the UI and check if every entity is healthy. We only show an alert if needed. The alert system is easy to see and catches your attention right away. We make sure that alerts are not too plentiful, cluttering the screen. We try to keep them organized. Here’s a glimpse of our alert system:

Alerts appear on top. When users click the alert icon, a drop down appears. We included action buttons for users so they can click to indicate that a notification has been addressed.

Our object-oriented UI also shows the alerts on the icon and in the context tray menu. The example above shows that the disk is bad.

We also show alerts on the datagrid right away with specific details.

Statistics

Statistical graphs are an important part of a system for troubleshooting and diagnostics problem solving. We made different charts for various metrics we thought would be important for troubleshooting:

1. Usage Chart

We’ve made a Google Finance-like charting system that allows you to slide through the history of the storage usage of hosts, disks, containers and storage pools. Users can select different time frames to look, zooming in and out of the graph for the level of granularity that is needed for either diagnosing the problem or just tracking the storage usage history.

2. ILM Chart (Information Lifecycle Management)

Our ILM charts split into IOPS and the different drives (SSD and HDD) in the system.

3. Provisioned Capacity and Usage Beaker Chart

This one shows the provisioned capacity of the system as well as actual usage.

Hope you enjoyed this sneak peek into the new alerts and stats functionality coming in our next UI release, and we look forward to sharing more ways we’re trying to empower the virtualization admin!

Josh O’Brien on Nutanix: “So far I love the platform.  It gave me what I needed in the price point I needed and offers huge scale out options considering it is based of the GFS files system that Google uses across their DC’s… Spinning up my first production ESXi Cluster of (4) Servers, Sphere and provisioning the storage was a bit nerve-racking.  Its a HUGE statement for all the vendors that it was actually pretty painless.  Maybe even a bit to much so.”

Josh O’Brien from Language Access Network wrote about installing Nutanix last week. Read about it here: http://www.staticnat.com/WP/2012/02/07/fake-it-till-ya-make-it/#more-516

What we, at Nutanix, would like to point out from this post is that Josh referred to us as partners. Building a long-term relationship with our customers and offering world-class customer support remains one of our biggest goals.

Look forward to a case study in the coming weeks on LAN and using Nutanix for building a private cloud!

http://www.youtube.com/watch?v=p3Qq6QU1WOk