Once upon a time, we asked you what your favorite binge-worthy Netflix show was. Many contenders battled it out, but Stranger Things led the pack. So, it’s safe to say IT has a bit of a fascination with the mysterious (and seriously scary) world of the Upside Down. And with season three here, we’re amping up to binge even more.
And while we won’t find any demogorgons in our datacenters any time soon, we might find cable-eating rodents.
If your datacenter is starting to feel a bit like the Upside Down (cue eerie music), you’re definitely not alone—and thankfully, you won’t need Eleven’s powers to banish your IT monsters.
What exactly is there to fear in the datacenter, you may ask? Here are some of the strangest stories we’ve heard.
Curiously Cataclysmic Critters
Rats, spiders, roaches—all the things you expect to find in a run-down home built in the ‘20s. But surprisingly, the datacenter is full of cable-gnawing, fear-inducing pests. Here are just a few horror stories.
Hungry, Hungry Rats
One hospital employee not-so-fondly remembers a time when a hungry rodent took down an electrical grid by chewing through the lines. Though the uninterruptible power supply (UPS) kicked in, the generator didn’t. During the 30-hour debacle, the hospital had to cancel all non-urgent appointments until the following Monday.
Takin’ a Dip
In other chew-worthy news, one small East Coast company found themselves facing a disaster some winters ago. Their Nutanix box was inexplicably soaked, prompting them to investigate the matter. The cause? Rodents had gnawed through their sprinkler system, leaking water all over the box. Luckily, Nutanix customer service overnighted a box and sent a support engineer to install it. Whew!
Squirrels, Snakes, and Scares!
But while rats are always the first pests to blame, lesser-known datacenter critters don’t get a free pass to cause mayhem. One customer reports that their campus had been terrorized by a single menacing squirrel who kept eating the fiber lines between their buildings. And to make matters worse, this squirrel was a master escape artist, miraculously evading capture.
And finally, we’ll leave you with this: Snakes under the floorboards. ‘Nuff said.
The Axeman of the Datacenter (And Other Strange Folks)
Is your IT team starting to get a case of cabin fever—or datacenter fever? Here are some cautionary tales of human error (or human oddities).
Security, We’ve got a Situation...
Let’s start off with a bang. It’s not everyday security steps in to handle a situation in a datacenter. But years ago, one organization in Las Vegas had to do just that. So, what was the cause of the ruckus? When security arrived, they found a naked man, armed with a pick axe he had smuggled in, hacking away at the company’s data library. (Far less charming than Stranger Things’ Steve and his nailed-up baseball bat). In the aftermath, the company only lost their backups. Had he gone after the production servers, the situation would have been far worse (though not necessarily stranger).
If you think one security scene is bad enough, here’s one more. On the first day of his new job, one enterprise architect walked into the datacenter to find yellow police tape blocking off the heat aisle of every row. His immediate thought? A datacenter murderer! Turns out that the building generator was offline, so portable UPS units were installed in the meantime. Oh, and the police tape? They didn’t want staff to trip over exposed power cables. (Don’t worry, we’ve got tales of datacenter clumsiness coming up).
“Congratulations, You’ve Won!”
Now, let’s move on to the wild (but hilarious) stories of human error. One administrator reports that when one of their users plugged her phone into her workstation to charge, she noticed 3 files on her desktop that hadn’t been there before. Here’s where you might be thinking, “Oh no! Don’t open it!” She clicked on one of the folders, and a slew of uncontrolled spamming begun. While the IT team was able to contain it, they did have to fight for a week to get their company’s URL off the blacklist!
Have You Tried Turning it Off and On Again?
It goes without saying that small mistakes can have wacky consequences. One company reports that their laptops and PCs lost their domain connections and started experiencing a bunch of other weird effects...just from someone plugging a data cable from their phone VLAN to their corporate VLAN, creating a loop. It took hours to track down the issue, but luckily, unplugging the cable brought everything back to normal.
What Does This Button Do?
Other note-worthy stories include one trigger-happy person who couldn’t resist pushing the red emergency UPS shut-off button, another person accidentally leaning back and bumping into a power switch that shut off a whole row of servers, and a clumsy fellow who fell in a tile hole in the datacenter twice—in the same day. Maybe police tape could’ve saved him!
But sometimes, things inexplicably go haywire in the datacenter, and that’s when you have to call in the tech exorcists.
Who You Gonna Call? Tech-Busters!
From old houses to historic prisons to abandoned hospitals, ghosts are notorious in a lot of spooky, tourist-worthy destinations. But what about the datacenter?
Some folks report losing their entire databases for no reason, or experiencing a building-wide power outage because the air conditioner wasn’t connected to the UPS, or finding wires randomly, inexplicably unplugged, or getting soaked by phantom sprinklers.
But while those may be the equivalent of a door opening and closing on its own—creepy, but not exorcist-worthy—we do have some eerie stories of poltergeists in the datacenter.
First up, one administrator recalls a time when a server simply refused to work in their datacenter. When they brought it back to the office, it worked just fine. When they took it back to the datacenter? No such luck. After 3 of these back-and-forth trips, it suddenly worked. After that event, the server never had a problem again. (Theory: Maybe the server just wanted a field trip).
Technology can certainly have a mind of its own. Take, for example, this story of a possessed fire suppression system. One night, it mysteriously went off on its own. When the team returned the next morning, all was seemingly normal—except for a noticeable layer of dust that had blown out from underneath the cabinets. Perhaps this ghost was starting its own eerie form of spring cleaning.
During a routine black power test, one team lead reports everything actually going black. They found out later this was due to defective batteries. But faulty batteries can’t explain why another community member’s SAN lost power...and somehow continued to still work. Or what about the story of the eerie Ethernet cables and access switches that look like a pathway into the Upside Down itself?
Nosferatu, is that You?
And of course, the datacenter is no stranger (hah!) to ghosts. So when one company realized their motion set off an alarm identifying that someone was in their datacenter, they checked the footage—and found nobody there. Clearly, there are stranger things in heaven and earth—and the datacenter than can be dreamt of in our wildest imaginations.
Equipment Gone Wild!
We’ve mentioned possessed technology, but now it’s time to move into stories of haywire, vengeful technology that might be out to get you.
Think that’s an overstatement? The person whose entire hard drive was killed by an exploded INERGEN gas cartridge would beg to differ. So would the recovery crew that had to deal with (and clean up) the blown up remains of a faulty main board in their fire suppression system. And the IT team that had to deal with weeks’ worth of flooding from a malfunctioning AC might also have a say in the matter.
Are you starting to notice a pattern with faulty air conditioning? These next two stories make us thankful for functioning AC units.
The Cold (and Leaky) Shoulder
While many businesses keep ACs off in buildings after hours, server rooms are at-risk of failure if the temperature climbs too high. So when one company’s IT team entered their 95-degree server room, they quickly argued for the AC to be left on in the server room. The team returned another morning to find that an AC unit was installed, but it was pumping out HOT air. As the servers’ fans desperately whirred and the room temperature rose to 100 and above, the team realized that maintenance hadn’t vented the exhaust out of the room.
In that blazing heat, taking a cool dip sounds tempting, but this next story might have you rethinking that wish. When an air conditioner broke in a small test lab over one weekend, the unit started leaking water, soaking the carpet and flooding the area just a few centimeters from the server racks. Interestingly, before this catastrophe, the team faced countless outages due to high ambient temperature and overheating servers, prompting them to install an AC unit. Oh, the irony!
Luckily, we don’t have to worry about failing AC units in the dead of winter. But for this company, their Christmas Eve was anything but cheery. After completing a generator exercise, the automatic transfer switch was supposed to fail back to power line. When it failed to do so, 2/3rds of their datacenter went down—and they weren’t even alerted since their monitor watched the one leg that was working. The moral of the story? No more holiday tests!
A Mind of Their Own
Finally, let’s wrap up this section with a tale of anthropomorphic technology. Before this company moved to Nutanix, they had 4 disk failures in just 6 months—a major upswell compared to the 1 failure a year they were used to. The service engineer who replaced the disks said hardwares have “feelings” and “throw tantrums” when they know they’ll be replaced soon. Can’t say we blame them!
Banish Angsty IT Spirits
You’ve heard the horror stories; now it’s time to lighten up the mood with some tales of datacenter triumphs! Check out some happy, ghoul-free stories here.
© 2019 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and the other Nutanix products and features mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).