Pages

Sunday, March 05, 2017

The Amazon Web Service Outage of 2017

Earlier this week, Amazon Web Services suffered a service disruption.  For me personally, that meant disruption in Autodesk Fusion 360 (which I thankfully wasn't using at the time), and Fusion Lifecycle, which I was.  Fortunately, while Fusion Lifecycle was disrupted, it wasn't offline.

So, what happened?!?!

Several sites were affected.  My girlfriend sent me an instant message to let me know their CRM (Contact Resource Management) was down.  According to NPR, Gizmodo and Apple iCloud were affected.

The amazing, and perhaps disconcerting part of this is how easily the service was disrupted.  It seems that a technician was troubleshooting, and took off more servers than intended with a typographical error.

That's right, a typo, fat fingers as us techies call them, were the cause of the disruption.

Added to that, there didn't seem to enough redundancy, or possibly any, to absorb the lost servers while they were brought back online.

Now I'm no expert on Internet architecture. I imagine I know more than some, less than others.  I'm not going to prognosticate on what Amazon could have, and should have done, or should be doing.

There are plenty of people smarter than I who can do that better than I can.

But what I will say, is what I learned, relearned, and what I can do in the future.

What I learned and (relearned)

  • The cloud is a great tool, it's saved my class grades when I didn't print it out by mistake.  But the technology is new, and new technology is often developing systems, and developing systems can be prone to failure.  
  • As a whole, the cloud is pretty robust, but when it fails, it can fail spectacularly.  Many can be affected, and word can get out quick!
What can I do in the future


  • I can't make Autodesk, Amazon, or anyone else change their system, at least not directly.  But I've heard rumors that improved offline capabilities are on their way.  I hope they're true, and that after this outage, the efforts have been increased.  I'll be keeping my eyes out for that development! 
  • To this end, I'll be doing a better job of backing things up locally.  That's right, in a 180 degree turn, I'll be working in the cloud and backing up locally.  Just in case. 
I won't be abandoning the cloud, what I will be doing is approaching it more cautiously, and more diligently.  

In many ways, the cloud is a new frontier filled with pioneers, and pioneers must face the hazards of a new frontier to pave the way for others.  

For my part, I'll choose to stay on that frontier, but I will make sure I'm planning ahead! 

On into the brave new world!

photo credit: NPS Park Cultural Landscapes Program Chilkoot Trail via photopin (license)