Keeping Up With External Development

I believe, maybe unlike many traditional sysadmins and operations professionals, that you need to keep upgrading your systems continuously. When developers have Continuous Integration why can’t operations have Continuous Deployment? The answer is not simple and here are my views.

What’s the primary reason for sticking with as static an environment as you can in production environments? To reduce the chances of disastrous events taking down your systems. This is a tradeoff between agility/advancement and stability. There’s a huge cost to upgrade any system, not just in say licensing but training of people operating or using these systems.

There are two major aspects to deploying to production frequently: (1) each deployment has a smaller delta than major deployments, (2) major changes are inevitable as the progress train rolls along.

Smaller changes are easier to revert. That’s their biggest benefit. That and they are generally easier to troubleshoot. You know what changed and you can revert it if you really need to. Contrast this to major changes after longer periods of time. Similarly, bite-sized changes are relatively easier to digest for the users of any system.

There are times when major changes are made. It’s inevitable. SysV to systemd, for example, is a huge change. You prepare for them by having put in place processes and practices that are able to handle changes in the first place. Transitions may not be smooth (are they ever?) but at least your team has the experience and confidence of handling any issues as they arise. Your team’s mantra should be Disaster Preparedness and Disaster Recovery not Disaster Prevention. There’s never a 100% chance of you having Disaster Prevention in effect; bad things will happen. Once you internalize this it’s easier to think about Disaster Preparedness and Disaster Recovery.

One more thing you need to consider is that while your little production silo is happily running its systems, the world outside of it is changing rapidly. Between 2006 (when I started with Ubuntu) and 2014 the GUI, kernel, init have all changed in Ubuntu. So while I could’ve been happily running Ubuntu 6.10 up to today it wouldn’t be a wise decision. Let’s extrapolate this example to an enterprise. Are you still running the same systems or following the same procedures in 2014 as you were in 2006? Things change at a rapid pace and you don’t want to emerge from an underground silo to realize the Cold War is over and you can pay at the store using a phone. In a sysadmin’s world there are no constants, just change.

Changes in technology are like board games. Each player gets a turn to advance their position. But while one player is actually moving ahead, others are preparing to move ahead. All systems you use follow a similar pattern to an extent. As one vendor brings some innovation to market, another is building on their next big announcement. You have the responsibility of managing and integrating these changes into your systems. Having a plan to deal with change will get you further than trying to stymie change in your organization.

Do you remember from your Computer Science or Software Engineering classes that the biggest cost of any system is maintenance? Well, why not turn this resource sink into a resource success? Keep changing your systems as the world changes around them.

Conclusion

The reason for this ranting was that although vendors like EMC and Red Hat thrive on long term support contracts, the world should move (and is moving) toward a continuous delivery cadence. It is cheaper for vendors to move onto new code bases for example and move their customers with them. These cost savings can even be passed on to the customer. I’m having issues where some environments I work with have such old versions of software I run into bugs that have been resolved ages ago in upstream. Upstream has also added new features that would make my life as a developer so much easier. This is because of a change-averse attitude still prevalent in technology circles. I think it’s time to embrace change.