This Blog is Moving

I have written on this blog since 2007 and it’s been a big part of my life. The things I learned were documented here for myself (primarily) and others (incidentally). I had never imagined this blog would be such a success. It has surpassed 1.5 million all-time views and 700+ comments. Not bad for something I only intended to use for my notes.

WordPress has been a superb platform for my purposes and wordpress.com has been priceless by providing me free hosting for so many years. I can never thank their team enough for allowing bloggers like me to have a say in our online world. Thank you, WordPress!

I am copying this blog over to my own VPS at Code Ghar. I’m trying out a static blog/site generator, Nikola, as the platform to create and manage this content. Let’s see how it works out.

I am not moving or deleting this blog from here because that would cause many links to be broken. I don’t want to be the cause of someone not finding the information they’re looking for because some forum post that linked back here now has a broken link. My plan is to continue to let this blog stand here as-is as long as wordpress.com continues to provide free hosting.

I’ll start migrating posts from here as and when I feel a need to do so. I’ll probably not migrate everything because I have lost interest in a lot of that stuff. I do have backups in case I ever need to resurrect any content.

Finally, thanks to all the people who have visited, read, commented on these pages. I hope you have learned as much as I did. I’m looking forward to adding more content on my new blog and continue this journey of discovery and learning.

Install Nikola in virtualenv

Nikola is a static website/blog generator written in Python. You can install it in a virtualenv. Although it supports Python 2 and Python 3 I had trouble using Nikola version 7.1 with Python 3.4. Python 2.7 (tested on Ubuntu 14.04) and Python 3.3 (tested on Mac OS X Mavericks) worked fine.

Install pre-requisites: sudo aptitude install build-essential python-dev python-pip python-virtualenv libxml2-dev libxslt-dev zlib1g-dev

I had to add a few dev libraries to get rid of some errors on Ubuntu.

Install libxml2-dev and libxslt-dev to get rid of error: “fatal error: libxml/xmlversion.h: No such file or directory”.

Install zlib1g-dev to get rid of error: “/usr/bin/ld: cannot find -lz”.

cg@host ~ $ virtualenv virt-nikola

cg@host ~ $ source virt-nikola/bin/activate

(virt-nikola)cg@host ~ $ pip install nikola

(virt-nikola)cg@host ~ $ pip install nikola[extras]

Hat tip: “/usr/bin/ld: cannot find -lz”, Ubuntu 11 : fatal error: libxml/xmlversion.h: No such file or directory

Create Sequence Diagrams Using seqdiag

Have you ever needed to create sequence diagrams? If you’re like me you constantly have to. Up to now I have always used a manual process using pen/paper or Visio. But I wanted an easier way and found seqdiag.

I installed it in a virtualenv on Ubuntu 14.04.

codeghar@host~$ virtualenv --system-site-packages virt

I used --system-site-packages because seqdiag relies on Python Imaging Library (PIL). Of course, make sure you have python-pil package installed in your OS. I ran into this problem (About the PIL Error — IOError: decoder zip not available) when I didn’t use --system-site-packages. The impression I got was it’s hard to get PIL installed in a virtualenv and easier to just use the system-wide PIL.

codeghar@host~$ source virt/bin/activate

(virt)codeghar@host~$ pip install seqdiag

To create a diagram you need to provide a dialog in a .diag file. You can look at the sample diagrams for more information. The following will create a new file (or overwrite an existing file) called example.png.

(virt)codeghar@host~$ seqdiag -Tpng --no-transparency example.diag

Hat tip to generate uml sequence diagrams with python or perl for bringing seqdiag to my attention.

Sysadmin Snowflakes

In a world dominated by cloud and scale the traditional snowflake of a sysadmin will be an increasingly rare breed. Sysadmins will basically use the community, hivemind, whatever as an outsourced resource to help them get jobs done. They will not feel the need to customize every tidbit out of their OS (Linux, BSD, etc.) because they will have tens of VMs (or more) running in private and public clouds controlled with config management and orchestration tools. Even if their Python is compiled with stuff they don’t need they wouldn’t care as much as they did a decade ago.

Any work done at scale will require pre-built binary packages instead of loving custom compiles on each server. Think of it as Ubuntu versus Gentoo. More teams will use Ubuntu to serve their needs because with a smaller team size their work has actually increased instead of decreasing. Their concerns have moved away from extracting every ounce of speed from one physical machine to extracting value out of $X.xx they are spending in their cloud assets.

It’s hard to change your ways but you just have to if you want to keep up. The ports collection in FreeBSD is a fantastic idea; I love the idea. But when you are creating and dropping VMs in your lab at the speed of your users’ requirements you can’t afford to wait for a VM to be usable in 10 minutes instead of 2. I believe the FreeBSD folks have realized this and have taken steps to rectify this. Poudriere is an example of this realization and effort.

Sysadmins who still want snowflake systems will now instead have to be happy with snowflake package repositories. Their level of focus has zoomed out a bit. Such sysadmins have two primary worries: setup proper config management and setup required package repositories. Gone will be the days when each server was handcrafted. Now servers will serve a purpose for a while and then be replaced by something else as soon as requirements change. The loving handcraftiness will instead need to be directed at service orchestration. The snowflake-loving sysadmins will need to become snowflake-loving service admins instead.

Keeping Up With External Development

I believe, maybe unlike many traditional sysadmins and operations professionals, that you need to keep upgrading your systems continuously. When developers have Continuous Integration why can’t operations have Continuous Deployment? The answer is not simple and here are my views.

What’s the primary reason for sticking with as static an environment as you can in production environments? To reduce the chances of disastrous events taking down your systems. This is a tradeoff between agility/advancement and stability. There’s a huge cost to upgrade any system, not just in say licensing but training of people operating or using these systems.

There are two major aspects to deploying to production frequently: (1) each deployment has a smaller delta than major deployments, (2) major changes are inevitable as the progress train rolls along.

Smaller changes are easier to revert. That’s their biggest benefit. That and they are generally easier to troubleshoot. You know what changed and you can revert it if you really need to. Contrast this to major changes after longer periods of time. Similarly, bite-sized changes are relatively easier to digest for the users of any system.

There are times when major changes are made. It’s inevitable. SysV to systemd, for example, is a huge change. You prepare for them by having put in place processes and practices that are able to handle changes in the first place. Transitions may not be smooth (are they ever?) but at least your team has the experience and confidence of handling any issues as they arise. Your team’s mantra should be Disaster Preparedness and Disaster Recovery not Disaster Prevention. There’s never a 100% chance of you having Disaster Prevention in effect; bad things will happen. Once you internalize this it’s easier to think about Disaster Preparedness and Disaster Recovery.

One more thing you need to consider is that while your little production silo is happily running its systems, the world outside of it is changing rapidly. Between 2006 (when I started with Ubuntu) and 2014 the GUI, kernel, init have all changed in Ubuntu. So while I could’ve been happily running Ubuntu 6.10 up to today it wouldn’t be a wise decision. Let’s extrapolate this example to an enterprise. Are you still running the same systems or following the same procedures in 2014 as you were in 2006? Things change at a rapid pace and you don’t want to emerge from an underground silo to realize the Cold War is over and you can pay at the store using a phone. In a sysadmin’s world there are no constants, just change.

Changes in technology are like board games. Each player gets a turn to advance their position. But while one player is actually moving ahead, others are preparing to move ahead. All systems you use follow a similar pattern to an extent. As one vendor brings some innovation to market, another is building on their next big announcement. You have the responsibility of managing and integrating these changes into your systems. Having a plan to deal with change will get you further than trying to stymie change in your organization.

Do you remember from your Computer Science or Software Engineering classes that the biggest cost of any system is maintenance? Well, why not turn this resource sink into a resource success? Keep changing your systems as the world changes around them.

Conclusion

The reason for this ranting was that although vendors like EMC and Red Hat thrive on long term support contracts, the world should move (and is moving) toward a continuous delivery cadence. It is cheaper for vendors to move onto new code bases for example and move their customers with them. These cost savings can even be passed on to the customer. I’m having issues where some environments I work with have such old versions of software I run into bugs that have been resolved ages ago in upstream. Upstream has also added new features that would make my life as a developer so much easier. This is because of a change-averse attitude still prevalent in technology circles. I think it’s time to embrace change.

Ubuntu and I

Ubuntu and I go way back, starting in 2006. I had built my own PC a couple years before that was running Windows. I decided to finally start using Linux on the desktop full time. I had dabbled with it in college labs but nothing serious. A few years before I had even unsuccessfully attempted to install some Red Hat version from CDs provided in the back of books I got from the library. This time I was wiser and more determined.

The biggest obstacle I remember is getting my PCI WiFi card to work. Using some tutorials and NDISwrapper I got it working flawlessly. Thus began my journey with Ubuntu. If you read older posts on this blog you’ll see how I brought my love for Ubuntu to my workplace. There were a bunch of old unused servers lying in storage. Just about then the network file share device died and I got to add a few hard drives to one of the old servers for the purposes of a network file share. I tried Ubuntu and it installed easily. Just to experiment a bit more I also tried installing Debian and Fedora (even CentOS) but the experience wasn’t as smooth as Ubuntu.

Around the same time I was tasked with building a web-based project. I got the latest Ubuntu release installed in a VM and started developing. I deployed the application on the same version of Ubuntu on another repurposed server. I was on top of the world because Ubuntu was giving me opportunities to get work done.

When my MacBook started showing its age I tried to inject some life into it by installing Linux. Ubuntu was the only distro that worked best with the hardware out of the box. Some things like camera needed some extra work but I didn’t bother because I didn’t need it to work.

We needed a system at work to drive the monitoring screen in our NOC. I found an abandoned desktop with a bad hard drive. Debian wouldn’t install, Fedora balked and Ubuntu installed fine. To my knowledge that desktop with the bad hard drive is still performing 4+ years later.

More recently I have been using Fedora and openSUSE for getting my work done. These distros have improved a lot and even outpace Ubuntu in some areas. But one thing is for sure: I keep recommending Ubuntu to new users to Linux because it’s well-rounded. I also keep coming back to Ubuntu every now and then because it is built for power users, too.