This Blog is Moving

I have written on this blog since 2007 and it’s been a big part of my life. The things I learned were documented here for myself (primarily) and others (incidentally). I had never imagined this blog would be such a success. It has surpassed 1.5 million all-time views and 700+ comments. Not bad for something I only intended to use for my notes.

WordPress has been a superb platform for my purposes and wordpress.com has been priceless by providing me free hosting for so many years. I can never thank their team enough for allowing bloggers like me to have a say in our online world. Thank you, WordPress!

I am copying this blog over to my own VPS at Code Ghar. I’m trying out a static blog/site generator, Nikola, as the platform to create and manage this content. Let’s see how it works out.

I am not moving or deleting this blog from here because that would cause many links to be broken. I don’t want to be the cause of someone not finding the information they’re looking for because some forum post that linked back here now has a broken link. My plan is to continue to let this blog stand here as-is as long as wordpress.com continues to provide free hosting.

I’ll start migrating posts from here as and when I feel a need to do so. I’ll probably not migrate everything because I have lost interest in a lot of that stuff. I do have backups in case I ever need to resurrect any content.

Finally, thanks to all the people who have visited, read, commented on these pages. I hope you have learned as much as I did. I’m looking forward to adding more content on my new blog and continue this journey of discovery and learning.

Sysadmin Snowflakes

In a world dominated by cloud and scale the traditional snowflake of a sysadmin will be an increasingly rare breed. Sysadmins will basically use the community, hivemind, whatever as an outsourced resource to help them get jobs done. They will not feel the need to customize every tidbit out of their OS (Linux, BSD, etc.) because they will have tens of VMs (or more) running in private and public clouds controlled with config management and orchestration tools. Even if their Python is compiled with stuff they don’t need they wouldn’t care as much as they did a decade ago.

Any work done at scale will require pre-built binary packages instead of loving custom compiles on each server. Think of it as Ubuntu versus Gentoo. More teams will use Ubuntu to serve their needs because with a smaller team size their work has actually increased instead of decreasing. Their concerns have moved away from extracting every ounce of speed from one physical machine to extracting value out of $X.xx they are spending in their cloud assets.

It’s hard to change your ways but you just have to if you want to keep up. The ports collection in FreeBSD is a fantastic idea; I love the idea. But when you are creating and dropping VMs in your lab at the speed of your users’ requirements you can’t afford to wait for a VM to be usable in 10 minutes instead of 2. I believe the FreeBSD folks have realized this and have taken steps to rectify this. Poudriere is an example of this realization and effort.

Sysadmins who still want snowflake systems will now instead have to be happy with snowflake package repositories. Their level of focus has zoomed out a bit. Such sysadmins have two primary worries: setup proper config management and setup required package repositories. Gone will be the days when each server was handcrafted. Now servers will serve a purpose for a while and then be replaced by something else as soon as requirements change. The loving handcraftiness will instead need to be directed at service orchestration. The snowflake-loving sysadmins will need to become snowflake-loving service admins instead.

Keeping Up With External Development

I believe, maybe unlike many traditional sysadmins and operations professionals, that you need to keep upgrading your systems continuously. When developers have Continuous Integration why can’t operations have Continuous Deployment? The answer is not simple and here are my views.

What’s the primary reason for sticking with as static an environment as you can in production environments? To reduce the chances of disastrous events taking down your systems. This is a tradeoff between agility/advancement and stability. There’s a huge cost to upgrade any system, not just in say licensing but training of people operating or using these systems.

There are two major aspects to deploying to production frequently: (1) each deployment has a smaller delta than major deployments, (2) major changes are inevitable as the progress train rolls along.

Smaller changes are easier to revert. That’s their biggest benefit. That and they are generally easier to troubleshoot. You know what changed and you can revert it if you really need to. Contrast this to major changes after longer periods of time. Similarly, bite-sized changes are relatively easier to digest for the users of any system.

There are times when major changes are made. It’s inevitable. SysV to systemd, for example, is a huge change. You prepare for them by having put in place processes and practices that are able to handle changes in the first place. Transitions may not be smooth (are they ever?) but at least your team has the experience and confidence of handling any issues as they arise. Your team’s mantra should be Disaster Preparedness and Disaster Recovery not Disaster Prevention. There’s never a 100% chance of you having Disaster Prevention in effect; bad things will happen. Once you internalize this it’s easier to think about Disaster Preparedness and Disaster Recovery.

One more thing you need to consider is that while your little production silo is happily running its systems, the world outside of it is changing rapidly. Between 2006 (when I started with Ubuntu) and 2014 the GUI, kernel, init have all changed in Ubuntu. So while I could’ve been happily running Ubuntu 6.10 up to today it wouldn’t be a wise decision. Let’s extrapolate this example to an enterprise. Are you still running the same systems or following the same procedures in 2014 as you were in 2006? Things change at a rapid pace and you don’t want to emerge from an underground silo to realize the Cold War is over and you can pay at the store using a phone. In a sysadmin’s world there are no constants, just change.

Changes in technology are like board games. Each player gets a turn to advance their position. But while one player is actually moving ahead, others are preparing to move ahead. All systems you use follow a similar pattern to an extent. As one vendor brings some innovation to market, another is building on their next big announcement. You have the responsibility of managing and integrating these changes into your systems. Having a plan to deal with change will get you further than trying to stymie change in your organization.

Do you remember from your Computer Science or Software Engineering classes that the biggest cost of any system is maintenance? Well, why not turn this resource sink into a resource success? Keep changing your systems as the world changes around them.

Conclusion

The reason for this ranting was that although vendors like EMC and Red Hat thrive on long term support contracts, the world should move (and is moving) toward a continuous delivery cadence. It is cheaper for vendors to move onto new code bases for example and move their customers with them. These cost savings can even be passed on to the customer. I’m having issues where some environments I work with have such old versions of software I run into bugs that have been resolved ages ago in upstream. Upstream has also added new features that would make my life as a developer so much easier. This is because of a change-averse attitude still prevalent in technology circles. I think it’s time to embrace change.

Ubuntu and I

Ubuntu and I go way back, starting in 2006. I had built my own PC a couple years before that was running Windows. I decided to finally start using Linux on the desktop full time. I had dabbled with it in college labs but nothing serious. A few years before I had even unsuccessfully attempted to install some Red Hat version from CDs provided in the back of books I got from the library. This time I was wiser and more determined.

The biggest obstacle I remember is getting my PCI WiFi card to work. Using some tutorials and NDISwrapper I got it working flawlessly. Thus began my journey with Ubuntu. If you read older posts on this blog you’ll see how I brought my love for Ubuntu to my workplace. There were a bunch of old unused servers lying in storage. Just about then the network file share device died and I got to add a few hard drives to one of the old servers for the purposes of a network file share. I tried Ubuntu and it installed easily. Just to experiment a bit more I also tried installing Debian and Fedora (even CentOS) but the experience wasn’t as smooth as Ubuntu.

Around the same time I was tasked with building a web-based project. I got the latest Ubuntu release installed in a VM and started developing. I deployed the application on the same version of Ubuntu on another repurposed server. I was on top of the world because Ubuntu was giving me opportunities to get work done.

When my MacBook started showing its age I tried to inject some life into it by installing Linux. Ubuntu was the only distro that worked best with the hardware out of the box. Some things like camera needed some extra work but I didn’t bother because I didn’t need it to work.

We needed a system at work to drive the monitoring screen in our NOC. I found an abandoned desktop with a bad hard drive. Debian wouldn’t install, Fedora balked and Ubuntu installed fine. To my knowledge that desktop with the bad hard drive is still performing 4+ years later.

More recently I have been using Fedora and openSUSE for getting my work done. These distros have improved a lot and even outpace Ubuntu in some areas. But one thing is for sure: I keep recommending Ubuntu to new users to Linux because it’s well-rounded. I also keep coming back to Ubuntu every now and then because it is built for power users, too.

Blending Linux with BSD

The past 24 hours have been a revelation: there’s no need to be entrenched in one camp of free software. There’s a much wider world outside of any one camp. For example, if you think Ubuntu is Linux, think again. If you think Linux is the bastion of free software, think harder. Free software is all around us and it’s only us who choose not to see it.

I’ve been re-introduced (with a new perspective) to MacPorts. It’s a fascinating and remarkable way to install and use free software on your Mac OS X. I had tried it some years ago but it was just too slow on a spinning hard drive. On my MacBook Air it runs much better (some slowness still because of the nature of compiling source). But the world of possibility it opens is fantastic.

Today I re-remembered pkgsrc from NetBSD and looked into it a bit more. It, too, provides fantastic opportunity to blend (Net)BSD with your favorite Linux distro or even Mac OS X. Go ahead and read pkgsrc: my favorite non-root package manager on linux” and see how simple it can make someone’s life. This article seeded this notion of blending Linux with BSD to benefit any user.

Take this theoretical possibility. There’s a user who wants a Linux-based desktop/notebook OS with great hardware support, wide application availability, cheap-ish or free of cost, great community support, in-depth documentation, etc. Some distros that come to mind immediately include your favorite distro as well. Now let’s say this user chose Ubuntu 14.04 LTS. It is promised to be version-stable with support for 5 years. The user can stick with it for 5 years or can migrate to 16.06 in two years. But for the foreseeable future the user is stuck on the same version of some software unless the distro is upgraded as a whole or by using third-party packages. Although PPAs are available for a variety of software, including updated versions of programming languages like Python, they can be hit and miss in terms of packaging quality and support. An upstream developer cannot be expected to be well-versed in the nuances of Ubuntu packaging. So the overall experience may not be ideal.

A possible workaround is to use something like pkgsrc to obtain and use updated software on a distro meant to provide stability above all else. This distro could be CentOS or Debian or whatever. Turn the concept of a Linux distro on its head to be a more FreeBSD-like “core v apps” architecture. Continue to use your Linux distro and all its great features and packages. And when you need to move beyond its supplied packages to something newer or different use something like pksrc.

May your blending be fruitful.

Self Exceptionalism in the Free and Open Software Communities

Recently, I have felt myself being pulled in three directions at once. There’s the RHEL ecosystem that I have immersed myself into over the past many months. Ubuntu 14.04 LTS Trusty Tahr released last week and is a very enticing option, especially since I started with Ubuntu many moons ago. Finally, there’s the BSD world beckoning, with its culture and technology. Which option do I pick for primarily two roles: my daily driver at work and a way for me to grow my knowledge and skills, prepared to take on tomorrow’s challenges?

The answer is not as simple as I had anticipated. And the reason is my belief in my own exceptionalism. I have convinced myself that my vote for any one of these three options will turn the tide of FOSS in that direction. That if I don’t support an option then that option will disappear and its community will wither away. That I need to be a part of the community so I can contribute my skills, preventing unforeseen catastrophes waiting to be unveiled. That the FOSS world is waiting for a hero: me.

Of course, this is an absurd way of thinking. One person does make a difference but not at the scale I have imagined. It doesn’t matter to Ubuntu or its community that I fired up Fedora in my VM today. It has no affect on FreeBSD if I don’t learn how to use it. It doesn’t matter to Fedora that I am slowly moving to using Ubuntu daily. In a sense it does matter to all three projects whether an individual stays within the community or leaves. But not in the way I have convinced myself it does.

I am no savior. I can’t save or condemn a FOSS project because of my participation or the lack thereof. The reason is simple: these projects are much bigger than an individual. They have existed before I became a part of their communities; not just existed but prospered. I am but one man.

And therein lies the crux of the matter. If I am not writing code, finding bugs, generating official documentation, etc. then I’m not really a part of the community. I may be a passive advocate at best or a user at worst; but not part of the community. The community is formed of people who actively participate in shaping a project not those who use the output of their efforts without giving back.

A lot of people like myself are delusional in this manner. Just because they can download Ubuntu for free does not give them the right to complain and criticize when the desktop does not behave the way they demand. Either they pitch in, take a leadership role in the community, and fix the problems they have, or they lose the right to criticize. It’s alright to point out bugs and papercuts in a constructive gesture. Anything else reeks of their own belief in their exceptionalism. They have to stop thinking that just because *they* don’t like something that the project’s priorities need to shift to cater to their whims.

No project or product can survive without users. No one disputes that. But it’s not necessary that a project has to be used by *everyone*. Those who participate in developing the product and nurturing the project are its users, too. As long as they use and are satisfied, that’s all it matters.

So all of you who think the world revolves around them and that somehow FOSS means their wishes are paramount, think again. Get off your high horse, throw away the cloak of self exceptionalism, and pitch in. Either you are part of the solution or you don’t matter.

A question now comes to mind: why did I think that I was exceptional? It’s because I *expect* a lot from myself. I want to excel in all things I do. If I’m using some FOSS application that needs contributors I expect of myself to become such a contributor. Alas, there’s only so much one person can do. I have so many other obligations taking up my time that it becomes impossible for me to participate in many platforms and forums concurrently. That’s just life. The problem, though, is that I continue to expect of myself anyways. This causes undue stress, a highly unhealthy trend.

Given this new light in which I see myself, I will attempt to stop worrying about what the Internet says is a viable project and which one to support. I will no longer allow myself to be bound to follow the herd. If Ubuntu on the desktop gives me a pleasurable experience, CentOS works when I need a server, and FreeBSD can protect my network better, then I’ll use the best tool for the job. All these projects deserve contributions from their users. I’ll try my best to become a contributor. Until I do so I know that I’m a user and only a user.