Sysadmin Snowflakes

In a world dominated by cloud and scale the traditional snowflake of a sysadmin will be an increasingly rare breed. Sysadmins will basically use the community, hivemind, whatever as an outsourced resource to help them get jobs done. They will not feel the need to customize every tidbit out of their OS (Linux, BSD, etc.) because they will have tens of VMs (or more) running in private and public clouds controlled with config management and orchestration tools. Even if their Python is compiled with stuff they don’t need they wouldn’t care as much as they did a decade ago.

Any work done at scale will require pre-built binary packages instead of loving custom compiles on each server. Think of it as Ubuntu versus Gentoo. More teams will use Ubuntu to serve their needs because with a smaller team size their work has actually increased instead of decreasing. Their concerns have moved away from extracting every ounce of speed from one physical machine to extracting value out of $X.xx they are spending in their cloud assets.

It’s hard to change your ways but you just have to if you want to keep up. The ports collection in FreeBSD is a fantastic idea; I love the idea. But when you are creating and dropping VMs in your lab at the speed of your users’ requirements you can’t afford to wait for a VM to be usable in 10 minutes instead of 2. I believe the FreeBSD folks have realized this and have taken steps to rectify this. Poudriere is an example of this realization and effort.

Sysadmins who still want snowflake systems will now instead have to be happy with snowflake package repositories. Their level of focus has zoomed out a bit. Such sysadmins have two primary worries: setup proper config management and setup required package repositories. Gone will be the days when each server was handcrafted. Now servers will serve a purpose for a while and then be replaced by something else as soon as requirements change. The loving handcraftiness will instead need to be directed at service orchestration. The snowflake-loving sysadmins will need to become snowflake-loving service admins instead.

Fix for “undefined reference to clock_gettime”

While running the build.sh script for SIPp I ran into this error:

src/sipp-time.o: In function getmicroseconds()': /chroot/root/sipp/src/time.cpp:57: undefined reference to clock_gettime'

The fix is to add LDFLAGS=”-ltr” like so:

LDFLAGS=" -lrt " ./configure --with-openssl --with-pcap --with-sctp --with-rtpstream

LDFLAGS=" -lrt " make

More information about this error: Ubuntu Linux C++ error: undefined reference to ‘clock_gettime’ and ‘clock_settime’; Untitled; [xymon] Can not compile 4.2.3 on CentOS 5.2

pkgsrc on Linux – Quickstart Guide

A much more opinionated and updated guide is now on my new blog location. You should head over there instead.

Do you want to try out pkgsrc on Linux? Here’s a quickstart guide (tested on Ubuntu 14.04). Of course, always refer to the pkgsrc guide for accurate information.

Get pkgsrc

Install prerequisities.

user@host [~] $ sudo apt-get install build-essential libncurses-dev

Create directory structure in your home directory.

user@host [~] $ mkdir ~/opt

user@host [~] $ cd ~/opt

You have two options for getting pkgsrc: (1) the official ways: by downloading a tar file and extracting it, or by using CVS; (2) by using git. The first way, by using a tar file, is this:

Download pkgsrc and extract.

user@host [~/opt] $ curl -O http://ftp.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.gz

user@host [~/opt] $ tar xvzf pkgsrc.tar.gz

The second way, to use git, is also easy. Jörg Sonnenberger has created a GitHub repo that automatically syncs with pkgsrc source.

user@host [~/opt] $ git clone https://github.com/jsonn/pkgsrc.git

I think I would prefer using git.

Install and Setup pkgsrc

Install pkgsrc.

user@host [~/opt] $ cd ~/opt/pkgsrc/bootstrap/

user@host [~/opt/pkgsrc/bootstrap] $ export SH=/bin/bash

user@host [~/opt/pkgsrc/bootstrap] $ ./bootstrap --unprivileged

Post install steps.

user@host [~/opt/pkgsrc/bootstrap] $ cat >> ~/pkg/etc/security.local << EOF
    if [ -x $HOME/pkg/sbin/pkg_admin ]; then
            $HOME/pkg/sbin/pkg_admin audit
    fi
    EOF

user@host [~/opt/pkgsrc/bootstrap] $ echo "export PATH=\$PATH:\$HOME/pkg/bin:\$HOME/pkg/sbin" >> ~/.bashrc

user@host [~/opt/pkgsrc/bootstrap] $ source ~/.bashrc

user@host [~/opt/pkgsrc/bootstrap] $ pkg_admin -K ~/pkg/var/db/pkg fetch-pkg-vulnerabilities

user@host [~/opt/pkgsrc/bootstrap] $ echo 'alias pkgupvuln="\$HOME/pkg/sbin/pkg_admin -K ~/pkg/var/db/pkg fetch-pkg-vulnerabilities >/dev/null 2>&1"' >> ~/.bash_aliases

If you used git to get pkgsrc then move on to the next section.

If you did not use git to get pkgsrc then this is the best way going forward. Install CVS and update pkgsrc.

user@host [~/opt/pkgsrc/bootstrap] $ cd ~/opt/pkgsrc/devel/scmcvs/

user@host [~/opt/pkgsrc/devel/scmcvs] $ echo "export CVSEDITOR=vim" >> ~/.bashrc

user@host [~/opt/pkgsrc/devel/scmcvs] $ echo "export CVS_RSH=ssh" >> ~/.bashrc

user@host [~/opt/pkgsrc/devel/scmcvs] $ source ~/.bashrc

user@host [~/opt/pkgsrc/devel/scmcvs] $ bmake install clean clean-depends

user@host [~/opt/pkgsrc/devel/scmcvs] $ cd ~/opt/pkgsrc

user@host [~/opt/pkgsrc] $ cvs up -dP

user@host [~/opt/pkgsrc] $ cd ~/opt/pkgsrc/devel/scmcvs/

user@host [~/opt/pkgsrc/devel/scmcvs] $ bmake update

user@host [~/opt/pkgsrc/devel/scmcvs] $ bmake clean clean-depends

Install Package

Let’s install golang.

user@host [~/opt/pkgsrc] $ cd ~/opt/pkgsrc/lang/go

user@host [~/opt/pkgsrc/lang/go] $ bmake install clean clean-depends

Update Package

Follow these steps when you want to update a package, say golang.

user@host [~] $ pkg_admin -K ~/pkg/var/db/pkg fetch-pkg-vulnerabilities

user@host [~] $ cd ~/opt/pkgsrc

If you used git:
user@host [~/opt/pkgsrc] $ git pull

If you didn’t use git: user@host [~/opt/pkgsrc] $ cvs up -dP

user@host [~/opt/pkgsrc] $ cd ~/opt/pkgsrc/lang/go

user@host [~/opt/pkgsrc/lang/go] $ bmake update

user@host [~/opt/pkgsrc/lang/go] $ bmake clean clean-depends

Delete Package

You can uninstall a package and all its dependencies that are not needed by any other package.

user@host [~] $ pkg_delete -R go

Blending Linux with BSD

The past 24 hours have been a revelation: there’s no need to be entrenched in one camp of free software. There’s a much wider world outside of any one camp. For example, if you think Ubuntu is Linux, think again. If you think Linux is the bastion of free software, think harder. Free software is all around us and it’s only us who choose not to see it.

I’ve been re-introduced (with a new perspective) to MacPorts. It’s a fascinating and remarkable way to install and use free software on your Mac OS X. I had tried it some years ago but it was just too slow on a spinning hard drive. On my MacBook Air it runs much better (some slowness still because of the nature of compiling source). But the world of possibility it opens is fantastic.

Today I re-remembered pkgsrc from NetBSD and looked into it a bit more. It, too, provides fantastic opportunity to blend (Net)BSD with your favorite Linux distro or even Mac OS X. Go ahead and read pkgsrc: my favorite non-root package manager on linux” and see how simple it can make someone’s life. This article seeded this notion of blending Linux with BSD to benefit any user.

Take this theoretical possibility. There’s a user who wants a Linux-based desktop/notebook OS with great hardware support, wide application availability, cheap-ish or free of cost, great community support, in-depth documentation, etc. Some distros that come to mind immediately include your favorite distro as well. Now let’s say this user chose Ubuntu 14.04 LTS. It is promised to be version-stable with support for 5 years. The user can stick with it for 5 years or can migrate to 16.06 in two years. But for the foreseeable future the user is stuck on the same version of some software unless the distro is upgraded as a whole or by using third-party packages. Although PPAs are available for a variety of software, including updated versions of programming languages like Python, they can be hit and miss in terms of packaging quality and support. An upstream developer cannot be expected to be well-versed in the nuances of Ubuntu packaging. So the overall experience may not be ideal.

A possible workaround is to use something like pkgsrc to obtain and use updated software on a distro meant to provide stability above all else. This distro could be CentOS or Debian or whatever. Turn the concept of a Linux distro on its head to be a more FreeBSD-like “core v apps” architecture. Continue to use your Linux distro and all its great features and packages. And when you need to move beyond its supplied packages to something newer or different use something like pksrc.

May your blending be fruitful.

I can do it better

We have all experienced this moment when we thought we could do something better than someone else. I want us all to think twice whenever we have this thought. You may be able to do things better but you should always ask yourself if you really can.

I have been thinking about this for a while now. Arch Linux is a prime example of users who think they know better than those building other distributions. Arch allows them to be lean and customized unlike, say, Ubuntu. They are right but consider this: you and I spend how many hours working on Linux? And how many of them are spent thinking about the nitty gritty of the OS? Pretty low compared to the high-level stuff we are mostly involved with. For example, I’m more concerned about getting my application developed and deployed into production and less about what flags were used to compile Apache httpd. For users in this situation it’s better to let others take care of stuff they’re not interested in.

The software you are running was developed by someone else. If we let that developer focus on one application so that all we had to do was run it, why not do the same for our OS? There comes a point in our careers when we cannot do everything in a vertical stack. We have to give up control over more things so we can focus on the things that matter to us.

pfSense is another great example. Sure I could run a minimal Fedora to serve as my firewall and VPN server. I’d have to dig in and learn how to do it and it wouldn’t be too difficult. But my knowledge of security best practices would not compare favorably to the developers of pfSense. It’s not because they are smarter but because they spend more hours working on a firewall than I do. Thus I trust them to do a better job.

I’m not saying Arch users or those who build their firewalls from scratch should move on to something else. I am saying that you should focus on the things you really want to do and trust others in the free(dom) software community to fill the remaining gaps. Thus when openSUSE makes a decision and you think “I could do it better”, re-think. You could do it better but would it serve you or the community if you did?

100% Linux is Impractical

I was reading a post on the Ubuntu sub-reddit on how a user has converted 100% to Ubuntu. This might be possible but can be fairly impractical. Allow me to explain.

I use Linux almost every day at home and work. I also use Windows every day at work. At home I use Mac OS X on a MacBook. My immediate family is all iOS and extended family is mostly iOS with some Android and BlackBerry OS thrown in. I also have a Nexus 7 tablet. I own an Apple TV and a Roku.

Mine may be a typical American household these days (for a geeky household). No one device or OS is able to solve every use case. This not only the world we live in but the world of our near future. Consumers have lots of choice but they don’t really use that choice. Phones are mostly either iOS or Android. Computers are mostly either Mac OS X or Windows. Servers are mostly either Linux or Windows. Tablets are mostly either iOS or Android. So even with the widest choice available most consumers have picked a couple of winners for each device and the rest use the remaining choices.

I haven’t made up my mind whether this is a good thing or not. On one hand you have the ability for consumers to help each other because they use the same device and OS. On the other hand consumers are creating de facto monopolies with their wallets by rejecting other viable options. So you have these large corporations clambering for supremacy in one facet of technology and relinquishing control in others. Eventually we’ll be left with some devices ruled by sub-par manufacturers or without manufacturers altogether. But I digress.

To expect Linux to rule all devices is a bit naive. It already has a large market share in many device segments and this share increases every day. But it cannot be expected to replace the latest iPhone. The reason is simple: market forces pull technology and consumers in all directions all at once. In addition, consumers are not all alike. I enjoy using Linux, my wife is not a big fan of technology, and kids care about Netflix, Angry Birds, and Temple Run.

Let’s assume Linux could be king for me, my wife, and kids. Who will produce this Linux? Already in the server and desktop market we have major distributions pulling Linux in different directions. This causes headaches for application developers and hardware manufacturers: which distribution should they support? No two major distribution producers can agree on all the things at the same time, from package management to release/support life cycles. How can we expect a single direction for Linux?

Even if Apple, Microsoft, Google, BlackBerry, and others decided to drop everything they are doing now and back Linux, they all have different visions of products and how they want to create them. There’s no uniting factor compelling enough for the thousands of developers to disregard their goals. Each one of those developers has their own vision. It’s hard enough to get them on the same page in a corporation, let alone in the free software world.

Linux can be king of all; it’s possible. But it’s just not practical. Being optimistic is good and working to make Linux king of all domains is great. But we also need to realize that sometimes our ideals might get in the way of making the lives of people better with technology. That should be our goal while we try to get there in our own ways.