Code Ghar Official Linux Desktop

So I decided to take the plunge and use Linux on the desktop. This post describes how the first ever official Code Ghar Linux desktop was created. As I make changes to the computer, I will keep this post updated.

Distribution

I first installed Fedora Rawhide (nightly build) using ext4 as the filesystem. After a couple days, I decided to add more memory to the system. Once this memory was installed successfully, the OS would not boot at all. I decided to not risk things further and went with Debian testing (Squeeze) instead. I am more comfortable with a Debian based system anyways so the choice wasn’t hard. I chose the netinstall CD and only installed the command line (did not choose the Desktop option during install).

Desktop Environment

I wanted to make best use of the 3GB RAM in my system. So I chose to install LXDE as my desktop environment. Since I was only running command line, I ran the following command to install LXDE.

sudo aptitude install xorg gdm lxde

Wine

I needed to run three main applications. I could either install a virtual machine running Windows to run these apps or install Wine. I decided to try out Wine. To install Wine, I ran the following command.

sudo aptitude install wine

Once installed, all you have to do is run the exe file to install your desired application and Wine takes care of the rest. But make sure you have searched that application on the Wine Application Database.

Virtualization

However much I want to move away from Windows, I just can’t because of some applications. I chose to use VirtualBox although I have been using VMware on Windows and Mac for a couple years now. To install VirtualBox on Debian, I followed their Wiki (VirtualBox Installation). Following are the commands I ran.

sudo aptitude install virtualbox-ose virtualbox-ose-modules-$(uname -r)

sudo invoke-rc.d udev reload

sudo modprobe vboxdrv

sudo addgroup vboxusers

sudo adduser codeghar vboxusers

I then logged out and logged back in for VirtualBox to be ready to be used.

Remote Access

To allow remote access to my machine using VNC, I chose to use X11VNC. To install it, I ran the following command:

sudo aptitude install x11vnc

To invoke it for my user (since I have a single-user machine), I ran the following command once I logged in. This command is all on one line (in case the formatting of the blog makes it appear different)

x11vnc -display :0 -xkb -bg -forever -passwd mypassword

What this does is that it allows you to access your actual desktop on port 5900, runs in the background (bg flag), does not stop after first remote client disconnects (forever flag) i.e. keeps running, and assigns a password (passwd) which must be used to authenticate yourself. The xkb flag was used because for some reason the shift key did not work when I ran a VNC client from another machine into this one. I forgot which resource I used to fix this issue but I will put up the link once I can find it again.

If my computer is restarted, I will have to run the same command again for me to be able to use VNC. This is why you need access to at least SSH to be able to run this command and start VNC. I added the following alias at the end of my ~/.bashrc file. This way I don’t have to retype the command all the time.

alias myvnc="x11vnc -display :0 -xkb -bg -forever -passwd mypassword"

Flash for Iceweasel (Firefox)

To install Adobe Flash, I followed the instructions on How to Install Adobe Flash in Debian Etch/Lenny/Sid. As they were very generic, I took only what I needed and did the following.

First I downloaded the debian-multilemedia-keyring package on the desktop. Then I ran the following command to add it to aptitude.

sudo dpkg -i ~/Desktop/debian-multimedia-keyring_2008.10.16_all.deb

The I modified my /etc/apt/sources.list and added the following line:

deb http://www.debian-multimedia.org testing main

I then ran the following command to update aptitude and then install Flash.

sudo aptitude update

sudo aptitude install flashplayer-mozilla

Remote File Access with SSHFS

I used the guide at Mount a remote file system through ssh Using sshfs to get this working. I strongly recommend that you read it thoroughly for better information. To install SSHFS, you also need to install FUSE, as below.

sudo aptitude install fuse-utils sshfs

Now you need to do two things: load the FUSE module (as below)

sudo modprobe fuse

and also add your user to the fuse group (as below)

sudo adduser codeghar fuse

I then had to log out and log back in for my user to be recognized as part of fuse group.

I then created a mount directory in my home folder, created folders for all remote file systems I need to access, and created aliases for mounting and un-mounting SSHFS “shares”.

mkdir /home/codeghar/.mount/
mkdir /home/codeghar/.mount/server1
mkdir /home/codeghar/.mount/server2

To mount server1, I ran the following (also added this as a bash alias to save typing):

sshfs codeghar@10.10.1.101:/home/remoteuser/ /home/codeghar/.mount/server1

To unmount server1, I ran the following (also added this as a bash alias to save typing):

fusermount -u /home/codeghar/.mount/server1

Miscellaneous Applications

Following are the applications I also installed.

OpenOffice: sudo aptitude install openoffice.org
Wireshark: sudo aptitude install wireshark
gvim: sudo aptitude install gvim
VNC client: sudo aptitude install vncviewer
Grsync: sudo aptitude install grsync
Pidgin: sudo aptitude install pidgin
PDF: sudo aptitude install epdfview
Samba: sudo aptitude install samba samba-client smbfs

Little sed Tricks

I am beginning to use sed more and more these days. As I learn, I want to document and share that knowledge for all, and to recognize the places from where I learned.

Remove Selective Newline

I had text similar to the following, with ~ as the delimiter. I wanted to replace certain newline characters so that two or more lines combined into one.

1~Some Text~123,456,789~Some More Text
2~Some Text~9223,321,
,789~Some More Text
3~Some Text~3556,36512,57567~Some More Text

In example above, line 2 and line 3 should be just one line. To remove a selective newline, and to bring two lines into one, I used the following (hat tip to sed and newline):

sed '/,$/{N;s/\n//;}'

We are searching for ,$ which means a pattern where the last character on the line is a comma. We then replace it with {N;s/\n//;}. N means “Read/append the next line of input into the pattern space“. After that we again do a search/replace by replacing newline with nothing. This means, in effect, that whenever you find a pattern, read the next line, and replace the newline with nothing.

This resulted in the following text, which is what I needed.

1~Some Text~123,456,789~Some More Text
2~Some Text~9223,321,,789~Some More Text
3~Some Text~3556,36512,57567~Some More Text

Delete Selective Line

If you want to delete or ignore one or more specific lines, sed can help you with that as well. Extending the example above, let’s say we have following text.

1~Some Text~123,456,789~Some More Text
2~Some Text~9223,321,,789~Some More Text
3~Some Text~3556,36512,57567~Some More Text
4~Some Text~(00:00 – 23:59)~Some More Text
5~Some Text~~Some More Text

Let’s say you want to ignore or delete the line starting with 4. As you can see, colon is the character in this line but not in any other. This helps us identify which lines to ignore. The idea, then, is to pick a unique characteristic which clearly identifies the line(s) to ignore. The command, then, is as below.

sed '/:/d'

Hat tip: delete line in file with sed; Ignore Lines Begining With #.

Load Data from Excel Format into Database

Task: Take as input name of a directory on a Windows machine containing Microsoft Excel (xls) documents. Convert them to csv files. Process these csv files to normalize them into a single, standard csv format. Load these files into a PostgreSQL database. Tools available: Windows, Linux (Debian), Python, and PostgreSQL. Result: success (as described below).

Mount Windows Share

It is very easy to mount a Windows shared folder under Debian. Hat tip to Ubuntu Linux Quick Tip – Mount a Samba (Windows) file share to a folder and How to mount remote windows partition (windows share) under Linux. First, install smbfs.

sudo aptitude install smbfs

Create a directory where you want to mount it. Initially, I had chosen /mnt/ but reading up on Linux File System Hierarchy, the proper place is /media instead of /mnt. So I have modified the code.

sudo mkdir /mnt/ntserver
sudo mkdir /media/ntserver

Actually mount the share using the following command. It’s a single command but the formatting may make it appear as if it’s on two lines.

sudo mount -t smbfs //192.168.0.111/shared/ /mnt/ntserver -o username=user,password=password
sudo mount -t smbfs //192.168.0.111/shared/ /media/ntserver -o username=user,password=password

Now you can easily browse and use the mounted share just like any directory. You do not need to mount it every time. Just edit your /etc/fstab so upon every reboot the share is mounted automatically. At the end of my /etc/fstab file, I added the following (hat tip to Permanent mount – fstab):

//192.168.0.111/shared /media/ntserver cifs username=user,password=password,rw,mand 0 2

Then I ran the following command:

sudo mount -a

Now with every reboot the shared directory was mounted automatically.

Convert XLS to CSV

Linux has a command line solution to it, called xls2csv. To install xls2csv in Debian, run the following command.

sudo aptitude install catdoc

Now you can use the command to convert any xls file to csv. I like to do the following, using ! as delimiter because the data I am working with usually has a list, separated by comma, in one column. To differentiate between columns and the list, I use another delimiter. With this tweak, I choose to use no quotes because it would only make things difficult by having to remove quotes myself before loading data.

xls2csv -x -q0 -c! myfile.xls

Since xls in all it’s wisdom can cause date and time problems when converting to csv, Python may need to be used. xlrd is a package for Python to extract Excel data. To install it in Debian, run the following.

sudo aptitude install python-xlrd

Normalize CSV Files

Normalize the resulting csv files, with their different formats, by using standard commands, like cut, sort, uniq, etc. This normalization should be done based on the table structure into which data has to be loaded in the next step.

Load Data into Database

You can use the copy command in PostgreSQL to load csv files into it.

Update 2009-08-25: Added Python to the tools because of problems with Excel (xls) date time format.

State of Linux

Linux’s world has seen very interesting recent developments. But two that stand out the most for me are Debian’s decision about scheduled freezes and the drama surrounding CentOS. Both can be seen positively or negatively but both, in my view, have improved the state of Linux.

First, let’s look at Debian. I had read Mark Shuttleworth’s interview on Gnome 3.0 some time ago. In it he had mentioned talks between Ubuntu and Debian on freeze dates. Now that Debian has confirmed this idea, it’s official: Ubuntu and Debian can now sit down every two years and discuss what to freeze together. It means that Debian can better plan itself by setting hard deadlines on features. Once that freeze occurs, they can focus on making things as perfect as possible before releasing.

Debian has always been known to release when they are ready to release. It was the Debian way, and worked for their users. If someone disagreed, they could use another distro. This wasn’t very helpful to enterprises as they like to know a semi-formal schedule for maintenance and support before committing to a thing as important as an operating system. All users get good quality but that extra bit – a schedule – especially helps enterprises.

Some would see this move as a betrayal of the Debian-ness of Debian. They would point to the idea of time-based freezes as compromising quality. But they would be mistaken. A freeze is not the same as a release. A freeze means for two years developers work on new features before they spend all their time improving the quality of these features.

Debian previously supported a release for roughly three or four years, not because they set out to do so, but because of their release schedule. Now we know they plan to pursue a two-year cycle and I don’t know how long they plan to support a release. If it’s two to two-point-five years, enterprises might not want to invest in Debian for production systems expected to run longer than that. They would rather than Red Hat’s promise of five to seven years. But if Debian can partially match Red Hat’s promise, it now becomes a very good alternative for enterprises.

This also means that Debian stable and Ubuntu LTS could be very closely related, and users get a choice of which goals they want to support. Do they want the Debian goals or Ubuntu goals? I support both because they both provide choices for different set of use cases. But I also love that I can now use Debian when appropriate and Ubuntu when it fits the bill.

I love Debian because of the sheer amount of choice it offers. Red Hat doesn’t match this choice (because it’s not financially feasible?). I am beginning to like Debian with LXDE as GUI (if required). Red Hat doesn’t have it by default. Yes, there are third parties offering this but if you are paying Red Hat you would want Red Hat to provide these things. They don’t and if you are going to third parties, why not just use something which provides these choices under one umbrella?

Combine Debian with the inroads Ubuntu and Canonical are making, and you have a tasty recipe for success. If we continue to support this alliance with our money, it will continue to become a viable alternative to Red Hat for enterprises. Once enterprises begin to support Debian+Ubuntu+Canonical, hardware vendors are more likely to support this alliance; an all-round victory for all.

I am not against Red Hat. In fact, I am a big fan of their success and contribution. I use RHEL and CentOS whenever I can. But I am also a big fan of using the right tools for the job. And for this idea to work, we need choices.

Now let’s look at the CentOS “debacle”. I have always trusted CentOS to provide the best product they can. Recent events have not dented my confidence. In fact, they have reinforced the idea of not relying on one person for anything. At least two or more people should have access and ability to manage all aspects of a group or project. You know, the “hit by the bus” scenario. CentOS can only improve from this moment.

CentOS provides a great service to all Linux users. It builds on the good work of Red Hat and introduces more users to the benefits of using Red Hat, thus supporting them. All contributors to CentOS should be lauded for their voluntary efforts.

And then there are those who had little confidence in open source to begin with and which has decreased significantly because of recent events. They see open source in general, and Linux in particular, as less reliable than proprietary solutions. I think the reason is not technical but commercial: who will support me if I use this product? I would rather not have such people using Linux then. If they don’t believe in open source, they don’t deserve it either.

I know of the fight between “open source” and “free”. I prefer “open source” because “free”, although a laudable goal, is not practical for all use cases. I would rather everything be “free” but if it’s not workable then I would rather support “open source” than proprietary.

In conclusion, I think Linux is better off today than it was a week ago. Debian is changing itself to keep pace with the changing world. CentOS has gone through a rough period to emerge stronger. Things don’t always remain the same but if they are getting better, it’s a victory for all of us.

What is ‘wheel’?

I came across a very interesting article titled Why GNU su does not support the ‘wheel’ group. I have seen wheel under Red Hat Enterprise Linux but never understood why such a special group existed. I didn’t even bother to find out. But the aforementioned article made me curious. So off to Google. The wheel Group was the one I was looking for. According to it, wheel is a special group which in conjunction with settings for su determines which users are allowed to use the su command. So if you have a wheel group set up then you may configure su so that only members of wheel are able to use the command. As one commentor on the post put it, this creates a circle of trust, and is called wheel.

Quick Introduction to Advanced SSH

I will not go through the complete overview of advanced SSH. For that, you may either use a search engine or read the following articles: SSH Advanced Techniques; Advanced SSH Usage; SSH Host Key Protection; ssh-keygen Tutorial; and SSH authorized_keys howto. What I will provide, though, are three quick commands:

Generate new public/private key pair, using RSA as encryption type and key strength as 4096 bits:
ssh-keygen -t rsa -b 4096

Change the passphrase used in the key
ssh-keygen -f id_rsa -p

Check the fingerprint of a public key
ssh-keygen -f id_rsa.pub -l