Convert pst to mbox

I was helping a friend migrate from Windows to Linux. I tried to install Outlook 2003 in Wine. It installed without problems, was able to read his original pst file, but when trying to create an email account so he could begin sending and receiving his emails, the screen was blank. The next issue was bringing in all his emails from Outlook 2003 to Thunderbird 3 so that he could use it. This is where readpst comes in handy. You can use it to convert pst to mbox which you can then copy/paste into your Thunderbird profile.

Install in Debian: sudo aptitude install readpst
Install in Fedora: sudo yum install libpst

Then you run the following command to convert your pst into mbox:

readpst -r Outlook.pst

Sort String Based on a Substring Key in Python

I have a scenario where I need to sort a file based on a substring in each line. The format of each line is fixed, each line has a fixed length (with some exceptions due to error, etc.), and the file has to be sorted based on a substring key. Example lines could be as below:

ABC800555121200000854112520100601002358000048
ABC800555548500000854112520100601070112000041
ABC800555751200000854112520100601125539000077

I tried to use sort in bash but since there was no delimiter to break the lines, the -k flag did not seem to work. It wouldn’t sort on the value of, say -k 22,36, but instead would sort from the first character. I wanted to give awk a try but didn’t have much time to look at it. Since I already work with Python, I thought why not create a function to sort a list of strings based on a substring. Following is the result:

def sort_on_substr(string_list, pos_start, pos_end=None):
    sort_key_list = []
    sort_key_dict = {}
    for one_string in string_list:
        if not pos_end:
            pos_end_here = len(one_string)
        elif pos_end > len(one_string) or pos_end  len(one_string):
            pos_start_here = len(one_string)
            pos_end_here = pos_start_here
        else:
            pos_start_here = pos_start
        sort_key = one_string[pos_start_here:pos_end_here]
        sort_key_list.append(sort_key)
        if sort_key in sort_key_dict:
            templist = sort_key_dict[sort_key]
            templist.append(one_string)
            sort_key_dict[sort_key] = templist
        else:
            templist = [one_string]
            sort_key_dict[sort_key] = templist
    sort_key_list_sorted = sorted(sort_key_list)
    return (sort_key_list_sorted, sort_key_dict)

To use this function you need to give a list of strings to sort, starting position of substring and ending position of substring (optional). It creates and returns two things: (1) a sorted list containing the substring keys; (2) a dictionary containing the string as value with the substring key as a key. Now you can iterate over the sorted list and stick it in the dictionary as a key to get the value. In my use case there could be one key in more than one strings so the dictionary stores the value as a list of strings. For me the order of this list does not matter because if the key is the same then the strings are equal and any one of them could come after the other in a sorted list.

An alternative to this approach is to forgo the sorted list of keys and instead sort the dictionary itself. Let’s leave it as an exercise for you in case you want to implement it that way.

Now all I have to do is supply a list of strings (maybe read from a file) and then give positions for the key. The result is a sorted list which I can then use as required.

Installing Linux on Dell Inspiron Mini 1012

I got a Dell Inspiron Mini 1012 recently just to use as a machine to distro-hop. I tried Live CDs of many distributions and following are my observations. Since there’s no CD or DVD drive in the Mini 1012, I used Windows, UNetbootin, and a Patriot Razzo USB thumb drive to create my installation media.

PCLinuxOS 2010

I downloaded the GNOME and KDE versions. All buttons worked in the KDE version, including brightness and audio keys. Wireless was detected automatically and I was able to connect to my home wireless network without problems. The only problem was that audio was too low, even after maximizing the audio level. Thinking it might be an issue with KDE, I ran the GNOME version.

In GNOME all keys worked except the audio keys. Audio itself was very good, unlike in KDE. But no matter what I tried I could not get the audio keys to work. Again, wireless worked out of the box and I was even able to watch videos in You Tube without having to install Flash. But because of the problems with either too low audio or the audio keys not working, I decided to skip PCLinuxOS.

gNewSense

I was aware that gNewSense provides free software only and would not work with hardware devices requiring proprietary drivers. I still gave it a shot. As I had expected, wireless did not work and I found no option to actually get it to work. Kudos to the project for sticking to their principles but I do need to use wireless. All keys, such as brightness and audio worked out of the box. I had to skip it because of the wireless issue.

Linux Mint

I downloaded Linux Mint 9, based on Ubuntu. All keys worked but wireless would not work. Since I have read in many places that Mint is supposed to be a friendlier version of Ubuntu I was a little disappointed. So I just left it at that.

Peppermint Linux

This is, I believe, based on Linux Mint. It had the same issue with wireless card not working. I skipped it for the same reason.

openSUSE

I downloaded the 11.2 version but when booting I got an error message saying Could not find kernel image: gfxboot. A little searching showed (Could not find kernel image: gfxboot) that the issue is with using UNetbootin to create the installer on a USB drive. I didn’t feel inclined to follow the official method and so skipped it.

Mandriva

I tried to get Mandriva to boot but it would get stuck on a splash screen of some sort. I searched around a bit but did not find a solid way to solve the problem. The result: skip it.

Ubuntu

Since Dell has been working with Ubuntu I expected it to work flawlessly. So instead of playing around with the Live CD I just went in there and started installing Ubuntu 10.04. The first thing that impressed me was during disk partitioning it recognized the original Windows installation and prepared a plan to keep it while installing Ubuntu. Once I booted for the first time I was shown a message saying there was a proprietary driver available for my wireless card. I installed it and everything worked from. All keys were working as expected. I kept it around for a few days and then decided to move on.

Fedora

Soon after I installed Ubuntu Fedora 13 was released. I booted its Live CD and found the same repeated problem: wireless card would not work. I searched around and found a solution: install RPM Fusion repo and then follow the instructions of How To: Wireless LAN with Broadcom BCM4312 in Fedora 11. This basically involved running the following command and then re-booting the machine: sudo yum install broadcom-wl wl-kmod.

During installation Fedora also found the Windows and Ubuntu partitions and allowed me to install over Ubuntu while keeping Windows intact.

I installed Flash using the instructions at Fedora 11 Flash. And from there You Tube videos started working. All keys worked out of the box; there were no issues with audio either.

Conclusion

As of this writing I am happily running Fedora 13 on my Dell Mini 1012. Although I tried a lot of distributions, I didn’t spend as much time trying to get them to work on this machine as I did with Ubuntu and Fedora. For this reason I would give all of them a try sometime in the future. Ubuntu has a netbook remix which I tried as well, I prefer the simple GNOME with just one bar at the top to better utilize my netbook’s screen space. If you have the same or similar netbook please share your experience with distro-hopping on it (if any) and which distro you are happily running these days.

lspci Output

I ran lspci and following is the output.

00:00.0 Host bridge: Intel Corporation N10 Family DMI Bridge
00:02.0 VGA compatible controller: Intel Corporation N10 Family Integrated Graphics Controller
00:02.1 Display controller: Intel Corporation N10 Family Integrated Graphics Controller
00:1b.0 Audio device: Intel Corporation N10/ICH 7 Family High Definition Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 1 (rev 02)
00:1c.1 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 2 (rev 02)
00:1d.0 USB Controller: Intel Corporation N10/ICH7 Family USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #4 (rev 02)
00:1d.7 USB Controller: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e2)
00:1f.0 ISA bridge: Intel Corporation NM10 Family LPC Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation N10/ICH7 Family SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation N10/ICH 7 Family SMBus Controller (rev 02)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 02)
07:00.0 Network controller: Broadcom Corporation BCM4312 802.11b/g LP-PHY (rev 01)

We are under attack

The heading may be alarmist but there’s nothing to be alarmed about. I have been collecting IPs and user names for SSH attacks over the past few weeks. Initially I did not use throttling so I got a whole bunch of names. Recently I have implemented iptables rules so that if a user enters wrong password three times in one minute when attempting to connect via SSH, further connections are blocked for a few minutes. This change means that usually I only see three attempts in logs and then nothing more. Unless, of course, the attacker is smart and re-tries after waiting a few minutes. There have been a handful of such smart attackers. Mostly I see just three attempts from an IP and then nothing more.

My interest lies in two statistics: (1) which user names are tried and how many times; and (2) which geographic locations are used for these attacks. I have compiled a list of unique user names with the counts of attempts. The top ten names and their counts are as below.

347 root
39 user
31 admin
30 nagios
23 test
18 oracle
17 webmaster
14 teste
13 stud
12 web

We could already have guessed that root would be at the top. So the first lesson here is to disable root login in SSH. The second lesson is to disable SSH access for all users except a few who really need to login. The third lesson is to use an unconventional user name. So instead of, say fred (your name), try to use your screen name, say fredinlondon. This will reduce the possibility of someone guessing your user name and password. Of course, if you disable password login and only allow public key authentication, you don’t even have to worry about this.

Now comes the geographic locations of attackers. The top ten countries, after doing whois $IP, were as below.

73 CN
10 US
8 TW
6 IN
6 BR
4 DE
4 CL
3 RU
3 PK
3 PH

In these numbers, China is the location from which most attacks originated. The first lesson learned here is if you don’t live in China, you can try to block all SSH traffic coming from there. Similarly, if you don’t live in Taiwan, India, Brazil, and so on, you can cut down on the number of attacks by blocking SSH access from those countries. A helpful resource is Country IP Blocks if you want to pursue this route. Of course, you could do the opposite and only allow traffic from countries from where you expect to access your machine. This means that you can still be targeted but by machines with IPs of these countries only.

In conclusion, this has been a fun exercise for me. There have been lessons learned which can help secure my machine against malicious would-be users. But more than that I have learned how to use my Linux machine to gather this data. You should give it a try sometime.

Install MoinMoin in Debian

I have read some good things about MoinMoin and decided to give it a shot. Some of its features I like are: built using Python, stores data in files (not in database), and Debian has a package available.

Install Software

To install the software, run the following command:

sudo aptitude install python python-moinmoin apache2

After installation, some paths related to MoinMoin are as below:

Source code for MoinMoin is located at: /usr/lib/pymodules/python2.5/MoinMoin/
Templates for MoinMoin are located at: /usr/share/moin/
Config for your wikis is located at: /etc/moin/

Create Your First Wiki

I want to thank these pages for helping me write this section. You should read them in this order as well: Basic Installation; WikiInstanceCreation.

I installed my wiki in the /opt/wiki/ path. So I needed to create the directory.

sudo mkdir /opt/wiki/

Once the directory has been created, you need to copy over files from the MoinMoin templates.

cd /opt/wiki/
sudo cp -R /usr/share/moin/data/ /opt/wiki/
sudo cp -R /usr/share/moin/underlay/ /opt/wiki/
sudo cp /usr/share/moin/config/wikiconfig.py /opt/wiki/
mkdir /opt/wiki/cgi-bin
cd /opt/wiki/cgi-bin/
cp /usr/share/moin/server/moin.cgi /opt/wiki/cgi-bin/

Since we installed in a location which is accessible to root by default only, we need to change its permissions. We change the owners: owned by user codeghar and owned by group www-data. www-data is the user and group under which Apache runs in Debian. This way I, as a user, and Apache, as a program, can edit files easily.

cd /opt/
sudo chown codeghar:www-data wiki
sudo chmod -R ug+rwx wiki
sudo chmod -R o-rwx wiki

The config for MoinMoin is stored in /etc/moin/. To install our first wiki, we need to make some changes.

cd /etc/moin/
sudo vim mywiki.py

Change the following settings in the file /etc/moin/mywiki.py

sitename = u'CodeGharWiki'
interwikiname = u'CodeGharWiki'
data_dir = '/opt/wiki/data/'
data_underlay_dir = '/opt/wiki/underlay/'
url_prefix_static = '/moin_static192' # here the value 192 depends on which version of MoinMoin you have installed. On my Debian Squeeze installation it was 1.9.2; hence, 192.

Configure Apache

Now you need to configure Apache to run your wiki. I want to thank Apache on Linux in helping with this section.

cd /etc/apache2/sites-available/
sudo vim wiki

Make sure the following exists in this file.

<VirtualHost *:80>
#comment the line below if you intend to use only http://localhost/mywiki
ServerName wiki.codeghar.com
DocumentRoot /opt/wiki/
Alias /moin_static192/ "/usr/share/moin/htdocs/"
ScriptAlias /wiki "/opt/wiki/cgi-bin/moin.cgi"
<Directory /opt/wiki/>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
allow from all
Options +ExecCGI
</Directory>
</VirtualHost>


Once you have created the file, you need to install the site in Apache by running the following commands.

sudo a2ensite /etc/apache2/sites-available/wiki
sudo /etc/init.d/apache2 reload

Browse Your Wiki

Everything should be installed and ready to go. Just point your browser to http://wiki.codeghar.com/wiki/ (or whatever your website is) and enjoy.

Hat tips: MoinMoin; Help on Installing.

vpnc Keeps Disconnecting

I had installed vpnc to connect to a Cisco VPN on an Ubuntu machine. The problem was the connection kept dropping. It would remain alive for five minutes or so and then vpnc would disconnect. I finally found an answer at vpnc disconnect every 5 minutes on Hardy. You have to enable Disable Dead Peer Detection. This should solve the problem.