Install Nikola in virtualenv

Nikola is a static website/blog generator written in Python. You can install it in a virtualenv. Although it supports Python 2 and Python 3 I had trouble using Nikola version 7.1 with Python 3.4. Python 2.7 (tested on Ubuntu 14.04) and Python 3.3 (tested on Mac OS X Mavericks) worked fine.

Install pre-requisites: sudo aptitude install build-essential python-dev python-pip python-virtualenv libxml2-dev libxslt-dev zlib1g-dev

I had to add a few dev libraries to get rid of some errors on Ubuntu.

Install libxml2-dev and libxslt-dev to get rid of error: “fatal error: libxml/xmlversion.h: No such file or directory”.

Install zlib1g-dev to get rid of error: “/usr/bin/ld: cannot find -lz”.

cg@host ~ $ virtualenv virt-nikola

cg@host ~ $ source virt-nikola/bin/activate

(virt-nikola)cg@host ~ $ pip install nikola

(virt-nikola)cg@host ~ $ pip install nikola[extras]

Hat tip: “/usr/bin/ld: cannot find -lz”, Ubuntu 11 : fatal error: libxml/xmlversion.h: No such file or directory

Advertisements

Create Sequence Diagrams Using seqdiag

Have you ever needed to create sequence diagrams? If you’re like me you constantly have to. Up to now I have always used a manual process using pen/paper or Visio. But I wanted an easier way and found seqdiag.

I installed it in a virtualenv on Ubuntu 14.04.

codeghar@host~$ virtualenv --system-site-packages virt

I used --system-site-packages because seqdiag relies on Python Imaging Library (PIL). Of course, make sure you have python-pil package installed in your OS. I ran into this problem (About the PIL Error — IOError: decoder zip not available) when I didn’t use --system-site-packages. The impression I got was it’s hard to get PIL installed in a virtualenv and easier to just use the system-wide PIL.

codeghar@host~$ source virt/bin/activate

(virt)codeghar@host~$ pip install seqdiag

To create a diagram you need to provide a dialog in a .diag file. You can look at the sample diagrams for more information. The following will create a new file (or overwrite an existing file) called example.png.

(virt)codeghar@host~$ seqdiag -Tpng --no-transparency example.diag

Hat tip to generate uml sequence diagrams with python or perl for bringing seqdiag to my attention.

SaltStack Quickstart Guide on Mac OS X

As always, refer to the official SaltStack documentation first.

To get up and running with SaltStack on Mac OS X can be a daunting task. I have found it much easier to use MacPorts to accomplish any such tasks.

Step 1: Install MacPorts on your Mac OS X. I find this to be an essential step for any developer using Mac OS X. Of course, there’s Homebrew as an alternative. I haven’t used it but you may check it out as well.

Once you have setup MacPorts successfully you are ready to begin. Of course, since MacPorts provides a variety of GCC versions I prefer it over the default gcc provided on my Mac. Install a newer GCC and link your system to it.

[user@host ~] $ sudo port install gcc48

[user@host ~] $ sudo mv /usr/bin/gcc /usr/bin/gcc.orig

[user@host ~] $ sudo ln -s /opt/local/bin/gcc-mp-4.8 /usr/bin/gcc

Install Python 2.7.

[user@host ~] $ sudo port install python27

Install Virtualenv.

[user@host ~] $ sudo port install py27-virtualenv

M2Crypto is a requirement for SaltStack and you need to install SWIG. Otherwise you may get errors[1].

[user@host ~] $ sudo port install swig

[user@host ~] $ sudo port install swig-python

Create your virtualenv.

[user@host ~] $ /opt/local/bin/virtualenv-2.7 ~/dev/virt-env/salt-virt

Activate the virtualenv.

[user@host ~] $ source ~/dev/virt-env/salt-virt/bin/activate

Install SaltStack pre-requisites.

(salt-virt)[user@host ~] $ pip install pyzmq PyYAML pycrypto msgpack-python jinja2 psutil

Install SaltStack.

(salt-virt)[user@host ~] $ pip install salt

Make sure things are working well.

(salt-virt)[user@host ~] $ python

Python 2.7.8 (default, Jul 13 2014, 17:11:32) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import salt
>>> ^D

To deactivate the virtualenv,

deactivate

Note: this post was created using Mac OS X 10.9 (Mavericks) with SaltStack 2014.1.7.


[1] Possible SWIG error

unable to execute 'swig': No such file or directory

error: command 'swig' failed with exit status 1

gevent Quickstart Guide

This is a simple list of steps on how to get started with installing gevent on openSUSE (tested on openSUSE 13.1 with gevent 1.0.1). We’ll use virtualenv instead of a system-wide install.

You need to have a compiler (gcc) and Python development library.

[user@host ~] $ sudo zypper install gcc python-devel

Install Python 2.7 virtualenv, since gevent 1.0.1 doesn’t appear to support Python 3.

[user@host ~] $ sudo zypper install python-virtualenv

Create virtual environment in your home directory (or anywhere else you want).

[user@host ~] $ virtualenv ~/virt

Activate the virtualenv.

[user@host ~] $ source virt/bin/activate

Install gevent.

(virt) [user@host ~] $ pip install gevent

Check it all works.

(virt) [user@host ~] $ python

Python 2.7.6 (default, Nov 21 2013, 15:55:38) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gevent import sleep
>>> sleep(2)
>>> 

When you’re done deactivate the virtualenv.

(virt) [user@host ~] $ deactivate

virtualenv with Python 3 using venv

Python 3 comes with its own virtualenv built in, and it’s called venv. Using it is fairly simple and quick. Make sure you have installed Python 3 on your system. Also make sure you have installed the Python 3 development library for your distribution. For example, install python3-devel on Fedora and python3-dev on Ubuntu.

Then run the following command to create your virtualenv.

cg@codeghar [~] $ pyvenv ~/myvenv

Note: In Ubuntu 14.04 (attempted on 2014-04-17) you may get an error:

cg@codeghar [~] $ pyvenv-3.4 ~/myvenv
Error: Command '['/home/cg/myvenv/bin/python3.4', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

To work around it, make sure you use the --without-pip flag, like so pyvenv-3.4 ~/myvenv --without-pip.

If you do an ls you’ll notice that the python you used to create this virtualenv is copied into your virtualenv. Thus your virtualenv will continue to use its local copy. You now need to “activate” this virtualenv.

cg@codeghar [~] $ source ~/myvenv/bin/activate

Sidebar: At some point you’ll want to “deactivate” your virtualenv. To do this just run the deactivate command.

Next step is to install setuptools. At one point Distribute was set to replace/supersede setuptools but that’s not the case anymore. Don’t worry, we’ll still use pip but to first get pip we need setuptools.

cg@codeghar [~] $ cd ~/myvenv

cg@codeghar [~/myvenv] $ mkdir pypioffline

cg@codeghar [~/myvenv/pypioffline] $ cd pypioffline

The latest version of setup tools at the time of writing was 1.1.6 so we’ll download it to the temporary/optional directory pypioffline.

cg@codeghar [~/myvenv/pypioffline] $ curl -O https://pypi.python.org/packages/source/s/setuptools/setuptools-1.1.6.tar.gz

cg@codeghar [~/myvenv/pypioffline] $ tar xvzf setuptools-1.1.6.tar.gz

cg@codeghar [~/myvenv/pypioffline] $ cd setuptools-1.1.6

cg@codeghar [~/myvenv/pypioffline/setuptools-1.1.6] $ python ez_setup.py

cg@codeghar [~/myvenv/pypioffline/setuptools-1.1.6] $ easy_install pip

That’s pretty much it.

venv — Creation of virtual environments

Install Python 3 locally under /home directory in CentOS 6.4

First off, why would you want to install local Python 3? There could be many reasons, some of which may be:

  • You don’t want to touch the system-installed Python
  • You don’t have root privileges
  • You want to run your private/personal scripts without affecting other users on the system
  • You want to run multiple versions of Python not available on your system

For whatever the reasons you want to install Python locally in your home directory, it’s not as difficult as I had first thought it to be. This is a good alternative especially since Python3 is not officially available in CentOS 6.4.

There are four major steps:

  1. Install Development Tools
  2. Download Python Source Code
  3. Compile and Install Python
  4. Modify your $PATH

Install Development Tools

You will need some tools to compile Python. The easiest way to obtain them in CentOS is to install the ‘Development Tools’ group.

cg@codeghar [~] $ su -c 'yum groupinstall "Development Tools"'

You also need some libraries that Python uses to provide “batteries included” support.

cg@codeghar [~] $ su -c 'yum install openssl-devel bzip2-devel expat-devel gdbm-devel readline-devel sqlite-devel'

Download Python Source Code

Head over to Python.org and download the version you want to install. For this post I used Python 3.3.2.

cg@codeghar [~] $ curl -O http://python.org/ftp/python/3.3.2/Python-3.3.2.tar.bz2

Now just un-tar the file so you can use its contents.

cg@codeghar [~] $ tar xvf Python-3.3.2.tar.bz2

You should see these two items in your home directory:

cg@codeghar [~] $ ls

Python-3.3.2  Python-3.3.2.tar.bz2

Compile and Install Python

You now have Python source and tools to compile it. Change directory to the un-tar’ed source code.

cg@codeghar [~] $ cd Python-3.3.2

Run configure.

cg@codeghar [~/Python-3.3.2] $ ./configure

Create a directory where you will install your local Python.

cg@codeghar [~/Python-3.3.2] $ mkdir -p ~/usr/local

Compile and install Python to the directory you just created.

cg@codeghar [~/Python-3.3.2] $ make altinstall prefix=$HOME/usr/local exec-prefix=$HOME/usr/local

Modify your $PATH

You now want to add ~/usr/local/bin to your $PATH so you don’t have to enter the full path to the Python binary every time you want to use it. Change directory to where local Python binary is located.

cg@codeghar [~/Python-3.3.2] $ cd ~/usr/local/bin

Create an alias so you can refer to it as python3. This is optional but I like to do it because I can then use python to refer to the Python that came with CentOS and python3 to refer to my locally installed Python3.

cg@codeghar [~/usr/local/bin] $ ln -s python3.3 python3

Now add this path to your $PATH environment variable. Notice that we add the new path before existing $PATH. This is so that our local Python is always used before any other.

cg@codeghar [~/usr/local/bin] $ echo "export PATH=\$HOME/usr/local/bin:\$PATH" >> ~/.bashrc

When you open a new terminal window, or log off and then log on, or (even better) run source ~/.bashrc, you can run the following commands to see that your local Python is now on your path.

cg@codeghar [~] $ which python3.3

~/usr/local/bin/python3.3

cg@codeghar [~] $ which python3

~/usr/local/bin/python3

cg@codeghar [~] $ which python

/usr/bin/python

You can even run your local Python.

cg@codeghar [~] $ python3

Python 3.3.2 (default, Sep 26 2013, 15:30:36) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print ("Hello World!")
Hello World!
>>> 

Conclusion

You can now run your local Python. There’s no need to keep the tar file and the source code around. You may want to delete them.

cg@codeghar [~] $ rm -rf ~/Python-3.3.2

cg@codeghar [~] $ rm ~/Python-3.3.2.tar.bz2

Another thing to consider is virtual environments, virtualenv or venv (in Python 3). Since Python3 comes with its built-in venv, just create an alias for its binary as well.

cg@codeghar [~] $ cd ~/usr/local/bin

cg@codeghar [~/usr/local/bin] $ ln -s pyvenv-3.3 pyvenv

There’s one more modification you need to make to get pyvenv to work. Open the file and change the path to your locally installed Python.

cg@codeghar [~/usr/local/bin] $ vim pyvenv-3.3

Original line is:

#!/usr/local/bin/python3.3

After changing it should look something like:

#!/home/cg/usr/local/bin/python3.3

You can now use this local pyvenv to create your virtualenvs.

Hat Tips

How to install locally python on linux home directory?; Installing Python 3 on CentOS/Redhat 5.x From Source; UNIX / Linux: Set your PATH Variable using set or export command; .bash_profile vs .bashrc; RHEL / CentOS Linux Install Core Development Tools Automake, Gcc (C/C++), Perl, Python & Debuggers

Beginning AES with Python3

Encryption is a vast field and one post can never do it justice. But I’ll try to provide code examples on how to use the PyCrypto library to work with AES.

Disclaimer: My programming skills might not be up to par when it comes to encryption. Try to learn from my mistakes (when I make them).

Install the library in Fedora:

yum install python3 python3-crypto

Let’s just dive into the code. I have tried to make it simple and clear.

#!/usr/bin/env python3
from Crypto.Cipher import AES
from Crypto import Random
from base64 import b64encode, b64decode
from Crypto.Util import Counter
from binascii import hexlify

print('```GENERAL```')
print('AES block size: {0}'.format(AES.block_size))
original_key = 'This is my k\u00eay!! The extra stuff will be truncated before using it.'
key = original_key.encode('utf-8')[0:32]
print('Original Key: {0}'.format(original_key))
print('Usable Key: {0}'.format(key))
print('Base64 Encoded key: {0}'.format(b64encode(key).decode('utf-8')))
message = '0123456789'.encode('utf-8')
print('Original Message: {0}'.format(message))

#````````````````````````````````

# MODE CFB
print('```MODE CFB```')
cfb_iv = Random.new().read(AES.block_size)
print('Base64 Encoded IV: {0}'.format(b64encode(cfb_iv).decode('utf-8')))



cfb_cipher_encrypt = AES.new(key, AES.MODE_CFB, cfb_iv)
cfb_msg_encrypt = b64encode(cfb_cipher_encrypt.encrypt(message))
print ('Mode CFB, Base64 Encoded, Encrypted message: {0}'.format( cfb_msg_encrypt.decode('utf-8')))



cfb_cipher_decrypt = AES.new(key, AES.MODE_CFB, cfb_iv)
cfb_msg_decrypt = cfb_cipher_decrypt.decrypt(b64decode(cfb_msg_encrypt)).decode('utf-8')
print('Mode CFB, Decrypted message: {0}'.format(cfb_msg_decrypt))



#````````````````````````````````

# MODE CTR
print('```MODE CTR```')
def ctr_pad_message(in_message):
    # http://stackoverflow.com/questions/14179784/python-encrypting-with-pycrypto-aes
    # We use PKCS7 padding
    length = 16 - (len(in_message) % 16)
    return (in_message + bytes([length])*length)
def ctr_unpad_message(in_message):
    return in_message[:-in_message[-1]]


ctr_iv = int(hexlify(Random.new().read(AES.block_size)), 16)
print('CTR IV (int): {0}'.format(ctr_iv))
ctr_encrypt_counter = Counter.new(128, initial_value=ctr_iv)
ctr_decrypt_counter = Counter.new(128, initial_value=ctr_iv)



ctr_padded_message = ctr_pad_message(message)
print('Mode CTR, Padded message: {0}'.format(ctr_padded_message))
ctr_cipher_encrypt = AES.new(key, AES.MODE_CTR, counter=ctr_encrypt_counter)
ctr_msg_encrypt = b64encode(ctr_cipher_encrypt.encrypt(ctr_padded_message))
print('Mode CTR, Base64 Encoded, Encrypted message: {0}'.format( ctr_msg_encrypt.decode('utf-8')))



ctr_cipher_decrypt = AES.new(key, AES.MODE_CTR, counter=ctr_decrypt_counter)
ctr_msg_decrypt = ctr_cipher_decrypt.decrypt(b64decode(ctr_msg_encrypt))
ctr_unpadded_message = ctr_unpad_message(ctr_msg_decrypt)
print('Mode CTR, Decrypted message: {0}'.format(ctr_msg_decrypt))
print('Mode CTR, Unpadded, Decrypted message: {0}'.format(ctr_unpadded_message))

Here I have provided examples for two modes: CFB and CTR. Although both of them should not use fixed blocks for some reason CTR expects you to use fixed blocks in this library.