Django, Apache, and X-SendFile in Debian

The idea for this post is to allow downloading files only to restricted users. The only constraint for now is that users are logged in. As I add more code to the Django app, I plan to include the ability to specify which users should have access to which files.

This post assumes that you have a working Apache server with VirtualHost file setup for a Django project. I also assume that you would want to access files using a URL scheme such as http://example.com/download/filename.ext where filename.ext represents the file you want to download.

To install X-SendFile in Apache2, run the following:

sudo aptitude install libapache2-mod-xsendfile

And to make sure you enable it, run the following

sudo a2enmod xsendfile

Add the following two lines in your VirtualHost file, like so:

<VirtualHost *:80>
...
XSendFile on
XSendFileAllowAbove on
...
</VirtualHost>

The code for this has been uploaded to Launchpad under the project Django X-SendFile Download or DjangoXSFD for short. Since it’s a work in progress, you can always get the latest code from there.

Hat tips: Djangocon X-Sendfile Lightning Talk; Fast File Transfer with X-send file; How to make a private download area with django?; Having Django serve downloadable files; Fast private file transfers for Drupal;

Install Trac on Debian Squeeze

This post aims to provide a step-by-step on how to install Trac in Debian and get it working. These “instructions” were created exactly the way I setup my server. So paths and setup information can be modified for your project. The idea was this: there are many guides out there which provide the same information but in a more generic way; this post provides exact path information and such so that there is concrete, real world example of what needs to be done.

CAUTION: This post is a work in progress so there are many things which might not be properly documented and could have adverse effect.

Install Necessary Applications

sudo aptitude install trac bzr trac-bzr trac-spamfilter postgresql python-psycopg2 apache2 libapache2-mod-wsgi

Prepare Database

sudo vim /etc/postgresql/8.4/main/pg_hba.conf
Add following lines:
local tracdb tracuser password
host tracdb tracuser 127.0.0.1/32 md5
host tracdb tracuser 192.168.1.0/24 md5

psql template1 -W postgres
create database tracdb with encoding = 'utf8';
create user tracuser password 'password';
grant all on database tracdb to tracuser;
sudo /etc/init.d/postgresql-8.4 reload

Prepare Directories

cd /media/shared/Admin/
cd /home/codeghar/
mkdir /home/codeghar/trac
cd trac
cd /home/codeghar/trac/
mkdir repo env
cd /home/codeghar/trac/repo/
bzr init

Create Trac Project

cd /media/shared/Admin/env/
cd /home/codeghar/trac/env/
trac-admin $(pwd) initenv
trac-admin /home/codeghar/trac/env/ initenv
Project Name [My Project]> codeghartrac
Database connection string [sqlite:db/trac.db]> postgres://tracuser:password@localhost/tracdb
Repository type [svn]> bzr
Path to repository [/path/to/repos]> /media/shared/Admin/trac/repo

Enable Bazaar with Trac

To get Trac working with bzr, edit the trac.ini file (Hat tip: bzr – trac integration)
vim /media/shared/Admin/trac/env/conf/trac.ini
And add the following two lines, if they don’t already exist
[components]
tracbzr.* = enabled

First Test

Check if everything is working as it should be

tracd --port 8000 /media/shared/Admin/trac/repo/

Trac with WSGI and Apache

cd /media/shared/Admin/trac/www/
trac-admin /media/shared/Admin/trac/env/ deploy /media/shared/Admin/trac/www/
trac-admin /home/codeghar/trac/env/ deploy /home/codeghar/trac/www/

Put the following in your Virtual Host:

WSGIScriptAlias / /home/codeghar/trac/www/cgi-bin/trac.wsgi

<Directory /home/codeghar/trac/www/cgi-bin/trac.wsgi>
WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all
</Directory>

And then reload Apache. Your site is now available at http://yourip/

Hat tip: View of /packages/trac/trunk/debian/README.Debian; bazaar & trac-bzr; Recipe Installing Trac with PostgreSQL on CentOS4.2; Trac and mod_wsgi;

When to use GET and POST

In HTTP and/or HTML, there are two (main?) types of submissions: GET and POST. I have always had a hard time determining when to use GET and when to use POST. In other words, what is the main difference between GET and POST?

Quite simply, if the submission is reading data, without making any changes, use GET. If your submission will be making changes, or causing side-effects, use POST. For example, when doing a query, such as a Google search, the form should use GET. If you are creating a new account with Yahoo! mail, the form should use POST. The Google search is reading data and new account in Yahoo! mail is changing things.

Simple enough, right? Not so, as there are some situations where either could work. One of the best descriptions of the problem I have read is Methods GET and POST in HTML forms – what’s the difference? And here is what I learned from it (you might learn something else reading the same words):

When doing a submission which will not make changes use GET. When making changes, use POST. But you may want to use POST instead of GET in the following (major?) case: when you don’t want the data you submit to become a part of the URL. When using GET, the form you submit becomes a part of the URL. So if you have a couple fields, say myname and mynumber, using GET your URL might look like http://www.mydomain.com/someform.html/?myname=codeghar&mynumber=1234 after submission. The biggest benefit is that you can bookmark this URL and visit it in future as just another link. The biggest disadvantage, in my opinion, is when you have multiple fields, the fields and their data becomes part of the URL, thus making for ugly URLs. So instead of a neat looking, small URL, you get one huge string.

Don’t get me wrong: a useful URL, even if ugly, should be preferred to an unfriendly, pretty URL. But if you want to pretty-fy your URLs, use POST instead of GET. And with a pretty URL, your user doesn’t have to know the inner workings of your form. They can still look at the source code of the page to see what’s going on, but only if they want to. Encoding the form data into a URL forces them to deal with the data even if they don’t want to.

The second benefit of using POST is that if your form contains non-ASCII data, it doesn’t form a part of the URL, which might be a good thing if your HTTP server is unable to handle this data in the URL. I don’t know, maybe all modern servers or intermediary devices can handle this stuff easily. But better safe than sorry, eh?

So from today my best practice is thus: if the form has a small number of fields, showing the submitted data in the URL is not a problem, and/or URL should be bookmark-able for future reference, use GET if there are no side-effects. If none of these requirements is met, use POST.

One concern I have is: if using HTTPS, is the URL sent in the secure tunnel or is it plaintext for all to see? According to the responses at HTTPS – is URL string itself secure??, the URL should be encrypted before being sent to the server as it is sent as part of the tunnel rather than a separate string. I think it would depend on the implementation of the client and server (I could be wrong in thinking this).

Don’t take my understanding of the situation as the final word. Read as much as you can on the subject to form your own best practice. And if you share your understanding and best practice with us, it would help us as well.

Good reads and hat tips: URIs, Addressability, and the use of HTTP GET and POST; Methods GET and POST in HTML forms – what’s the difference?; Post/Redirect/Get;

Apache 2.0 in Ubuntu

This post will deal with setting up a web server in Ubuntu using Apache 2.0. I will not be able to cover every scenario but I hope I can cover the most common ones. Default paths and so on are based on Ubuntu 7.10, Gutsy Gibbon.

Install Apache

sudo apt-get install apache2

Make sure it has been installed and is working properly by going to the server’s IP address using any web browser.

Custom Configurations

Since Apache allows you to include other files for configuration, it might be a good idea to create a file with all your custom configurations in it. Then you just have to save that file and use it whenever needed without messing around with the default configurations.

sudo mkdir /etc/apache2/customconfig/

sudo touch /etc/apache2/customconfig/server.conf

And then add the following line to /etc/apache/apache2.conf

Include /etc/apache2/customconfig/

And from then on all custom configurations will be loaded by Apache.

Give Apache a Server Name

Check that the configuration files have a server name or not.

grep -n ServerName /etc/apache2/apache2.conf

grep -n ServerName /etc/apache2/httpd.conf

grep -n ServerName /etc/apache2/ports.conf

Usually, if a server name has not been set, Apache will complain when you start it. So just add a server name to your config file. If you have created a custom config file, such as server.conf above, just add the following line

sudo vim /etc/apache2/customconfig/server.conf

If you haven’t created a custom config file, add the following line to the main config file

sudo vim /etc/apache2/apache2.conf

And add the line:

ServerName localhost

Now when you restart Apache, it will not complain that it doesn’t have a server name.

Listen on Ports

You need to make sure that Apache is listening on at least two ports: 80 (HTTP) and 443 (HTTPS). Your /etc/apache2/ports.conf file should look something like this:

Listen 80
<IfModule mod_ssl.c>
Listen 443
</IfModule>

Create Site

To create a site, you first have to decide where to store the files. I like to store them under my home directory, with read access to the user under which Apache runs, which is more likely than not www-data in Ubuntu.

mkdir /home/me/web/ /home/me/logs/

sudo chown -R www-data /home/me/logs/

chmod g+rwx /home/me/logs

VirtualHost Configuration

Since I may need to host more than one sites on the server, I want to use VirtualHost and its associated settings. I prefer to keep separate files for each virtual host.

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/sitedomain

sudo vim /etc/apache2/sites-available/sitedomain

And then get to editing the file. After editing, my first virtual host config looks like this:

NameVirtualHost *:80
<VirtualHost *:80>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
</VirtualHost>

Enable New Site

After creating the config file for new web site, you need to enable it.

sudo a2ensite sitedomain

But to use it, you must also disable the default site

sudo a2dissite default

Next you have to reload configurations in Apache

sudo /etc/init.d/apache2 reload

Now when you navigate to the server, it will show your new site.

Install SSL

Make sure apache has SSL module available into it:

ls /etc/apache2/mods-available | grep ssl.conf

ls /etc/apache2/mods-available | grep ssl.load

Install OpenSSL and ssl-cert

sudo apt-get install openssl ssl-cert

Your Ubuntu installation may not have one critical piece of software: apache2-ssl-certificate. You may download it from Launchpad (apache2-ssl) and do the following steps (instructions courtesy of Apache2-SSL-Certificate for Ubuntu Feisty..How to install this missing script to get the SSL certificates up):

cd ~/

wget http://librarian.launchpad.net/7477840/apache2-ssl.tar.gz

tar xvzf apache2-ssl.tar.gz

sudo mv ssleay.cnf /usr/share/apache2/

sudo mv apache2-ssl-certificate /usr/sbin

sudo mkdir /etc/apache2/ssl

Now you may proceed with the next steps.

Configure SSL

First you have to generate an SSL certificate.

sudo apache2-ssl-certificate -days 365

It started asking a bunch of questions and here are sample answers:

Country name: US
State: New York
Locality: New York
Organization Name: My Organization
Organizational Unit Name: .
Server name: ssl.mysite.net
Email: myemail@mysite.net

To enable the SSL module in Apache, run the following

sudo a2enmod ssl

Site Configuration

Your virtual host config will change and be something like:

NameVirtualHost *:80
<VirtualHost *:80>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
</VirtualHost>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
SSLEngine On
SSLCertificateFile /etc/apache2/ssl/apache.pem
</VirtualHost>

Now you may navigate your site without problems.

Configure SSL with OpenSSL

If you want to use OpenSSL to create and maintain your certificates, and also use them with Apache, read my guide titled Create a Certificate Authority and Certificates with OpenSSL. Once you have created server certificates, you have to make just a few changes in your virtual host config: location of the key and certificate files. For example, your config file might now look like

NameVirtualHost *:80
<VirtualHost *:80>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
</VirtualHost>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
SSLEngine On
SSLCertificateFile /home/ca/certs/mydomain.com.cert.pem
SSLCertificateKeyFile /home/ca/private/mydomain.com.key.pem
SSLCACertificateFile /home/ca/certs/cacert.pem
</VirtualHost>

Redirect Traffic to SSL

And if you want to redirect all your port 80 traffic to the secure, SSL-enabled part of your web server, you first have to enable the Rewrite engine

sudo a2enmod rewrite

Then you have to enable it in your virtual host configuration and change it to

NameVirtualHost *:80
<VirtualHost *:80>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
SSLEngine On
SSLCertificateFile /home/ca/certs/mydomain.com.cert.pem
SSLCertificateKeyFile /home/ca/private/mydomain.com.key.pem
SSLCACertificateFile /home/ca/certs/cacert.pem
</VirtualHost>

Now all traffic on regular port will be redirected to secure port.

Access Using Client Certificates

If you want only selected people to be able to browse your website, you can create certificates for clients and set Apache to only allow them access. To create client certificates, read my guide titled Create a Certificate Authority and Certificates with OpenSSL.

On the Apache side, you have to change your virtual host config and make it look something like:

NameVirtualHost *:80
<VirtualHost *:80>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName mydomain.com
DocumentRoot /home/me/web/
ErrorLog /home/me/logs/error.log
SSLEngine On
SSLCertificateFile /home/ca/certs/mydomain.com.cert.pem
SSLCertificateKeyFile /home/ca/private/mydomain.com.key.pem
SSLCACertificateFile /home/ca/certs/cacert.pem
<Directory /home/me/web>
SSLVerifyClient require
SSLVerifyDepth 1
</Directory>
</VirtualHost>

Hat Tips

To be honest, my starting point was Ubuntu LAMP Server Guide, from where I took steps and then added more details where I thought they were needed.

Also helpful were: Apache2 SSL in Ubuntu; Apache2-SSL-Certificate for Ubuntu Feisty..How to install this missing script to get the SSL certificates up; SSL FAQ;

Django in Ubuntu

As I use Django in Ubuntu, I would like to point out little quirks here.

Install Django

Installing Django in Ubuntu is quite simple:

sudo apt-get install python-django

Django is installed in the path /var/lib/python-support/python2.5/django

Since I will be using MySQL, I needed to install another package which would let Python connect to it.

sudo apt-get install python-mysqldb

Start New Project

While trying to create a new project, run the following command:

cd /home/me/

django-admin startproject newprojectname

Although the documentation says to use django-admin.py, in Ubuntu Gutsy, you should not use the .py extension. Instead, just use django-admin. The source of this information is a post on Ubuntu Forums: Path to Django Installation.

Start a New Application

Go into the directory of your project. For example, if your project is called newprojectname, then

cd /home/me/newprojectname

Then run the following command

python manage.py startapp newappname

Inspect the Database

If you need to generate a model of your database automatically, you can easily inspect the database by first going into the directory of the project

cd /home/me/newprojectname

and then running this

python manage.py inspectdb

I usually save the output to a file so that I can look at it and make changes to it as required.

python manage.py inspectdb > ~/inspectdb-date.py

In my example, inspectdb-date.py can be used as a models.py file after rearranging the contents (Cleaning Up Generated Models).

Deploy Django with Apache

Although the Django Book has a very good way of explaining this stuff (Using Django with Apache and mod_python), I would like to explain the same thing step-by-step.

Install Apache: sudo apt-get install apache2

Install mod_python: sudo apt-get install libapache2-mod-python

Check if mod_python is enabled: ls /etc/apache2/mods-enabled/mod_python.load and if you see mod_python.load as a result, it is enabled

If mod-python is not enabled: sudo a2enmod mod_python

Restart Apache server: /etc/init.d/apache2 restart

Configure Virtual Host

Your virtual host config file should look something like this

sudo vim /etc/apache2/sites-available/mysite

NameVirtualHost *:80
<VirtualHost *:80>
ServerAdmin admin@mysite.com
ServerName mysite.com
ServerAlias http://www.mysite.com
DocumentRoot /home/me/mysite/
ErrorLog /home/me/logs/error.log
<Location "/">
SetHandler mod_python
PythonHandler django.core.handlers.modpython
PythonPath "['/home/me/'] + sys.path"
SetEnv DJANGO_SETTINGS_MODULE mysite.settings
PythonDebug On
</Location>
</VirtualHost>

Of course, if it is a production site, PythonDebug On should be omitted.

If you are moving your Django site from one server to another, it might be a good idea to first create a new site on the new server with the same name as the site being transferred. Then just copy all the files and folders into the new site. I don’t know how Django works in this regard but it is possible it keeps a list of projects created.

How Do I do it?

I have created a directory for templates and static (css, images, etc.) files: /home/me/templates/ and /home/me/templates/static/. My Django application resides in /home/me/mysite/. My file looks like

NameVirtualHost 192.168.0.10:80
<VirtualHost 192.168.0.10:80>
ServerAdmin myemail@mydomain.com
ServerName mysite.mydomain.com
DocumentRoot /home/me/templates/
ErrorLog /home/me/logs/error.log
LogFormat "%v %l %u %t \"%r\" %>s %b" comonvhost
CustomLog /home/me/logs/custom.log comonvhost
<Directory /home/me/templates/*>
Options -Indexes
</Directory>
<Location "/">
SetHandler mod_python
PythonHandler django.core.handlers.modpython
PythonPath "['/home/me/'] + sys.path"
SetEnv DJANGO_SETTINGS_MODULE myapp.settings
PythonDebug On
</Location>
<Location "/static">
SetHandler None
</Location>
<Location "/media">
SetHandler None
</Location>
</VirtualHost>

I also created a symlink to the Django admin media files in /home/me/templates/ (my apache root directory) so that http://mydomain.com/admin/ would look the way it’s supposed to with all its GUI intact.

cd /home/me/templates/
ln -s /var/lib/python-support/python2.5/django/contrib/admin/media media

Hat tips to: Django Installation on Apache; Serving the Admin Files;