Introduction to Python subprocess module
December 9, 2011
First off, head over to subprocess — Subprocess management to get all the details. This post will try to provide a gentle introduction to subprocess and my experience using it. There will be some suggestions here that I *think* are correct but be careful when you implement them in your code. Also remember that I wrote and tested this code using Python 3.1 on Debian Squeeze.
First off, I found it better to just use the Popen class and not the convenience functions provided. Using it helped me get a better handle on what’s going on. Second, learn the difference between Popen.wait() and Popen.communicate(). wait() basically sets Popen.returncode but keeps the stdout and stderr pipes as is. communicate() sets Popen.returncode but also returns stdout and stderr and closes the pipes so you can’t use them again as stdin for another command.
Third, use the shlex module so that you don’t have to fight with the command while creating a list to feed to args in Popen.
#!/usr/bin/env python3 # -*- coding: utf-8 -*- import subprocess import sys import shlex command_line = "sed -e 's/^import dev as settings_file$/import production as settings_file/' test -i" command_to_run = shlex.split(command_line) print (command_to_run) try: command_run = subprocess.Popen(command_to_run, stdout=subprocess.PIPE, stderr=subprocess.PIPE) except: raise command_run_stdout, command_run_stderr = command_run.communicate() print (command_run.returncode, command_run_stderr.decode('utf-8')) print (command_run_stdout.decode('utf-8'))
The preceeding code sample is pretty self-explanatory. I used shlex to create a list from my command string, a list used in the Popen class. I set both stdout and stderr to send their output to pipes. command_run is an object representing the command I ran. Using communicate(), I get three things: returncode (set automatically), stdout (returned by communicate), and stderror (returned by communicate). Since command_run_stdout and command_run_stderr are byte strings, I convert them into UTF-8 before printing.
I will modify the preceeding code so that I can use the stdout and stderr as stdin for another command.
#!/usr/bin/env python3 # -*- coding: utf-8 -*- import subprocess import sys import shlex command_line = "ls -l" command_to_run = shlex.split(command_line) print (command_to_run) try: command_run = subprocess.Popen(command_to_run, stdout=subprocess.PIPE, stderr=subprocess.PIPE) except: raise command_run.wait() print (command_run.returncode) command_to_run_2 = ["grep", "-i", "TOTAL"] try: command_run_2 = subprocess.Popen(command_to_run_2, stdin=command_run.stdout) except: raise
The biggest difference here was that I used wait() instead of communicate() so that I could use stdout as stdin for the second command.
If you are able to understand these things, I believe you are on your way to writing basic scripts that call out to the shell to do some tasks it’s best suited to do: run commands.