Just a little Python

I've been thinking a bit about macros and what use they might be in Python. Basically, I was contemplating writing an import hook that would allow you to use code quoting and unquoting and stuff for your Python modules. My motive was just that Lisp people seem to rave about how awesome macros are all the time, so I figured they must be cool.

As I sat down to actually start figuring out what macro definitions and uses should look like in Python, I thought, hey, I'll just throw together a use case. But I haven't been able to come up with one (yet).

Most of the examples I found on the web focused on "hey, you can implement a 'while' loop with macros in Lisp!" or "hey, look at all the cool stuff the 'setf' macro can do!" So I started to wonder whether maybe Lisp people love macros because it allows them to extend Lisp's minimalist syntax with new constructs (like object-oriented programming with CLOS, while loops, etc.) Python, OTOH, has pretty rich syntax. It has a nice OOP system with syntactic support, while and for loops, generators, iterators, context managers, primitive coroutines, comprehensions, destructuring bind,.... -- What would I use macros for? (OK, depending on the syntax, I could add a "switch" statement, but that hardly seems worth the trouble.)

I should mention that I also saw some examples of people using macros for performance; you basically get rid of a function call and you can potentially make the inner loop of some critical function run really fast. But if that's all it buys me in Python-land (well, that and a switch statement), my motivation is pretty low. Because let's face it -- if your critical inner loop is written in pure Python, you can pretty easily throw it at Cython and get better performance than Python macros could ever provide.

So here's the question: does anyone out there have an idea of what macros would add to Python's power or expressiveness? Or maybe some Lisp, Meta OCAML, or Template Haskell hackers who can enlighten me as to what macros can add to a language with already rich syntax?

Today I had a need to create a property on an object "lazily." The Python builtin property does a great job of this, but it calls the getter function every time you access the property. Here is how I ended up solving the problem:

First of all, I had (almost) the behavior I wanted by using the following pattern:

class Foo(object):
    def __init__(self):
        self._bar = None
    @property
    def bar(self):
        if self._bar is None:
            print 'Calculating self._bar'  
            self._bar = 42    
        return self._bar      

class LazyProperty(object):

    def __init__(self, func):
        self._func = func
        self.__name__ = func.__name__
        self.__doc__ = func.__doc__

    def __get__(self, obj, klass=None):
        if obj is None: return None
        result = obj.__dict__[self.__name__] = self._func(obj)
        return result

class Foo(object):
    @LazyProperty
    def bar(self):
        print 'Calculating self._bar'  
        return 42

I just wanted to let you all know that I've changed from the blogger domain to my own blog.pythonisito.com. You should be redirected there automatically, but if you've noticed some hiccups in feeds or weird redirects from Reddit or Delicious, now you know why.

One of the things I do in my "spare" time is work on building web applications that will (hopefully) earn some spare money on the side without too much maintenance on my part. Those who have read The Four Hour Work Week will recognize this as my "muse" business.

In working on these web apps, I needed a place to host them, so I went with WebFaction due to their excellent support for TurboGears. I'm using a shared hosting environment with WebFaction, so my usual method of getting stuff up can't involve any scripts in /etc/init.d like I'd use at work, so I used to do the old nohup python start-appname.py >> output.log 2>&1 & to "daemonize" the process and then ps -furick446 to figure out which processes I needed to kill/restart when updating code. This was irritatingly verbose, so I figured I'd build a "userspace daemonizer" which I'll describe below.

The requirements of my daemonizer were pretty straightforward:

• I should be able to add/remove services from the command line with a minimum amount of typing


• I should be able to start/stop/restart any service just by listing it by name (no more ps -furick446)


• It should do "real" daemonization (fork, fork, dup stuff if you're familiar with it) rather than the nohup garbage I'd been using


• It should perform some sort of verification or status check on processes to make sure that they're successfully started/stopped/running/etc.

#!/usr/bin/env python2.5
import os, sys, signal, shlex, time
from optparse import OptionParser

from sqlalchemy import *
from sqlalchemy.orm import *

HOME=os.environ.get('HOME')
DBFILE=os.path.abspath(os.environ.get(
    'SERVER_FILE',
    os.path.join(HOME, 'server.sqlite')))

DBURI='sqlite:///' + DBFILE

engine = create_engine(DBURI)
metadata = MetaData(engine)
session = scoped_session(sessionmaker(bind=engine))

process = Table(
    'process', metadata,
    Column('name', String(255), primary_key=True),
    Column('pid', Integer),
    Column('command_line', String(255)),
    Column('working_directory', String(255)),
    Column('stdin', String(255), default='/dev/null'),
    Column('stdout', String(255), default='/dev/null'),
    Column('stderr', String(255), default='/dev/null'))

class Process(object):
    def __repr__(self):
        return '%s(PID %s, WD %s): %s < %s >> %s 2>> %s' % (
            self.name, self.pid, self.working_directory,
            self.command_line, self.stdin, self.stdout, self.stderr)
session.mapper(Process, process)      

OPTPARSE_OPTIONS = {
    'working-directory':(['-w', '--working-directory'],
                         dict(dest='ws', default=HOME,
                              help='Set working directory to DIR',
                              metavar='DIR')),
    'stdin':(['-i', '--stdin'],
             dict(dest='stdin', default='/dev/null',
                  help='Use FILE as stdin', metavar='FILE')),
    'stdout':(['-o', '--stdout'],
             dict(dest='stdout', default='/dev/null',
                  help='Use FILE as stdout', metavar='FILE')),
    'stderr':(['-e', '--stderr'],
             dict(dest='stderr', default='/dev/null',
                  help='Use FILE as stderr', metavar='FILE')),
    }      

@Command
def add(service, command, working_directory,
        stdin, stdout, stderr):
    p = Process(name=service,
                command_line=command,
                working_directory=working_directory,
                stdin=stdin,
                stdout=stdout,
                stderr=stderr)
    session.commit()      

• Add the function name to a command list so I can get a list of commands from the command line


• Build an optparse parser based on the OPTPARSE_OPTIONS dict and the named arguments to the function


• Wrap the function in a new function with a signature like function(args) so it can be called with sys.argv[2:]


• In the wrapper function, use the optparse parser to initialize the argument list to the function and then call with a named argument dict
  

class Command(object):
    commands = {}
    def __init__(self, func):
        Command.commands[func.__name__] = self
        options = self.get_options(func)
        self.func = func
        self.optparse_options = []
        self.optparse_option_names = []
        self.positional_options = []
        for o in options:
            if o in OPTPARSE_OPTIONS:
                self.optparse_option_names.append(o)
                self.optparse_options.append(
                    OPTPARSE_OPTIONS[o])
            else:
                self.positional_options.append(o)
        positional_string = ' '.join(
            '<%s>' % o for o in self.positional_options)
        self.parser = OptionParser('%%prog [options] %s' % positional_string,
                                   prog=func.__name__)
        for args, kwargs in self.optparse_options:
            self.parser.add_option(*args, **kwargs)

    def get_options(self, func):
        code = func.func_code
        return [ vn for vn in code.co_varnames[:code.co_argcount] ]

    @classmethod
    def run(klass, args):
        if args and args[0] in klass.commands:
            return klass.commands[args[0]](args[1:])
        else:
            print 'Unrecognized command'
            print 'Acceptable commands:'
            for name in klass.commands:
                print '  -', name

    def __call__(self, args):
        (opts,a) = self.parser.parse_args(args)
        if len(a) != len(self.positional_options):
            self.parser.error('Wrong number of arguments')
        o = {}
        for name in self.optparse_option_names:
            o[name] = getattr(opts, name, None)
        for name, value in zip(self.positional_options, a):
            o[name] = value
        return self.func(**o)      

@Command
def initialize():
    try:
        metadata.drop_all()
    except:
        pass
    metadata.create_all()
    print 'Service database initialized'

@Command
def list():
    q = Process.query()
    if q.count():
        for p in Process.query():
            print p
    else:
        print 'No processes'

@Command
def add(service, command, working_directory,
        stdin, stdout, stderr):
    p = Process(name=service,
                command_line=command,
                working_directory=working_directory,
                stdin=stdin,
                stdout=stdout,
                stderr=stderr)
    session.commit()

@Command
def remove(service):
    p = Process.query.get(service)
    session.delete(p)
    print 'Removed %s from the service list' % service
    session.commit()

@Command
def status(service):
    p = Process.query.get(service)
    print p      

def is_running(pid):
    try:
        os.kill(pid, 0)
        return True
    except Exception, ex:
        return False      

def daemonize(p):
    pid = os.fork()
    if pid: return # exit first parent
    os.chdir(p.working_directory)
    os.umask(0)
    os.setsid()
    pid = os.fork()
    if pid:
        Process.query.get(p.name).pid = pid
        session.commit()
        sys.exit(1) # exit second parent
    si = open(p.stdin, 'r')
    so = open(p.stdout, 'a+')
    se = open(p.stderr, 'a+', 0)
    os.dup2(si.fileno(), sys.stdin.fileno())
    os.dup2(so.fileno(), sys.stdout.fileno())
    os.dup2(se.fileno(), sys.stderr.fileno())
    args = shlex.split(p.command_line.encode('utf-8'))
    os.execvp(args[0], args)      

@Command
def start(service):
    print 'Starting %s ...' % service
    p = Process.query.get(service)
    if p.pid is not None:
        if is_running(p.pid):
            print '... %s already running with PID %s' % (
                service, p.pid)
            return
    p.pid = 0
    session.commit()
    daemonize(p)
    print '... started %s' % service

@Command
def stop(service):
    print 'Stopping %s ...' % service
    p = Process.query.get(service)
    if not p.pid or not is_running(p.pid):
        print '... service %s is already stopped' % service
        p.pid = None
        session.commit()
        return
    for retry in range(5):
        print '... sending SIGTERM to %s' % p.pid
        os.kill(p.pid, signal.SIGTERM)
        time.sleep(0.5)
        if not is_running(p.pid):
            break
    else:
        print '... sending SIGKILL to %s' % p.pid
        os.kill(p.pid, signal.SIGKILL)
        time.sleep(0.5)
        if is_running(p.pid):
            print '... process %s could not be killed' % p.pid
            return
    p.pid = None
    session.commit()
    print '... %s is stopped' % service

@Command
def restart(service):
    stop([service])
    start([service])        

def main():
    Command.run(sys.argv[1:])

if __name__ == '__main__':
    main()        

One of the painful things about working with any database-oriented project in production is that you can't just drop the database and re-create every time you have a schema change. (Of course, you could do that, but your users might get a little miffed when their data disappears.) Rails and Django have for this reason had support for migrating from one schema version to another. Well, now SQLAlchemy has a automatic migrations tool by the name of Miruku. I haven't tried it, and it's an extremely early version (0.1a7), but it looks promising. Have a look here, and let me know what you think.

Mark Ramm and I were talking about how to differentiate GET, POST, PUT, and DELETE in TurboGears 2 and we came up with a syntax that's pretty cool. That's not the reason for this post, though. This morning, I noticed that our syntax is completely compatible with TurboGears 1.x -- so here's how you do it.

What we wanted was to expose a RESTful method like this:

class Root(controllers.RootController):

    class index(RestMethod):
        @expose('json')
        def get(self, **kw):
            return dict(method='GET', args=kw)
        @expose('json')
        def post(self, **kw):
            return dict(method='POST', args=kw)
        @expose('json')
        def put(self, **kw):
            return dict(method='PUT', args=kw)
        # NOT exposed, for some reason
        def delete(self, **kw):
            return dict(method='DELETE', args=kw)

• it's callable, and


• it has a property exposed which is true (or
  "truthy")

• Classes are callable in Python (calling a class == instantiating an object)


• Classes can have exposed attributes

class RestMeta(type):
    def __new__(meta,name,bases,dct):
        cls = type.__new__(meta, name, bases, dct)
        allowed_methods = cls.allowed_methods = {}
        for name, value in dct.items():
            if callable(value) and getattr(value, 'exposed', False):
                allowed_methods[name] = value
        return cls

import cherrypy as cp
class ExposedDescriptor(object):
    def __get__(self, obj, cls=None):
        if cls is None: cls = obj
        allowed_methods = cls.allowed_methods
        cp_methodname = cp.request.method
        methodname = cp_methodname.lower()
        if methodname not in allowed_methods:
            raise cp.HTTPError(405, '%s not allowed on %s' % (
                cp_methodname, cp.request.browser_url))
        return True

class RestMethod(object):
    __metaclass__ = RestMeta

    exposed = ExposedDescriptor()

    def __init__(self, *l, **kw):
        methodname = cp.request.method.lower()
        method = self.allowed_methods[methodname]
        self.result = method(self, *l, **kw)

    def __iter__(self):
        return iter(self.result)

• CherryPy traverses along to the root.index class and looks up root.index.exposed


• root.index.exposed is intercepted by the descriptor which checks the CherryPy request method to see if it's valid for this controller, and if it is, returns True


• CherryPy says, "Great! root.index is exposed. So now I'll call root.index(...)." This calls index's constructor, which in turn calls the appropriate method and saves the result in self.result


• CherryPy says "Cool! root.index returned me an iterable object. I'll iterate over it to get the text to send back to the browser." This calls root.index(...).__iter__, which is just delegated to the result that the real controller gave.
  

import cherrypy as cp
class ExposedDescriptor(object):
    def __get__(self, obj, cls=None):
        if cls is None: cls = obj
        cp_methodname = cp.request.method
        methodname = cp_methodname.lower()
        method = getattr(cls, methodname, None)
        if callable(method) and getattr(method, 'exposed', False):
            return True
        raise cp.HTTPError(405, '%s not allowed on %s' % (
            cp_methodname, cp.request.browser_url))

Last night at the PyAtl meeting, there was a question about how you define your cascade rules in SQLAlchemy mappers. I'll confess that it confused me at first, too, but here's all you need to know:

What's "cascading" in the mapper is session-based operations. This includes putting an object into the session (saving it), deleting an object from the session, etc. Generally, you don't care about all that stuff, because it Just Works most of the time, as long as you specify cascade="all" on your relation() properties in your mappers. What this means is "whatever session operation you do to the mapped class, do it to the related class as well".

One little confusing thing is that there's another thing you'll often want to specify in your cascade rules, and that's the "delete-orphan". In fact, most of my 1:N relation()s look like:

mapper(ParentClass, parent, properties=dict(
    children=relation(ChildClass, backref='parent', 
                      cascade='all,delete-orphan')
    )
)
      

photo = Table(
    'photo', metadata,
    Column('id', Integer, primary_key=True))
tag = Table(
    'tag', metadata,
    Column('id', Integer, primary_key=True),
    Column('photo_id', None, ForeignKey('photo.id')),
    Column('tag', String(80)))

class Photo(object): pass
        
class Tag(object): pass
        
session.mapper(Photo, photo, properties=dict(
    tags=relation(Tag, backref='photo', cascade="all"),
    session.mapper(Tag, tag)
      

p1 = Photo(tags=[
    Tag(tag='foo'),
    Tag(tag='bar'),
    Tag(tag='baz') ])
p2 = Photo(tags=[
    Tag(tag='foo'),
    Tag(tag='bar'),
    Tag(tag='baz') ])
session.flush()
session.clear()
      

>>> for t in Tag.query():
...     print t.id, t.photo_id, t.tag
... 
1 1 foo
2 1 bar
3 1 baz
4 2 foo
5 2 bar
6 2 baz
>>> session.delete(Photo.query.get(1))
>>> session.flush()
>>> for t in Tag.query():
...     print t.id, t.photo_id, t.tag
... 
4 2 foo
5 2 bar
6 2 baz
      

>>> p2 = Photo.query.get(2)
>>> del p2.tags[0]
>>> session.flush()
>>> for t in Tag.query():
...     print t.id, t.photo_id, t.tag
... 
4 None foo
5 2 bar
6 2 baz
       

>>> p2 = Photo.query.get(2)
>>> del p2.tags[0]
>>> session.flush()
>>> for t in Tag.query():
...     print t.id, t.photo_id, t.tag
... 
5 2 bar
6 2 baz
       

Well, last night was the Python Atlanta user group meeting (PyAtl). It had been a while since I've been, and I'd forgotten how fun it can be. The theme was SQLAlchemy, and the speaker lineup was me, Brandon Craig Rhodes, and James Fowler.

The meeting started off with "shooting the breeze" as usual, and then moved into my presentation "Essential SQLAlchemy", which gives a 30 minute overview of the basics of SQLAlchemy. Here are the usual links to slides and the video:



After my talk, Brandon Craig Rhodes (who is, by the way, an incredibly lively presenter, using nothing but emacs!) gave a talk "SQLAlchemy Advanced Mappings" that focused on using the ORM layer in SQLAlchemy. It really was more of a mini-tutorial that took you through basic mappings all the way through relations, backrefs, and more. SQLAlchemy is an amazingly rich library, and it's hard to squeeze a talk into half an hour. Here's the video:



After Brandon, James Fowler did a "now for something completely different" kind of talk on wxPython, "WxPython Quick Bite", focusing on how you can make wxPython (designed to be event-driven and single-threaded) play nicely in a multi-threaded environment. Unfortunately the start of the video was cut off as I feverishly tried to download the other two videos to make room for James's talk. I'll post the video as soon as it gets uploaded.

I'd be remiss if I didn't thank O'Reilly for "sponsoring" the meetup with a giveaway of a number of books (including 9 copies of Essential SQLAlchemy, which I stuck around afterwards to sign). We also had a couple of copies of Beautiful Code, Beginning Development with Python Gaming, and Hackerteen to give away. A great time was had by all!