Web Services in Python - Part I - Setting up an MVC Pattern

So if you followed along in the last blog post you should have all the tools and frameworks you need to build our Python web service. We're going to create a passport service which will give us single sign on capabilities for any applications we write in the future. This service can be responsible for as little as simply keeping all our passwords in one place or as much as the entire group and permission system it really just depends on your needs. For now we're just going to handle usernames and passwords. Of course this application is going to require a database so let's start there. Unlike Microsoft SQL Server MySQL doesn't come with a GUI to assist with schema generation so the next best alternative is to write some scripts. These files can be located anywhere but hang on to them because we'll be updating them as we move along.

passport_schema.sql


DROP DATABASE IF EXISTS passport;
CREATE DATABASE passport;

USE passport;

CREATE TABLE application_info (
    applicationid INT NOT NULL AUTO_INCREMENT,
    name VARCHAR(50) NOT NULL,
    PRIMARY KEY (applicationid),
    UNIQUE(name)
) ENGINE=InnoDB;

Later on we will add tables to store permissions and user groups but for now we're just creating a single table so we can demonstrate proof of concept with regard to the application design. Let's add some initialization information so we have something to look up.

passport_init.sql


USE passport;

INSERT INTO application_info (name) VALUES ('Passport');

Since we're defining an authentication system it would make sense that the system would handle it's own authentication right? Now all we need is some credentials for our application to use.

passport_user.sql

GRANT USAGE ON passport.* TO 'passport'@'localhost';
DROP USER 'passport'@'localhost';
CREATE USER 'passport'@'localhost' IDENTIFIED BY 'somepass';
GRANT ALL ON passport.* TO 'passport'@'%';
FLUSH PRIVILEGES;

I recommend using a password other than somepass of course. Each one of these scripts will need to be executed in order. You can do that with the following Linux commands.

mysql --user=root --password=YourPassword < passport_schema.sql
mysql --user=root --password=YourPassword < passport_init.sql
mysql --user=root --password=YourPassword < passport_user.sql

That's fairly painless enough. Now let's get on to the web service. The Werkzeug tutorial offers a glimpse on how to create a basic web service so I started off by simply copying that code. The tutorial is good for getting familiar with Werkzeug but it's a very brute force approach so in this phase we'll be spending some time on the design and not just service itself. I'm a pretty big fan of the ASP.Net MVC framework so I figured I'd see what concepts and design decisions I could borrow from that starting with the folder structure. I've been very impressed with how Python handles multiple script files all while allowing you to override the default behavior if you so desire. Here's the project structure we'll be starting out with.

utilities/
    __init__.py
controllers/
    provider/
        __init__.py
    __init__.py
models/
    __init__.py
passport.py

The __init__.py file tells Python that these folders are Packages which means we can import them in other scripts. You can include special instructions in these files but for now their existence is all that's necessary. An easy way to create a blank file in Linux is to use the echo command in conjunction with a pipe operator.

echo "" > __init__.py

Now let's get on to coding some Python.

passport.py


from controllers.provider.controllerprovider import ControllerProvider
from werkzeug.wrappers import Request
from werkzeug.routing import Map, Rule
from werkzeug.wsgi import SharedDataMiddleware

class Passport(object):

    def __init__(self):
        self.url_map = Map([
            Rule('/'),
            Rule('/<controller_name>'),
            Rule('/<controller_name>/'),
            Rule('/<controller_name>/<method_name>'),
            Rule('/<controller_name>/<method_name>/')
        ])

    def __call__(self, environ, start_response):
        response = self.dispatch_request(Request(environ))
        return response(environ, start_response)

    def dispatch_request(self, request):
        adapter = self.url_map.bind_to_environ(request.environ)
        endpoint, values = adapter.match()

        controllerName = 'controller_name' in values and values['controller_name'or 'Service'
        methodName = 'method_name' in values and values['method_name'or 'Index'

        return ControllerProvider.processRequest(request, controllerName, methodName)

if __name__ == '__main__':
    from werkzeug.serving import run_simple
    run_simple('0.0.0.0'8000Passport(), use_debugger=True, use_reloader=True)

The first thing you may notice is that Python uses the encapsulating double underscore naming convention in a number of different contexts. For instance Python defines the __name__ variable in all scripts and sets that value to the string '__main__' when it is the executing script as opposed to being called from another script. This is our entry point and allows us to make a call to the Werkzeug method run_simple. The string '0.0.0.0' opens up a socket on all interfaces. For now I've chosen port 8000 as opposed to the typical HTTP port because to open a port in the reserved range you need to be running as root. This is problematic because I don't want the entire webservice running with administrative privileges so for now 8000 will work just fine. We'll come back to this later on in addition to adding support for hostname binding.

Classes and Behaviors of Python

As you can see, an instance of the Passport class is created using the syntax Passport() which calls the constructor __init__(self). Like most languages, constructors can be supplied with parameters but unlike other languages Python shows you the class instance parameter. Convention seems to dictate that this be named self but you can treat it like any other parameter and change the name if you wish. Python, much like Javascript, lets you add instance members on the fly. As you can see we're creating a map of URL rules to help assist with routing and setting it to a previously undefined instance member. Once that's done our instance of Passport sits in memory until __call__ is triggered by Werkzeug. The best way I know how to describe this function is a default method. If you "call" an instance of an object as if it were a method __call__ is what gets triggered. The rest of the code for this file simply isolates the controller and method names which are defaulted to 'Service' and 'Index' respectively before passing them to a static method processRequest of our ControllerProvider. But before we dive into that let's add some utility classes. If you've been following along with the ASP.Net MVC tutorial the next two files that will contain some very familiar structures.

/utilities/exceptions.py


class ServerException(Exception):
    def __init__(self, message):
        super(ServerException, self).__init__(message)

    def toSerializable(self):
        return { 'Message' : self.message }

class ClientException(Exception):
    def __init__(self, source = None, value = None, message = None, number = None):
        self.source = source
        self.value = value
        self.number = number
        super(ClientException, self).__init__(message)

    def toSerializable(self):
        return { 
            'Message' : self.message,
            'Source' : self.source,
            'Value' : self.value,
            'Number' : str(self.number) }

/utilities/messages.py


import json
from werkzeug.wrappers import Response

class Message(object):
    def __init__(self, 
            data = None, 
            success = True, 
            loggedIn = True, 
            clientExceptions = [], 
            serverExceptions = []):
        self.data = data
        self.success = success
        self.loggedIn = loggedIn
        self.clientExceptions = clientExceptions
        self.serverExceptions = serverExceptions

    def output(self, jsonpCallback = '', cookies = {}):
        config = {
            'Success': self.success,
            'LoggedIn': self.loggedIn,
            'Data': self.data,
            'ClientExceptions'map(lambda x: x.toSerializable(), self.clientExceptions),
            'ServerExceptions'map(lambda x: x.toSerializable(), self.serverExceptions) }
        responseStr = json.dumps(config)
        if jsonpCallback != '':
            responseStr = jsonpCallback + '(' + responseStr + ')'
        response = Response(responseStr)
        for name in cookies:
            if cookies[name] is None:
                response.delete_cookie(key = name)
            else:
                response.set_cookie(key = name,
                            value = cookies[name],
                            httponly = True)
        return response

I'm not going to talk about these since much the syntax is fairly straight forward and the concepts of standardized messages, exception handling and jsonp requests have already been demonstrated. We won't be using them in this phase but it's important to notice that the Message.output method will be where we add the response cookies.

/utilities/database.py


from storm.locals import create_database, Store

def dbconnect(func):
    def inner(*args, **kwargs):
        _DataConnectionProvider.getConnection()
        try:
            result = func(*args, **kwargs)
        finally:
            _DataConnectionProvider.closeConnection()
        return result
    return inner

def staticquerymethod(func):
    @staticmethod
    @querymethod
    def inner(*args, **kwargs):
        return func(*args, **kwargs)
    return inner

def querymethod(func):
    def inner(*args, **kwargs):
        return func(_DataConnectionProvider.getConnection().store, *args, **kwargs)
    return inner

class _DataConnectionProvider(object):
    _providerInstance = None

    @classmethod
    def getConnection(cls):
        if not cls._providerInstance:
            cls._providerInstance = _DataConnection(
                'mysql://passport:somepass@localhost:3306/passport')
        return cls._providerInstance

    @classmethod
    def closeConnection(cls):
        if not cls._providerInstance is None:
            cls._providerInstance.store.close()
        cls._providerInstance = None

class _DataConnection(object):

    def __init__(self, connStr):
        self.database = create_database(connStr)
        self.store = Store(self.database)

You'll notice that the dbconnect, querymethod and staticquerymethod functions look a little bit strange. These are known as wrapper functions and they accept a function as a parameter while the inner function accepts the actual arguments. Essentially the purpose of the dbconnect wrapper is to establish a connection to the database if one does not already exist and close it once the inner function has completed. In this case we're using Storm so we make the create_database call and pass in our connection string. You can read more about the Storm store object in the documentation. The querymethod wrapper will be responsible for providing our connected Store instance to the method by injecting _DataConnectionProvider.getConnection().store as the first parameter. The *args and **kwargs are used to provide any additional parameters that were passed in by the calling function. Python refers to this kind of parameter handling as packing and unpacking.

/models/data.py


from utilities.database import staticquerymethod 
from storm.locals import *

class Application(Storm):
    __storm_table__ = 'application_info'
    applicationid = Int(primary=True)
    name = Unicode()

    @staticquerymethod
    def findByName(store, name):
        return store.find(Application, Application.name == name).one()

As you can see we've prepended staticquerymethod with an @ symbol and placed it just before a function. This is known as a decorator. No, decorators are not attributes but as you can see they are extremely useful as it allows us to provide parameters to the function in a way that is transparent to the caller. As you already saw this store parameter is provided automatically by the wrapper function so we won't have to supply an argument for it when making the method call. The other important detail to notice is how simple Storm makes it to define an entity. All the mapping details are handled in the first three lines of the class. We can see the class name, table name that we created earlier in MySQL and both fields being initialized as their appropriate type. For now we're just using one query method to find an application to demonstrate proof of concept.

Now that we have all the utility classes in place we can take a look at how we handle the controller name and method name that we separated out in the passport.py file. While it would certainly be possible to implement a long list of if statements I wanted to go with something a bit more dynamic. Python has two very useful functions, dir() and getattr() which will help us do just that.

/controllers/provider/controllerprovider.py


import traceback
import controllers
from werkzeug.urls import url_decode
from utilities.database import dbconnect
from utilities.messages import Message
from utilities.exceptions import ClientException, ServerException

class ControllerProvider(object):
    @staticmethod
    def processRequest(request, controllerName, methodName):
        package = None      
        controller = None
        method = None

        if controllerName.lower() != 'base':
            package = ControllerProvider._safeGetAttr(controllers, controllerName + 'Controller')
            controller = ControllerProvider._safeGetAttr(package, controllerName + 'Controller')
            method = ControllerProvider._safeGetAttr(controller, methodName + 'Method')

        @dbconnect
        def runMethod(success=False, loggedIn=False, clientExceptions=[], serverExceptions=[], data=None):
            if method is None:
                clientExceptions = [ ClientException(   
                                source = 'url',
                                value = controllerName + '/' + methodName,
                                message = 'No method \'' + controllerName + '/' + methodName + '\' found.',
                                number = 404) ]
            else:
                try:
                    instance = controller(request)
                except:
                    print 'Unhandled exception in controller constructor: ' + \
                        str(traceback.format_exc())
                    serverExceptions = [ ServerException('Error written to console') ]

            if len(serverExceptions) + len(clientExceptions) != 0:
                return success, False, clientExceptions, serverExceptions, None, request.cookies.copy()

            try:
                data = method(instance)
                success = True
            except ClientException as e:
                clientExceptions = [ e ]
            except:
                print 'Unhandled exception in controller method: ' + \
                    str(traceback.format_exc())
                serverExceptions = [ ServerException('Error written to console') ]

            return success, loggedIn, clientExceptions, serverExceptions, data, request.cookies.copy()

        try:
            success, loggedIn, clientExceptions, serverExceptions, data, cookies = runMethod()
        except:
            print 'Unhandled exception calling runMethod(): ' + \
                str(traceback.format_exc())
            success = False
            loggedIn = False
            clientExceptions = []
            serverExceptions = [ ServerException('Error written to console') ]
            data = None
            cookies = request.cookies.copy()

        message = Message(success = success, loggedIn = loggedIn, clientExceptions = clientExceptions, 
                            serverExceptions = serverExceptions, data = data)
        return message.output(ControllerProvider._getjsonpCallback(request), cookies)

    @staticmethod
    def _safeGetAttr(entity, match):
        if entity is None:
             return None
        results = filter(lambda x: x.lower() == match.lower(), dir(entity))
        if len(results) == 0:
            return None 
        return getattr(entity, results[0])

    @staticmethod
    def _getjsonpCallback(request):
        querystring = dict((k.lower(), v) for k,v in url_decode(request.query_string).iteritems())
        return 'callback' in querystring and querystring['callback'or ''

This class will serve as our controller factory. In order to translate the request into an instance of a class we needed to protect the use of getattr() against a malicious user. Letting the user pass in a value to a function that allows them to grab a function seems like a recipe for disaster so I used two approaches to lock it down. The first is that I use the results from dir() so I can determine what values would even make sense and we only use the matched result as a parameter for getattr(). Still this could potentially allow the user to access some of the built in methods present on every Python object like __getattr__ so I made a rule that every controller file and class must end with the word 'Controller' and all web methods within that controller must end with the word 'Method'. We've also added some code that prevents anyone from requesting the BaseController class which we'll examine momentarily. The rest of the code is dedicated to handling four possible failures.

  • The requested service method does not exist. In this case we simply throw back a ClientException explaining this.
  • A failure occurs inside the @dbconnect wrapper which as we saw earlier handles our database connection.
  • Our controller constructor experiences a failure.
  • There is a failure in the method call itself.

After trapping all of these potential exceptions we are able to guarantee that our service will always provide a response in the standard Message format.

In order to avoid having to specify which controllers to import every time we create a new one we can have Python handle this for us automatically by editing our initialization file.

/controllers/init.py


import os
for module in os.listdir(os.path.dirname(__file__)):
    if module[-13:] != 'controller.py':
        continue
    __import__(module[:-3], globals(), locals())
del module

This piece of code iterates through all the files in the current folder and imports them. As you can see, we enforce the Controller naming convention here as well. Now let's take a look at the BaseController.

/controllers/basecontroller.py


from werkzeug.urls import url_decode

class BaseController(object):
    def __init__(self, request):
        self._request = request
        self._querystring = url_decode(request.query_string)

    @staticmethod
    def _get_args(data, func):
        argcount = func.func_code.co_argcount
        if argcount == 0:
            return dict()
        args = [arg.lower() for arg in func.func_code.co_varnames[0:argcount] if arg[0:1] != '_']
        result = dict((k.lower(), v) for k, v in data.iteritems() if k.lower() in args)
        for arg in filter(lambda x: x not in result, args):
            result[arg] = None
        return result

    @classmethod
    def httpget(cls, func):
        def inner(self, *args, **kwargs):
            return func(self, *args, **BaseController._get_args(self._querystring, func))
        return inner

Here we're defining one last important wrapper function httpget. Using Pythons handy unpacking capability we can map querystring parameters to the web method parameters directly for convenience. The func.func_code.co_argcount member gives us the number of arguments in the method which will be the first in the list of func.func_code.co_varnames. We can separate those using [0:argcount]. From there all we have to do is populate any unmatched method arguments with the value None and the ** operator handles the rest. We could define an httppost wrapper but this service will most likely be used exclusively in cross-domain requests which don't support the POST verb.

All that's left to do is create our first controller!

/controllers/servicecontroller.py


from controllers.basecontroller import BaseController
from utilities.exceptions import ClientException
from models.data import Application

class ServiceController(BaseController):
    @BaseController.httpget
    def getApplicationMethod(_self, name):
        if name is None:
            raise ClientException( source = 'name'
                        message = 'Application name must be specified.'
                        number = 1 )

        app = Application.findByName(name)
        if app is None:
            raise ClientException( source = 'name',
                        message = 'Could not find application \'' + name + '\'.',
                        number = 2)
        return { 
            'Message' : 'Application ID for \'' + app.name + '\' is: ' + str(app.applicationid),
            'Application' : { 'ID' : app.applicationid } }

Notice how thanks to our handy httpget wrappers this code is relatively simple and looks strikingly similar to our ASP.Net MVC service methods. All the dirty work of query string parsing and database session management occur behind the scenes which allow us to concentrate on what the service method does rather than how. There's still room for improvement but we can revisit this later.

Phase I Links: Download | Demo

Conclusion

As with any uncharted territory I think it makes sense to start with what you know. In this foreign environment we were able to successfully mimic some of the great design features provided in ASP.Net MVC. Particularly with respect to the way it routes and maps a url request to the controller, method and it's parameters. We were also able to borrow a lot of the custom design decisions we made on handling exceptions and a standard messaging scheme. Give it a try by executing python passport.py from the command line and hitting port 8000 with a browser. Remember to activate the virtual environment. In the next phase we'll back off a bit from the design of the service itself and concentrate more on it's behavior. One step at a time. :)

Quick Links: Next: Authentication and Login Sessions >>