Python Relative Imports and Module Distribution

If you've been following along in the previous posts you may have noticed that each phase of both the ASP.Net MVC and Python MVC projects has it's own demo. While this is pretty easy to do in IIS I wasn't entirely sure how to get it to work in Python using Werkzeug. As I said in an earlier entry I expect to have to do a bit of house cleaning as I learn more about what Python is capable of. My discovery of Python's support for relative imports and the distributions caused me to rethink some of my earlier design decisions and so I believe it is best to address these changes now rather than later.

Relative Imports

Relative imports in Python allow you to traverse the packages and modules relative to the script's location. As an example let's take a look at one of the files from the Passport project.

/passport.py


from controllers.provider.controllerprovider import ControllerProvider

This import line works so long as we don't try to import our Passport class from an external script. If we did our absolute package path would be incorrect and the script would fail. Python's relative import capability takes care of this for us.

from .controllers.provider.controllerprovider import ControllerProvider

Prepending the namespace with a period means that Python will look for the packages relative to the directory where the script resides. For each directory above we just add another period. Here are a few other files we'll need to alter.

/models/data.py


from ..utilities.database import staticquerymethod, querymethod

/controllers/basecontroller.py


from ..models.data import UserLogin
from ..utilities.messages import Message
from ..utilities.exceptions import ClientException

/controllers/servicecontroller.py


from .basecontroller import BaseController
from ..utilities.exceptions import ClientException
from ..models.data import Application, User, Challenge, UserLogin

/controllers/provider/controllerprovider.py


from ... import controllers
from ...utilities.database import dbconnect
from ...utilities.messages import Message
from ...utilities.exceptions import ClientException, ServerException

And that's it. Now we can load our Passport class from any script and it will run correctly. However there still is an issue due to the fact that we have hard coded the connection string in one of our utility files. What would be better is if we could pass in a collection of settings which include a connection string. In order to do this we'll need to make changes to our dbconnect wrapper function as well as the getConnection method of the _DataConnectionProvider class.

/utilities/database.py


def dbconnect(settings):
    if not 'connstr' in settings:
        raise Exception('No database connection string configuration found.')
    def wrapper(func):
        def inner(*args, **kwargs):
            _DataConnectionProvider.getConnection(settings['connstr'])
            try:
                result = func(*args, **kwargs)
            finally:
                _DataConnectionProvider.closeConnection()
            return result
        return inner
    return wrapper

You'll notice that we've added another layer to the wrapper function. Where dbconnect used to accept the function as the parameter that logic has been pushed downward into a new function called wrapper which we return just as we do the inner function. We'll take a look at how we can use this in a moment. As you can see we pass the settings value for the key connstr to the getConnection method so let's take a look at that code.

@classmethod
def getConnection(cls, connstr=None):
    if not cls._providerInstance:
        cls._providerInstance = _DataConnection(connstr)
    return cls._providerInstance

All we needed to do here is add an optional parameter and use it as our connection string. Now let's take a look at how we can use our altered dbconnect wrapper as a decorator.

/controllers/provider/controllerprovider.py


class ControllerProvider(object):
    @staticmethod
    def processRequest(request, controllerName, methodName, settings):
        package = None      
        controller = None
        method = None

        if controllerName.lower() != 'base':
            package = ControllerProvider._safeGetAttr(controllers, controllerName + 'Controller')
            controller = ControllerProvider._safeGetAttr(package, controllerName + 'Controller')
            method = ControllerProvider._safeGetAttr(controller, methodName + 'Method')

        @dbconnect(settings)
        def runMethod(success = False, loggedIn = False, clientExceptions = [], serverExceptions = [], data = None):

Now our @dbconnect decorator can accept the settings parameter that is passed into the processRequest method. All that's left to do is alter our passport.py file which I've decided to rename.

main.py


from werkzeug.wrappers import Request
from werkzeug.routing import Map, Rule
from werkzeug.wsgi import SharedDataMiddleware

from .controllers.provider.controllerprovider import ControllerProvider

class Passport(object):

    def __init__(self, settings):
        self.url_map = Map([
            Rule('/'),
            Rule('/<controller_name>'),
            Rule('/<controller_name>/'),
            Rule('/<controller_name>/<method_name>'),
            Rule('/<controller_name>/<method_name>/')
        ])
        self.settings = settings

    def __call__(self, environ, start_response):
        response = self.dispatch_request(Request(environ))
        return response(environ, start_response)

    def dispatch_request(self, request):
        adapter = self.url_map.bind_to_environ(request.environ)
        endpoint, values = adapter.match()

        controllerName = 'controller_name' in values and values['controller_name'or 'Service'
        methodName = 'method_name' in values and values['method_name'or 'Index'

        return ControllerProvider.processRequest(request, controllerName, methodName, self.settings)

Notice that in addition to adding the settings parameter in the constructor we've removed the main entry sub since we won't be calling this file directly any longer. We can now easily manage multiple phases of the project relatively easily. In order to demonstrate this I'm moving this whole project into the netortech/passport/v2 folder so our structure will look something like this.

netortech/passport/v2/<passport folders>
netortech/passport/v2/__init__.py
netortech/passport/v2/main.py
netortech/passport/__init__.py
netortech/__init__.py
app.py

For testing purposes I've created a file at the root that we'll use to run the project.

/app.py


from netortech.passport.v2.main import Passport

if __name__ == '__main__':
    from werkzeug.serving import run_simple
    run_simple('0.0.0.0'
            8000
            Passport({ 'connstr''mysql://passport:somepass@localhost:3306/passport' }), 
            use_debugger=True, 
            use_reloader=True)

Python Distributions

Now that we have relative imports and our database connection string is passed in our project is primed and ready to be placed into a distribution. Python has a really handy method for doing this using the distutils.core.setup method. The documentation provides a comprehensive how-to guide on writing a setup script for your project. The setup script for the Passport project is relatively simple.

/setup.py


from distutils.core import setup

setup(name='netortech.passport.v2',
    version='1.0',
    description='Python MVC Passport Service - Phase II',
    author='Spencer Ruport',
    author_email='spam_me_1' + '@' + 'netortech.com',
    url='http://www.netortech.com/',
    packages=[
        'netortech',
        'netortech.passport',
        'netortech.passport.v2'
        'netortech.passport.v2.utilities'
        'netortech.passport.v2.models'
        'netortech.passport.v2.controllers'
        'netortech.passport.v2.controllers.provider'],
    data_files=[
        ('databases', [
            'netortech/passport/v2/databases/passport_schema.sql',
            'netortech/passport/v2/databases/passport_function.sql',
            'netortech/passport/v2/databases/passport_init.sql',
            'netortech/passport/v2/databases/passport_user.sql'])]
    )

The only part of this script that gave me some trouble was for some reason I had to include netortech and netortech.passport in the packages list for the installation to work despite the fact that there are no modules in those packages. Without explicitly specifying these packages the installation would complete but the modules would not be found when executing a script. To create a distribution package simply type the command python setup.py sdist or to install use python setup.py install. Remember to activate your virtual environment before installing.

Conclusion

While the changes we've made may not directly benefit us with regard to the Passport project goals it is always important to keep publishing and portability in mind. Python has some extremely convenient ways of assisting us with this but only if the application design allows it. Now that we know a few pitfalls that can plague these mechanisms it will be that much easier to avoid them in the future. In the next article we'll be taking a look at how we can host multiple Werkzeug based applications using a lightweight application server.