[patch]: simple XML-RPC support for Django

[hugo <gb@…>]

The idea of this patch is to add support to Django for basic XML-RPC capabilities. It works mostly like URL patterns in that there is a ROOT_RPCCONF setting that points to a module that defines a rpc call registry like this:

rpccalls = {
        'anton': 'gallery.test.anton',
}

So the registry is a simple dictionary that maps method names (they can be structured with "." - for example blogger.post) to functions in modules (function "anton" in module gallery.test in this case).

Additionally you need to add a mapping for some URL to the rpc view:

urlpatterns = patterns('',
    (r'^RPC/', 'django.core.rpc.call'),
)

The last part is the django.core.rpc module:

import sys
import time
import xmlrpclib

from django.utils.httpwrappers import HttpResponse
from django.conf import settings

dispatch = {}

def call(request):
        """
        This is the view you need to map into your URL space to process RPC
        calls.
        """
        p, u = xmlrpclib.getparser()
        p.feed(request.raw_post_data)
        p.close()
        args = u.close()
        method = u.getmethodname()
        func = dispatch.get(method, None)
        if func is not None:
                result = func(*args)
                xml = xmlrpclib.dumps((result,), methodresponse=1)
        else:
                xml = xmlrpclib.dumps(xmlrpclib.Fault(-32601, 'method unknown: %s' % method), methodresponse=1)
        return HttpResponse(xml, mimetype='text/xml; charset=utf-8')

# build the dispatch table for rpc calls out of ROOT_RPCCONF
for (fn, fc) in __import__(settings.ROOT_RPCCONF, '', '', ['']).rpccalls.items():
        p = fc.rfind('.')
        modname = fc[:p]
        funcname = fc[p+1:]
        dispatch[fn] = getattr(__import__(modname, '', '', ['']), funcname)

After setting up all this (django.core.rpc should go into the django source - that's the actual patch), you could use your new RPC call like this:

import xmlrpclib
srv = xmlrpclib.Server('http://your.server.here/RPC/')
print srv.anton(5,6)
print srv.anton('anton','berta')
print srv.anton([1,2,3,4],[5,6,7,8})

I think this would allow people for much easier integration of simple XML-RPC webservices into django while keeping the framework idea of django intact.

[garthk@…]

+1 if you allow urlconf style chaining :)

[hugo <gb@…>]

first things first, this one has better error handling :-)

import sys
import time
import xmlrpclib

from django.utils.httpwrappers import HttpResponse
from django.conf import settings

dispatch = {}

def call(request):
        """
        This is the view you need to map into your URL space to process RPC
        calls.
        """
        global dispatch
        if dispatch == {}:
                # build the dispatch table for rpc calls out of ROOT_RPCCONF
                for (fn, fc) in __import__(settings.ROOT_RPCCONF, '', '', ['']).rpccalls.items():
                        p = fc.rfind('.')
                        modname = fc[:p]
                        funcname = fc[p+1:]
                        dispatch[fn] = getattr(__import__(modname, '', '', ['']), funcname)
        p, u = xmlrpclib.getparser()
        p.feed(request.raw_post_data)
        p.close()
        args = u.close()
        method = u.getmethodname()
        func = dispatch.get(method, None)
        if func is not None:
                try:
                        result = func(*args)
                        xml = xmlrpclib.dumps((result,), methodresponse=1)
                except Exception, e:
                        xml = xmlrpclib.dumps(xmlrpclib.Fault(-32400, 'system error: %s' % e), methodresponse=1)
        else:
                xml = xmlrpclib.dumps(xmlrpclib.Fault(-32601, 'method unknown: %s' % method), methodresponse=1)
        return HttpResponse(xml, mimetype='text/xml; charset=utf-8')

[hugo <gb@…>]

actually I think that urlpattern-style chaining isn't really usefull here - it's not as if the namespaces are really "chaine" - for example for metaWeblogAPI you nave to use the predefined metaWeblog.newPost method name. So it's not a real hierarchical namespace but just names with dot's in them ...

[Jason Huggins]

I also have a patch for XML-RPC. It is a little less 'simple' than this one, but is more flexible and promotes code re-use, I believe.

My patch includes the following:

1) An XmlRpcMiddleware module to marshall and unmarshall XML-RPC requests. SOAP or RESTful middleware can be added later.

2) Generic 'web service views' to handle basic CRUD data access. The generic functions are: get_list, get_object, create, update, and delete. These new views are required because the current "generic views" shipped with Django assume they are being called by a web browser and are not "generic" enough to be called by an xml-rpc or soap client program.

By separating the web service protocol into middleware, and the data access into generic views, additional protocols should be easier to implement in the future, by just adding additional middleware. For example, SOAP or RESTful APIs with HTTP+XML, HTTP+YAML, or HTTP+JSON.

My patch also plugs into the existing url dispatching scheme.

Perhaps, I'll add my patch as "less simple xml-rpc support for Django" :-) I'll add a ticket with my code as soon as possible.

[hugo <gb@…>]

Maybe a use-case helps. That's what I cooked up to make access to the model of an application via XML-RPC:

# this would be the module I hook into the RPC dispatcher
# its stored as myproject/dispatcher.py and is referenced in the
# myproject.settings.rpc.main module

from django.models.polls import polls

# this is just a handy way to turn any function with flexible
# parameters into one that just accepts one list and one hash
# and maps those back to the original function parameters.
def wrapper(func):
   return lambda args, kw: func(*args, **kw)

# these are the methods of the model I want to export.
# the wrapper changes from the positional parameters and
# named parameters to a simpler format that can be easily
# transported over XML-RPC
get_object = wrapper(polls.get_object)
get_list = wrapper(polls.get_list)

# this is the myproject.settings.rpc.main module. It's similar
# to the urlpatterns stuff, only that it's just a one-to-one map.
# the RPC dispatch table looks like this:

{
   'polls.getObject': 'myproject.dispatcher.get_object',
   'polls.getList': 'myproject.dispatcher.get_list',
}

# from the client you then could do:

srv = xmlrpclib.Server('http://somewhere.com/some/RPC/')
srv.getObject([], {'id__exact': 1})

[hugo <gb@…>]

Oh, and this is of course an oversimplification: usually you don't get basic types back from your functions but get internal objects and stuff like that - and those usually can't be converted into usefull return values and parameters can't be simply constructed if they need to be of complex types. Usually you do much more highlevel stuff than just exposing the data model via XML-RPC. For a more fullfeatures sample you can look into the ​code for my gallery software - there I implemented parts of the metaWeblogAPI, the bloggerAPI and the MoveableType API.

[hugo <gb@…>]

Ok, after sleeping over it, here are my thoughts why I don't think XMLRPC stuff (and other RPC style marshalling/unmarshalling) should be done with middleware:

• middleware is a broad solution - every request is passed through it to check wether there needs some work to be done. RPC style connection points are a rather small number in any application, though - usually you only have one. It's overkill to have a middleware that just sits there to look for one access point.

• middleware can't easily handle exceptions - RPC protocolls usually have a notion of an exception (SOAP has the faultType and xmlrpc has something similar, too) and it would be needed to change exceptions in something the protocol client understands. the normal way for Django is just to produce a 500, but that isn't really helpfull for the RPC client.

• RPC called functions are no views. The middleware solution (at least the version jrhuggins put up at pastebin) handles request and response objects. Parameters are passed via the request.POST variable, not as explicit parameters to the function. Ideally a RPC module should be testable without the need of the RPC protocol - that's how it is with my solution, you can just import the module and call the functions like you would call them via XML-RPC.

So I think using a middleware for RPC style webservices would be overengineering. This is totally different with REST style APIs, as those are much more like the current view mechanism of Django. With REST webservices you give every object a unique URI and then use the standard HTTP verbs GET/PUT/DELETE/POST to manipulate them. GET will fetch the object in some external representation, PUT will modify an existing object, DELETE will delete it and POST to some action URI will create a new object and return a Location-header with this objects URI. Based on that I think that REST style webservices and RPC style services should be handled differently - they are too different in how they work.

RPC style webservices only "tunnel" through very few specifically defined URIs and dispatch based on the method name in the XML payload. So it's much better to provide a standard dispatcher with Django that can be hooked to those URIs (the accesspoints).

[hugo <gb@…>]

And now my idea on what garthk requested. It's not yet implemented, but I think it's quite easy to do - maybe I upload a new version today that implements this idea.

A two layer dispatch table could look like this:

# this is myproject.settings.rpc.main
from django.core.rpc import dispatch, include

rpccalls = dispatch('',
    ('system.methodlist', 'django.core.rpc.methodlist'),
    include('myproject.apps.weblog.rpc.metaweblog'),
)

# this is myproject.apps.weblog.rpc.metaweblog
from django.core.rpc import dispatch

rpccalls = dispatch('myproject.apps.metaweblog.views',
   ('metaWeblog.newPost', 'newPost'),
   ('metaWeblog.editPost', 'editPost'),
)

So we would have the same two-layer dispatching as with urlpatterns and we would have the same typing-reducing base name. This would make registration of app specific calls easier and would make apps more portable in the same way as urlpatterns do it already.

[hugo <gb@…>]

And just another reason against using a middleware and the normal Django URLspace for RPC style APIS: you might have other middleware, too. For example the admin required middleware or some other kind of authentication. All those middleware stacks will be run over the XMLRPC requests, too - and might break the system. For example authentication middleware will use some HTTP authentication scheme, but many XMLRPC client libraries don't support HTTP authentication and most XMLRPC protocols just transport user and password as parameters. Yes, that's not really good architecture - but it's what existing APIs like metaWeblogAPI, bloggerAPI and MoveableTypeAPI provide, and many people will want to implement those for their Django app.

This again is different with REST style APIs: since those are just normal HTTP requests with HTTP verbs, they should definitely be routed through all middleware and will definitely make use of HTTP authentication. An example would be the Atom API, although they defined a special HTTP authentication scheme that is only used by Atom.

[hugo <gb@…>]

rpc connector source (this is the view function you need to hook into urls)

[hugo <gb@…>]

this is the dispatchtable loader stuff

[hugo <gb@…>]

Ok, I added two files that implement the multi level dispatching. Use them as follows:

# this is myproject.settings.rpc.main
from django.core.rpcdispatch import *

rpccalls = dispatch('',
        include('gallery.apps.picturefolder.rpc.metaweblog'),
        include('gallery.apps.picturefolder.rpc.blogger'),
        include('gallery.apps.picturefolder.rpc.moveabletype'),
)

# this is gallery.apps.picturefolder.rpc.metaweblog, others look similar
from gallery.core.rpcdispatch import *

rpccalls = dispatch('gallery.metaweblogapi',
        ('metaWeblog.newPost', 'newPost'),
        ('metaWeblog.getCategories', 'getCategories'),
        ('metaWeblog.newMediaObject', 'newMediaObject'),
)

This way you can easily decouple the dispatch table for applications. You still only have a flat namespace for RPC call names, but that's a limitation of the RPC style APIs.

[hugo <gb@…>]

it should be 'from django.core.rpcdispatch import *' in the myproject.apps.picturefolder.rpc.metaweblog, of course.

[garthk@…]

How about include('metaWeblog', 'myproject.apps.weblog.rpc.metaweblog'), with that module then only having to dispatch newPost and editPost? You'd have to figure out sensible dispatch behaviour across inclusions (first match wins? best qualified match wins?), but I'd prefer to be as consistent with the existing urlconf behaviour as possible.

[hugo <gb@…>]

The problem is that different apps might share a single RPC namespace. Think of metaWeblogAPI: you have a getCategories call and a getRecentPosts call. the getCategories call might be implemented by your global category mnagement app while the other is implemented by your weblogging app, or even on a project level with an internal dispatch to several blog-like apps running in your project (like a weblog app and a static pages app)

[garthk@…]

If you've got everyone and their dog trying to serve parts of metaWeblogAPI, surely you're going to need to explicitly target the calls anyway to avoid problems with overlap?

[hugo <gb@…>]

from the IRC log:

	hugo-	you just move the code for metaWeblog.newPost to the project level and dispatch in that function on the blogid
	hugo-	that would be one way to solve it - just give each apps "blog" a unique blogid and use that to decide to what app to forward the newPost
	hugo-	but that's nothing that a framework can factor out, because that dispatching will allways depend on parameter data and neither URI data nor method name.
	hugo-	so it's RPCAPI specific
	hugo-	one solution for metaWeblog, one for blogger, one for MoveableType ...
	hugo-	REST style APIs like the Atom API are much nicer in that regard, because they use the object URIs to directly interface with objects - you don't have collisions that way
	hugo-	REST = one object, one URI
	hugo-	RPC = one URI, one method name, possibly multiple objects, depending on parameter data

[hugo <gb@…>]

Since I am using the stuff in my gallery (insert jokes about eating your own dogfood here), the most current versions of rpc.py and rcpdispatch.py can be found in the ​subversion repository.

[upadhyay@…]

Oops, looks like overstepped on some work done already in ticket ​http://code.djangoproject.com/ticket/547. What we least want is more than one way of doing xmlrpc in django. I tend to believe it should not go in core, as XMLRPC is fine, its support is included in python standard distribution, but having core.rpc but not supporting SOAP would be misleading, either it should be core.xmlrpc and support just that, or it should be core.rpc and support all common rpc modes, and SOAP has dependency requirements on SOAPpy, which is not in standard python distribution.

They should both be in contrib, xmlrpc as described in TIcket # 547 and SOAP as in my work in progress ​http://nerdierthanthou.nfshost.com/soap.txt [rename it to py, and put it in django/contrib]:

from django.contrib.soap import SimpleSOAPView
soap = SimpleSOAPView()
def f2():
    return 'f2'
soap.registerFunction(f2)

when urlconf is set like:

urlpatterns = patterns('',
     ...
     (r'^soap/$', 'djangoprojects.apps.xrpc.views.xrpc.soap'),
)

What do you think?

[Jacob]

This is a subset of #115.