Models CRUD via web services

Some thoughts on this: http://blog.ianbicking.org/its-not-another-damn-framework-comment-2.html

Excerpt: "I've been playing around with CherryPy?'s filters and that description makes sense. You can write a function or class method once, and use filters to repurpose the functions/methods for different contexts (HTTP/HTML vs XML-RPC) without touching your original function/method definitions."

...

If there was a standard implementation of "filters" that other frameworks could use... then Django could benefit from this work for free without having to reimplement CherryPy?'s filters."

This is a great idea up to a point - the question is, how should authentication / access control work? There are a number of scenarios:

* "Public" web services, no authentication needed, modification (basically save() and delete() model methods) is not available. * Private web services, which require authentication include the ability to update and delete. Note that in Django the permissions model is meant to restrict users to certain activities, but it is currently not tied directly to the model. Should the permissions system be changed to directly control model access, or should some other solution (such as adding a set of web service API permissions) be pursued instead? * A bit of both - unauthenticated users can view data, authenticated users can change things.

Next question: SOAP, REST or XML-RPC? REST could be interesting, partly because many model objects in Django end up with their own URL on the public site. We could make it so that, when web services are enabled, GETs (and other HTTP verbs) to those URLs with either an ?xml argument in the query string or some special HTTP request header get served XMl representations instead of HTML.

Lots to think about here.

Maybe don't create only admin-alike CRUD interface available via web services, but allow creating views for many protocols, like HTTP, XML-RPC and REST at all?

To expand maurycy's idea of using views.... How about standard delivered Django Views that can introspect the model and enforce security where needed? I like Simon's thoughts on the public vs. private services.

It's probably not wise to overload the model too much with this stuff. However, a new model declaration might be handy:

class Widget(meta.Model):
    fields = ...
    admin = meta.Admin()
    web_services = ('XML-RPC', 'SOAP', 'REST/XML','REST/YAML','REST/JSON','JSON-RPC') 

By default, web service access to the Model API would be enabled with a declaration of "web_services"... What would be even better is if you could autogenerate the web service source code and store it in the project's app directory in case you want to override the default view, or add methods, or change security. (e.g. 'django-admin.py createwebservice <service_type> <app>')

see also #356

Jason, I'm a big fan of your web_services idea; doing WS as a framework instead of individual SOAP/REST/whathaveyou views is a very good idea. To that end, I've marked #356, #547, and #552 as duplicates of this one even though I'll certainly use some/all of the code attached to those tickets.

The big question I have at this point is how we should deal with authentication? Ideally we'd do it at the HTTP layer so that the individual REST/SOAP/XML-RPC interfaces wouldn't have to worry about it (and so it would be implemented consistently). I've thought about just use HTTP-Basic auth and putting a big honking note that if you want your web services to actually be secure you've got to run them over HTTPS... but that sounds somewhat scary to me. Thoughts, anyone?

I'm dealing with authentication by expecting an HTTP Basic Auth header for any XML-RPC, SOAP, or REST (currently JSON and ATOM) request. Yup, it's implemented consistently, I actually have one more Auth middleware piece that runs right before anything else that looks for the auth header and translates that into a user object and puts that user object on the request. Conventional wisdom is that Basic Auth + SSL is "secure enough". When I need SSL, I just front my app with Apache... Could/does the WSGI python server offer HTTPS as an option? That would be *really* useful when developing and testing... (Creating and installing quick and dirty self-signed certs is sometimes more difficult that it needs to be. Would be cool if Django did this for me during development.)

On a side note, requiring an HTTP Basic Auth header seams to go against most weblog APIs, like Blogger and MetaWeblog?. I don't know why those APIs required ids and passwords to be passed as parameters. Is it that at the time they were created, XML-RPC client libraries couldn't handle putting extra headers on the request?

I have a fair enough amount of code already written for this ticket. If possible, I'd like a branch in subversion set up so I can check in my code and we can start talking about real example code. In the meantime, I'm working on writing doctests and a tutorial for my web services stuff as an extension to the Poll tutorial... eventually posting this as blog post soon.

One caveat, Hugo, in ticket #356, says he's against the idea of middlware for services... I'd like to fully flesh out the arguments pro and con before a "final" decision is made for Django... because I think it's important to get this right the first time... For example, A JSON middleware version of web services could be the base of any AJAX/Django work going forward. If we can get the middleware stable, lots of fun stuff at the AJAX/UI level can start happening.

-Jason

OK, I've got a bunch of thoughts about this I've been trying to turn into rational discourse. It's still pretty scattered, though, so in the aim of getting my thoughts on "paper" here are some data points I've been thinking about:

Point: authentication is *very* important to me. I don't know why so many weblog APIs use plaintext passwords as parameters, but there's no way in a million years such a weak authentication method is going into Django. I'm OK with HTTP-Basic + SSL, but I am worried that people wouldn't use SSL and would open up their databases to malicious users. If there's a more secure auth method out there that could be used I'd be thrilled to use it. The point is that by default I'd like the interfaces to do the right thing.

Point: Using middleware for this is the wrong approach. Middleware is designed to be run on *any* request, but RPC endpoints can co-exist with "normal" URLs. The right way to do RPC is with a dispatch view similar to the way the RSS framework works. That is, in your urlconf you'd do something like:

    urlpatterns = patterns('',
        '^/rpc/', include('django.contrib.rpc.urls.rpc'))
    )

... and the RPC views would handle the rest.

Point: allowing multiple types of web services is a good idea. However, I'm not sure I like tying the config into the model a la Jason's suggestion above; our commercial CMS has over 100 models and I don't fancy adding a web_services declaration to every single one of them, nor do I fancy updating that clause when we add support for SOAP-2 or whatever. I'm not totally sure how to work out which models to include -- simply exposing everything is a bad idea, I think -- but I'll bet it'll be pretty easy to do in the config file or in the url pattern extra params dictionary.

Point: there should be a way to indicate that certain APIs are "public" and require no authentication.

Point: by default RPC bindings for everything should be available in the admin to admin users. This would let admin views do cool AJAX-y stuff. I'm not sure if this would use the same system or just parts of it, but it's something to think about.

OK, there it is. Jason (and others): if you've got code that's not attached to this ticket or to #356, #547 or #552 please attach it to this one as I'm going to try to get some work done on this Any Day Now™.

>  Point: authentication is *very* important to me.  I don't know why so many
>  weblog APIs use plaintext passwords as parameters, but there's no way in a
>  million years such a weak authentication method is going into Django.  I'm
>  OK with HTTP-Basic + SSL, but I am worried that people wouldn't use SSL
>  and would open up their databases to malicious users.  If there's a more
>  secure auth method out there that could be used I'd be thrilled to use it.

Authentication is very important, I may want to authenticate based on non model 
data, for example against LDAP, I am having difficulty imagining how would you 
handle that, and a hundread thousand other ways of possible auths people may be
interested in. 

HTTP-Basic + SSL has a huge disadvantage of being slower than HTTP, so if you 
are planning for a webservice that might be you know, used, you should not use 
django? For my current project I am using homegrown HMAC like protocol for 
doing secure authentication over plain text protocol, my webservice is currently 
written in PHP, client in DotNet [which I am not sure supports HTTP-Basic]. 

Check out the beuty of the useage I have described in #347 or #352, can be 
described in 3 lines and stays out of the way. That is power.

>  ... and the RPC views would handle the rest.

This line just doesnt sound right in my head. RPC is [method] name-resolution+
marshalling, it should not handle anything other than that. 

Just a note from me on this: please keep in mind that webservices is not only for implementing new webservices where you can choose what architecture (and what way of authentication) you want. It's about implementing all those shitty webservices out there - like metaWeblogAPI, bloggerAPI, Pingback and whatever is there. So people should be able to implement those - that calls out for flexible authentication management and rules out most "consitently defined" authentication ideas. Mostly because all those APIs aren't very consistently defined, to start with. And it's not just about CRUD services via some webservice infrastructure - as I said, there are loads of webservices out there people might want to implement that (hopefully) don't have anything to do with their models. And that's why authentication can't be handled by some abstract infrastructure - it might just not know where to look for authentication data. It's all nice and dandy if you build your own webservices, but that's often not the case.

A second note: +1 on RPC style webservices as a standard view and some dispatching based on method names. That's actually what I tried to implement with #356 and what I am already using with my gallery :-)

And please make RPC style webservices and REST style webservices different beasts - they really don't have much in common and should be discussed and implemented independendly.

All those "shitty webservices" aren't part of this. Django isn't a blog app, or a ping/track/call/whatever-back service; it's a framework. All those other APIs belong in individual apps, not in the core. Ditto with the "loads of other webservices" -- this framework is about a generic CRUD web service akin to the generic views, not implementing a kitchen-sink WS-* stack.

Obviously RPC and REST services will have different code, but as they're both data-over-HTTP it makes sense to share what they can -- authentication is the obvious piece to share.

Speaking of authentication, that's good point about supporting multiple authentication methods. I'm thinking there should be a configurable authentication method so you could plug in an LDAP authenticator or whatever.

Sure, the services aren't part of Django. But the infrastructure to build services is - and if that infrastructure doesn't allow building webservices against protocol definitions that are floating around, it's rather weird: because it would say that you can't write a metaWeblogAPI-enabled blogging software or a ping server with django :-)

Of course people can roll their own RPC solution, but all I am asking for is to have the stuff flexible enough so that they don't _need_ to roll their own XMLRPC handling stuff. So if there is a standard auth scheme in place, make it so that it can be not-used, if the user needs that.

Because, I say it again: I wouldn't like django to be some framework where it is explicitely made harder to write blogging software that implements metaWeblogAPI, just because it's a shitty protocol :-)

I agree with Hugo... Django should provide the goods to implement any web service protocol. Of course, the ability to do that now is there, but it's very low-level (and roll-you-own-ish)... The code that I have for web services is not generic enough, it's tied closely to the model. My starting point was adapting any data structure sent in via XML-RPC or SOAP into the data structure that a models' Manipulator expects. So my web service code is as tightly coupled to the model as the generic views are... So I fulfilled the intent of this ticket, but it might not be generically useful enough. What Hugo is asking for (rightly) is to not promote this tight coupling... Make it easy to do XML-RPC for any protocol (which might be tied to a Django view and not directly to a Model) and make it easy to expose interfaces to Django Models... However, making those things easy is not easy. Right now, I'm reading Rails' ActionWebServices? docs for inspiration...

A simple, although kinda crude way to solve this seems to me to make an app with the following model classes: Change, Read, Update, Delete. Each of these has a ForeignKey?(User). Now, in any model you want to use CRUD authentication you can extend some base class that handles the lookup to these tables, and you add the field "read_access = models.ManyToManyField?(Read)" and you get an admin interface... I think at least, I gotta implement this to see if I haven't made a mistake in my thinking :P

adding me to cc.

added myself to cc

Are there any new on this ticket? some branch or some ongoing work? about authentication, it could/should integrate with django admin permissions and login via HTTP BASIC for example.

Thanks!

Added myself to cc

Removed self from CC

Added myself to cc.

CC'd myself

#547 looks interesting, it has a patch that implements xmlrpc and has been closed as duplicate of this ticket.

For the record:
In this thread on django-users that was CC'ed here on django-developers David Larlet referenced some of the efforts/solutions related to this ticket.

Please be carefull when touching thinks, you removed almost all CC's.

added myself to CC

What's the status of this ticket? :D Is there any plan to support this for version 1.0?

That's probably a question best asked on one of the mailing lists.

In general, "what is the status of this", "when will this be merged", etc. aren't useful questions to ask in a ticket -- they draw any existing discussion off-topic and distract from any real issues, so let's keep them on the lists :)

I've been working on using the decorator patttern to expose individual views. My reasoning is that REST services are URI-based which excludes the models themselves. Using decorators on views lets me expose a URI to various methods on a case-by-case basis. It also allows me to hook into the authentication module.

My original implementation so far:

http://agentultra.com/?p=58

But I recently came across: http://trac.turbogears.org/wiki/RouteByHttpMethod

Which has some neat ideas I should probably integrate (ie: being method-aware and dispatching to alternate views via decorator parameters).

Ultimately, I'd like to see PUT integrated for handling file uploads in Django rather than POST. The performance benefits of PUT are too good to pass up.

This has been a bit of a discussion, rather than any clear path towards resolving anything. There's django-rest-interface for those who REST-based stuff. And I think there's enough experience around now for people to be able to build most of these things without needing extra core support (except for PUT, which is #5682).

If/when anybody builds something sufficiently general it's worth proposing for django/contrib/, that can be brought up on django-dev.