New Manipulators

[This section is a lot of explanation, to get right to some code scroll down.]

Although validation aware models are great for database integrity, and should be leveraged within the manipulator system, it is still necessary for manipulators to do most of the work. There are three main steps to the process of taking post data and updating or creating a model object with that data:

1. Gather and convert the post data.
2. Validate the converted data.
3. Apply the data to the object and save, aka do something with the data.

A validation aware model alone cannot complete these tasks as information needs to be provided that details the sorts of fields that should be used, and processes are needed to convert from string / post data to the pythonic data. Furthermore many manipulation and form tasks require form fields that do not map directly to a model field, for instance a password changing form that requires a password to be given twice. This example also illuminates how a form field might map directly to a model field, but may not want to give its direct value to the field. As in a password, one probably wouldn't want: user.password = data['password'], but rather: user.set_password(data['password']).

So we change Manipulators some, and then add model validation. But we need to make our manipulator system leverage it. In other words, when our model throws a ValidationError, we need to add it to our errors and then present our form back to the user. This should be made simple for the developer so that they don't have to create a try/except block every time they want to save an object.

This means that validation exceptions could be thrown at any point in our manipulation process: conversion, validation, and model object saving. There might even be more steps in the future, or with manipulators that aren't HTML based but something else like JSON.

There are a few ways of doing this, from the developer's perspective. Currently manipulator process in a custom view for the developer is needlessly complicated. Compare the ​current process to the ​rather basic one that added to the cookbook. But with model validation it complicates both with even more code for checking for errors after the object.save().

So it occured to me that a form was a lot like exception. At various points during the manipulation process it becomes evident that user input is needed, and that we cannot continue on. This comes to a head at one point in the view where it returns a render_to_response('form_template', {'form': form}). So why not make the Form class an actual Exception? This way we can raise the form all the way back to the view. Here then is the basic form of my proposed view:

def create_poll(request):
    try:
        m = Poll.CreateManipulator()
        poll = m.process(request)
        return HttpResponseRedirect('/poll/%d/' % poll.id)
    except Form, form:
        return render_to_response('poll/create.html', {'form': form})

As you can see, our form is now more of an exception. And our manipulation process is very simple, most of it takes place in process(), and if at any point a form is raised, it is presented. Otherwise the manipulator calls and returns save(). In this case it creates and returns a new poll object. It will also raise a form if the model validation returns an error. If no form is raised, we are done and return our Response.

Benefits

The benefits of this system include:

• Custom form views become very simple, and intuitive.
• Validation aware models are harnessed in this system.
• Custom Manipulators could be defined in the model itself, which hightens the sense of everything in one place. Likewise, it can be seperated out if it's too much code in one spot.
• Defining Custom Manipulators allows one greater control of how the Admin deals with objects. Having a field automatically update with the author's User object would be a snap. Also, some options could move from the class Admin: into the Custom Manipulator, which would tighten things up a great deal.
• Making your own Admin app now becomes a lot easier. People often want to be able to simply plop a form down, and this gives them that ability.
• Because the form is raised, like an exception, it allows the writer to assume the data is correct the whole way through. But when there is a validation error, it is handled easily.

Compatibility

There would be many changes needed in some other django code for all of this to work out right:

• The FormWrapper class would be destroyed, and would be supplanted by the easier Form.
• Form fields would be altered, both cosmetically "html2python" -> "to_python", and functionally for optimization / simplicity
• Models would have to gain the validation awareness that we've been talking about.
• The Model metaclass would need to create default Create and Edit manipulators if they weren't supplied (this would be very easily done).

Comments

Add wiki-comments here, also note there is a thread on the Django-devel mailing list: ​New Manipulators and Validation Aware models.

Code

There is a lot more code to this, as FormWrapper and FormFieldWrapper and some other pieces have been rewritten. Here is the most of it, though:

class Manipulator(object):
    """     
    A Manipulator "manipulates" data and objects.  
    Sample usage with Poll.CreateManipulator:
    
    def create_poll(request):
    try:
        m = Poll.CreateManipulator()
        poll = m.process(request)
        return HttpResponseRedirect('/poll/%d/' % poll.id)
    except Form, form:
        return render_to_response('poll/create.html', {'form': form})
    
    As you can see, if a Manipulator needs to get information with a form, it 
    will raise it's form, which can then be placed right into a template context.

    To create a custom manipulator you have four functions that you may 
    want to override, each is a step in the maniplator process:
        * __init__
        * convert
        * validate
        * save
        
    And there are four more attributes that you will deal with:
        * fields - A sequence that defines the fields of the Manipulator
        * errors - A dictionary that holds a mapping of field-name -> error list
        * data - A dictionary of the current data.  Remember that this may be
                    raw data if conversion has not yet been done.
        * form - This is actually a property that returns the form to be given
                    in the template context.
    """

    def __getitem__(self, k):
        "Allows access of the fields via the brackets, like: manipulator['field_name']"
        for f in self.fields:
            if (f.name == k):
                return f
        raise KeyError, "Could not find field with the name %r." % k
        
    def __repr__(self):
        return "Manipulator <%r, %r, %r>" % (self.fields, self.data, self.errors)
    
    def __init__(self):
        # Properties that must be created:
        self.fields = ()
        # Properties that can optionally be created:
        self.data = {}  # The default data given
        self.errors = {}
    
    def convert(self, data):
        """
        This conversion function will happen AFTER the data is already converted 
        by the fields.  More conversion can be done here.
        """
        pass
    
    def validate(self, data):
        """
        Here you can further validation after the fields have already had their
        go at it.
        """
        pass
    
    def save(self, data):
        """
        Once the data has been converted and validated it is ready to be 
        processed.  Usually this will entail creating or updating an model instance.
        To aid in that task, there are convenience functions edit() and create(), 
        see below for more details.
        """
        raise NotImplementedError
    
    ### Interneal functions ###
    def process(self, request):
        "Perform the manipulation process."
        if (not request.POST):
            raise self.form
        self.request = request
        self.data = copy_dict(request.POST) # copy_dict == lambda m: dict((k, v) for k, v in m.items())
        self.errors = {}
        self._convert()
        self._validate()
        return self._save()
    
    def _get_form(self):
        return Form(self.fields, self.data, self.errors)
    form = property(_get_form)
    
    def _convert(self):
        """
        Goes through each field and converts the data in place on the data dictionary.
        This will raise a Form if there are validation errors by the end.
        """
        
        for field in self.fields:
            name = field.name
            if (name in self.data):
                try:
                    #~ self.data[name] = field.to_python(self.data[name])
                    self.data[name] = field.to_python(self.data[name])
                except (validators.ValidationError, validators.CriticalValidationError), e:
                    self.errors.setdefault(name, []).extend(e.messages)
        if (self.errors):
            raise self.form
        self.convert(self.data)
        if (self.errors):
            raise self.form
    
    def _validate(self):
        """
        Goes through each field and validates the data.  This will raise a form if there are 
        validation errors by the end.
        """
        
        for field in self.fields:
            name = field.name
            #~ for validator in field.validators:
                #~ try:
                    #~ validator(self.data[name])
                #~ except (validators.ValidationError, validators.CriticalValidationError), e:
                    #~ self.errors.setdefault(name, []).extend(e.messages)
            errors = field.validate(self.data)
            if (errors):
                self.errors.setdefault(name, []).extend(errors)
        if (self.errors):
            raise self.form
        self.validate(self.data)
        if (self.errors):
            raise self.form
    
    def _save(self):
        try:
            return self.save(self.data)
        except (validators.ValidationError, validators.CriticalValidationError), e:
            name = "Some field"
            self.errors.setdefault(name, []).extend(e.messages)
            raise self.form
    
    ### Default Properties ###
    data = {}
    errors = {}
    fields = ()

### Manipulator Factories / Helpers ###
def create_manipulator(model_cls):
    class CreateManipulator(Manipulator):
        def __init__(self):
            self.fields = generate_fields(model_cls)
            # generate_fields() would need to be built.  It creates a list of fields based
            # on the Model's fields.
        
        def save(self, data):
            return create(Poll, data)
    return CreateManipulator

def edit_manipulator(model_cls):
    class EditManipulator(Manipulator):        
        def __init__(self, object):
            self.fields = generate_fields(model_cls)
            self.object = object
        
        def save(self, data):
            return forms.edit(self.object, data)
    return EditManipulator

def create(Model, data):
    """
    Usefull in the .save of a custom manipulator, this will quickly create an object for you.
    """
    m = Model(**data)
    # In a validation aware model, any problems would arise here.
    m.save()
    return m
    
def edit(i, data):
    """
    Like create() above, this will update an object for you.
    """
    i.__dict__.update(data)
    # Also validation errors would arise here.
    i.save()
    return i

### Attach to classes ###
Poll.CreateManipulator = create_manipulator(Poll)
Poll.EditManipulator = edit_manipulator(Poll)

### Custom Manipulator ###
class CreateManipulator(Manipulator):
    def __init__(self, request):
        self.fields = generate_fields(Poll)   # Generate fields creates default fields for a Model

        # Fields have an "attributes" attribute, that adds itself to the tag that is created for it.
        # Here, we change our class to "vPopupDateField" so that a javascript function will know to
        # add a Date popup window for when the field gets focus.
        self.fields['pub_date'].attributes = {'class': 'vPopupDateField'}
    
    def save(self):
        poll = create(Poll, data)
        # Here we'll also save our author as the current user.
        # This assumes the Poll model has an author field
        poll.author = self.request.user
        poll.save()
        return poll

### Custom views ###
def create_poll(request):
    try:
        m = Poll.CreateManipulator()
        poll = m.process(request)
        return HttpResponseRedirect('/polls/%d/' % poll.id)
    except Form, form:
        return render_to_response('polls/create.html', {'form': form})
            
def update_poll(request, id):
    poll = get_object_or_404(Poll, pk=id)
    try:
        m = Poll.EditManipulator(request, poll)
        m.process(request)
        return HttpResponseRedirect('/polls/%d/' % poll.id)
    except Form, form:
        return render_to_response('polls/update.html', {'form': form, 'poll': poll})

### Template Tag ###
#
# Forms should be renderable with a simple tag, that plops a form down for the user:
#
# <form action='.' method='post'>
#    {% render-form form %}
#    <input type='submit' value='Create Poll'/>
# </form>
#
# or {% render-form-row form.slug %} could be used to create just a row.  Many more options could be made.
# 

### Generic Views ###
def create_view(model, template=None):
    """
    Generic create view.  Redirects to object.get_update_url() or 
    object.get_absolute_url() when it's done.
    
    TODO: Add all the other stuff already in the create view.
    """
    if template == None:
        template = "%s/%s_create.html" % (model._meta.app_label, model._meta.object_name.lower())
    def create_func(request):
        try:
            m = model.CreateManipulator()
            object = m.process(request)
            if hasattr(object, 'get_update_url'):
                return HttpResponseRedirect(object.get_update_url())
            else:
                return HttpResponseRedirect(object.get_absolute_url())
        except Form, form:
            return render_to_response(template, {'form': form})
    return create_func
    
def update_view(model, template=None, pk=None):
    """
    Generic update view.  Redirects to object.get_absolute_url() when it's done.
    
    TODO: Add all the other stuff already in the update view.
    """
    if template == None:
        template = "%s/%s_update.html" % (model._meta.app_label, model._meta.object_name.lower())
    def update_func(request, pk):
        object = get_object_or_404(model, pk=pk)
        try:
            m = model.EditManipulator(object)
            m.process(request)
            return HttpResponseRedirect(object.get_absolute_url())
        except Form, form:
            return render_to_response(template, {'form': form, 'object': object})
    return update_func   

### urls.py generic usage ###
from app.models import Poll
('^polls/create/%d/$', 'django.views.generic.create_update.create_view', {'model':Poll})
('^polls/update/$',    'django.views.generic.create_update.update_view', {'model':Poll})