[patch] newforms: choices checking should unicode choices list before comparing to a value

quick fix

Oh, yet another unicode bug ;-)

Hmm, since newforms is unicode country, I guess this is the wrong fix. formfield.choices should contain unicode strings, not bytestrings. IMHO, the bug is rather in formfield() (probably ForeignKey.formfield() ), it should decode the options to unicode during formfield construction, using settings.DEFAULT_CHARSET as encoding. Can you please specify how you created the ChoiceField??

A unit test would be great (it's needed anyway, no matter how this turns out).

But I leave this to Adrian --> decision needed.

Michael Radziej, you seem misunderstood it.

clean() always converts value to unicode and then unicode value is compared against list of choices and if this list is not in unicode, this check fails. My patch converts choices list to unicode if needs just for comparing because unicode should be compared against unicode, not bytestrings.

And again: take a look a few lines below my patch, there is absolutely the same code for MultipleChoiceField? and it does exactly the same as my patch does

Detailed.

line 354: value = smart_unicode(value) the value is in unicode after this line

line 357 (original): valid_values = set([str(k) for k, v in self.choices]) here we create the list of available choices.

line 358: if value not in valid_values: any ideas how this may work if valid_values contains a list of bytestrings ?

And finally, compare my new line 357: valid_values = set([smart_unicode(k) for k, v in self.choices]) to original line 383: valid_values = set([smart_unicode(k) for k, v in self.choices]) :)

I think I understood you perfectly. I argue that formfield.choices should be a list of tuples of unicode strings. Well, you can do it this way or either, it's Adrian's decisions, anyway.

But leaving this aside, I tried to reproduce your bug, but I cannot:

>>> u"bla" == "bla"
True
>>> u"a" in set(["a"])
True

Then I try with newforms:

>>> from django.newforms import *
>>> f=ChoiceField(choices=[("a","a"),("b","b")])
>>> f.clean(u"a")
u'a'
>>> f.clean(u"c")
django.newforms.util.ValidationError                           Traceback (most recent call last)

/home/mir/src/kundebunt/<console> 

/home/mir/lib/python/django/newforms/fields.py in clean(self, value)
    357         valid_values = set([str(k) for k, v in self.choices])
    358         if value not in valid_values:
--> 359             raise ValidationError(gettext(u'Select a valid choice. %s is not one of the available choices.') % value)
    360         return value
    361 

ValidationError: [u'Select a valid choice. c is not one of the available choices.']

So, I this works for me. Without converting the choices to unicode. Can you show me a way how to get this bug?

Then try this:

>>> u"превед" == "превед"
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
>>> u"превед" in ["превед"]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

and the code that fails:

class InvoiceBody(models.Model):
    ....
    UNITS = (
        ('шт.', 'шт.'),
        ('мес.', 'мес.'),
    )
    units = models.CharField(maxlength=10, choices=UNITS)

and then try to validate a form built using form_for_model()

And btw, you suggest to define choices as unicode strings. Then this line (original) would do:

>>> str(u"превед")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)

Doesn't work either.

I'm convinced ;-)

I meant 'Accepted'! Probably all this unicode stuff should be written by Russians ;-)

Not my day. But it still needs test.

Michael, if Adrian won't merge this patch to svn without tests until tomorrow (local time) I'd add tests too.

But I am happy anyway that you have changed your mind about all my work :)

Umm there is a problem with tests: because it touches local (national) environments, it seems I need a new test directory similar to tests/modeltests/model_forms/models.py but with sources in coding: utf-8 or windows-1251. Is that ok ?

I really don't know, better ask on the list. Perhaps you can also use unicode escapes like '\u1234', but better discuss this on the list.

(In [5236]) unicode: Changed a few more places in newforms where str() was being used with potential non-ASCII arguments. Refs #3406 (and added a test for the latter).

This bug is fixed in the above commit -- [5236] -- to the unicode branch. I'll close this ticket once that branch is merged back into trunk.

(In [5609]) Merged Unicode branch into trunk (r4952:5608). This should be fully backwards compatible for all practical purposes.

Fixed #2391, #2489, #2996, #3322, #3344, #3370, #3406, #3432, #3454, #3492, #3582, #3690, #3878, #3891, #3937, #4039, #4141, #4227, #4286, #4291, #4300, #4452, #4702

seduction Here's my post on how to get laid in the next month or two. This post does not maximize your chances of "getting really good at pickup" - in fact, it does the opposite.