Opened 13 years ago

Closed 11 years ago

#16123 closed Bug (fixed)

DateTimeField, DateField fails covert unicode date string with non-ascii characters

Reported by: Gene Owned by: Claude Paroz <claude@…>
Component: Forms Version: 1.3
Severity: Normal Keywords: unicode, date, forms
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

it's old python bug - [python-Bugs-1280061]
in my case (ru_RU)

>>>from sys import platform
>>>from locale import setlocale, LC_ALL
>>>locale = 'russian' if platform == 'win32' else ''
>>>setlocale(LC_ALL, locale) #set locale for '%d %b %Y' convertion
>>>import time, datetime
>>>datetime.date(*time.strptime(u'31 мая 2011', '%d %b %Y')[:3]) # like in django/forms/fields.py, Line 362
ValueError: time data u'31 \u043c\u0430\u044f 2011' does not match format '%d %b %Y'

for strings contains non-ascii characters, you need

'string'.encode('utf8')
>>>datetime.date(*time.strptime(u'31 мая 2011'.encode('utf8'), '%d %b %Y')[:3])
datetime.date(2011, 5, 31)

Attachments (2)

patch.diff (1.3 KB ) - added by Gene 13 years ago.
patch_for_ticket16123.diff (1.9 KB ) - added by Wim Feijen 13 years ago.
Patch for ticket 16123

Download all attachments as: .zip

Change History (9)

by Gene, 13 years ago

Attachment: patch.diff added

comment:1 by Jonas H., 13 years ago

Triage Stage: UnreviewedAccepted

Seems like the right solution to me.

comment:2 by Ramiro Morales, 13 years ago

Needs tests: set
Patch needs improvement: set

Patch need to be generated from the root of the source code tree, needs tests.

comment:3 by Wim Feijen, 13 years ago

Owner: changed from nobody to Wim Feijen
UI/UX: unset

Workon on this during DjangoCon Europe 2011 sprints

comment:4 by Wim Feijen, 13 years ago

Bug is confirmed against trunk.

Suggested patch wirks, but has to be updated to trunk; and tests are needed. I'll try to write those in the upcoming hour.

# in settings.py:
LANGUAGE_CODE = 'ru'

# BEFORE PATCH:
>>> from django.forms.fields import DateField
>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, 'ru_RU.utf-8')
'ru_RU.utf-8'
>>> DateField().strptime(u'31 мая 2011', '%d %b %Y')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/django/forms/fields.py", line 376, in strptime
    return datetime.date(*time.strptime(value, format)[:3])
  File "/usr/lib/python2.7/_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data u'31 \u043c\u0430\u044f 2011' does not match format '%d %b %Y'

# AFTER PATCH:
>>> from django.forms.fields import DateField
>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, 'ru_RU.utf-8')
'ru_RU.utf-8'
>>> DateField().strptime(u'31 мая 2011', '%d %b %Y')
datetime.date(2011, 5, 31)

by Wim Feijen, 13 years ago

Attachment: patch_for_ticket16123.diff added

Patch for ticket 16123

comment:5 by Wim Feijen, 13 years ago

Ok, I included a new patch which fixes the problem.

Before committing, there are three things to look at:

  1. You need to have the Russian language installed to run the test...
  2. The tests import DateField and DateTimeField. Is this the proper way to do it? If so, the imports should be placed on top op the file. If not, the test should be rewritten.
  3. In class BaseTemporalField, method to_python (django/forms/fields.py, line 347) a similar statement is used:
    datetime.datetime(*time.strptime(datetime_str, format[:-3])[:6]+(usecs,))
    

which might need to be converted to

datetime.datetime(*time.strptime(datetime_str.encode('utf-8'), format[:-3])[:6]+(usecs,))

but I cannot write a test case for this, because I do not know where this is being used.

comment:6 by Wim Feijen, 13 years ago

Owner: Wim Feijen removed

comment:7 by Claude Paroz <claude@…>, 11 years ago

Owner: set to Claude Paroz <claude@…>
Resolution: fixed
Status: newclosed

In 4b3f7110ae1e9ef7862c67a38431780080f0062c:

Fixed #16123 -- Ensured strptime receive proper string type

strptime generates an UnicodeEncodeError when using a non-ascii
unicode string on Python 2.

Note: See TracTickets for help on using tickets.
Back to Top