Opened 9 years ago

Closed 9 years ago

Last modified 8 years ago

#1945 closed defect (worksforme)

Problem following the tutorial when using non English characters

Reported by: gradha@… Owned by: jacob
Component: Internationalization Version:
Severity: minor Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

I'm following http://www.djangoproject.com/documentation/tutorial1/ and its second part. Everything goes OK. Just, to try things, I modify most of the examples of code. So when I modified the polls/models.py to create the Poll object, instead of using pub_date's tutorial provided description, I wrote:

pub_date = models.DateTimeField('Fecha de publicación')

To make the script a valid python script, I added at the top:

# -*- coding: latin1 -*-

So later, much much later, when I'm adding the administrative interface, I reach the point where I have added the "class Admin: pass" to Poll and web browsed my way to see the first poll. Here, the poll is displayed, and the date description shows a broken character (the accented o): "Fecha de publicaci�n:" (not sure if that will get through, it shows a question mark inside a rotated square).

Well, I guess maybe Django wants unicode characters or stuff. So I modified my Poll class to read (note the unicode string):

pub_date = models.DateTimeField(u'Fecha de publicación')

Reloading the admin page I get:

UnicodeDecodeError at /admin/polls/poll/1/
'ascii' codec can't decode byte 0xc3 in position 75: ordinal not in range(128)

...along with a hell long of a callstack and environment dump. Since I'm used to these kind of things, I tried the following change:

pub_date = models.DateTimeField(u'Fecha de publicación'.encode("utf8"))

...which worked and didn't crash, but strikes me as unusually cumbersome to do. AFAICS the model doesn't properly treat unicode strings and the reason I get this OK with the utf8 encoding is just because that's what the output page charset is, and the result is processed by the browser. I guess if I switched that to latin1 now it would break visually as well, right? Is Django unable to handle internationalised stuff in the model?

If this is the case, I would add a note on the kind of strings you can use for the model.

Change History (5)

comment:1 Changed 9 years ago by asmodai@…

I cannot reproduce this.

I used:

# -*- coding: latin1 -*-

with a verbose name of:

models.CharField('áóéíàèìò', maxlength = 64)

and in my admin I see:

áóéíàèìò:

Did you check your browser to see if it selected/auto-detected the right page encoding? It should be UTF-8.

Note that I am using the latest SVN version and you didn't specify yours though.

comment:2 Changed 9 years ago by adrian

  • Resolution set to fixed
  • Status changed from new to closed

Looks like this isn't a bug. (See previous comment.)

comment:3 Changed 9 years ago by ramiro

  • Component changed from Documentation to Internationalization
  • Resolution fixed deleted
  • Status changed from closed to reopened

I'm reopening this ticket because I'm experiencing the same issue and to post some additional information:

SVN revision 3065

first line of models.py has:

  # -*- coding: iso-8859-1 -*-

a model:

  class Project(models.Model):
    ...
    description = models.CharField('descripcion', blank=True, maxlength=255, db_column='DESCRIPTION')
    ...
    class Admin:
        fields = ((None, { 'fields': ('title', 'description', 'client', 'start_date', 'leader', 'status')} ), )
        list_display=('title', 'client', 'status', 'leader')

Django conf/global_settings.py has

  DEFAULT_CHARSET = 'utf-8'

Project settings.py has

  DEBUG = True
  LANGUAGE_CODE = 'es-ar'
$ file models.py
models.py: ASCII English text

If I edit the field verbose name to add a non-ASCII char

    description = models.CharField('descripción', blank=True, maxlength=255, db_column='DESCRIPTION')
$ file models.py
models.py: ISO-8859 English text

And then whe trying to access the edit page for that model by selecting it from the change list page of the admin app, I get a 'Descripci�n:' field label.

I'm using the bundled Django development Web server.

Server-side info:
Linux, python 2.3.5

Client-side info:

Mozilla Firefox 1.5.0.4 / Win32 / Windows 2000 + SP4 English

View -> Character Encoding -> UTF-8 is checked, if I change it to Western (ISO-8859-1) the
problem goes away.

When I change the verbose name to include the u prefix

    description = models.CharField(u'descripción', blank=True, maxlength=255, db_column='DESCRIPTION')

I also get the exception + back trace debug page:

Request Method:  	GET
Request URL: 	http://<my server DNS name>:8000/admin/intro/project/81/
Exception Type: 	UnicodeDecodeError
Exception Value: 	'ascii' codec can't decode byte 0xc3 in position 222: ordinal not in range(128)
Exception Location: 	/usr/lib/python2.3/site-packages/django/template/__init__.py in render, line 686
Traceback (most recent call last):
File "/usr/lib/python2.3/site-packages/django/template/__init__.py" in render_node
  701. result = node.render(context)
File "/usr/lib/python2.3/site-packages/django/template/defaulttags.py" in render
  113. nodelist.append(node.render(context))
File "/usr/lib/python2.3/site-packages/django/template/defaulttags.py" in render
  115. return nodelist.render(context)
File "/usr/lib/python2.3/site-packages/django/template/__init__.py" in render
  686. return ''.join(bits)

  UnicodeDecodeError at /admin/intro/project/81/
  'ascii' codec can't decode byte 0xc3 in position 222: ordinal not in range(128)

Why is the text getting decoded with the ascii codec?.

Feel free to ask for further info bacause I have the environment at hand.

Regards,

--

Ramiro

comment:4 Changed 9 years ago by ramiro

More info:

"Request information - META" section of the backtrace:

CONTENT_LENGTH
CONTENT_TYPE'text/plain'
DJANGO_SETTINGS_MODULE'tang.settings'
GATEWAY_INTERFACE'CGI/1.1'
HOME'/home/ramiro'
HTTP_ACCEPT'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING'gzip,deflate'
HTTP_ACCEPT_LANGUAGE'en-us,en;q=0.7,es-ar;q=0.3'
HTTP_CONNECTION'keep-alive'
HTTP_COOKIE'sessionid=145006e2d5e41bb0ae6e740c5c3e4811; SITESERVER=ID=f9aa77d7c40371371daebd95787b0042'
HTTP_HOST'<my Django server DNS name>:8000'
HTTP_IF_MODIFIED_SINCE'Fri, 02 Jun 2006 18:46:50 GMT'
HTTP_IF_NONE_MATCH'1a18096f877784f328f0045a20f26e9a'
HTTP_KEEP_ALIVE'300'
HTTP_REFERER'http://<my Django server DNS name>:8000/admin/intro/project/'
HTTP_USER_AGENT'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4'
LANG'en_US'
LANGUAGE'en_AR:en_US:en_GB:en'
LOGNAME'ramiro'
PATH_INFO'/admin/intro/project/81/'
PWD'/home/ramiro/src/tang'
QUERYSTRING
REMOTE_ADDR'<my workstation IP address>'
REMOTE_HOST'<my workstation DNS name>'
REQUEST_METHOD'GET'
RUN_MAIN'true'
SCRIPT_NAME
SERVER_NAME'<my Django server DNS name>'
SERVER_PORT'8000'
SERVER_PROTOCOL'HTTP/1.1'
SERVER_SOFTWARE'WSGIServer/0.1 Python/2.3.5'
TZ'America/Cordoba'
USER'ramiro'
_'./manage.py'
wsgi.errors<open file '<stderr>', mode 'w' at 0xb7d890a0>
wsgi.file_wrapper<class django.core.servers.basehttp.FileWrapper at 0xb7a9e7dc>
wsgi.input<socket._fileobject object at 0xb7031c34>
wsgi.multiprocessFalse
wsgi.multithreadTrue
wsgi.run_onceFalse
wsgi.url_scheme'http'
wsgi.version(1, 0)

comment:5 Changed 9 years ago by ramiro

  • Resolution set to worksforme
  • Status changed from reopened to closed

I'm seeing now that a solution to this problem was posted in one of the lasts comments of ticket #170 (I've tested it and it works).

So I'm closing this ticket again. Perhaps I will open another targeted at the documentation component because it seems a somewhat common pitfall for non English users (see #1715, http://groups.google.com/group/django-users/browse_thread/thread/d536902362a36b9c/0d5b920adc628005#0d5b920adc628005). If the recommended solution is in fact the right one then maybe a note could be added to the models docs stating that for non ASCII languages it is beter to encode the models.py file in UTF-8 and flag it so by using magic comments as per PEP 0263

Regards,

Note: See TracTickets for help on using tickets.
Back to Top