Django

Code

Ticket #3630 (closed: invalid)

Opened 1 year ago

Last modified 10 months ago

Newforms clean_data causes encoding issue with database

Reported by: Baptiste Assigned to: adrian
Component: django.newforms Version: SVN
Keywords: accents Cc: mir@noris.de
Triage Stage: Unreviewed Has patch: 0
Needs documentation: 0 Needs tests: 0
Patch needs improvement: 0

Description

"form" is a newform and "Article" is a model.

new_article = Article(text=form.clean_data['article'])

If the field article contains accents, they are replaced by "?" in the database (same thing if they are displayed with a view/template). Encoding of the page is utf8.

That would be interesting to test if it is a newforms issue by trying to make an Article with accents in the admin of thenewforms-admin branche. I couldn't make it work (admin said I didn't have rights) to test by myself.

Attachments

Change History

03/04/07 02:34:21 changed by Baptiste

  • needs_better_patch changed.
  • needs_tests changed.
  • needs_docs changed.

Some informations : If I "print" the POST informations, accents are encoded like that : \xc3\xa9 (é) I tried to convert them to UTF8 :

		for i in data :
			data[i] = data[i].decode("utf8","replace")
		print data

But now the string recorded in the db stops to the first accent."Espérance" is recorded as "Esp" ; and the server gives me this message :

/usr/lib/python2.4/site-packages/django/db/backends/mysql/base.py:42: Warning: Data truncated for column 'username' at row 1
  return self.cursor.execute(sql, params)

03/04/07 02:35:38 changed by Baptiste

Oops, I have forgotten to say that the "\xc3\xa9" was transformed to "\xe9" with the decode.

03/04/07 07:12:56 changed by anonymous

If I transform the fields of my database in UTF8 and that I transform the encoding of the data :

		for i in data :
			data[i] = data[i].decode("latin1","replace")

The form created with these data is okay and I can do :

		   	new = Article(
			        	name_user=form.clean_data['author_name'],
                                        ...
                                         )

Accents are well recorded in the database. But not well displayed anymore, so I need to keep a reference to the old data and to display a form which call them. Not sexy...

03/08/07 04:15:24 changed by Nuno Mariz

For me works this way:

for i in data :
	if isinstance(data[i], unicode):
		data[i] = data[i].encode(settings.DEFAULT_CHARSET)

Although is ugly.

03/08/07 07:09:20 changed by anonymous

  • cc set to mir@noris.de.

03/27/07 17:08:04 changed by adrian

  • summary changed from Newforms and accents don't like each others to Newforms clean_data causes encoding issue with database.

07/30/07 06:19:01 changed by Simon G. <dev@simon.net.nz>

  • status changed from new to closed.
  • resolution set to invalid.

This should be fixed since after the unicode merge - can someone reopen this, if this is NOT fixed.


Add/Change #3630 (Newforms clean_data causes encoding issue with database)




Change Properties
Action