== Unicode and Django ==

This page is to make an impact analysis on the Django source to see what parts of it need what changes if we want to switch Django from using utf-8 bytestrings internally to fully use unicode strings internally.

Just a pin-down of things that spring to mind, all of them need more complete checking:

 * database backends need to handle unicode vs. DATABASE_CHARSET translations
 * special casing: the psycopg backend will need type handlers for string types (just as it already has type handlers for date/time types)
 * the HTTPResponse sending machinery needs to do the unicode to DEFAULT_CHARSET translation
 * the HTTPRequest creation process needs to turn outside strings into unicode strings, using the provided charset (if given) or defaulting to DEFAULT_CHARSET (as that is what was sent to the browser when the form was transmitted)
 * special casing: what happens with GET parameters? those don't provide charsets, what should we do if DEFAULT_ENCODING is utf-8, but the GET parameters aren't valid utf-8? The clean way would be to throw an exception (like with all other places, too)
 * internal usage of str() needs to be checked and supposedly changed over to unicode() usage
 * debugging stuff needs to use repr() on strings, not str() (or use unicode() and let the HTTP response handling stuff handle the conversion - most debugging stuff is working with the response machinery anyway)
 * mail sending functions need to do the right thing with the MIME type
 * we should decide wether to normalize the input unicode data so that at the database or application level we can match strings regardless of their decomposition (see the standard lib’s [http://docs.python.org/lib/module-unicodedata.html unicodedata module ] with its `normalize()` function). I would go for NFC, if there’s consensus around normalizing.
 * Lazy evaluated method calls do not currently work with Unicode return values, see #1664.

Please either complete the above list or add headlines with more detailed discussions of the points above. Please only post results here, discussion should take place on the django-developer list.