Code

Opened 7 years ago

Closed 3 years ago

#5877 closed Uncategorized (wontfix)

Force manage.py to output UTF-8 to avoid UnicodeEncodeErrors

Reported by: anonymous Owned by: nobody
Component: Core (Management commands) Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Got the following Unicode error while trying to run: ./manage sqlall ws with the workstyle-py application.
(Btw the application is writen for Django-0.91 and i found this error while porting it to Django-trunk)

The application tries to insert some Japanese (UTF-8) data into the database while running sqlall.
But the console running the ./manage.py command default to ASCII not UTF-8, thus raises an UnicodeEncodeError.
I figure that since Django should be full unicode now this should work and print out UTF-8 (or at least not raise an EncodeError)

The attached path forces the output print stream into UTF-8.
Hopefully this helps other people with Unicode problems on the console with SQL.

Traceback (most recent call last):
  File "./manage.py", line 11, in ?
    execute_manager(settings)
  File "/var/www/site/webapp/__init__.py", line 274, in execute_manager
    
  File "/var/www/site/webapp/__init__.py", line 224, in execute
    
  File "/usr/lib/python2.4/site-packages/django/core/management/base.py", line 71, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.4/site-packages/django/core/management/base.py", line 94, in execute
    print(txt)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 66-68: ordinal not in range(128)

Attachments (1)

patch-django-core-management-base_unicode_printing_#5877.patch (798 bytes) - added by anonymous 7 years ago.

Download all attachments as: .zip

Change History (6)

comment:1 Changed 7 years ago by anonymous

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

FYI: On the console of my workstation this error does not get raised and utf-8 get's printed correctly.
But when logging in on a remote server (with ssh) i need this (or a similar) patch to make the project work with manage sql* commands

comment:2 Changed 7 years ago by Simon G <dev@…>

  • Component changed from Uncategorized to django-admin.py
  • Summary changed from UnicodeEncodeError in manage.py sqlall on project with utf8 data to Force manage.py to output UTF-8 to avoid UnicodeEncodeErrors
  • Triage Stage changed from Unreviewed to Design decision needed

comment:3 Changed 4 years ago by russellm

  • Resolution set to worksforme
  • Status changed from new to closed

This is a very old report, with a diff that no longer applies, without a test case that actually describes how the problem can actually be replicated (I have no idea what workstyle-py is, and the report doesn't tell me where I can get it from).

I'm also reasonably certain that this has been resolved as a side effect of other changes (i.e., I've fixed other bugs about unicode in management output, like #12849). Closing worksforme; please reopen if you have a concrete example that fails.

comment:4 Changed 3 years ago by joni

  • Easy pickings unset
  • Resolution worksforme deleted
  • Severity set to Normal
  • Status changed from closed to reopened
  • Type set to Uncategorized
  • UI/UX unset

This doesn't work for me with django 1.3.1, I get this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 0: ordinal not in range(128)

#coding=utf-8
from django.core.management.base import BaseCommand

class Command(BaseCommand):
    def handle(self, *args, **options):
        self.stdout.write(u'á')

comment:5 Changed 3 years ago by aaugustin

  • Resolution set to wontfix
  • Status changed from reopened to closed

Django doesn't have enforce an output encoding, Python provides everything you need to get output in your teminal's charset!

You must set either your locale, or the PYTHONIOENCODING environment variable, according to your terminal's encoding — in most cases, utf-8.

Here's a demo:

% cat test.py
# coding: utf-8
print u"café!"


% unset LANG
% unset PYTHONIOENCODING
% python test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    print u"café!"
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 3: ordinal not in range(128)


% unset LANG
% export PYTHONIOENCODING=utf-8
% python test.py
café!


% export LANG=en_US.UTF-8
% unset PYTHONIOENCODING
% python test.py
café!

If you still get errors or mangled output, it means that Django is outputting encoded data instead of unicode. If this happens, it's a bug, please open a separate ticket.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.