Opened 9 years ago

Closed 5 years ago

#5877 closed Uncategorized (wontfix)

Force manage.py to output UTF-8 to avoid UnicodeEncodeErrors

Reported by: anonymous Owned by: nobody
Component: Core (Management commands) Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Got the following Unicode error while trying to run: ./manage sqlall ws with the workstyle-py application.
(Btw the application is writen for Django-0.91 and i found this error while porting it to Django-trunk)

The application tries to insert some Japanese (UTF-8) data into the database while running sqlall.
But the console running the ./manage.py command default to ASCII not UTF-8, thus raises an UnicodeEncodeError.
I figure that since Django should be full unicode now this should work and print out UTF-8 (or at least not raise an EncodeError)

The attached path forces the output print stream into UTF-8.
Hopefully this helps other people with Unicode problems on the console with SQL.

Traceback (most recent call last):
  File "./manage.py", line 11, in ?
    execute_manager(settings)
  File "/var/www/site/webapp/__init__.py", line 274, in execute_manager
    
  File "/var/www/site/webapp/__init__.py", line 224, in execute
    
  File "/usr/lib/python2.4/site-packages/django/core/management/base.py", line 71, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.4/site-packages/django/core/management/base.py", line 94, in execute
    print(txt)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 66-68: ordinal not in range(128)

Attachments (1)

patch-django-core-management-base_unicode_printing_#5877.patch (798 bytes) - added by anonymous 9 years ago.

Download all attachments as: .zip

Change History (6)

comment:1 Changed 9 years ago by anonymous

FYI: On the console of my workstation this error does not get raised and utf-8 get's printed correctly.
But when logging in on a remote server (with ssh) i need this (or a similar) patch to make the project work with manage sql* commands

comment:2 Changed 9 years ago by Simon G <dev@…>

Component: Uncategorizeddjango-admin.py
Summary: UnicodeEncodeError in manage.py sqlall on project with utf8 dataForce manage.py to output UTF-8 to avoid UnicodeEncodeErrors
Triage Stage: UnreviewedDesign decision needed

comment:3 Changed 6 years ago by Russell Keith-Magee

Resolution: worksforme
Status: newclosed

This is a very old report, with a diff that no longer applies, without a test case that actually describes how the problem can actually be replicated (I have no idea what workstyle-py is, and the report doesn't tell me where I can get it from).

I'm also reasonably certain that this has been resolved as a side effect of other changes (i.e., I've fixed other bugs about unicode in management output, like #12849). Closing worksforme; please reopen if you have a concrete example that fails.

comment:4 Changed 5 years ago by Joni

Easy pickings: unset
Resolution: worksforme
Severity: Normal
Status: closedreopened
Type: Uncategorized
UI/UX: unset

This doesn't work for me with django 1.3.1, I get this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 0: ordinal not in range(128)

#coding=utf-8
from django.core.management.base import BaseCommand

class Command(BaseCommand):
    def handle(self, *args, **options):
        self.stdout.write(u'á')

comment:5 Changed 5 years ago by Aymeric Augustin

Resolution: wontfix
Status: reopenedclosed

Django doesn't have enforce an output encoding, Python provides everything you need to get output in your teminal's charset!

You must set either your locale, or the PYTHONIOENCODING environment variable, according to your terminal's encoding — in most cases, utf-8.

Here's a demo:

% cat test.py
# coding: utf-8
print u"café!"


% unset LANG
% unset PYTHONIOENCODING
% python test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    print u"café!"
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 3: ordinal not in range(128)


% unset LANG
% export PYTHONIOENCODING=utf-8
% python test.py
café!


% export LANG=en_US.UTF-8
% unset PYTHONIOENCODING
% python test.py
café!

If you still get errors or mangled output, it means that Django is outputting encoded data instead of unicode. If this happens, it's a bug, please open a separate ticket.

Note: See TracTickets for help on using tickets.
Back to Top