Clarify pytz requirement for time zone support

[z]

A basic Django application:

models.py

from django.db import models

class test(models.Model):
    slug = models.SlugField(unique_for_date='pub_date')
    pub_date = models.DateTimeField()

admin.py

from django.contrib import admin
from app.models import test

admin.site.register(test)

Creating new "test" entry using Django admin results in UnicodeDecodeError on save:

UnicodeDecodeError at /admin/app/test/add/
'utf8' codec can't decode byte 0xcc in position 225: invalid continuation byteRequest Method:	POST
Request URL:	http://127.0.0.1:8000/admin/app/test/add/
Django Version:	1.6.3
Exception Type:	UnicodeDecodeError
Exception Value:	'utf8' codec can't decode byte 0xcc in position 225: invalid continuation byte
.....
Unicode error hint

The string that could not be encoded/decoded was: ONE '������

This code works fine up to Django 1.5.*.
Using DateField instead of DateTimeField also works in current version of Django.

[Tim Graham]

I wasn't able to reproduce this. A full traceback would be helpful. I guess you may be using a non-English locale so there is some unicode character in the DateTime field? Please reopen if you can provide more details.

[z]

I am new to Django but what I have been able to understand is that this is a time zone support issue.
From Django docs:​https://docs.djangoproject.com/en/1.6/topics/i18n/timezones/:

Time zone support is disabled by default. To enable it, set USE_TZ = True in your settings file. Installing pytz is highly recommended, but not mandatory.

But when I create a project in Django 1.6.3, USE_TZ = True by default.
Disabling time zone support solves the problem, as does installing the pytz while setting USE_TZ = True.

As far as I can understand this shouldn't be the case.

Here is full traceback:

Environment:


Request Method: POST
Request URL: http://localhost:8000/admin/app/test/add/

Django Version: 1.6.3
Python Version: 2.7.6
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'app')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')


Traceback:
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\core\handlers\base.py" in get_response
  114.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\contrib\admin\options.py" in wrapper
  432.                 return self.admin_site.admin_view(view)(*args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  99.                     response = view_func(request, *args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\views\decorators\cache.py" in _wrapped_view_func
  52.         response = view_func(request, *args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\contrib\admin\sites.py" in inner
  198.             return view(request, *args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\utils\decorators.py" in _wrapper
  29.             return bound_func(*args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  99.                     response = view_func(request, *args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\utils\decorators.py" in bound_func
  25.                 return func(self, *args2, **kwargs2)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\transaction.py" in inner
  371.                 return func(*args, **kwargs)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\contrib\admin\options.py" in add_view
  1113.             if form.is_valid():
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\forms\forms.py" in is_valid
  129.         return self.is_bound and not bool(self.errors)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\forms\forms.py" in errors
  121.             self.full_clean()
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\forms\forms.py" in full_clean
  275.         self._post_clean()
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\forms\models.py" in _post_clean
  419.             self.validate_unique()
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\forms\models.py" in validate_unique
  428.             self.instance.validate_unique(exclude=exclude)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\models\base.py" in validate_unique
  755.         date_errors = self._perform_date_checks(date_checks)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\models\base.py" in _perform_date_checks
  881.             if qs.exists():
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\models\query.py" in exists
  512.             return self.query.has_results(using=self.db)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\models\sql\query.py" in has_results
  409.         return bool(compiler.execute_sql(SINGLE))
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\models\sql\compiler.py" in execute_sql
  786.         cursor.execute(sql, params)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\backends\util.py" in execute
  73.             sql = self.db.ops.last_executed_query(self.cursor, sql, params)
File "c:\dj_proj\Test\Test\test_env\lib\site-packages\django\db\backends\postgresql_psycopg2\operations.py" in last_executed_query
  214.             return cursor.query.decode('utf-8')
File "c:\dj_proj\Test\Test\test_env\lib\encodings\utf_8.py" in decode
  16.     return codecs.utf_8_decode(input, errors, True)

Exception Type: UnicodeDecodeError at /admin/app/test/add/
Exception Value: 'utf8' codec can't decode byte 0xcc in position 225: invalid continuation byte

Here is a local var input from 16. return codecs.utf_8_decode(input, errors, True)

input	'SELECT (1) AS "a" FROM "app_test" WHERE ("app_test"."pub_date" BETWEEN \'2014-01-01T00:00:00+04:00\'::timestamptz and \'2014-12-31T23:59:59.999999+04:00\'::timestamptz AND EXTRACT(\'month\' FROM "app_test"."pub_date" AT TIME ZONE \'\xcc\xee\xf1\xea\xee\xe2\xf1\xea\xee\xe5 \xe2\xf0\xe5\xec\xff (\xe7\xe8\xec\xe0) \') = 4 AND EXTRACT(\'day\' FROM "app_test"."pub_date" AT TIME ZONE \'\xcc\xee\xf1\xea\xee\xe2\xf1\xea\xee\xe5 \xe2\xf0\xe5\xec\xff (\xe7\xe8\xec\xe0) \') = 28 AND "app_test"."slug" = \'ddad\' ) LIMIT 1'

codecs.utf_8_decode can't decode this part \xcc\xee\xf1\xea\xee\xe2\xf1\xea\xee\xe5 \xe2\xf0\xe5\xec\xff (\xe7\xe8\xec\xe0) \

OS is windows 7.

[Aymeric Augustin]

You're seeing the intended and documented behaviour. (I know because I wrote that code and its documentation.)

If you don't specify USE_TZ, the default is False. Since it's probably a better choice for most sites to turn it on, startproject creates a settings.py that contains USE_TZ = True.

Then, on Windows, you usually have install pytz for things to work correctly, unless you're lucky enough to operate in a locale whose time zone name matches MySQL's definitions. On Unix/Linux you can get away, to some extent, with not installing pytz. Hence the vague advice.

If you remove the declaration of USE_TZ in the settings file, you're back to the default of False and you no longer need pytz.

I'm leaving this ticket open in case you want to suggest a better wording for the documentation. (I'm not a native speaker.) Otherwise, it can be closed.

[Tim Graham]

How about something like this: "Installing pytz is highly recommended, but may not be mandatory depending on your particular operating system and time zone. If you encounter an exception querying dates or times, please try installing it before filing a bug."

[Aymeric Augustin]

It also depends on the database backend. I'll commit your proposal.

[Aymeric Augustin <aymeric.augustin@…>]

In 0fb7570c5cc89e6a8adaa5052574de29ccf6adde:

Fixed #22523 -- Clarified pytz requirement.

Thanks z for the report and timo for the proposal.

[Aymeric Augustin <aymeric.augustin@…>]

In 95c5e244fef906c688642f555292635e1b10a8c3:

[1.7.x] Fixed #22523 -- Clarified pytz requirement.

Thanks z for the report and timo for the proposal.

Backport of 0fb7570 from master

[Aymeric Augustin <aymeric.augustin@…>]

In f6e9127483fe09601ed176d0e56bb77bd5cf9b8c:

[1.6.x] Fixed #22523 -- Clarified pytz requirement.

Thanks z for the report and timo for the proposal.

Backport of 0fb7570 from master

[z]

Thanks! By the way, I got to the bottom of this. Django's timezone.py uses standard time module in python (when pytz is unavailable), and in particular it uses "time.tzname". And in windows using time.tzname is unreliable, because it acquires time zone names from the registry and apparently

Microsoft stores the the name of the time zone as displayed in the dialog box where you change time zones ...
that value will be translated into the language of the installed Windows.

So this stuff "\xcc\xee\xf1\xea\xee\xe2\xf1\xea\xee\xe5 \xe2\xf0\xe5\xec\xff (\xe7\xe8\xec\xe0) \" is the name of the time zone, the value of time.tzname.

So in fact it doesn't depend on the database backend.

[Aymeric Augustin]

Yes, but other parts of Django may depend or not on pytz depending on your database backend. For instance PostgreSQL on Linux almost works without pytz.