"USE_TZ = True" causes error

[tomas_00]

When setting USE_TZ to True, this error appears:

'NoneType' object has no attribute 'replace'

Environment:


Request Method: GET
Request URL: http://domain.com/admin/news/image/

Django Version: 1.6.dev20130227090207
Python Version: 2.7.3
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.sites',
 'django.contrib.flatpages',
 'south',
 'djcelery',
 'gunicorn',
 'sorl.thumbnail',
 'template_utils',
 'compressor',
 'tagging',
 'ckeditor',
 'debug_toolbar',
 'mptt',
 'news',)
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'myapp.CookieMiddleware.CookieMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')


Template error:
In template /home/USER/.virtualenvs/SITE/downloads/django-trunk/django/contrib/admin/templates/admin/change_list.html, error at line 73
   'NoneType' object has no attribute 'replace'
   63 :       {% endif %}


   64 :     {% endblock %}


   65 :     {% if cl.formset.errors %}


   66 :         <p class="errornote">


   67 :         {% blocktrans count cl.formset.errors|length as counter %}Please correct the error below.{% plural %}Please correct the errors below.{% endblocktrans %}


   68 :         </p>


   69 :         {{ cl.formset.non_form_errors }}


   70 :     {% endif %}


   71 :     <div class="module{% if cl.has_filters %} filtered{% endif %}" id="changelist">


   72 :       {% block search %}{% search_form cl %}{% endblock %}


   73 :       {% block date_hierarchy %} {% date_hierarchy cl %} {% endblock %}


   74 : 


   75 :       {% block filters %}


   76 :         {% if cl.has_filters %}


   77 :           <div id="changelist-filter">


   78 :             <h2>{% trans 'Filter' %}</h2>


   79 :             {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}


   80 :           </div>


   81 :         {% endif %}


   82 :       {% endblock %}


   83 : 


Traceback:
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/core/handlers/base.py" in get_response
  130.                     response = response.render()
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/response.py" in render
  105.             self.content = self.rendered_content
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/response.py" in rendered_content
  82.         content = template.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  140.             return self._render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  837.                 bit = self.render_node(node, context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/debug.py" in render_node
  78.             return node.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/loader_tags.py" in render
  123.         return compiled_parent._render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  837.                 bit = self.render_node(node, context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/debug.py" in render_node
  78.             return node.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/loader_tags.py" in render
  123.         return compiled_parent._render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  837.                 bit = self.render_node(node, context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/debug.py" in render_node
  78.             return node.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/loader_tags.py" in render
  62.             result = block.nodelist.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  837.                 bit = self.render_node(node, context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/debug.py" in render_node
  78.             return node.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/loader_tags.py" in render
  62.             result = block.nodelist.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  837.                 bit = self.render_node(node, context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/debug.py" in render_node
  78.             return node.render(context)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/template/base.py" in render
  1192.                     _dict = func(*resolved_args, **resolved_kwargs)
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/contrib/admin/templatetags/admin_list.py" in date_hierarchy
  362.                 } for year in years]
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/db/models/query.py" in _result_iter
  125.                 self._fill_cache()
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/db/models/query.py" in _fill_cache
  962.                     self._result_cache.append(next(self._iter))
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/db/models/query.py" in _safe_iterator
  348.             for item in iterator:
File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/db/models/sql/compiler.py" in results_iter
  1065.                     datetime = datetime.replace(tzinfo=None)

Exception Type: AttributeError at /admin/news/image/
Exception Value: 'NoneType' object has no attribute 'replace'

[Aymeric Augustin]

This is weird. A few questions:

• What database are you using?
• Was the table for this model created by Django?
• Can you give me the definition of the field used in date_hierarchy?

Thanks.

[tomas_00]

Mysql: 5.5.29-0ubuntu0.12.04.1

news_image table, created with South:

Column	Type	Null	Standard
id	int(11)	No
image	varchar(100)	Yes	NULL
title	varchar(255)	No
caption	longtext	Yes	NULL
creation_date	datetime	No
crop_value	varchar(10)	No

date_hierarchy = 'creation_date'

[Aymeric Augustin]

creation_date isn't nullable, that rules out my first hypothesis.

Next question: could you give me the output of:

>>> Image.objects.datetimes('creation_date', 'year')

in ./manage.py shell, with USE_TZ set to False and True respectively? It should raise an execption in some cases, please paste the full traceback.

Thanks!

EDIT: the output for kind='year' should suffice.

[anonymous]

USE_TZ = True

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/db/models/query.py", line 79, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/home/USER/.virtualenvs/SITE/downloads/django-trunk/django/db/models/query.py", line 118, in _result_iter
    upper = len(self._result_cache)
TypeError: object of type 'NoneType' has no len()

USE_TZ = False

[datetime.datetime(2012, 1, 1, 0, 0), datetime.datetime(2013, 1, 1, 0, 0)]

[Aymeric Augustin]

len swallowed the useful portion of the traceback :(

Here are my current conclusions:

• Django explicitly excludes NULL values from datetimes queries; this behavior is tested; and anyway your field isn't nullable here.
• For a reason I can't explain, the list of year values returned by the database contains None, which triggers this exception.

I could paper over this problem by skipping None values, but that doesn't address the root cause, and the problem may resurface somewhere else.

New theory: does MySQL knows about your timezone? In ./manage.py shell, this will tell you the name of your current time zone:

>>> from django.utils import timezone
>>> timezone.get_current_timezone_name()

Then, in ./manage.py dbshell, run:

mysql> SELECT * from mysql.time_zone_name where Name = "<your current timezone name>";

Does this return a row?

[tomas_00]

No, it does not return a row! This is probably not a bug then?

>>> timezone.get_current_timezone_name()
'Europe/Oslo'

mysql> SELECT * from mysql.time_zone_name where Name = "Europe/Oslo";
Empty set (0.00 sec)

[Aymeric Augustin]

Yes, that explains the problem.

It's documented as a backwards incompatible change in the release notes for 1.6: ​https://docs.djangoproject.com/en/dev/releases/1.6/#backwards-incompatible-changes-in-1-6

See also the note here: ​https://docs.djangoproject.com/en/dev/ref/models/querysets/#datetimes

I propose to:

• add a note in the docs for date_hierarchy pointing out this caveat,
• test for None values and raise an exception with an explicit message ("check that your database has a definition for your current time zone...")

Does that sound sufficient?

[tomas_00]

Yes, sound good!

Thank you!

[Aymeric Augustin <aymeric.augustin@…>]

In 0c82b1dfc48b4870e8fbcfb782ae02cdca821e1f:

Fixed #19929 -- Improved error when MySQL doesn't have TZ definitions.

Thanks tomas_00 for the report.

["Quique" <cronopios@…>]

From the error message ("Are time zone definitions and pytz installed?") I didn't get the time zone definitions should be 'installed' _in_ MySQL. I thought it was talking about the tzdata system package.

I think it would be useful to add in the message a link to ​https://docs.djangoproject.com/en/dev/ref/models/querysets/#datetimes , which explains one should use the mysql_tzinfo_to_sql program to LOAD the time zone tables in the MySQL database.