Opened 9 years ago

Closed 5 years ago

#23714 closed Bug (fixed)

`date` filter raises an exception for naive datetimes during DST change

Reported by: Markus Bertheau Owned by:
Component: Template system Version: 1.7
Severity: Normal Keywords: timezone
Cc: tomas.ehrlich@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Markus Bertheau)

When an exception, that would normally - with DEBUG=True - be displayed in the detailed 500 page, occurs during the DST/no-DST ambiguous hour of the geographical area that is set as TIMEZONE, the handler crashes with AmbiguousTimeError. This seriously limits the ability of many developers to develop and Django applications for a time span of 1/8760 of the year. This is unacceptable! ;)

2014-10-26 02:40:14,104    ERROR django.request                     :handle_uncaught_exception  231 Internal Server Error: /dashboard/suppliers/
Traceback (most recent call last):
  File "/home/markus/src/django/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/markus/src/django/django/db/transaction.py", line 394, in inner
    return func(*args, **kwargs)
  File "/home/markus/src/django/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/markus/src/django/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/markus/src/django/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/markus/src/django/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/markus/src/django/django/views/generic/list.py", line 145, in get
    self.object_list = self.get_queryset()
  File "/home/markus/src/machtfit/apps/dashboard/supplier/views.py", line 44, in get_queryset
    .annotate(num_orders=Count('partner__order_line')))
  File "/home/markus/src/django/django/db/models/query.py", line 802, in annotate
    is_summary=False)
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1012, in add_aggregate
    field_list, opts, self.get_initial_alias())
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1419, in setup_joins
    names, opts, allow_many, fail_on_missing=True)
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1383, in names_to_path
    self.raise_field_error(opts, name)
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1389, in raise_field_error
    "Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword u'order_line' into field. Choices are: <irrelevant>
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/markus/src/django/django/contrib/staticfiles/handlers.py", line 64, in __call__
    return self.application(environ, start_response)
  File "/home/markus/src/django/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/home/markus/src/django/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/home/markus/src/django/django/core/handlers/base.py", line 236, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/markus/src/django/django/views/debug.py", line 91, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/markus/src/django/django/views/debug.py", line 350, in get_traceback_html
    return t.render(c)
  File "/home/markus/src/django/django/template/base.py", line 148, in render
    return self._render(context)
  File "/home/markus/src/django/django/test/utils.py", line 88, in instrumented_test_render
    return self.nodelist.render(context)
  File "/home/markus/src/django/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/home/markus/src/django/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/home/markus/src/django/django/template/debug.py", line 90, in render
    output = self.filter_expression.resolve(context)
  File "/home/markus/src/django/django/template/base.py", line 624, in resolve
    new_obj = func(obj, *arg_vals)
  File "/home/markus/src/django/django/template/defaultfilters.py", line 769, in date
    return format(value, arg)
  File "/home/markus/src/django/django/utils/dateformat.py", line 343, in format
    return df.format(format_string)
  File "/home/markus/src/django/django/utils/dateformat.py", line 35, in format
    pieces.append(force_text(getattr(self, piece)()))
  File "/home/markus/src/django/django/utils/dateformat.py", line 268, in r
    return self.format('D, j M Y H:i:s O')
  File "/home/markus/src/django/django/utils/dateformat.py", line 35, in format
    pieces.append(force_text(getattr(self, piece)()))
  File "/home/markus/src/django/django/utils/dateformat.py", line 136, in O
    seconds = self.Z()
  File "/home/markus/src/django/django/utils/dateformat.py", line 189, in Z
    offset = self.timezone.utcoffset(self.data)
  File "/home/markus/.virtualenvs/machtfit/local/lib/python2.7/site-packages/pytz/tzinfo.py", line 406, in utcoffset
    dt = self.localize(dt, is_dst)
  File "/home/markus/.virtualenvs/machtfit/local/lib/python2.7/site-packages/pytz/tzinfo.py", line 349, in localize
    raise AmbiguousTimeError(dt)
AmbiguousTimeError: 2014-10-26 02:40:14.190834

Change History (26)

comment:1 by Markus Bertheau, 9 years ago

Description: modified (diff)

comment:2 by Aymeric Augustin, 9 years ago

Good catch.

Either django.utils.dateformat.TimeFormat.Z or django.template.defaultfilters.date should deal with that exception — probably the latter, as failing silently on errors is a behavior of the Django Template Language.

comment:3 by Aymeric Augustin, 9 years ago

Component: UncategorizedTemplate system
Triage Stage: UnreviewedAccepted
Type: UncategorizedBug

comment:4 by Tomáš Ehrlich, 9 years ago

Cc: tomas.ehrlich@… added
Resolution: invalid
Status: newclosed

I can't reproduce this bug at all.

The debug page formats datetime.datetime.now() with {{ server_time|date:'r' }}. Date format r is equivalent to D, j M Y H:i:s O as seen in traceback and O call Z whitch causes the exception. I tried to simply format datetime from traceback, but I didn't get any exception:

from django.conf import settings
settings.configure(USE_TZ=True, TIME_ZONE='Europe/Amsterdam')  # required for DateFormat, doesn't raise exception for USE_TZ=False neither

import datetime
from django.utils.dateformat import DateFormat

try:
    DateFormat(datetime.datetime(2014, 10, 26, 2, 30)).format('Z')
    DateFormat(datetime.datetime(2014, 10, 26, 2, 30, tzinfo=None)).format('Z')
except AmiguousTimeError:
    pass
else:
    print('Exception wasn't raised')

# Exception wasn't raised...

There seems to be missing information about timezone, which makes date&time 2014-10-26 2:30 really ambiguous, but even when I try to format timezone-naive datetime, it never raises exception.

This bug can actually affects debug page only once a year, when the time is switched back by one hour. So it's after all just 1/4380 of the year ;)

Could you please provide more information? What TIME_ZONE, USE_TZ settings do you use and etc.

comment:5 by Markus Bertheau, 9 years ago

Resolution: invalid
Status: closednew

You have to install pytz to see the error.

comment:6 by Tomáš Ehrlich, 9 years ago

Keywords: timezone added
Summary: Django-1.7 doesn't display 500 error pages during DST change`date` filter raises an exception for naive datetimes during DST change

That's right, with installed pytz I can see the exception.

Minimal code to reproduce the problem:

from django.conf import settings
settings.configure(USE_TZ=True, TIME_ZONE='Europe/Amsterdam')

import datetime
from django.utils.dateformat import DateFormat

right_now = datetime.datetime.now().replace(2014, 10, 26, 2, 30)
print(DateFormat(right_now).format('Z'))

In my opinion it's more serious because it doesn't affect just 500 debug page, but all templates that uses date filter with r, O or Z.

comment:7 by Tomáš Ehrlich, 9 years ago

Needs tests: set
Owner: changed from nobody to Tomáš Ehrlich
Status: newassigned

Tests are still missings. Should be simple though.

I don't like the fix, but can't figure out anything better. As mentioned in PR: The except block catching all exceptions is ugly, but I can't rely on pytz library, which is optional, and therefore I can't catch pytz.exceptions.AmbiguousTimeError exception.

comment:8 by Markus Bertheau, 9 years ago

I think handling AmbiguousTimeError in dateformat essentially means we're making a business logic decision on behalf of the implementor that we shouldn't make for him - his business logic might dictate a different decision.

Instead the template code should handle AmbiguousTimeError and silently return an empty string.

Kudos for fixing the root issue where a naive datetime was put in the context :) This part can surely be commit separately, because it solves the first problem that the technical 500 page doesn't work during DST change.

in reply to:  8 comment:9 by Aymeric Augustin, 9 years ago

Replying to mbertheau:

Instead the template code should handle AmbiguousTimeError and silently return an empty string.

Yes, that is the correct fix (or at least the fix consistent with how Django handles ambiguous datetimes in general).

comment:10 by Tomáš Ehrlich, 9 years ago

I understand that usually templates handle exceptions silently, printing nothing on error. I just don't like the idea that for one hour a year, the template won't display datetime at all (for naive datetimes). Yes, we're making a decision instead of users, but they won't know about this issue, since its occurrence is so rare. This isn't a common scenario which is easily tested.

Maybe at least show warning so user can see in logs what happened and why?

Last edited 9 years ago by Tomáš Ehrlich (previous) (diff)

comment:11 by Tomáš Ehrlich, 9 years ago

Alright, I simplified the solution a little bit. Added explanation into docs.

comment:12 by Markus Bertheau, 9 years ago

Reading the code more closely, I'm convinced, that the correct fix is to change TimeFormat.Z to return an empty string if given a naive datetime. This is in line with python's strftime, which exhibits the same behaviour.

If you agree, then the scope of this is actually bigger: TimeFormat shouldn't make up a timezone for anything, if given a naive datetime. This then concerns TimeFormat.e, TimeFormat.O, TimeFormat.T, TimeFormat.Z and DateFormat.I.

comment:13 by Markus Bertheau, 9 years ago

I'm not aware of all the circumstances that TimeFormat is given a naive datetime - with that caveat I think showing a warning, similar to what tricoder42 suggested, whenever TimeFormat gets a naive datetime. The warning could be similar to the datetime model field warning.

comment:14 by Tomáš Ehrlich, 9 years ago

Just to be precise: The problem occurs only for naive datetime during DST change. For other cases the timezone functions/formats works fine even for naive datetimes.

And yes, we should fix all other formats as well. I'll do it later.

comment:15 by Tomáš Ehrlich, 9 years ago

Has patch: set

I've updated the pull request, fixed also other TimeFormat formats.

comment:16 by Carl Meyer, 9 years ago

I'm not seeing a link to a pull request anywhere in this thread.

in reply to:  16 comment:17 by Tomáš Ehrlich, 9 years ago

Replying to carljm:

I'm not seeing a link to a pull request anywhere in this thread.

It's in the ticket's header (https://github.com/django/django/pull/3576)

comment:18 by Tomáš Ehrlich, 9 years ago

Needs documentation: set
Patch needs improvement: set

After discussion on PR, we're waiting for approval from @aaugustin:
Naive datetimes returns empty string for any timezone-related format specifiers (for the whole year). This is backward-incompatible change which will be mentioned in docs.

comment:19 by Aymeric Augustin, 9 years ago

Owner: changed from Tomáš Ehrlich to Aymeric Augustin

comment:20 by Aymeric Augustin <aymeric.augustin@…>, 9 years ago

In c7a6996df7e77bc3b9c5e581e67d766627ebabec:

Fixed a crash of the debug page.

During the autumn DST change, the template engine would fail to convert
the naive representation of now when USE_TZ = True. Passing now in UTC
eliminates the issue.

Thanks mbertheau and tricoder42.

Refs #23714.

comment:21 by Tim Graham <timograham@…>, 9 years ago

In c6a49d4f17db4579389e3f0477436f2444dead10:

[1.8.x] Fixed a crash of the debug page.

During the autumn DST change, the template engine would fail to convert
the naive representation of now when USE_TZ = True. Passing now in UTC
eliminates the issue.

Thanks mbertheau and tricoder42.

Refs #23714.

Backport of c7a6996df7e77bc3b9c5e581e67d766627ebabec from master

comment:22 by Aymeric Augustin, 9 years ago

Owner: Aymeric Augustin removed
Status: assignednew

comment:23 by Aymeric Augustin <aymeric.augustin@…>, 8 years ago

In 1014ba02:

Fixed debug view crash during autumn DST change.

This only happens if USE_TZ = False and pytz is installed (perhaps not
the most logical combination, but who am I to jugde?)

Refs #23714 which essentially fixed the same problem when USE_TZ = True.

Thanks Florian and Carl for insisting until I wrote a complete patch.

comment:24 by Aymeric Augustin <aymeric.augustin@…>, 8 years ago

In 94d1341:

[1.8.x] Fixed debug view crash during autumn DST change.

This only happens if USE_TZ = False and pytz is installed (perhaps not
the most logical combination, but who am I to jugde?)

Refs #23714 which essentially fixed the same problem when USE_TZ = True.

Thanks Florian and Carl for insisting until I wrote a complete patch.

Backport of 1014ba026e from master

comment:25 by Tim Graham <timograham@…>, 8 years ago

In ca0278f4:

[1.9.x] Fixed debug view crash during autumn DST change.

This only happens if USE_TZ = False and pytz is installed (perhaps not
the most logical combination, but who am I to jugde?)

Refs #23714 which essentially fixed the same problem when USE_TZ = True.

Thanks Florian and Carl for insisting until I wrote a complete patch.

Backport of 1014ba026e879e56e0f265a8d9f54e6f39843348 from master

comment:26 by Mariusz Felisiak, 5 years ago

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top