Bug in admin with date_hierarchy and Daylight savings

[Hugo Tácito]

When I have a date_hierarchy in admin and I select a range with a record with the day equal the day of a daylight savings I got an error of NonExistentTimeError.

I discovered that indeed 2016-10-16 00:00:00 does not exist in my Timezone ('America/Sao_Paulo') because that was when Daylight Savings started.

So when I have a record with that date in my database, it will brake the date_hierarchy feature.

[Tim Graham]

I'm not sure what the expected behavior is. What could Django do if you have bad data?

[Hugo Tácito]

Hello Tim,

I'm sorry for my lack of information and clarity, but even if I do have an existing datetime the problem still occur.

For example I have a User with date_joined equals 2016-10-16 12:17:44.270221-03 and when I select a date_hierarchy filter, that still raise the same exception of:

[12/Nov/2016 13:28:10] "GET /admin/auth/user/?date_joined__year=2016 HTTP/1.1" 200 6265
Internal Server Error: /admin/auth/user/
Traceback (most recent call last):
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/core/handlers/base.py", line 174, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/core/handlers/base.py", line 172, in get_response
    response = response.render()
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/response.py", line 160, in render
    self.content = self.rendered_content
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/response.py", line 137, in rendered_content
    content = template.render(context, self._request)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/backends/django.py", line 95, in render
    return self.template.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 206, in render
    return self._render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 197, in _render
    return self.nodelist.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 992, in render
    bit = node.render_annotated(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated
    return self.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py", line 173, in render
    return compiled_parent._render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 197, in _render
    return self.nodelist.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 992, in render
    bit = node.render_annotated(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated
    return self.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py", line 173, in render
    return compiled_parent._render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 197, in _render
    return self.nodelist.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 992, in render
    bit = node.render_annotated(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated
    return self.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py", line 69, in render
    result = block.nodelist.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 992, in render
    bit = node.render_annotated(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated
    return self.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py", line 69, in render
    result = block.nodelist.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 992, in render
    bit = node.render_annotated(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py", line 959, in render_annotated
    return self.render(context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/library.py", line 223, in render
    _dict = self.func(*resolved_args, **resolved_kwargs)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/contrib/admin/templatetags/admin_list.py", line 380, in date_hierarchy
    } for day in days]
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/query.py", line 158, in __iter__
    for row in compiler.results_iter():
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 808, in results_iter
    row = self.apply_converters(row, converters)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 792, in apply_converters
    value = converter(value, expression, self.connection, self.query.context)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/expressions.py", line 950, in convert_value
    value = timezone.make_aware(value, self.tzinfo)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/utils/timezone.py", line 364, in make_aware
    return timezone.localize(value, is_dst=is_dst)
  File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/pytz/tzinfo.py", line 327, in localize
    raise NonExistentTimeError(dt)
NonExistentTimeError: 2016-10-16 00:00:00

Even with an existing datetime.

The admin I did to test:

class UserAdmin(admin.ModelAdmin):
    date_hierarchy = 'date_joined'
    list_display = ('email', 'first_name', 'last_name', 'date_joined')
    list_filter = ('is_staff', 'is_superuser')


admin.site.unregister(User)
admin.site.register(User, UserAdmin)

The internationalization settings:

LANGUAGE_CODE = 'pt-br'

TIME_ZONE = 'America/Sao_Paulo'

USE_I18N = True

USE_L10N = True

USE_TZ = True

With TIME_ZONE = 'UTC' the problem doesn't occur.

Edit:
I've did this in a new clean project, with just this object at the User model.

[Hugo Tácito]

This also happens if we set a datetime to 2015-10-18. For example: 2015-10-18 10:45:29.270221-03
In 2015 at 10-18 was started the daylight savings at Brazil too.

​https://www.timeanddate.com/time/change/brazil/sao-paulo?year=2015

[Tim Graham]

Any idea what Django should do to resolve this?

[Hugo Tácito]

Hello Tim,

It seems that the datetime truncation is too hard in the date_hierarchy templatetag.

Changing this line:
​https://github.com/django/django/blob/master/django/contrib/admin/templatetags/admin_list.py#L385

to:
days = getattr(days, dates_or_datetimes)(field_name, 'second')

Probably will fix it.
I can also create a patch if you want.

[Aymeric Augustin]

The problem seems to stem from the fact that Django wants dates, but fetches them as datetimes, and applies the converter for datetimes, which causes an exception in that case.

It isn't possible to represent "2016-10-16 in local time in São Paulo" as datetime.datetime(2016, 10, 16, tzinfo=pytz.timezone('America/Sao_Paulo')) because that value doesn't exist. The only proper way to represent it is datetime.date(2016, 10, 16).

I believe the resolution should be along the lines of making the year / month / day truncation expressions cast their result to a DateField instead of a DateTimeField. Reducing the problem to a lower-level test case may help.

It seems possible that the same problem could occur with the hour truncation expression if some time zones have their DST between 1:30 and 2:30. I'm not sure what we could do about that; it seems much harder to fix.

[Anton Samarchyan]

I've attached a django project to reproduce the problem. Admin credentials - admin/AiNeuw3uOhvei7da
To reproduce, open /admin/app/test/?date__month=10&date__year=2016

[Anton Samarchyan]

Added ​PR

[Tim Graham <timograham@…>]

In e88d2df:

Fixed #27475 -- Fixed NonExistentTimeError crash in ModelAdmin.date_hierarchy.