Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#27308 closed Cleanup/optimization (fixed)

BytesWarning exception raised when running with python 3 -bb option

Reported by: Gustavo J. A. M. Carneiro Owned by: nobody
Component: Core (Other) Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When your template file does not exist, naturally Django gives an exception. But if TEMPLATE_DEBUG is True, it also tries to generate a pretty html traceback, and this is where the bug is, if you try to run with the python 3 -bb option, which complains when you try to do str() on a bytes value.

I have Django version 1.10.1.

$ python3 -bb
Python 3.5.2 (default, Sep 10 2016, 08:21:44) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> str(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BytesWarning: str() on a bytes instance
>>> repr(b'')
"b''"
>>>

Traceback:

2016-10-03 17:47:29,582 ERROR production.py:138 GET /login -> TemplateDoesNotExist(b'loginxxx.html',)
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>
Internal Server Error: /login
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>
2016-10-03 17:47:29,583 ERROR exception.py:124 Internal Server Error: /login
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>
2016-10-03 17:47:29,599 ERROR production.py:138 GET /login -> BytesWarning('str() on a bytes instance',)
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/mollybet/apps/home/geoipblocking.py", line 17, in middleware
    return get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = response_for_exception(request, exc)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 86, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 84, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 316, in get_traceback_html
    c = Context(self.get_traceback_data(), use_l10n=False)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 308, in get_traceback_data
    c['exception_value'] = smart_text(self.exc_value, errors='replace')
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 41, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 76, in force_text
    s = six.text_type(s)
BytesWarning: str() on a bytes instance
Internal Server Error: /login
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/mollybet/apps/home/geoipblocking.py", line 17, in middleware
    return get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = response_for_exception(request, exc)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 86, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 84, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 316, in get_traceback_html
    c = Context(self.get_traceback_data(), use_l10n=False)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 308, in get_traceback_data
    c['exception_value'] = smart_text(self.exc_value, errors='replace')
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 41, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 76, in force_text
    s = six.text_type(s)
BytesWarning: str() on a bytes instance
2016-10-03 17:47:29,601 ERROR exception.py:124 Internal Server Error: /login
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/mollybet/apps/home/geoipblocking.py", line 17, in middleware
    return get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = response_for_exception(request, exc)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 86, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 84, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 316, in get_traceback_html
    c = Context(self.get_traceback_data(), use_l10n=False)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 308, in get_traceback_data
    c['exception_value'] = smart_text(self.exc_value, errors='replace')
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 41, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 76, in force_text
    s = six.text_type(s)
BytesWarning: str() on a bytes instance
2016-10-03 17:47:29,687 ERROR production.py:138 GET /login -> BytesWarning('str() on a bytes instance',)
Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/gjc/projects/molly/mollybet/apps/home/views.py", line 124, in login
    new_molly_url=settings.NEW_MOLLY_URL)
  File "/home/gjc/projects/molly/mollybet/webutils.py", line 167, in render
    t = loader.get_template((name + '.html').encode("ascii"))
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/template/loader.py", line 25, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: <unprintable TemplateDoesNotExist object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/mollybet/apps/home/geoipblocking.py", line 17, in middleware
    return get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = response_for_exception(request, exc)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 86, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 84, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 316, in get_traceback_html
    c = Context(self.get_traceback_data(), use_l10n=False)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 308, in get_traceback_data
    c['exception_value'] = smart_text(self.exc_value, errors='replace')
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 41, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/utils/encoding.py", line 76, in force_text
    s = six.text_type(s)
BytesWarning: str() on a bytes instance

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/home/gjc/projects/molly/mollybet/apps/home/auth.py", line 113, in middleware
    return get_response(request)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = response_for_exception(request, exc)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 86, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 84, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/gjc/projects/molly/env/lib/python3.4/site-packages/django/views/debug.py", line 317, in get_traceback_html
    return t.render(c)
[...]

Change History (8)

comment:1 by Tim Graham, 8 years ago

I'm not familiar with the -bb flag. Can a bug happen as a result of this warning?

Should we be trying to get Django's test suite to pass with it enabled? Currently there are 2 failures and 14 errors with a few exceptions coming from PIL rather than Django itself.

comment:2 by Gustavo J. A. M. Carneiro, 8 years ago 

       -b     Issue   warnings    about    str(bytes_instance),    str(bytear‐
              ray_instance)  and  comparing  bytes/bytearray  with  str. (-bb:
              issue errors)

The main point is to catch bugs due to mixing bytes vs str. Also, how many times have you done in Python 3 this type of bug:

>>> def create_redis_key(param1):
...     return "foo:bar:" + str(param1)
... 
>>> create_redis_key("zbr")
'foo:bar:zbr'

But if param1 comes from e.g. a network socket and you forget to decode the bytes:

>>> create_redis_key(b"zbr")
"foo:bar:b'zbr'"

So I like running my app with -bb (just one -b just gives a warning, but warnings often pass silently). So it is useful if Django works with -bb. I am certainly running a production Django app with -bb, for a few weeks, and this is the first problem that I see caused by it.

Last edited 8 years ago by Gustavo J. A. M. Carneiro (previous) (diff)

comment:3 by Tim Graham, 8 years ago

Component: Uncategorized → Core (Other)
Triage Stage: Unreviewed → Accepted
Type: Uncategorized → Cleanup/optimization

The goal of this ticket could be to get Django's test suite passing with -bb, but I'm not sure if it's feasible due to errors coming from dependencies like Pillow.

comment:4 by Claude Paroz, 8 years ago

In the reported issue, I think the warning and the problem come from that line in the user's project where the template name is unexpectedly converted to bytes:
loader.get_template((name + '.html').encode("ascii")).

I see we are giving some bytestrings to some debug view tests in the Django test suite. It's probably fine to test that as long as we are supporting Python 2, but after dropping that support, we could simply assume exception messages are strings, not bytes. Unless Django is itself creating exception with bytes content on Python 3, which would be something to fix, I don't think we should do anything more now on Django's side.

comment:5 by Tim Graham, 8 years ago

Has patch: set

​PR to fix BytesWarnings identified by Django's test suite.

comment:6 by Claude Paroz, 8 years ago

Triage Stage: Accepted → Ready for checkin
Version: 1.10 → master

comment:7 by Tim Graham <timograham@…>, 8 years ago

Resolution: → fixed
Status: new → closed

In 75f0070a:

Fixed #27308 -- Fixed BytesWarnings in the test suite.

comment:8 by Tim Graham <timograham@…>, 8 years ago

In 0a66aa15:

Refs #27308, #27753 -- Removed obsolete cookie test mixing bytes with str.

Python 3's SimpleCookie treats all values as strings. Passing a bytes
object coerces to the repr value.

Note: See TracTickets for help on using tickets.
Back to Top