CsrfViewMiddleware UnicodeDecodeError

[codeitloadit]

The csrf middleware is raising an exception when the HTTP_REFERER contains non-ascii characters. Since this code is attempting to validate the referer, I would expect these request to just be rejected and return 403.

Here is the code in django/middleware/csrf.py:

                referer = request.META.get('HTTP_REFERER')
                if referer is None:
                    return self._reject(request, REASON_NO_REFERER)

                # Note that request.get_host() includes the port.
                good_referer = 'https://%s/' % request.get_host()
                if not same_origin(referer, good_referer):
                    reason = REASON_BAD_REFERER % (referer, good_referer)
                    return self._reject(request, reason)

This issue is very similar to https://code.djangoproject.com/ticket/20356 which was patched by ​https://github.com/django/django/commit/8fd44b2551b9cca765b216a31306f9c6935f1492 which just encodes the referer like so:

                referer = force_text(request.META.get('HTTP_REFERER', ''), errors='replace')

[Grzegorz Ślusarek]

Cannot repeat that error, I've tested it with several different unicode chars in REFERER and it works correctly.

[Daniel Harding]

I've seen this in production: Django 1.6.5, Python 2.7, and here are some examples of REFERER causing the issue:

'\xd8B\xf6I\xdf'

'|\xcaH'

Let me know if I can be of any help resolving this issue.

[Tim Graham]

Can you test with Django 1.7 and/or master? 1.6 is only receiving security fixes so if this issue has been fixed since then we can close the ticket.

[Daniel Harding]

I was able to reproduce with Django 1.7.2/Python 2.7.9. Reproducing requires accessing Django using HTTPS, because the affected code is behind if request.is_secure():. To achieve this, I used the django-sslserver application (​https://github.com/teddziuba/django-sslserver) in conjunction with a simple project with the Django admin enabled. Using the requests module to supply a bad REFERER header when POST-ing to the admin login page:

import requests

requests.post('https://localhost:8000/admin/login/',
              headers={'referer': '\xd8B\xf6I\xdf'},
              verify=False).text

I get the UnicodeDecodeError.

[Claude Paroz]

Reproducible with:

tests/csrf_tests/tests.py

@@ -300 +300 @@
         req2 = CsrfViewMiddleware().process_view(req, post_form_view, (), {})
         self.assertNotEqual(None, req2)
         self.assertEqual(403, req2.status_code)
+        # Non-ASCII
+        req.META['HTTP_REFERER'] = b'\xd8B\xf6I\xdf'
+        req2 = CsrfViewMiddleware().process_view(req, post_form_view, (), {})
+        self.assertNotEqual(None, req2)
+        self.assertEqual(403, req2.status_code)
 
     @override_settings(ALLOWED_HOSTS=['www.example.com'])
     def test_https_good_referer(self):

[Claude Paroz]

​https://github.com/django/django/pull/3843

[Claude Paroz <claude@…>]

In 27dd7e727153cbf12632a2161217340123687c44:

Fixed #23815 -- Prevented UnicodeDecodeError in CSRF middleware

Thanks codeitloadit for the report, living180 for investigations
and Tim Graham for the review.

[Claude Paroz <claude@…>]

In d8fb557a519a419a53d648ce1ef12dad8673151f:

[1.7.x] Fixed #23815 -- Prevented UnicodeDecodeError in CSRF middleware

Thanks codeitloadit for the report, living180 for investigations
and Tim Graham for the review.
Backport of 27dd7e7271 from master.