Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#23815 closed Bug (fixed)

CsrfViewMiddleware UnicodeDecodeError

Reported by: codeitloadit Owned by: nobody
Component: HTTP handling Version: master
Severity: Normal Keywords: middleware CsrfViewMiddleware UnicodeDecodeError
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The csrf middleware is raising an exception when the HTTP_REFERER contains non-ascii characters. Since this code is attempting to validate the referer, I would expect these request to just be rejected and return 403.

Here is the code in django/middleware/csrf.py:

                referer = request.META.get('HTTP_REFERER')
                if referer is None:
                    return self._reject(request, REASON_NO_REFERER)

                # Note that request.get_host() includes the port.
                good_referer = 'https://%s/' % request.get_host()
                if not same_origin(referer, good_referer):
                    reason = REASON_BAD_REFERER % (referer, good_referer)
                    return self._reject(request, reason)

This issue is very similar to https://code.djangoproject.com/ticket/20356 which was patched by https://github.com/django/django/commit/8fd44b2551b9cca765b216a31306f9c6935f1492 which just encodes the referer like so:

                referer = force_text(request.META.get('HTTP_REFERER', ''), errors='replace')

Change History (10)

comment:1 Changed 5 years ago by Simon Charette

Triage Stage: UnreviewedAccepted
Type: UncategorizedBug
Version: 1.6master

comment:2 Changed 5 years ago by Grzegorz Ślusarek

Resolution: needsinfo
Status: newclosed

Cannot repeat that error, I've tested it with several different unicode chars in REFERER and it works correctly.

comment:3 Changed 5 years ago by Daniel Harding

Resolution: needsinfo
Status: closednew

I've seen this in production: Django 1.6.5, Python 2.7, and here are some examples of REFERER causing the issue:

'\xd8B\xf6I\xdf'

'|\xcaH'

Let me know if I can be of any help resolving this issue.

comment:4 Changed 5 years ago by Tim Graham

Can you test with Django 1.7 and/or master? 1.6 is only receiving security fixes so if this issue has been fixed since then we can close the ticket.

comment:5 Changed 5 years ago by Daniel Harding

I was able to reproduce with Django 1.7.2/Python 2.7.9. Reproducing requires accessing Django using HTTPS, because the affected code is behind if request.is_secure():. To achieve this, I used the django-sslserver application (https://github.com/teddziuba/django-sslserver) in conjunction with a simple project with the Django admin enabled. Using the requests module to supply a bad REFERER header when POST-ing to the admin login page:

import requests

requests.post('https://localhost:8000/admin/login/',
              headers={'referer': '\xd8B\xf6I\xdf'},
              verify=False).text

I get the UnicodeDecodeError.

comment:6 Changed 5 years ago by Claude Paroz

Reproducible with:

  • tests/csrf_tests/tests.py

    diff --git a/tests/csrf_tests/tests.py b/tests/csrf_tests/tests.py
    index 9c6e2e5..f22cddb 100644
    a b class CsrfViewMiddlewareTest(TestCase): 
    300300        req2 = CsrfViewMiddleware().process_view(req, post_form_view, (), {})
    301301        self.assertNotEqual(None, req2)
    302302        self.assertEqual(403, req2.status_code)
     303        # Non-ASCII
     304        req.META['HTTP_REFERER'] = b'\xd8B\xf6I\xdf'
     305        req2 = CsrfViewMiddleware().process_view(req, post_form_view, (), {})
     306        self.assertNotEqual(None, req2)
     307        self.assertEqual(403, req2.status_code)
    303308
    304309    @override_settings(ALLOWED_HOSTS=['www.example.com'])
    305310    def test_https_good_referer(self):

comment:8 Changed 5 years ago by Tim Graham

Triage Stage: AcceptedReady for checkin

comment:9 Changed 5 years ago by Claude Paroz <claude@…>

Resolution: fixed
Status: newclosed

In 27dd7e727153cbf12632a2161217340123687c44:

Fixed #23815 -- Prevented UnicodeDecodeError in CSRF middleware

Thanks codeitloadit for the report, living180 for investigations
and Tim Graham for the review.

comment:10 Changed 5 years ago by Claude Paroz <claude@…>

In d8fb557a519a419a53d648ce1ef12dad8673151f:

[1.7.x] Fixed #23815 -- Prevented UnicodeDecodeError in CSRF middleware

Thanks codeitloadit for the report, living180 for investigations
and Tim Graham for the review.
Backport of 27dd7e7271 from master.

Note: See TracTickets for help on using tickets.
Back to Top