Opened 15 months ago

Closed 14 months ago

Last modified 14 months ago

#28488 closed Bug (fixed)

Django 1.11+ raises CSRF verification failed if settings.DEBUG is False and an intermediate 404 page is requested

Reported by: Ruben Alves Owned by: nobody
Component: CSRF Version: master
Severity: Release blocker Keywords: csrf failed settings debug false production
Cc: rubenanapu@…, Nicola, Jay Cox, Florian Apolloner, Berker Peksag Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Ruben Alves)

Hi all,

I'm using Django1.11 (and made tests also with Django1.11.4) and having problems when submitting a form with POST method.

I'm calling the {% csrf_token %} inside of the form, so, this is not the problem.
The problem when submitting the form is:

Forbidden (403)
CSRF verification failed. Request aborted.
More information is available with DEBUG=True.

Then, in order to see "more information", I've enabled settings.DEBUG to True and submitted the form again. At this moment, the problem didn't happens anymore.

So I've disabled settings.DEBUG, submitted again, and the problem was there. Enabled DEBUG=True again, problem has gone.

Initially I thought that could be some error in my code, but the same happens when I try to reset my password wit the django.contrib.auth.views.password_reset view.

In my settings.py, I have the following changes that were made recently:

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

I use AWS (Amazon Web Service) Elastic Beanstalk with https enabled.

The worst part is that I've discovered this only on production because I make all tests in my local machine with DEBUG=True, and on production, we set DEBUG=False. Also, the error message should be shown only if DEBUG=True.

It's something related to the same error mentioned on Google Groups https://groups.google.com/forum/#!searchin/django-users/CSRF$20verification$20failed.$20Request$20aborted.$20More$20information$20is$20available$20with$20DEBUG$3DTrue.%7Csort:relevance/django-users/ISoJ6CwHOXQ/pirLih0jBgAJ

Attachments (3)

error.png (9.9 KB) - added by Ruben Alves 15 months ago.
Error message on production with settings.DEBUG=False
ticket28488.zip (18.4 KB) - added by Nicola 15 months ago.
sample project that show the issue
t28488.diff (1.2 KB) - added by Florian Apolloner 15 months ago.

Download all attachments as: .zip

Change History (44)

Changed 15 months ago by Ruben Alves

Attachment: error.png added

Error message on production with settings.DEBUG=False

comment:1 Changed 15 months ago by Ruben Alves

Description: modified (diff)

comment:2 Changed 15 months ago by Tim Graham

Did you experience a change in Django 1.11 compared to Django 1.10 or earlier? I think you'll need to debug the problem further and let us know why Django is at fault.

comment:3 Changed 15 months ago by Ruben Alves

Before Django 1.11, I was using Django 1.8 and everything was working perfectly.

In order to use Django 1.11, I did all the changes mentioned on https://docs.djangoproject.com/en/1.11/releases/1.11/

What I've done now was add settings.CSRF_FAILURE_VIEW = 'courses.views.csrf_failure' with DEBUG=False

I made my courses.views.csrf_failure raise an exception, so we could see the traceback.

The error message is:

Lets see where this shit happens: {'template_name': 'dooesntmatter', 'reason': u'CSRF token missing or incorrect.', 'request': <WSGIRequest: POST '/en/accounts/password_reset/'>}

The traceback is the following:

File "/opt/python/run/venv/lib64/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/opt/python/run/venv/lib64/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/opt/python/run/venv/lib64/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)

File "/opt/python/run/venv/lib64/python2.7/site-packages/django/middleware/csrf.py" in process_view
  314.                 return self._reject(request, REASON_BAD_TOKEN)

File "/opt/python/run/venv/lib64/python2.7/site-packages/django/middleware/csrf.py" in _reject
  163.         return _get_failure_view()(request, reason=reason)

File "/opt/python/current/app/courses/views/__init__.py" in csrf_failure
  49.     raise Exception('Lets see where this shit happens: ' + str(locals()))
Last edited 15 months ago by Tim Graham (previous) (diff)

comment:4 Changed 15 months ago by Tim Graham

The traceback doesn't give me any ideas about the cause. I think you'll need to add further logging in django/middleware/csrf.py to find the root cause.

A large CSRF change between Django 1.8 and 1.11 is 5112e65ef2df1dbb95ff83026b6a962fb2688661. Perhaps you can test with Django 1.9 and 1.10 to help narrow the problem.

In the backwards-incompatible changes section of the 1.10 release notes, there's a note, "CSRF token values are now required to be strings of 64 alphanumerics; values of 32 alphanumerics, as set by older versions of Django by default, are automatically replaced by strings of 64 characters. Other values are considered invalid. This should only affect developers or users who replace these tokens." Maybe it's relevant to you.

Last edited 15 months ago by Tim Graham (previous) (diff)

comment:5 Changed 15 months ago by Ruben Alves

Ok, I've added some logs on django.middleware.csrf, and this is what I have:

Between lines 208 and 209 (https://github.com/django/django/blob/stable/1.11.x/django/middleware/csrf.py#L208) I've added a log and the value of csrf_token is vcdodHkRb8jlxQP5fnTjuQp5i3PMWYYEBOYfpOFkCEjWWwHMpJ9uqaPGI6vGi6hS

The if request.is_secure(): is True on line 228 (https://github.com/django/django/blob/stable/1.11.x/django/middleware/csrf.py#L228)

The value of request_csrf_token on line 311 (https://github.com/django/django/blob/stable/1.11.x/django/middleware/csrf.py#L311) before we call request_csrf_token = _sanitize_token(request_csrf_token) is tMgl4aTrdjOEShUax3Gz1CJLbvnhBWiEVzmzgTxSclDuP01lBfwoE2R6dDyljaCQ

After _sanitize_token be called, the value of request_csrf_token is the same, and the value of csrf_token is vcdodHkRb8jlxQP5fnTjuQp5i3PMWYYEBOYfpOFkCEjWWwHMpJ9uqaPGI6vGi6hS.

Finally _compare_salted_tokens(request_csrf_token, csrf_token) on line 313 returns False (https://github.com/django/django/blob/stable/1.11.x/django/middleware/csrf.py#L313).

comment:6 Changed 15 months ago by Tim Graham

Maybe you can try to minimize the middleware, etc. in your project and see if something like that is causing the issue.

comment:7 Changed 15 months ago by Ruben Alves

I've being testing to reload the page and see the Cookie named csrftoken.

On the system that runs with Django1.8, every time I reload the page, the csrftoken Cookie has the same value.

On the system that runs with Django1.11, every time I reload the page, the csrftoken Cookie has a different value.

Do you know if on Django1.11 is expected to have a different csrftoken Cookie every time I reload the page? If yes, why is expected to be different in Django1.11 and not in Django1.8?

comment:8 Changed 15 months ago by Tim Graham

Yes, that's expected after 5112e65ef2df1dbb95ff83026b6a962fb2688661.

comment:9 Changed 15 months ago by Ruben Alves

But even if they are different, the function _unsalt_cipher_token (https://github.com/django/django/blob/stable/1.11.x/django/middleware/csrf.py#L62) should return always the same value with the different values on the Cookie, right?

I've called the _unsalt_cipher_token function with the different values of the Cookie, and the returned values are ALWAYS different.

comment:10 Changed 15 months ago by Ruben Alves

I've created a template tag named uncypher that calls _unsalt_cipher_token as you can see below:

from django.middleware.csrf import _unsalt_cipher_token
from django.template import Library

register = Library()


@register.simple_tag
def uncypher(token):
    """
    Testing the CSRF problem
    """
    return _unsalt_cipher_token(token)

Then, on a html template I've added

var parser = new DOMParser();
var csrf = parser.parseFromString("{% csrf_token %}", "text/xml").firstChild;
console.error('token: {% uncypher csrf_token %}');

After set settings.DEBUG=True and reload the page 3 times, on the JavaScript Console was printed the same value lKnMEqisHw8mxMIXZgZTKt9cirypfv1f on the 3 times that I've reloaded the page.

Then I've set settings.DEBUG=False and reloaded the page 4 times again.
On the first time it printed the same value lKnMEqisHw8mxMIXZgZTKt9cirypfv1f. On the second YgnNvwBJJwAID17nl7FLhtA3rObtvGLX, third time printed hB1plGV688FpDPNu9waO1H959c04YLPM and fourth time printed C2sJ9e7coe74juO6HxAY4RIppBY4yi1Y.

comment:11 Changed 15 months ago by Ruben Alves

Tested the system with Django1.10 and Django1.11.

With Django1.10 everything works. With Django1.11 we have the error.

I'm going to see the diff between the two versions: https://github.com/django/django/compare/stable/1.10.x...stable/1.11.x

Last edited 15 months ago by Ruben Alves (previous) (diff)

comment:12 Changed 15 months ago by Ruben Alves

I finally solved my problem.

The problem is: If a user just get a 404 page, a new CSRF Token is generated, invalidating the CSRF Token that was originally loaded with {% csrf_token %}.

How I've discovered

On the WebSite that I work, we have a page with the URI /en/courses/.

On that page, on the .html file we include a JavaScript: <script src="/static/js/app.js"></script>

The app.js was trying to load a file named assets/js/particles.json using the relative path, so the final URL of the static file was /en/courses/assets/js/particles.json. Load the particles.json is the only thing that app.js was doing. Nothing else.

That particles.json doesn't exist on our system.

After successfully load /en/courses/, the Django Server was receiving a request for /en/courses/assets/js/particles.json that raises a 404 error for the static file, but this error 404 is not even noticed by the users because this JS file was doing nothing.

After I remove this <script src="/static/js/app.js"></script>, everything worked fine.

Then with debugs on the django.middleware.csrf.CsrfViewMiddleware I could confirm that if I reload a page 1000 times, the CSRF Token returned by _unsalt_cipher_token (​https://github.com/django/django/blob/stable/1.11.x/django/middleware/csrf.py#L62) is always the same. But if I just try to access a page that doesn't exist, then _unsalt_cipher_token returns a different value (a new token is generated).

Unbelievable that a single javascript that doesn't even exist has broken my system and took me 3 days to find out.

comment:13 Changed 15 months ago by KwonHan Bae

so it's not a bug?
I think if it a bug we can make another explicit ticket

comment:14 Changed 15 months ago by Ruben Alves

I consider a bug.

If a user has a form, and on the same page tries to load an image, css, or javascript with a wrong URI, the form will never be accepted.

comment:15 Changed 15 months ago by Mark Lavin

Based on the root cause of this problem, it sounds to be very similar to #22961 (StaticFilesHandler should not run middleware on 404). It has a patch to make that change but didn't include any tests for that new behavior.

comment:16 Changed 15 months ago by Claude Paroz

Is it right to assume that you are using the runserver server on your production instance?

comment:17 Changed 15 months ago by Ruben Alves

No, ./manage.py runserver is used only in localhost for development purposes.

In AWS we use Elastic Beanstalk with a .ebextensions/django.config file according to the instructions found on http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html

comment:18 Changed 15 months ago by Claude Paroz

Ah sorry, I didn't notice at first that the missing file has a path outside of the static directory. So it's not a StaticFilesHandler issue.

comment:19 Changed 15 months ago by Claude Paroz

I think this is clearly a bug if someone is able reproduce the issue in a test.

comment:20 Changed 15 months ago by Tim Graham

Resolution: needsinfo
Status: newclosed
Summary: Django 1.11 to 1.11.4 raises CSRF verification failed if settings.DEBUG is FalseDjango 1.11+ raises CSRF verification failed if settings.DEBUG is False and an intermediate 404 page is requested

If someone can provide a sample project with steps to reproduce, please reopen the ticket.

comment:21 Changed 15 months ago by Nicola

same problem here, in my case the problem is with the django admin site and the 404 page is caused by chrome trying to get /favicon.ico

comment:22 Changed 15 months ago by Nicola

Cc: Nicola added

comment:23 Changed 15 months ago by Nicola

this bug is really difficult to track down, for my problem I added the workaround in apache config, something like this:

ProxyPass /favicon.ico !
Alias /favicon.ico /path/to/favicon.ico
<Location "/favicon.ico">
    Require all granted
</Location> 

after this change the django admin site worked again, hope this helps someone other.

I think I have this issue on my web site since a favicon is defined for all pages but not /admin (django admin site) and so chrome try to get the /favicon.ico for admin site (the favicon is in /static/images/favicon.ico), django return 404 and admin site login does not work.

A fix in django itself would be really appreciated, thanks!

Changed 15 months ago by Nicola

Attachment: ticket28488.zip added

sample project that show the issue

comment:24 Changed 15 months ago by Nicola

Resolution: needsinfo
Status: closednew

attached a sample project that show the issue with the admin site

you can reproduce using runserver

visit the page (tested with firefox and chromium on linux)

http://127.0.0.1:8000/admin

look at the runserver logs and ensure that you see

GET /favicon.ico HTTP/1.1" 404 230

try to login to the django admin site with any user and pass

you'll get a csrf error

comment:25 Changed 15 months ago by Tim Graham

Thanks for the sample project. I bisected the changed in behavior to eef95ea96faef0b7dbbe0c8092202b74f68a899b which first appeared in Django 1.9, yet the report is about Django 1.11, so I'm not sure if everyone reporting this issue is encountering the same problem. It looks like the key to reproducing it is that the custom handler404 uses render() rather than render_to_response() or a plain HttpResponse(). I'll look more later.

comment:26 Changed 15 months ago by Tim Graham

Cc: Jay Cox added

Jay, as the author of eef95ea96faef0b7dbbe0c8092202b74f68a899b, can you look at this? It looks like before your patch, this condition in CsrfViewMiddleware.process_response() prevented a CSRF token from being set for the 404 requests:

if request.META.get("CSRF_COOKIE") is None:
    return response

After the patch, a cookie is being set for those requests.

comment:27 Changed 15 months ago by Florian Apolloner

Cc: Florian Apolloner added

@tim: If you run over similar security related bugs, please always CC me

I also agree that https://github.com/django/django/commit/eef95ea96faef0b7dbbe0c8092202b74f68a899b#diff-a3be722ce2831a8d11438021d44cedf1L200 probably should never have been dropped. I'll look into the root cause later this day.

Last edited 15 months ago by Florian Apolloner (previous) (diff)

comment:28 Changed 15 months ago by Florian Apolloner

I think the main issue here is that a 404 handler does not run through process_view of the csrf middleware and therefor does not know that a token already exists and creates a new one (more details later)

Changed 15 months ago by Florian Apolloner

Attachment: t28488.diff added

comment:29 Changed 15 months ago by Florian Apolloner

I've attached a patch, it is still not perfect and the underlying problem will still be there if there is a middleware doing something stupid before the CSRF middleware is run (ie run the http404 handler in process_request).

As for https://github.com/django/django/commit/eef95ea96faef0b7dbbe0c8092202b74f68a899b#diff-a3be722ce2831a8d11438021d44cedf1L200 -- I was wrong there, it is afaik not needed since if the "CSRF_COOKIE_USED" basically takes care of that too.

comment:30 Changed 14 months ago by Florian Apolloner

Triage Stage: UnreviewedAccepted

comment:31 Changed 14 months ago by Florian Apolloner

Version: 1.11master

comment:33 Changed 14 months ago by Berker Peksag

Cc: Berker Peksag added

comment:34 Changed 14 months ago by Berker Peksag

I have the same problem, but disabling the custom 404 handler didn't work for me as described in #28143 (closed as a duplicate of this issue)

Removing the custom 404 view from urls.py, or setting DEBUG to True (obviously), fixes the Django admin failed csrf error.

Test URL: https://staging.python.org/admin/
Source code: https://github.com/python/pythondotorg
Installation document: https://pythondotorg.readthedocs.io/install.html

comment:35 Changed 14 months ago by Florian Apolloner

Does my PR fix it for you?

comment:36 Changed 14 months ago by Berker Peksag

Does my PR fix it for you?

I can't reproduce the problem in development environment even if I set settings.DEBUG to False. I applied your patch manually on the server and unfortunately I still get 403 when I try to submit a form.

comment:37 Changed 14 months ago by Berker Peksag

I just tried Nicola's sample project and I was able to reproduce it locally.

For some strange reason, I can't reproduce it locally on my own project even if I made the same request:

[19/Sep/2017 02:00:34] "GET / HTTP/1.1" 200 42428
[19/Sep/2017 02:00:36] "GET /static/js/main-min.js HTTP/1.1" 404 31942
[19/Sep/2017 02:00:50] "GET /admin/login/?next=/admin/ HTTP/1.1" 200 1625
[19/Sep/2017 02:00:50] "GET /static/admin/css/base.css HTTP/1.1" 404 31950
[19/Sep/2017 02:00:50] "GET /static/admin/css/login.css HTTP/1.1" 404 31952

I removed all custom middlewares and copied the MIDDLEWARE setting from Nicola's project and I was still able to login successfully.

On both local and production environment, I see different csrfmiddlewaretoken values after each refresh but the value of csrfmiddlewaretoken doesn't match with the csrftoken cookie (it also stays same after I refreshed the page) However, the csrftoken cookie changes each request in Nicola's project.

comment:38 Changed 14 months ago by Berker Peksag

[...] I see different csrfmiddlewaretoken values after each refresh but the value of csrfmiddlewaretoken doesn't match with the csrftoken cookie [...]

Turns out I was reading an older version of the document and this behavior is expected in Django 1.11.

Also, I get 403 even if there was no 404 request in the middle so I'm beginning to think that my problem is different than this ticket.

comment:39 Changed 14 months ago by Berker Peksag

[...] so I'm beginning to think that my problem is different than this ticket.

Turns out the hostname of the referer should be an exact match of CSRF_COOKIE_DOMAIN and in my case is_same_domain('staging.python.org', 'staging.python.org:9000') returns False. I set CSRF_TRUSTED_ORIGINS = ['staging.python.org'] and everything works fine now.

Sorry for the noise!

comment:40 Changed 14 months ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In c4c128d:

Fixed #28488 -- Reallowed error handlers to access CSRF tokens.

Regression in eef95ea96faef0b7dbbe0c8092202b74f68a899b.

comment:41 Changed 14 months ago by Tim Graham <timograham@…>

In 42847327:

[1.11.x] Fixed #28488 -- Reallowed error handlers to access CSRF tokens.

Regression in eef95ea96faef0b7dbbe0c8092202b74f68a899b.

Backport of c4c128d67c7dc2830631c6859a204c9d259f1fb1 from master

Note: See TracTickets for help on using tickets.
Back to Top